python朴素贝叶斯的文本分类_基于Python 朴素贝叶斯--文本分类

￡神魔★判官ぃ 2022-11-05 11:57 118阅读 0赞

基于Python 朴素贝叶斯--文本分类

\# coding: utf-8

利用jupter book在线运行code。

步骤：

准备分类文档内容和分类标签，停用词文档

利用Jieba(中文)/NTLK(英文)将文档中单词分词

加载停用词文件，生成TFIDF向量，计算单词的TFIDF,(TF:词频，IDF：逆向文档频率=

![890de4b03afb87f031aa834603a481e6.gif][](文档数/(单词出现的文档数+1))

使用多项式贝叶斯算法生成分类器

预测结果并计算分类器的准确率

\# 中文文本分类

import os

import jieba

import warnings

from sklearn.feature\_extraction.text import TfidfVectorizer

from sklearn.naive\_bayes import MultinomialNB

from sklearn import metrics

def cut\_words(file\_path):

"""

对文本进行切词

:param file\_path: txt文本路径

:return: 用空格分词的字符串

"""

text\_with\_spaces = ''

text=open(file\_path, 'r', encoding='gb18030').read()

textcut = jieba.cut(text)

for word in textcut:

text\_with\_spaces += word +' '

return text\_with\_spaces

def loadfile(file\_dir, label):

"""

将路径下的所有文件加载

:param file\_dir: 保存txt文件目录

:param label: 文档标签

:return: 分词后的文档列表和标签

"""

file\_list = os.listdir(file\_dir)

words\_list = \[\]

labels\_list = \[\]

for file in file\_list:

file\_path = file\_dir + '/' + file

words\_list.append(cut\_words(file\_path))

labels\_list.append(label)

return words\_list, labels\_list

pathdir=...

\# 训练数据

train\_words\_list1, train\_labels1 = loadfile(pathdir+'text classification/train/女性', '女性')

train\_words\_list2, train\_labels2 = loadfile(pathdir+'text classification/train/体育', '体育')

train\_words\_list3, train\_labels3 = loadfile(pathdir+'text classification/train/文学', '文学')

train\_words\_list4, train\_labels4 = loadfile(pathdir+'text classification/train/校园', '校园')

train\_words\_list = train\_words\_list1 + train\_words\_list2 + train\_words\_list3 + train\_words\_list4

train\_labels = train\_labels1 + train\_labels2 + train\_labels3 + train\_labels4

\# 测试数据

test\_words\_list1, test\_labels1 = loadfile(pathdir+'text classification/test/女性', '女性')

test\_words\_list2, test\_labels2 = loadfile(pathdir+'text classification/test/体育', '体育')

test\_words\_list3, test\_labels3 = loadfile(pathdir+'text classification/test/文学', '文学')

test\_words\_list4, test\_labels4 = loadfile(pathdir+'text classification/test/校园', '校园')

test\_words\_list = test\_words\_list1 + test\_words\_list2 + test\_words\_list3 + test\_words\_list4

test\_labels = test\_labels1 + test\_labels2 + test\_labels3 + test\_labels4

\#加载停用词

stop\_words = open(pathdir+'text classification/stop/stopword.txt', 'r', encoding='utf-8').read()

print(stop\_words)

stop\_words = stop\_words.encode('utf-8').decode('utf-8-sig') \# 列表头部\\ufeff处理

stop\_words = stop\_words.split('\\n') \# 根据分隔符分隔

new\_stopword=\['ain','aren', 'couldn', 'didn', 'doesn', 'don', 'hadn', 'hasn', 'haven', 'isn', 'll', 'mon', 'shouldn', 've', 'wasn', 'weren', 'won', 'wouldn'\]

stop\_words=stop\_words.append(new\_stopword)

print(stop\_words)

\# 计算单词权重

tf2 = TfidfVectorizer(stop\_words=stop\_words, max\_df=0.5)

train\_features = tf2.fit\_transform(train\_words\_list)

print(train\_features)

print('不重复的词:',len(tf2.get\_feature\_names()))

print('每个单词的ID:', tf2.vocabulary\_)

print('每个单词的tfidf值:', train\_features.toarray())

\# 上面fit过了，这里transform

test\_features = tf2.transform(test\_words\_list)

\# 多项式贝叶斯分类器

from sklearn.naive\_bayes import MultinomialNB

clf = MultinomialNB(alpha=0.001).fit(train\_features, train\_labels)

predicted\_labels=clf.predict(test\_features)

\# 计算准确率

print('准确率为：', metrics.accuracy\_score(test\_labels, predicted\_labels))

\#准确率为：0.92

[890de4b03afb87f031aa834603a481e6.gif]: /images/20221023/0c189053c0f7458bb37644c3bf568af4.png

python朴素贝叶斯的文本分类_基于Python 朴素贝叶斯--文本分类

发表评论取消回复

还没有评论，来说两句吧...

相关阅读

相关朴素贝叶斯分类算法

相关朴素贝叶斯（naive bayes）分类

相关贝叶斯分类（轻松理解朴素贝叶斯与半朴素贝叶斯）

相关 python朴素贝叶斯的文本分类_基于Python 朴素贝叶斯--文本分类

相关朴素贝叶斯分类

相关文本分类-朴素贝叶斯分类器

相关朴素贝叶斯分类：使用案例

相关朴素贝叶斯实战-文本主题分类

相关给代码分类-朴素贝叶斯

相关 python3机器学习朴素贝叶斯分类