应用scikit-learn做文本分类提feature tfidg共享词库分类聚类

浅浅的花香味﹌ 2022-09-25 00:23 178阅读 0赞

http://blog.csdn.net/abcjennifer/article/details/23615947/

文本挖掘的paper没找到统一的benchmark，只好自己跑程序，走过路过的前辈如果知道20newsgroups或者其它好用的公共数据集的分类（最好要所有类分类结果，全部或取部分特征无所谓）麻烦留言告知下现在的benchmark，万谢！

嗯，说正文。[20newsgroups官网][20newsgroups]上给出了3个数据集，这里我们用最原始的[20news-19997.tar.gz][]。

分为以下几个过程：

*  加载数据集
 *  提feature
 *  分类
    
     *  Naive Bayes
     *  KNN
     *  SVM

*  聚类

说明： [scipy官网][scipy] 上有参考，但是看着有点乱，而且有bug。本文中我们分块来看。

Environment： Python 2.7 + Scipy （scikit-learn）

**1.加载数据集**

从[20news-19997.tar.gz][]下载数据集，解压到scikit\_learn\_data文件夹下，加载数据，详见code注释。

**\[python\]** [ view plain][view plain] [ copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  \#first extract the 20 news\_group dataset to /scikit\_learn\_data
2.  from sklearn.datasets import fetch\_20newsgroups
3.  \#all categories
4.  \#newsgroup\_train = fetch\_20newsgroups(subset='train')
5.  \#part categories
6.  categories = \['comp.graphics',
7.  'comp.os.ms-windows.misc',
8.  'comp.sys.ibm.pc.hardware',
9.  'comp.sys.mac.hardware',
10. 'comp.windows.x'\];
11. newsgroup\_train = fetch\_20newsgroups(subset = 'train',categories = categories);

可以检验是否load好了：

**\[python\]** [ view plain][view plain] [ copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  \#print category names
2.  from pprint import pprint
3.  pprint(list(newsgroup\_train.target\_names))

结果：

\['comp.graphics',  
'comp.os.ms-windows.misc',  
'comp.sys.ibm.pc.hardware',  
'comp.sys.mac.hardware',  
'comp.windows.x'\]

**2. 提feature：**

刚才load进来的newsgroup\_train就是一篇篇document，我们要从中提取feature，即词频啊神马的，用fit\_transform

Method 1. HashingVectorizer，规定feature个数

**\[python\]** [ view plain][view plain] [ copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  \#newsgroup\_train.data is the original documents, but we need to extract the 
2.  \#feature vectors inorder to model the text data
3.  from sklearn.feature\_extraction.text import HashingVectorizer
4.  vectorizer = HashingVectorizer(stop\_words = 'english',non\_negative = True,
5.  n\_features = 10000)
6.  fea\_train = vectorizer.fit\_transform(newsgroup\_train.data)
7.  fea\_test = vectorizer.fit\_transform(newsgroups\_test.data);
8.  
9.  
10. \#return feature vector 'fea\_train' \[n\_samples,n\_features\]
11. print 'Size of fea\_train:' \+ repr(fea\_train.shape)
12. print 'Size of fea\_train:' \+ repr(fea\_test.shape)
13. \#11314 documents, 130107 vectors for all categories
14. print 'The average feature sparsity is \{0:.3f\}%'.format(
15. fea\_train.nnz/float(fea\_train.shape\[0\]\*fea\_train.shape\[1\])\*100);

结果：

Size of fea\_train:(2936, 10000)  
Size of fea\_train:(1955, 10000)  
The average feature sparsity is 1.002%

因为我们只取了10000个词，即10000维feature，稀疏度还不算低。而实际上用TfidfVectorizer统计可得到上万维的feature，我统计的全部样本是13w多维，就是一个相当稀疏的矩阵了。

\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*

上面代码注释说TF-IDF在train和test上提取的feature维度不同，那么怎么让它们相同呢？有两种方法：

Method 2.CountVectorizer+TfidfTransformer

让两个CountVectorizer共享vocabulary：

**\[python\]** [ view plain][view plain] [ copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  \#----------------------------------------------------
2.  \#method 1:CountVectorizer+TfidfTransformer
3.  print '\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\\nCountVectorizer+TfidfTransformer\\n\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*'
4.  from sklearn.feature\_extraction.text import CountVectorizer,TfidfTransformer
5.  count\_v1= CountVectorizer(stop\_words = 'english', max\_df = 0.5);
6.  counts\_train = count\_v1.fit\_transform(newsgroup\_train.data);
7.  print "the shape of train is "\+repr(counts\_train.shape)
8.  
9.  count\_v2 = CountVectorizer(vocabulary=count\_v1.vocabulary\_);
10. counts\_test = count\_v2.fit\_transform(newsgroups\_test.data);
11. print "the shape of test is "\+repr(counts\_test.shape)
12. 
13. tfidftransformer = TfidfTransformer();
14. 
15. tfidf\_train = tfidftransformer.fit(counts\_train).transform(counts\_train);
16. tfidf\_test = tfidftransformer.fit(counts\_test).transform(counts\_test);

结果：

\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*

CountVectorizer+TfidfTransformer  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
the shape of train is (2936, 66433)  
the shape of test is (1955, 66433)

Method 3. TfidfVectorizer

让两个TfidfVectorizer共享vocabulary：

**\[python\]** [ view plain][view plain] [ copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  \#method 2:TfidfVectorizer
2.  print '\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\\nTfidfVectorizer\\n\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*'
3.  from sklearn.feature\_extraction.text import TfidfVectorizer
4.  tv = TfidfVectorizer(sublinear\_tf = True,
5.  max\_df = 0.5,
6.  stop\_words = 'english');
7.  tfidf\_train\_2 = tv.fit\_transform(newsgroup\_train.data);
8.  tv2 = TfidfVectorizer(vocabulary = tv.vocabulary\_);
9.  tfidf\_test\_2 = tv2.fit\_transform(newsgroups\_test.data);
10. print "the shape of train is "\+repr(tfidf\_train\_2.shape)
11. print "the shape of test is "\+repr(tfidf\_test\_2.shape)
12. analyze = tv.build\_analyzer()
13. tv.get\_feature\_names()\#statistical features/terms

结果：

\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
TfidfVectorizer  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
the shape of train is (2936, 66433)  
the shape of test is (1955, 66433)

此外，还有sklearn里封装好的抓feature函数，fetch\_20newsgroups\_vectorized

Method 4. fetch\_20newsgroups\_vectorized

但是这种方法不能挑出几个类的feature，只能全部20个类的feature全部弄出来：

**\[python\]** [ view plain][view plain] [ copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  print '\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\\nfetch\_20newsgroups\_vectorized\\n\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*'
2.  from sklearn.datasets import fetch\_20newsgroups\_vectorized
3.  tfidf\_train\_3 = fetch\_20newsgroups\_vectorized(subset = 'train');
4.  tfidf\_test\_3 = fetch\_20newsgroups\_vectorized(subset = 'test');
5.  print "the shape of train is "\+repr(tfidf\_train\_3.data.shape)
6.  print "the shape of test is "\+repr(tfidf\_test\_3.data.shape)

结果：

\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
fetch\_20newsgroups\_vectorized  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
the shape of train is (11314, 130107)  
the shape of test is (7532, 130107)

**3. 分类**

**3.1 Multinomial Naive Bayes Classifier**

见代码&comment，不解释

**\[python\]** [ view plain][view plain] [ copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#
2.  \#Multinomial Naive Bayes Classifier
3.  print '\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\\nNaive Bayes\\n\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*'
4.  from sklearn.naive\_bayes import MultinomialNB
5.  from sklearn import metrics
6.  newsgroups\_test = fetch\_20newsgroups(subset = 'test',
7.  categories = categories);
8.  fea\_test = vectorizer.fit\_transform(newsgroups\_test.data);
9.  \#create the Multinomial Naive Bayesian Classifier
10. clf = MultinomialNB(alpha = 0.01)
11. clf.fit(fea\_train,newsgroup\_train.target);
12. pred = clf.predict(fea\_test);
13. calculate\_result(newsgroups\_test.target,pred);
14. \#notice here we can see that f1\_score is not equal to 2\*precision\*recall/(precision+recall)
15. \#because the m\_precision and m\_recall we get is averaged, however, metrics.f1\_score() calculates
16. \#weithed average, i.e., takes into the number of each class into consideration.

注意我最后的3行注释，为什么f1≠2\*（准确率\*召回率）/（准确率+召回率）

其中，函数calculate\_result计算f1：

**\[python\]** [ view plain][view plain] [ copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  def calculate\_result(actual,pred):
2.  m\_precision = metrics.precision\_score(actual,pred);
3.  m\_recall = metrics.recall\_score(actual,pred);
4.  print 'predict info:'
5.  print 'precision:\{0:.3f\}'.format(m\_precision)
6.  print 'recall:\{0:0.3f\}'.format(m\_recall);
7.  print 'f1-score:\{0:.3f\}'.format(metrics.f1\_score(actual,pred));
8.

**3.2 KNN：**

**\[python\]** [ view plain][view plain] [ copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#
2.  \#KNN Classifier
3.  from sklearn.neighbors import KNeighborsClassifier
4.  print '\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\\nKNN\\n\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*'
5.  knnclf = KNeighborsClassifier()\#default with k=5
6.  knnclf.fit(fea\_train,newsgroup\_train.target)
7.  pred = knnclf.predict(fea\_test);
8.  calculate\_result(newsgroups\_test.target,pred);

**3.3 SVM:**

**\[cpp\]** [ view plain][view plain] [ copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#
2.  \#SVM Classifier
3.  from sklearn.svm import SVC
4.  print '\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\\nSVM\\n\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*'
5.  svclf = SVC(kernel = 'linear')\#default with 'rbf'
6.  svclf.fit(fea\_train,newsgroup\_train.target)
7.  pred = svclf.predict(fea\_test);
8.  calculate\_result(newsgroups\_test.target,pred);

结果：

\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*

Naive Bayes  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
predict info:  
precision:0.764  
recall:0.759  
f1-score:0.760  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
KNN  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
predict info:  
precision:0.642  
recall:0.635  
f1-score:0.636  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
SVM  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
predict info:  
precision:0.777  
recall:0.774  
f1-score:0.774

**4. 聚类**

**\[cpp\]** [ view plain][view plain] [ copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#
2.  \#KMeans Cluster
3.  from sklearn.cluster import KMeans
4.  print '\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\\nKMeans\\n\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*'
5.  pred = KMeans(n\_clusters=5)
6.  pred.fit(fea\_test)
7.  calculate\_result(newsgroups\_test.target,pred.labels\_);

结果：

\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
KMeans  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
predict info:  
precision:0.264  
recall:0.226  
f1-score:0.213

本文全部代码下载：[在此][Link 1]

貌似准确率好低……那我们用全部特征吧……结果如下：

\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
Naive Bayes  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
predict info:  
precision:0.771  
recall:0.770  
f1-score:0.769  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
KNN  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
predict info:  
precision:0.652  
recall:0.645  
f1-score:0.645  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
SVM  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
predict info:  
precision:0.819  
recall:0.816  
f1-score:0.816  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
KMeans  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
predict info:  
precision:0.289  
recall:0.313  
f1-score:0.266

[20newsgroups]: http://qwone.com/%7Ejason/20Newsgroups/
[20news-19997.tar.gz]: http://qwone.com/%7Ejason/20Newsgroups/20news-19997.tar.gz
[scipy]: http://scikit-learn.org/stable/auto_examples/document_classification_20newsgroups.html
[view plain]: http://blog.csdn.net/abcjennifer/article/details/23615947/#
[CODE]: https://code.csdn.net/assets/CODE_ico.png
[CODE_CODE]: https://code.csdn.net/snippets/290287
[ico_fork.svg]: https://code.csdn.net/assets/ico_fork.svg
[ico_fork.svg 1]: https://code.csdn.net/snippets/290287/fork
[Link 1]: http://download.csdn.net/detail/abcjennifer/7187999