sklearn CountVectorizer\TfidfVectorizer\TfidfTransformer函数详解

妖狐艹你老母 2022-06-03 03:20 176阅读 0赞

*  sklearn CountVectorizer函数详解

from sklearn.feature_extraction.text import CountVectorizer
    
    texts=["dog cat fish","dog cat cat","fish bird", 'bird']
    cv = CountVectorizer()
    cv_fit=cv.fit_transform(texts)
    
    print(cv.get_feature_names())
    print(cv_fit.toarray())
    print(cv_fit)

*  返回的结果为稀疏矩阵

['bird', 'cat', 'dog', 'fish']
    [[0 1 1 1] [0 2 1 0] [1 0 0 1] [1 0 0 0]]
      (0, 3)    1
      (0, 1)    1
      (0, 2)    1
      (1, 1)    2
      (1, 2)    1
      (2, 0)    1
      (2, 3)    1
      (3, 0)    1

*  sklearn TfidfTransformer函数详解

from sklearn.feature_extraction.text import CountVectorizer
    from sklearn.feature_extraction.text import TfidfTransformer
    
    texts=["dog cat fish","dog cat cat","dog fish", 'dog pig pig 中国']
    cv = CountVectorizer()
    cv_fit=cv.fit_transform(texts)
    
    transformer = TfidfTransformer()
    tfidf = transformer.fit_transform(cv_fit)
    tfidf.toarray()

array([[ 0.64043405, 0.42389674, 0.64043405, 0. , 0. ], [ 0.94936136, 0.31418628, 0. , 0. , 0. ], [ 0. , 0.55193942, 0.83388421, 0. , 0. ], [ 0. , 0.22726773, 0. , 0.8710221 , 0.43551105]])

*  sklearn TfidfVectorizer函数详解
 *  TfidfVectorizer函数的功能相当于下面这四行代码的功能，即CountVectorizer＋TfidfTransformer

cv = CountVectorizer()
    cv_fit=cv.fit_transform(texts)
    
    transformer = TfidfTransformer()
    tfidf = transformer.fit_transform(cv_fit)

*  上代码,TfidfVectorizer

from sklearn.feature_extraction.text import TfidfVectorizer
    
    tv = TfidfVectorizer(max_features=100,
                         ngram_range=(1, 1),
                         stop_words='english')
    X_description = tv.fit_transform(texts)
    print(X_description.toarray())

[[ 0.64043405 0.42389674 0.64043405 0. 0. ] [ 0.94936136 0.31418628 0. 0. 0. ] [ 0. 0.55193942 0.83388421 0. 0. ] [ 0. 0.22726773 0. 0.8710221 0.43551105]]

*  可观察到输出的结果和上面的结果是一毛一样的。
 *  ngram\_range=(1, 1)也可以改为（２，３），这就是２-gram.
 *  stop\_words暂时只支持英文，即”english”