【备忘录】词性标注工具调研

た入场券 2023-01-21 05:24 90阅读 0赞

## NLTK ##

# NLTK分词
    words = tokenize.word_tokenize(result, language='portuguese')
    # NLTK词性标注
    words_tagger = nltk.pos_tag(words, lang='pt')

![在这里插入图片描述][2021051417065075.png]nltk只支持**英语、俄语**的词性标注

## spacy ##

在下载和使用上废了一些功夫，特在此记录下install流程。

pip install -U pip setuptools wheel
    pip install -U spacy
    python -m spacy download pt_core_news_sm

然而报错：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpX2ppYW95YW5n_size_16_color_FFFFFF_t_70]  
这是因为在国内的网络问题。只好自行下载pt\_core\_news\_sm  
[下载地址][Link 1]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpX2ppYW95YW5n_size_16_color_FFFFFF_t_70 1]

pip install pt_core_news_sm-3.0.0.tar.gz

下载完成！

### 使用 ###

import spacy
    
    nlp = spacy.load("en_core_web_sm")
    doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
    
    for token in doc:
        print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,
                token.shape_, token.is_alpha, token.is_stop)

实现了词性标注。pos\_就是对应的词性。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpX2ppYW95YW5n_size_16_color_FFFFFF_t_70 2]  
词性的定义如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpX2ppYW95YW5n_size_16_color_FFFFFF_t_70 3]

[2021051417065075.png]: /images/20221020/d86e3631aab04968af3e39ddf29551c8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpX2ppYW95YW5n_size_16_color_FFFFFF_t_70]: /images/20221020/9df327738a5a49c4b239292529cef3b4.png
[Link 1]: https://github.com/explosion/spacy-models/releases/tag/pt_core_news_sm-3.0.0
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpX2ppYW95YW5n_size_16_color_FFFFFF_t_70 1]: /images/20221020/fe8d9d7e2c764f8c91e1a89904049029.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpX2ppYW95YW5n_size_16_color_FFFFFF_t_70 2]: /images/20221020/4eb79850416b4b8bbb661cc56a296192.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpX2ppYW95YW5n_size_16_color_FFFFFF_t_70 3]: /images/20221020/1b649faa92764c37a1e0820099acd679.png