Kashgari:一个方便快捷的命名实体识别、文本分类工具包

太过爱你忘了你带给我的痛 2022-12-10 04:58 132阅读 0赞

## **Kashgari:一个方便快捷的命名实体识别、文本分类工具包** ##

最近小编做一个项目时发现的一个很好用的工具包（很适合小白直接上手）  
**Kashgari安装**  
（1）利用pip直接安装，这里为了方便小伙伴更加快捷的安装，使用了清华镜像源。

pip install kashgari -i https://pypi.tuna.tsinghua.edu.cn/simple/

（2）如果装了Anaconda环境就用conda安装（用pip也行），安装速度慢的话也可以先配置为清华镜像源。

conda install kashgari

**命名实体识别任务（NER）**  
模型目录:我的这个小demo使用了Bert预训练模型，也就是第一个文件夹，train\_data里为数据，格式为。  
四 B-Region  
川 I-Region  
区 B-Organization  
域 I-Organization  
内 I-Organization  
医 I-Organization  
院 I-Organization  
的 O  
骨 B-Department  
科 I-Department  
收 B-Result  
入 I-Result  
多 I-Result  
少 I-Result  
？ O

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI3NDkyNzM1_size_16_color_FFFFFF_t_70_pic_center]  
**data\_load.py**

from typing import Tuple, List
    
    
    # 定义一个读取数据的类
    class DataReader(object):
        def read_file(file_path: str,
                      text_index: int = 0,
                      label_index: int = 1) -> Tuple[List[List[str]], List[List[str]]]:
            """
            根据文件路径读取训练数据、测试数据以及验证数据的text和label
            """
            x_data, y_data = [], []
            with open(file_path, 'r', encoding='utf-8') as f:
                lines = f.read().splitlines()  # 以splitlines的方式获取数据
                x, y = [], []
                for line in lines:
                    rows = line.split(' ')  # 以空格进行切分
                    if len(rows) == 1:
                        x_data.append(x)
                        y_data.append(y)
                        x = []
                        y = []
                    else:
                        x.append(rows[text_index])
                        y.append(rows[label_index])
            return x_data, y_data

**train\_model.py**  
这里可选的模型有CNN\_LSTM\_Model、BiLSTM\_Model、BiLSTM\_CRF\_Model、BiGRU\_Model、BiGRU\_CRF\_Model，小编这里使用的是BiLSTM\_CRF\_Model。

import kashgari
    from kashgari import utils
    from kashgari.embeddings import BERTEmbedding,BERTEmbeddingV2
    from kashgari.tasks.labeling import BiLSTM_CRF_Model
    from data_load import DataReader
    
    kashgari.config.use_cudnn_cell = False
    
    
    #加载训练数据、验证数据以及测试数据
    
    train_x, train_y = DataReader.read_file('./train_data/train.txt')
    test_x, test_y = DataReader.read_file('./train_data/test.txt')
    valid_x, valid_y = DataReader.read_file('./train_data/new_val.txt')
    
    print("train data count: {len(train_x)}")
    print("validate data count: {len(valid_x)}")
    print("test data count: {len(test_x)}")
    
    #利用kashgari创建Bert Embedding
    
    bert_embed = BERTEmbedding('./chinese_wwm_ext_L-12_H-768_A-12',
                               task=kashgari.LABELING,
                               sequence_length=100)
    
    
    #调用BiLSTM_CRF_Model模型并进行训练
    
    model = BiLSTM_CRF_Model(bert_embed)
    
    model.fit(x_train=train_x, y_train=train_y,
              x_validate=valid_x, y_validate=valid_y,
              batch_size=128, epochs=2)
    model.save('./model/NER.h5')
    
    model.evaluate(test_x,test_y)

**predict.py**

# -*- coding: utf-8 -*-
    import kashgari
    import re
    import warnings
    warnings.filterwarnings('ignore')
    #模型加载
    loaded_model = kashgari.utils.load_model('./model/NER.h5')
    
    def cut_text(text):
        textArr = []
        textArr.append(text)
        return textArr
    
    #标签抽取
    def extract_labels(text, ners):
        ner_reg_list = []
        if ners:
            new_ners = []
            for ner in ners:
                new_ners += ner
            for word, tag in zip([char for char in text], new_ners):
                if tag != 'O':
                    ner_reg_list.append((word, tag))
    
        # 输出模型的NER识别结果
        labels = {}
        if ner_reg_list:
            for i, item in enumerate(ner_reg_list):
                if item[1].startswith('B'):
                    label = ""
                    end = i + 1
                    while end <= len(ner_reg_list) - 1 and ner_reg_list[end][1].startswith('I'):
                        end += 1
    
                    ner_type = item[1].split('-')[1]
    
                    if ner_type not in labels.keys():
                        labels[ner_type] = []
    
                    label += ''.join([item[0] for item in ner_reg_list[i:end]])
                    labels[ner_type].append(label)
    
        return labels
    
    
    while True:
        text_input = input('sentence: ')
        print(text_input)
        texts = cut_text(text_input)
        ners = loaded_model.predict([[char for char in text] for text in texts])
        # print(ners)
        labels = extract_labels(text_input, ners)
        for key,value in labels.items():
            new_value = ''.join(value)
            labels[key] = new_value
        print(labels)

**文本分类任务**  
别的就不说了，直接上代码。  
数据格式为：  
【文本+空格+标签】---->查询昨天下午店铺的收入情况？ 收入

**train.py**

import kashgari
    from kashgari.tasks.classification import CNN_GRU_Model
    from kashgari.callbacks import KashgariModelCheckpoint
    #获取数据集
    def load_data(path):
        with open(path, 'r', encoding='utf-8') as f:
            content = [_.strip() for _ in f.readlines() if _.strip()]
    
        x, y = [], []
        for line in content:
            text, label = line.split(' ')
            y.append(label)
            x.append([_ for _ in text])
    
        return x, y
    
    #获取数据
    train_x, train_y = load_data('./data/train_new.txt')
    valid_x, valid_y = load_data('./data/valid_new.txt')
    test_x, test_y = load_data('./data/test_new.txt')
    # print('train_x:',len(train_x))
    # print('train_y:',len(train_y))
    
    filepath = './model/model'
    epoch = 5
    # 训练模型
    model = CNN_GRU_Model()
    
    
    checkpoint = KashgariModelCheckpoint(filepath=filepath,
                                         monitor='val_loss',save_weights_only=False,
                                         verbose=0,save_best_only=True,
                                         mode='min',save_freq='epoch',period=0)
    new_checkpoint = [checkpoint]
    
    # model = kashgari.utils.load_model('./models/model.h5')
    # 若成功加载前面保存的参数，输出下列信息
    print("checkpoint_loaded")
    
    model.fit(train_x, train_y, valid_x, valid_y, batch_size=128,
                  epochs=epoch)
    
    
    model.save('./model/model')

**predict.py**

import kashgari
    import numpy as np
    #加载模型
    
    load_model = kashgari.utils.load_model('./models/classification_model')
    
    while True:
        text = input('Text:')
        x = [[_ for _ in text]]
        pro_label = load_model.predict(x)
        pro = np.max(pro_label[0])
        label = pro_label[1][0]
        print(float(pro),type(pro))
        print(label,type(label))

这个工具包的简单使用方法就到这，感兴趣的小伙伴可以去看一下作者在[GitHub][]上的源码。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI3NDkyNzM1_size_16_color_FFFFFF_t_70_pic_center]: /images/20221123/cb09dc72bf644e48ae05edca40e62d77.png
[GitHub]: https://github.com/BrikerMan/Kashgari