Python语料预处理操作，并保存中间结果

比眉伴天荒 2023-02-20 13:57 8阅读 0赞

#### doc1.txt ####

原始语料：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw_size_16_color_FFFFFF_t_70_pic_center]

#### doc2.txt ####

预处理后：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw_size_16_color_FFFFFF_t_70_pic_center 1]

#### doc3.txt ####

结果为：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw_size_16_color_FFFFFF_t_70_pic_center 2]

--------------------

代码为：

import re
    def yuchuli():
    
        filename = './中间结果/doc1.txt'
        f = open(filename, 'r', encoding='utf-8')
        context = f.read()
    
        pattern = ",|\\.|\\?|!|:|;|~|，|：|。|！|；|？| "
        sentence = [i.replace('\n', '##').strip() for i in re.split(pattern, context)]
    
        g = open('./中间结果/doc2.txt', 'w', encoding='utf-8')
    
        for word in sentence:
            a = str(word)
            a = a.replace('##', '\r\n')
            print(a, file=g)
    
        f.close()
        g.close()
    
        k = open('./中间结果/doc2.txt', 'r', encoding='utf-8')
        out = open('./中间结果/doc3.txt', 'w', encoding='utf-8')
    
        for eachline in k.readlines():
            if len(eachline) > 5:
                out.writelines(eachline)
        k.close()
        out.close()
        
    
    yuchuli()

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw_size_16_color_FFFFFF_t_70_pic_center]: https://img-blog.csdnimg.cn/20200627224242296.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw_size_16_color_FFFFFF_t_70_pic_center 1]: https://img-blog.csdnimg.cn/20200627224412626.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw_size_16_color_FFFFFF_t_70_pic_center 2]: https://img-blog.csdnimg.cn/20200627224553669.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw==,size_16,color_FFFFFF,t_70#pic_center