英文原始文本的读取与处理

- 日理万妓 2021-09-17 05:30 409阅读 0赞

声明：代码的运行环境为Python3。Python3与Python2在一些细节上会有所不同，希望广大读者注意。本博客以代码为主，代码中会有详细的注释。相关文章将会发布在我的个人博客专栏[《Python自然语言处理》][Python]，欢迎大家关注。

**一、在线下载txt文档**

import nltk, re, pprint
    from nltk import word_tokenize
    from urllib.request import urlopen
    
    # txt在线文档下载
    url = "http://www.gutenberg.org/files/2553/2553.txt"  # 链接路径
    response = urlopen(url)  # 下载路径中的文件
    raw = response.read().decode('utf8')  # 读取文件并进行解码操作
    print(type(raw))  # 打印出文本的类型
    print(len(raw))  # 打印出文本的长度
    print(raw[:10])  # 打印出文本前10个字符
    
    <class 'str'>
    653642
    The Projec

#分词
    tokens = word_tokenize(raw)
    print(type(tokens))
    print(len(tokens))
    print(tokens[:10])
    
    <class 'list'>
    131869
    ['The', 'Project', 'Gutenberg', 'EBook', 'of', 'Jeanne', "d'Arc", ',', 'by', 'Mrs.']

#创建text
    text = nltk.Text(tokens)
    print(type(text))
    print(text[1024:1062])
    print(text.collocations())  # 查看常用的搭配
    
    <class 'nltk.text.Text'>
    ['century', '.', 'A', 'strong', 'and', 'splendid', 'kingdom', ',', 'to', 'which', 'in', 'early', 'ages', 'one', 'great', 'man', 'had', 'given', 'the', 'force', 'and', 'supremacy', 'of', 'a', 'united', 'nation', ',', 'had', 'fallen', 'into', 'a', 'disintegration', 'which', 'seems', 'almost', 'incredible', 'when', 'regarded']
    Project Gutenberg-tm; St. Catherine; would seem; Project Gutenberg;
    St. Margaret; Frere Isambard; St. Michael; Literary Archive; St.
    Denis; Gutenberg-tm electronic; Archive Foundation; electronic works;
    Gutenberg Literary; fifteenth century; United States; Church militant;
    Holy Father; set forth; fifteen days; Jacques d'Arc
    None

print(raw.find("Frere Isambard"))
    
    415623
    
    
    print(raw.rfind("fifteen days"))
    
    502248
    
    
    raw = raw[415623:502248]
    raw
    
    Out[14]: 'Frere Isambard, who was\r\nthe person in question, speaks at a later period he tells us that "the\r\nquestions put to Jeanne were too difficult, subtle, and dangerous, so\r\nthat the great clerks and learned men who were present scarcely would\r\nhave known

**二、HTML格式下载文本**

# HTML下载
    url = "http://news.bbc.co.uk/2/hi/health/2284783.stm"
    html = urlopen(url).read().decode('utf8')
    
    #HTML解析
    from bs4 import BeautifulSoup
    raw = BeautifulSoup(html,'lxml').get_text()  # 使用lxml解析器，get_text()获取到所有的文本信息
    tokens = word_tokenize(raw)  # 分词
    
    bs = BeautifulSoup(html,'lxml')
    print(bs.find("div",class_='bodytext').get_text())  # 找到第一个属性为bodytext的div标签，bs.find_all()找到所有的属性为bodytext的div标签
    
    #过滤无关内容
    tokens = tokens[110:390]
    text = nltk.Text(tokens)

**三、读取本地文件**

raw = open('F:/document.txt').read()
    print(type(raw))
    tokens = word_tokenize(raw)
    print(type(tokens))
    words = [w.lower() for w in tokens]
    print(type(words))
    vocab = sorted(set(words))
    print(type(vocab))
    
    <class 'str'>
    <class 'list'>
    <class 'list'>
    <class 'list'>

vocab.append('blog')
    # raw.append('blog')  # 追加的时候一定要注意类型
    
    
    # query = 'Who knows?'
    # beatles = ['john', 'paul', 'george', 'ringo']
    # query + beatles  # 连接的时候也需要注意类型

字符串常用的方法：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dka3l4eTIwMTM_size_16_color_FFFFFF_t_70][]

**四、Unicode字符**

path = nltk.data.find('corpora/unicode_samples/polish-lat2.txt')  # 波兰语的文件
    f= path.open(encoding='latin2')  # 采用拉丁语的方式打开
    for line in f:
        line = line.strip()
        print(line)
        
    Pruska Biblioteka Państwowa. Jej dawne zbiory znane pod nazwą
    "Berlinka" to skarb kultury i sztuki niemieckiej. Przewiezione przez
    Niemców pod koniec II wojny światowej na Dolny Śląsk, zostały
    odnalezione po 1945 r. na terytorium Polski. Trafiły do Biblioteki
    Jagiellońskiej w Krakowie, obejmują ponad 500 tys. zabytkowych
    archiwaliów, m.in. manuskrypty Goethego, Mozarta, Beethovena, Bacha.

ord('a')  # 查找字母的整数序号
    
    Out[21]: 97

a=u'\u0061' # 表示Unicode编码
    print(a)
    
    a

**五、正则表达式**

常用的正则表达式符号：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dka3l4eTIwMTM_size_16_color_FFFFFF_t_70 1][]

# 正则表达式
    import re  # 导入正则表达式的包
    wordlist = [w for w in nltk.corpus.words.words('en') if w.islower()]
    #查找ed结尾的词汇
    print([w for w in wordlist if re.search('ed$', w)])
    
    ['abaissed', 'abandoned', 'abased', 'abashed', 'abatised', 'abed', 'aborted', 'abridged', 'abscessed', 'absconded', 'absorbed', 'abstracted', 'abstricted', 'accelerated', 'accepted', 'accidented', ...]

#字谜：8个字母，第3个字母是j，第6个字母是t
    print([w for w in wordlist if re.search('^..j..t..$', w)])
    ['abjectly', 'adjuster', 'dejected', 'dejectly', 'injector', 'majestic', 'objectee', 'objector', 'rejecter', 'rejector', 'unjilted', 'unjolted', 'unjustly']

#9宫格输入判断
    print([w for w in wordlist if re.search('^[ghi][mno][jlk][def]$', w)])
    ['gold', 'golf', 'hold', 'hole']

#正则表达式中的+
    chat_words = sorted(set(w for w in nltk.corpus.nps_chat.words()))
    print([w for w in chat_words if re.search('^m+i+n+e+$', w)])
    ['miiiiiiiiiiiiinnnnnnnnnnneeeeeeeeee', 'miiiiiinnnnnnnnnneeeeeeee', 'mine', 'mmmmmmmmiiiiiiiiinnnnnnnnneeeeeeee']

# 提取字符块
    word = 'supercalifragilisticexpialidocious'
    print(re.findall(r'[aeiou]', word))  # 匹配正则表达式的时候通常在字符前面加上r
    print(len(re.findall(r'[aeiou]', word)))
    
    ['u', 'e', 'a', 'i', 'a', 'i', 'i', 'i', 'e', 'i', 'a', 'i', 'o', 'i', 'o', 'u']
    16

regexp = r'^[AEIOUaeiou]+|[AEIOUaeiou]+$|[^AEIOUaeiou]'
    def compress(word):
        pieces = re.findall(regexp, word)
        return ''.join(pieces)
    english_udhr = nltk.corpus.udhr.words('English-Latin1')
    print(nltk.tokenwrap(compress(w) for w in english_udhr[:75]))
    
    Unvrsl Dclrtn of Hmn Rghts Prmble Whrs rcgntn of the inhrnt dgnty and
    of the eql and inlnble rghts of all mmbrs of the hmn fmly is the fndtn
    of frdm , jstce and pce in the wrld , Whrs dsrgrd and cntmpt fr hmn
    rghts hve rsltd in brbrs acts whch hve outrgd the cnscnce of mnknd ,
    and the advnt of a wrld in whch hmn bngs shll enjy frdm of spch and

rotokas_words = nltk.corpus.toolbox.words('rotokas.dic')
    cvs = [cv for w in rotokas_words for cv in re.findall(r'[ptksvr][aeiou]', w)]
    cfd = nltk.ConditionalFreqDist(cvs)
    cfd.tabulate()  # 运用正则表达式获取交叉表
    
        a   e   i   o   u 
    k 418 148  94 420 173 
    p  83  31 105  34  51 
    r 187  63  84  89  79 
    s   0   0 100   2   1 
    t  47   8   0 148  37 
    v  93  27 105  48  49

re.findall(r'^.*(ing|ly|ed|ious|ies|ive|es|s|ment)$', 'processing')  # 提取括号里面的内容
    
    Out[41]: ['ing']
    
    
    re.findall(r'^.*(?:ing|ly|ed|ious|ies|ive|es|s|ment)$', 'processing')  # 匹配整个单词
    
    Out[42]: ['processing']
    
    
    re.findall(r'^(.*)(ing|ly|ed|ious|ies|ive|es|s|ment)$', 'processing')  # 分成单词和后缀
    
    Out[43]: [('process', 'ing')]
    
    
    re.findall(r'^(.*)(ing|ly|ed|ious|ies|ive|es|s|ment)$', 'processes')  # 如果以es或s结尾，可能会出现问题
    
    Out[44]: [('processe', 's')]
    
    
    re.findall(r'^(.*?)(ing|ly|ed|ious|ies|ive|es|s|ment)$', 'processes')  # 加?，匹配出完整的后缀和词语
    
    Out[45]: [('process', 'es')]

def stems(word):  # 去除文本中的后缀
        regexp = r'^(.*?)(ing|ly|ed|ious|ies|ive|es|s|ment)?$'
        stem, suffix = re.findall(regexp, word)[0]
        return stem
    raw = """DENNIS: Listen, strange women lying in ponds distributing swords
        is no basis for a system of government.  Supreme executive power derives from
        a mandate from the masses, not from some farcical aquatic ceremony."""
    tokens = word_tokenize(raw)
    [stems(t) for t in tokens]
    
    Out[47]: 
    ['DENNIS',
     ':',
     'Listen',
     ',',
     'strange',
     'women',
     'ly',
     'in',
     'pond',
     'distribut',
     'sword',
     'i',
     'no',
     'basi',
     'for',
     'a',
     'system',
     'of',
     'govern',
     '.',
     'Supreme',
     'execut',
     'power',
     'deriv',
     'from',
     'a',
     'mandate',
     'from',
     'the',
     'mass',
     ',',
     'not',
     'from',
     'some',
     'farcical',
     'aquatic',
     'ceremony',
     '.']

**六、规范化文本**

# 词干提取器(两种不同的方法)
    porter = nltk.PorterStemmer()
    lancaster = nltk.LancasterStemmer()
    [porter.stem(t) for t in tokens]
    
    Out[54]: 
    ['denni',
     ':',
     'listen',
     ',',
     'strang',
     'women',
     'lie',
     'in',
     'pond',
     'distribut',
     'sword',
     'is',
     'no',
     'basi',
     'for',
     'a',
     'system',
     'of',
     'govern',
     '.',
     'suprem',
     'execut',
     'power',
     'deriv',
     'from',
     'a',
     'mandat',
     'from',
     'the',
     'mass',
     ',',
     'not',
     'from',
     'some',
     'farcic',
     'aquat',
     'ceremoni',
     '.']

[lancaster.stem(t) for t in tokens]
    Out[55]: 
    ['den',
     ':',
     'list',
     ',',
     'strange',
     'wom',
     'lying',
     'in',
     'pond',
     'distribut',
     'sword',
     'is',
     'no',
     'bas',
     'for',
     'a',
     'system',
     'of',
     'govern',
     '.',
     'suprem',
     'execut',
     'pow',
     'der',
     'from',
     'a',
     'mand',
     'from',
     'the',
     'mass',
     ',',
     'not',
     'from',
     'som',
     'farc',
     'aqu',
     'ceremony',
     '.']

wnl = nltk.WordNetLemmatizer()  # 词性归并
    [wnl.lemmatize(t) for t in tokens]
    Out[56]: 
    ['DENNIS',
     ':',
     'Listen',
     ',',
     'strange',
     'woman',
     'lying',
     'in',
     'pond',
     'distributing',
     'sword',
     'is',
     'no',
     'basis',
     'for',
     'a',
     'system',
     'of',
     'government',
     '.',
     'Supreme',
     'executive',
     'power',
     'derives',
     'from',
     'a',
     'mandate',
     'from',
     'the',
     'mass',
     ',',
     'not',
     'from',
     'some',
     'farcical',
     'aquatic',
     'ceremony',
     '.']

**七、分割**

# 例子
    text = nltk.corpus.gutenberg.raw('chesterton-thursday.txt')
    sents = nltk.sent_tokenize(text)
    pprint.pprint(sents[79:89])
    
    ['"Nonsense!"',
     'said Gregory, who was very rational when anyone else\nattempted paradox.',
     '"Why do all the clerks and navvies in the\n'
     'railway trains look so sad and tired, so very sad and tired?',
     'I will\ntell you.',
     'It is because they know that the train is going right.',
     'It\n'
     'is because they know that whatever place they have taken a ticket\n'
     'for that place they will reach.',
     'It is because after they have\n'
     'passed Sloane Square they know that the next station must be\n'
     'Victoria, and nothing but Victoria.',
     'Oh, their wild rapture!',
     'oh,\n'
     'their eyes like stars and their souls again in Eden, if the next\n'
     'station were unaccountably Baker Street!"',
     '"It is you who are unpoetical," replied the poet Syme.']

def segment(text, segs):  # 分词例子
        words = []
        last = 0
        for i in range(len(segs)):
            if segs[i] == '1':
                words.append(text[last:i+1])
                last = i+1
        words.append(text[last:])
        return words
    text = "doyouseethekittyseethedoggydoyoulikethekittylikethedoggy"
    seg1 = "0000000000000001000000000010000000000000000100000000000"
    seg2 = "0100100100100001001001000010100100010010000100010010000"
    segment(text, seg1)
    
    Out[63]: ['doyouseethekitty', 'seethedoggy', 'doyoulikethekitty', 'likethedoggy']

segment(text, seg2)
    
    Out[64]: 
    ['do',
     'you',
     'see',
     'the',
     'kitty',
     'see',
     'the',
     'doggy',
     'do',
     'you',
     'like',
     'the',
     'kitty',
     'like',
     'the',
     'doggy']

def evaluate(text, segs):  # 分词例子
        words = segment(text, segs)
        text_size = len(words)
        lexicon_size = len(' '.join(list(set(words))))
        return text_size + lexicon_size
    text = "doyouseethekittyseethedoggydoyoulikethekittylikethedoggy"
    seg1 = "0000000000000001000000000010000000000000000100000000000"
    seg2 = "0100100100100001001001000010100100010010000100010010000"
    seg3 = "0000100100000011001000000110000100010000001100010000001"
    print(evaluate(text, seg3))
    
    46
    
    
    print(evaluate(text, seg2))
    
    47
    
    
    print(evaluate(text, seg1))
    
    63

# 模拟退火算法
    from random import randint
    def flip(segs, pos):
        return segs[:pos] + str(1-int(segs[pos])) + segs[pos+1:]
    def flip_n(segs, n):
        for i in range(n):
            segs = flip(segs, randint(0, len(segs)-1))
        return segs
    def anneal(text, segs, iterations, cooling_rate):
        temperature = float(len(segs))
        while temperature > 0.5:
            best_segs, best = segs, evaluate(text, segs)
            for i in range(iterations):
                guess = flip_n(segs, int(round(temperature,0)))
                score = evaluate(text, guess)
                if score < best:
                    best, best_segs = score, guess
            score, segs = best, best_segs
            temperature = temperature / cooling_rate
            print(evaluate(text, segs), segment(text, segs))
        print()
        return segs
    text = "doyouseethekittyseethedoggydoyoulikethekittylikethedoggy"
    seg1 = "0000000000000001000000000010000000000000000100000000000"
    anneal(text, seg1, 5000, 1.2)
    
    63 ['doyouseethekitty', 'seethedoggy', 'doyoulikethekitty', 'likethedoggy']
    63 ['doyouseethekitty', 'seethedoggy', 'doyoulikethekitty', 'likethedoggy']
    63 ['doyouseethekitty', 'seethedoggy', 'doyoulikethekitty', 'likethedoggy']
    63 ['doyouseethekitty', 'seethedoggy', 'doyoulikethekitty', 'likethedoggy']
    63 ['doyouseethekitty', 'seethedoggy', 'doyoulikethekitty', 'likethedoggy']
    63 ['doyouseethekitty', 'seethedoggy', 'doyoulikethekitty', 'likethedoggy']
    63 ['doyouseethekitty', 'seethedoggy', 'doyoulikethekitty', 'likethedoggy']
    63 ['doyouseethekitty', 'seethedoggy', 'doyoulikethekitty', 'likethedoggy']
    63 ['doyouseethekitty', 'seethedoggy', 'doyoulikethekitty', 'likethedoggy']
    61 ['doyousee', 'thekitty', 'seeth', 'edoggy', 'doyou', 'l', 'i', 'ke', 'thekitty', 'liketh', 'edoggy']
    57 ['doyou', 'see', 'thekitty', 'se', 'eth', 'edoggy', 'doyou', 'l', 'ike', 'thekitty', 'liketh', 'edoggy']
    54 ['doyou', 'see', 'thekitty', 'se', 'eth', 'edoggy', 'doyou', 'l', 'ik', 'e', 'thekitty', 'l', 'ik', 'eth', 'edoggy']
    52 ['doyou', 'se', 'e', 'thekitty', 'se', 'eth', 'edoggy', 'doyou', 'like', 'thekitty', 'lik', 'eth', 'edoggy']
    52 ['doyou', 'se', 'e', 'thekitty', 'se', 'eth', 'edoggy', 'doyou', 'like', 'thekitty', 'lik', 'eth', 'edoggy']
    51 ['doyou', 'se', 'e', 'thekitty', 'se', 'eth', 'edoggy', 'doyou', 'like', 'thekitty', 'like', 'th', 'edoggy']
    51 ['doyou', 'se', 'e', 'thekitty', 'se', 'eth', 'edoggy', 'doyou', 'like', 'thekitty', 'like', 'th', 'edoggy']
    48 ['doyou', 'se', 'e', 'thekitty', 'se', 'e', 'th', 'edoggy', 'doyou', 'like', 'thekitty', 'like', 'th', 'edoggy']
    45 ['doyou', 'see', 'thekitty', 'see', 'th', 'edoggy', 'doyou', 'like', 'thekitty', 'like', 'th', 'edoggy']
    42 ['doyou', 'see', 'thekitty', 'see', 'thedoggy', 'doyou', 'like', 'thekitty', 'like', 'thedoggy']
    42 ['doyou', 'see', 'thekitty', 'see', 'thedoggy', 'doyou', 'like', 'thekitty', 'like', 'thedoggy']
    42 ['doyou', 'see', 'thekitty', 'see', 'thedoggy', 'doyou', 'like', 'thekitty', 'like', 'thedoggy']
    42 ['doyou', 'see', 'thekitty', 'see', 'thedoggy', 'doyou', 'like', 'thekitty', 'like', 'thedoggy']
    42 ['doyou', 'see', 'thekitty', 'see', 'thedoggy', 'doyou', 'like', 'thekitty', 'like', 'thedoggy']
    42 ['doyou', 'see', 'thekitty', 'see', 'thedoggy', 'doyou', 'like', 'thekitty', 'like', 'thedoggy']
    42 ['doyou', 'see', 'thekitty', 'see', 'thedoggy', 'doyou', 'like', 'thekitty', 'like', 'thedoggy']
    42 ['doyou', 'see', 'thekitty', 'see', 'thedoggy', 'doyou', 'like', 'thekitty', 'like', 'thedoggy']
    Out[68]: '0000100100000001001000000010000100010000000100010000000'

**八、链表与字符串**

#链表到字符串
    silly = ['We', 'called', 'him', 'Tortoise', 'because', 'he', 'taught', 'us', '.']
    print(' '.join(silly))
    
    We called him Tortoise because he taught us .

fdist = nltk.FreqDist(['dog', 'cat', 'dog', 'cat', 'dog', 'snake', 'dog', 'cat'])
    for word in sorted(fdist):
        print(word, '->', fdist[word], '; ')
        
    cat -> 3 ; 
    dog -> 4 ; 
    snake -> 1 ;

for word in sorted(fdist):
        print('%s->%d;' % (word, fdist[word]))
        
    cat->3;
    dog->4;
    snake->1;

# 例子：定义并使用交叉表
    def tabulate(cfdist, words, categories):
        print('%-16s' % 'Category')
        for word in words: # column headings
            print('%6s' % word)
        print()
        for category in categories:
            print('%-16s' % category) # row heading
            for word in words: # for each word
                print('%6d' % cfdist[category][word]) # print table cell
            print()                                              # end the row
    from nltk.corpus import brown
    cfd = nltk.ConditionalFreqDist(
        (genre, word)
        for genre in brown.categories()
        for word in brown.words(categories=genre))
    genres = ['news', 'religion', 'hobbies', 'science_fiction', 'romance', 'humor']
    modals = ['can', 'could', 'may', 'might', 'must', 'will']
    tabulate(cfd, modals, genres)

[Python]: https://blog.csdn.net/gdkyxy2013/column/info/30938
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dka3l4eTIwMTM_size_16_color_FFFFFF_t_70]: /images/20210726/6e1f2cfbf1214e0b87a0bef01e69cf34.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dka3l4eTIwMTM_size_16_color_FFFFFF_t_70 1]: /images/20210726/66d24856035e46758fb5495f95fb7996.png