Python爬取糗事百科段子

超、凢脫俗 2022-10-23 12:42 204阅读 0赞

# Python爬取糗事百科段子 #

--------------------

糗事百科是一个原创的糗事笑话分享社区,糗百网友分享的搞笑段子，使用Python爬取段子  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ1MTc2NTQ4_size_16_color_FFFFFF_t_70]

翻页操作

http://www.qiushibaike.com/text/
    http://www.qiushibaike.com/text/page/2/?s=4964629
    http://www.qiushibaike.com/text/page/3/?s=4964629
    http://www.qiushibaike.com/text/page/4/?s=4964629

> 观察可知，我们只需要修改page/\{i\}参数即可

这里使用正则表达式来提取数据

import requests
    import re
    from fake_useragent import UserAgent
    headers= {
        'User-Agent':str(UserAgent().random)}
    
    def get_info(url):
        res = requests.get(url,headers=headers)
        ids = re.findall('<h2>(.*?)</h2>',res.text,re.S)
        levels = re.findall('<div class="articleGender \D+Icon">(.*?)</div>',res.text,re.S)
        contents = re.findall('<div class="content">.*?<span>(.*?)</span>',res.text,re.S)
        laughs = re.findall('<span class="stats-vote"><i class="number">(\d+)</i>',res.text,re.S)
        comments = re.findall('<i class="number">(\d+)</i> 评论',res.text,re.S)
        for id,level,sex,content,laugh,comment in zip(ids,levels,sexs,contents,laughs,comments):
            info = {
        
                'id':id,
                'level':level,
                'content':content,
                'laugh':laugh,
                'comment':comment
            }
            print(info)
    
    if __name__ == '__main__':
        urls = ['http://www.qiushibaike.com/text/page/{}/'.format(str(i)) for i in range(1,15)]
        for url in urls:
            get_info(url)

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ1MTc2NTQ4_size_16_color_FFFFFF_t_70]: /images/20221023/242b5e00051d4b7390c8abfa5ec23a85.png