利用Python中的BeautifulSoup库爬取豆瓣读书中书本信息

淩亂°似流年 2022-09-16 14:21 162阅读 0赞

利用BeautifulSoup库，获取前250本图书的信息，需要爬取的信息包括书名、书名的URL链接、作者、出版社和出版时间、书本价格、评分和评论，把这些信息存到txt文件，要求将这些信息对齐，排列整齐 (我是刚学习网络爬虫，代码如有错误，望指正)  
网址为：https://book.douban.com/top250  
代码如下：

import requests
    from bs4 import BeautifulSoup
    import time
    headers={ "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36"}
    book_list=[]
    
    def book_info(url):
        book_data=requests.get(url,headers=headers)
        soup=BeautifulSoup(book_data.text,"lxml")
        book_is=soup.select("div.pl2 > a")
        lines=soup.select("p.pl")
        marks=soup.select("span.rating_nums")
        comtents=soup.select("span.inq")
        
        for book_i,line,mark,comtent in zip(book_is,lines,marks,comtents):
            line=line.get_text().split("/")
            data={ 
                "book_name":book_i.get_text().replace('\n','').replace(' ',''),
                "book_url":book_i['href'],
                "line":' '.join(line),
                "mark":mark.get_text(),
                "comtent":comtent.get_text()
                }
            book_list.append(data)
            
            
    
    if __name__=='__main__':
        urls=["https://book.douban.com/top250?start={}".format(str(i)) for i in range(0,250,25)]
        for url in urls:
            book_info(url)
            time.sleep(1)
        for word in book_list:
            fo=open(r'D:\Python爬虫\doubanbook.txt',"a+",encoding='utf-8')
            try:
                fo.write(word["book_name"]+" "+word["book_url"]+" "+word["line"]+" "+word["mark"]+"分"+" "+word["comtent"]+"\n")
                fo.write("\n")
                fo.close()
            except UnicodeDecodeError:
                pass

结果部分截图：  
![在这里插入图片描述][watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5aSn56KX5ZCD5LiN5LiL_size_20_color_FFFFFF_t_70_g_se_x_16]

[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5aSn56KX5ZCD5LiN5LiL_size_20_color_FFFFFF_t_70_g_se_x_16]: /images/20220828/d0de7c26a48d4f54824f6209c0c956d1.png