Python下基于requests及BeautifulSoup构建网络爬虫

喜欢ヅ旅行 2022-06-08 13:40 229阅读 0赞

# 功能说明 #

在Python下面可使用requests模块请求某个url获取响应的html文件，接着使用BeautifulSoup解析某个html。

# 案例 #

假设我要[http://maoyan.com/board/4][http_maoyan.com_board_4]猫眼电影的top100电影的相关信息，如下截图：  
![这里写图片描述][SouthEast]

获取电影的标题及url。

# 安装requests和BeautifulSoup #

使用pip工具安装这两个工具。

pip install requests

pip install beautifulsoup4

运行结果如下：

![这里写图片描述][SouthEast 1]  
![这里写图片描述][SouthEast 2]

# 程序 #

__author__ = 'Qian Yang'
    # -*- coding:utf-8 -*-
    import requests
    from bs4 import BeautifulSoup
    def get_one_page(url):
        response= requests.get(url)
        if response.status_code == 200:
            return response.content.decode("utf8","ignore").encode("gbk","ignore")
    #采用BeautifulSoup解析
    def bs4_paraser(html):
        all_value = []
        value = {}
        soup = BeautifulSoup(html,'html.parser')
        # 获取每一个电影
        all_div_item = soup.find_all('div', attrs={
       'class': 'movie-item-info'})
        for r in all_div_item:
            # 获取电影的名称和url
            title = r.find_all(name="p",attrs={
       "class":"name"})[0].string
            movie_url = r.find_all('p', attrs={
       'class': 'name'})[0].a['href']
            value['title'] = title
            value['movie_url'] = movie_url
            all_value.append(value)
            value = {}
        return all_value
    
    def main():
        url = 'http://maoyan.com/board/4'
        html = get_one_page(url)
        all_value = bs4_paraser(html)
        print(all_value)
    
    if  __name__  == '__main__':
        main()

# 程序输出结果 #

![这里写图片描述][SouthEast 3]

[http_maoyan.com_board_4]: http://maoyan.com/board/4
[SouthEast]: /images/20220608/6f3e26820130401bb28681e8c8f2d95f.png
[SouthEast 1]: /images/20220608/59f43ca0b072423eb63ea3cf1c4993f2.png
[SouthEast 2]: /images/20220608/54030058ec9343678afd7b223b37550a.png
[SouthEast 3]: /images/20220608/db19bea6b3bd4bcca00893f3b7d3b265.png