python爬取豆瓣电影Top250

拼搏现实的明天。 2021-09-28 03:40 973阅读 0赞

### 爬取豆瓣电影Top250 ###

*   *   *  首先来看看我们要爬什么
         *  环境
         *  爬取的步骤
         *  爬取所有250条数据

### 首先来看看我们要爬什么 ###

红线框起来的是我们要爬取的目标  
![爬取目的][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3MTYyMDkw_size_16_color_FFFFFF_t_70]

### 环境 ###

> 这里我用的是python3.6.0、pyCharm、Google浏览器。

### 爬取的步骤 ###

> 我们需要爬取的地址：https://movie.douban.com/top250  
> 通过谷歌浏览器可以清楚的看到所有的元素DOM树结构。

![google浏览器][google]

> 先导入我们所需的包。

import requests
    from lxml import etree
    import urllib
    
    movic_title = []  # 电影标题
    movic_pic = []  # 电影海报
    movic_score = []    # 电影打分
    movic_quote = []  # 电影经典语句
    movic_info = []   # 电影基本信息

> 再给出所需的基本信息

url = 'https://movie.douban.com/top250  # url
    headers = { 'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}  # 给出头文件，否则请求会被拒绝。 
    res = requests.get(url, headers=headers)  # 使用requests库的方法获取响应。
    html = etree.HTML(res.text)  # 使用etree加载HTML树。

> 电影标题  
> ![电影标题][20190124114926913.png]

res_xpath_title = html.xpath("//span[@class='title'][1]/text()")  # 使用lxml语法方式来获取电影标题节点，不懂可以百度或者看看案例就可以 挺简单的。
    for tit in range(0, len(res_xpath_title)):
        movic_title.append(res_xpath_title[tit])
        print(res_xpath_title[tit], end=" ")

> 电影海报

res_xpath = html.xpath('//img/@src')
    for i in range(0, len(res_xpath)):
        # print(res_xpath[i]) # 输出到控制台
        movic_pic.append(res_xpath[i])  # 放到列表里

> 电影评分

res_xpath_score = html.xpath("//div[@class='star']//span[2]/text()")
    for j in range(0, len(res_xpath_score)):
        # print(res_xpath_score[j])
        movic_score.append(res_xpath_score[j])

> 电影精髓

# 电影精髓语句
    res_xpath_quote = html.xpath("//p[@class='quote']/span/text()")
    for quote in range(0, len(res_xpath_quote)):
        # print(res_xpath_quote[quote])
        movic_quote.append(res_xpath_quote[quote])

> 电影基本信息

res_xpath_info = html.xpath("//div[@class='bd']//p[1]/text()[1]")
    for info in range(0, len(res_xpath_info)):
        # print(res_xpath_info[info])
        movic_info.append(res_xpath_info[info])

> 但这样仅仅只是爬取了第一页的数据，如何爬取分页的数据呢？  
> 首先来观察这个网页。  
> ![分页][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3MTYyMDkw_size_16_color_FFFFFF_t_70 1]  
> ![分页2][2]  
> 可以看到他分页的规律了吧，每次start+25，我们只需要添加一个循环，使得url每次动态更改就行啦。下面附上源码。

### 爬取所有250条数据 ###

import requests
    from lxml import etree
    import urllib
    
    movic_title = []  # 电影标题
    movic_pic = []  # 电影海报
    movic_score = []    # 电影打分
    movic_quote = []  # 电影经典语句
    movic_info = []   # 电影基本信息
    
    
    def main(start):
        url = 'https://movie.douban.com/top250?start='+str(start)+'&filter='
        headers = { 'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}
        res = requests.get(url, headers=headers)
    
        # 电影标题
        html = etree.HTML(res.text)
        res_xpath_title = html.xpath("//span[@class='title'][1]/text()")
        for tit in range(0, len(res_xpath_title)):
            movic_title.append(res_xpath_title[tit])
            print(res_xpath_title[tit], end=" ")
    
        # pic 电影图片
        res_xpath = html.xpath('//img/@src')
        for i in range(0, len(res_xpath)):
            # print(res_xpath[i])
            movic_pic.append(res_xpath[i])
    
            urllib.request.urlretrieve(res_xpath[i], filename="G:\\top250\\" + str(start//25) + str(i) + ".jpg")  # 电影图片下载到本地
    
        # 电影评分
        res_xpath_score = html.xpath("//div[@class='star']//span[2]/text()")
        for j in range(0, len(res_xpath_score)):
            # print(res_xpath_score[j])
            movic_score.append(res_xpath_score[j])
    
        # 电影精髓语句
        res_xpath_quote = html.xpath("//p[@class='quote']/span/text()")
        for quote in range(0, len(res_xpath_quote)):
            # print(res_xpath_quote[quote])
            movic_quote.append(res_xpath_quote[quote])
    
        # 电影导演等基本信息
        res_xpath_info = html.xpath("//div[@class='bd']//p[1]/text()[1]")
        for info in range(0, len(res_xpath_info)):
            # print(res_xpath_info[info])
            movic_info.append(res_xpath_info[info])
    
    
    for i in range(10):
        main(i * 25)
    
    print(movic_title)
    print(movic_pic)
    print(movic_score)
    print(movic_quote)
    print(movic_info)

> 最终爬取的图片  
> ![图片][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3MTYyMDkw_size_16_color_FFFFFF_t_70 2]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3MTYyMDkw_size_16_color_FFFFFF_t_70]: /images/20210923/7a1ed4566f9a4708b15fd4c8ba2cf463.png
[google]: /images/20210923/0cbd42916f7c449f850d6920fddfe48c.png
[20190124114926913.png]: /images/20210923/f63b98e085324c8fae377eefbce51808.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3MTYyMDkw_size_16_color_FFFFFF_t_70 1]: /images/20210923/d537f2cf28994962b6641699167c37f7.png
[2]: /images/20210923/c11a6d024f154521967a26d0a15ddf34.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3MTYyMDkw_size_16_color_FFFFFF_t_70 2]: /images/20210923/8e6d0c11733a442593c904156b0eea85.png