scrapy框架里加ip代理加headers

女爷i 2024-04-18 17:24 58阅读 0赞

最近在熟悉爬虫框架的知识，针对代理和模拟请求头的设置有一些疑问，在网上查询了一些文章，总结一下：

【设置代理ip】

根据最新的scrapy官方文档,scrapy爬虫框架的代理配置有以下两种方法:  
  
一.使用中间件DownloaderMiddleware进行配置  
使用Scrapy默认方法scrapy startproject创建项目后项目目录结构如下,spider中的crawler是已经写好的爬虫程序:   
   
settings.py文件其中的DOWNLOADER\_MIDDLEWARES用于配置scrapy的中间件.我们可以在这里进行自己爬虫中间键的配置,配置后如下:  
  
DOWNLOADER\_MIDDLEWARES = \{  
    'WandoujiaCrawler.middlewares.ProxyMiddleware': 100,  
\}  
  
其中WandoujiaCrawler是我们的项目名称,后面的数字代表中间件执行的优先级,官方文档中默认proxy中间件的优先级编号是750,我们的中间件优先级要高于默认的proxy中间键.中间件middlewares.py的写法如下(scrapy默认会在这个文件中写好一个中间件的模板,不用管它写在后面即可):

# -*- coding: utf-8 -*-
    class ProxyMiddleware(object):
        def process_request(self, request, spider):
            request.meta['proxy'] = "http://proxy.yourproxy:8001"

这里有两个问题:   
一是proxy一定是要写号http://前缀的否则会出现to\_bytes must receive a unicode, str or bytes object, got NoneType的错误.   
二是官方文档中写到process\_request方法一定要返回request对象,response对象或None的一种,但是其实写的时候不用return,乱写可能会报错.   
另外如果代理有用户名密码等就需要在后面再加上一些内容:

# Use the following lines if your proxy requires authentication
    proxy_user_pass = "USERNAME:PASSWORD"
    # setup basic authentication for the proxy
    encoded_user_pass = base64.encodestring(proxy_user_pass)
    request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

二.直接在爬虫程序中设置proxy字段  
我们可以直接在自己具体的爬虫程序中设置proxy字段,代码如下,直接在构造Request里面加上meta字段即可:

class QuotesSpider(scrapy.Spider):
        name = "quotes"
        def start_requests(self):
            urls = [
                'http://quotes.toscrape.com/page/1/',
                'http://quotes.toscrape.com/page/2/',
            ]
            for url in urls:
                yield scrapy.Request(url=url, callback=self.parse, meta={'proxy': 'http://proxy.yourproxy:8001'})
     
        def parse(self, response):
            for quote in response.css('div.quote'):
                yield {
                    'text': quote.css('span.text::text').extract_first(),
                    'author': quote.css('span small::text').extract_first(),
                    'tags': quote.css('div.tags a.tag::text').extract(),
                }

第二种

1.在Scrapy工程下新建“middlewares.py”

1 # Importing base64 library because we'll need it ONLY in case if the proxy we are going to use requires authentication
     2 import base64
     3  
     4 # Start your middleware class
     5 class ProxyMiddleware(object):
     6     # overwrite process request
     7     def process_request(self, request, spider):
     8         # Set the location of the proxy
     9         request.meta['proxy'] = "http://YOUR_PROXY_IP:PORT"
    10  
    11         # Use the following lines if your proxy requires authentication
    12         proxy_user_pass = "USERNAME:PASSWORD"
    13         # setup basic authentication for the proxy
    14         encoded_user_pass = base64.encodestring(proxy_user_pass)
    15         request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

2.在项目配置文件里(./project\_name/settings.py)添加

1 DOWNLOADER_MIDDLEWARES = {
    2     'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
    3     'project_name.middlewares.ProxyMiddleware': 100,
    4 }

只要两步，现在请求就是通过代理的了。测试一下^\_^

1 from scrapy.spider import BaseSpider
     2 from scrapy.contrib.spiders import CrawlSpider, Rule
     3 from scrapy.http import Request
     4  
     5 class TestSpider(CrawlSpider):
     6     name = "test"
     7     domain_name = "whatismyip.com"
     8     # The following url is subject to change, you can get the last updated one from here :
     9     # http://www.whatismyip.com/faq/automation.asp
    10     start_urls = ["http://xujian.info"]
    11  
    12     def parse(self, response):
    13         open('test.html', 'wb').write(response.body)
    14

增加文件middlewares.py放置在setting.py平行的目录下

1 import base64
     2 class ProxyMiddleware(object):
     3 # overwrite process request
     4 def process_request(self, request, spider):
     5     # Set the location of the proxy
     6     request.meta['proxy'] = "http://YOUR_PROXY_IP:PORT"
     7 
     8     # Use the following lines if your proxy requires authentication
     9     proxy_user_pass = "USERNAME:PASSWORD"
    10     # setup basic authentication for the proxy
    11     encoded_user_pass = base64.b64encode(proxy_user_pass)
    12     request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

很多网上的答案使用base64.encodestring来编码proxy\_user\_pass，有一种情况，当username太长的时候，会出现错误，所以推荐使用b64encode编码方式

然后在setting.py中，在DOWNLOADER\_MIDDLEWARES中把它打开，projectname.middlewares.ProxyMiddleware: 1就可以了

【设置headers和cookies】

scrapy中有三种方式设置headers，cookies  
  
setting中设置cookie  
middlewares中设置cookie  
sipder文件中重写start\_requests方法  
这里记录第三种，重写start\_requests方法，这里以豆瓣网为例  
  
一、设置请求头headers  
在start\_request中新增  
  
headers = \{  
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'  
    \}  
二、设置cookies  
1、登录豆瓣网并获取cookie  
  
首先在chrome中登录豆瓣网  
按F12调出开发者工具  
查看cookie  
  
  
2、在start\_requests中新增  
  
cookies = \{  
    'key':'value',  
    'key':'value',  
    'key':'value'  
    \}

一行代码把字符串的cookies转字典

#bakooie装换成字典模式方便scrapy使用
    cookid = "_ga=GA1.2.1937936278.1538889470; __gads=ID=1ba11c2610acf504:T=1539160131:S=ALNI_MZwhotFaAA6KsIVzHG-ev0RnU4OIQ; .CNBlogsCookie=7F3B19F5204038FAE6287F33828591011A60086D0F4349BEDA5F568571875F43E1EED5EDE24E458FAB8972604B4ECD19FC058F5562321A6D87ABF8AAC19F32EC6C004B2EBA69A29B8532E5464ECD145896AA49F1; .Cnblogs.AspNetCore.Cookies=CfDJ8J0rgDI0eRtJkfTEZKR_e81dD8ABr7voOOlhOqLJ7tzHG0h7wCeF8EYzLUZbtYueLnkUIzSWDE9LuJ-53nA6Lem4htKEIqdoOszI5xWb4PUZHJtM1qgEjI1E1Q8YLz8cU3jts5xoHMzq7qq7AmtrlCYYqvBMgEX8GACn8j61WrxZfKe9Hmh4akC9AxcODmAPP--axDI0w6LTSQYKl4GnKihmxM6DQ3RDCXXzWukG-3xiPfKv5vdSNFBTvj7b2qOeTmy45RWkQT9dqf_bXjniWnhPHRnGq8uNHqN2bpzUlCOxsrjwuZlhbAPPLCnX90XJaA; _gid=GA1.2.201165281.1540104585"
    coolies = dict(i.split('=',1) for i in cookid.split(';'))
    print(coolies)

3、修改方法返回值
    
    yield scrapy.Request(url=url, headers=headers, cookies=cookies, callback=self.parse)
    4、修改COOKIES_ENABLED
    
    当COOKIES_ENABLED是注释的时候scrapy默认没有开启cookie
    当COOKIES_ENABLED没有注释设置为False的时候scrapy默认使用了settings里面的cookie
    当COOKIES_ENABLED设置为True的时候scrapy就会把settings的cookie关掉，使用自定义cookie
    因此在settings.py文件中设置COOKIES_ENABLED = True
    
    5、修改COOKIES_ENABLED
    
    在settings.py文件中设置ROBOTSTXT_OBEY = False
    
    三、测试
    1、新建scrapy爬虫项目
    
    scrapy stratproject douban
    
    2、在./doouban/spiders/下新建short_spider.py文件

# -*- coding: utf-8 -*-
    import scrapy
     
    class ShortSpider(scrapy.Spider):
        name = 'short'
        allow_domains = ['movie.douban.com']
     
        # 重写start_requests方法
        def start_requests(self):
     
            # 浏览器用户代理
            headers = {
                'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'
            }
            # 指定cookies
            cookies = {
                'key':'value',
                'key':'value',
                'key':'value'
            }
            urls = [
                'https://movie.douban.com/subject/26266893/comments?start=250&limit=20&sort=new_score&status=P'
            ]
            for url in urls:
                yield scrapy.Request(url=url, headers=headers, cookies=cookies, callback=self.parse)
     
        def parse(self, response):
            file_name = 'data.html'
            with open(file_name, 'wb') as f:
                f.write(response.body)

因为豆瓣电影短评论对未登录用户有限制，因此这里使用豆瓣电影段评论数比较靠后的页数进行测试
    
    3、进入douban目录执行scrapy crawl short
    
    状态码200，爬取成功（无权限返回403）