Scrapy第九篇：User-Agent中间件

待我称王封你为后i 2024-04-08 08:44 100阅读 0赞

####  ####

#### 1.安装依赖 ####

pip install fake-useragent -i https://pypi.tuna.tsinghua.edu.cn/simple
    pip install -U fake-useragent -i https://pypi.tuna.tsinghua.edu.cn/simple

#### 2.编写中间件：打开middlewares.py文件，新建中间件RandomUserAgentMidddlware ####

from fake_useragent import UserAgent
    
    
    class RandomUserAgentMidddlware(object):
        def __init__(self, crawler):
            super(RandomUserAgentMidddlware, self).__init__()
            self.ua = UserAgent()
            self.ua_type = crawler.settings.get('RANDOM_UA_TYPE', 'random')
    
        @classmethod
        def from_crawler(cls, crawler):
            return cls(crawler)
    
        def process_request(self, request, spider):
            request.headers['User-Agent'] = getattr(self.ua, self.ua_type)

#### 3.开启中间件：编辑settings.py。 ####

DOWNLOADER_MIDDLEWARES = {
        'announcement.middlewares.RandomUserAgentMidddlware': 542,
    }

#### 4.验证一下 ####

![65fd0b178d05447d9e42f957d6730ce1.png][]

#### 5.如果使用报错：参考 ####

[fake\_useragent.errors.FakeUserAgentError: Maximum amount of retries reached\_苍穹之跃的博客-CSDN博客pip install -U fake-useragent![favicon32.ico][]https://blog.csdn.net/wenxingchen/article/details/118164460][fake_useragent.errors.FakeUserAgentError_ Maximum amount of retries reached_-CSDN_pip install -U fake-useragent_favicon32.ico_https_blog.csdn.net_wenxingchen_article_details_118164460]

[65fd0b178d05447d9e42f957d6730ce1.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/355e57b0d15046a5b92e11ebd28bd17b.png
[favicon32.ico]: https://g.csdnimg.cn/static/logo/favicon32.ico
[fake_useragent.errors.FakeUserAgentError_ Maximum amount of retries reached_-CSDN_pip install -U fake-useragent_favicon32.ico_https_blog.csdn.net_wenxingchen_article_details_118164460]: https://blog.csdn.net/wenxingchen/article/details/118164460