fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached

比眉伴天荒 2022-10-09 04:53 169阅读 0赞

pip install -U fake-useragent

*  如果以上方法不成功，则需要fake\_useragent\_0.1.11.json文件来处理了
 *  参考[https://blog.csdn.net/xc\_zhou/article/details/106412377][https_blog.csdn.net_xc_zhou_article_details_106412377]
 *  下载链接[https://download.csdn.net/download/wenxingchen/20354836?spm=1001.2014.3001.5501][https_download.csdn.net_download_wenxingchen_20354836_spm_1001.2014.3001.5501]
 *  将fake\_useragent\_0.1.11.json文件复制到项目中,使用UserAgent类进行读取
 *      from fake_useragent import UserAgent
        
        
        # 配置headers
        class RandomUserAgentMidddlware(object):
            # 随机更换user-agent
            def __init__(self, crawler):
                super(RandomUserAgentMidddlware, self).__init__()
                self.ua = UserAgent(path="spiders/fake_useragent_0.1.11.json")
                # 从配置文件读取随机类型
                self.ua_type = crawler.settings.get('RANDOM_UA_TYPE', 'random')
        
            @classmethod
            def from_crawler(cls, crawler):
                return cls(crawler)
        
            def process_request(self, request, spider):
                # 通过配置文件的随机类型进行调用
                def get_ua():
                    return getattr(self.ua, self.ua_type)
        
                request.headers.setdefault('User-Agent', get_ua())

[https_blog.csdn.net_xc_zhou_article_details_106412377]: https://blog.csdn.net/xc_zhou/article/details/106412377
[https_download.csdn.net_download_wenxingchen_20354836_spm_1001.2014.3001.5501]: https://download.csdn.net/download/wenxingchen/20354836?spm=1001.2014.3001.5501