使用Scrapy爬取知乎网站马化腾所有粉丝的信息

àì夳堔傛蜴生んèń 2023-10-11 13:39 40阅读 0赞

采集网址：[https://www.zhihu.com/people/ponyma/followers][https_www.zhihu.com_people_ponyma_followers]

### 爬虫文件 ###

import scrapy
    import json
    
    
    class ZhihuSpider(scrapy.Spider):
        name = 'zhihu'
        allowed_domains = ['zhihu.com']
        start_urls = [
            f'https://www.zhihu.com/api/v4/members/ponyma/followers?include=data%5B*%5D.answer_count%2Carticles_count%2Cgender%2Cfollower_count%2Cis_followed%2Cis_following%2Cbadge%5B%3F(type%3Dbest_answerer)%5D.topics&offset={offset}0&limit=20'
            for offset in range(0, 200, 20)]
    
        def parse(self, response):
            # 打印请求头
            # print(response.request.headers)
            json_data = json.loads(response.text)
            if json_data.get("data", False):
                for data in json_data['data']:
                    item = data
                    yield item

![1568603-20190717161539482-785754123.png][]

此网址采用抓包方式爬取

步骤一：打开采集网址 右边右键选择检查或者F12打开开发者模式

步骤二：

![1568603-20190717162056762-406223913.png][]

步骤三：点击页面中的下一页，注意不用刷新整个页面

![1568603-20190717162254965-1243758244.png][]

![1568603-20190717162353494-1746962175.png][]

步骤四：点击Headers找到对应请求网址

![1568603-20190717162523502-2084639372.png][]

### 管道文件（pipelines.py） ###

import json
    
    
    class ZhihuspiderPipeline(object):
    
        def __init__(self):
            # 打开文件
            self.file = open("zhihu.json", "a", encoding="utf-8")
    
        def process_item(self, item, spider):
            self.file.write(json.dumps(item, ensure_ascii=False) + "\n")
            return item
    
        def close_spider(self, spider):
            # 关闭文件
            self.file.close()

### 配置文件(settings.py) ###

步骤一：关闭网址的robotstxt协议

![1568603-20190717162914310-1263231664.png][]

步骤二：设置请求头

![1568603-20190717162947267-1252279078.png][]

步骤三：激活管道文件

![1568603-20190717163218687-1766636543.png][]

### 运行程序 ###

方法① ：使用终端执行文件

![1568603-20190717163514067-2057663347.png][]

进入文件的目录下使用命令：scrapy crawl 爬虫名

方法② ：使用文件运行

在文件的目录下创建run\_spider.py 文件，这里的文件名可以随意起

![1568603-20190717163627743-1090478176.png][]

转载于:https://www.cnblogs.com/renshaoqi/p/11201675.html

[https_www.zhihu.com_people_ponyma_followers]: https://www.zhihu.com/people/ponyma/followers
[1568603-20190717161539482-785754123.png]: https://img2018.cnblogs.com/blog/1568603/201907/1568603-20190717161539482-785754123.png
[1568603-20190717162056762-406223913.png]: https://img2018.cnblogs.com/blog/1568603/201907/1568603-20190717162056762-406223913.png
[1568603-20190717162254965-1243758244.png]: https://img2018.cnblogs.com/blog/1568603/201907/1568603-20190717162254965-1243758244.png
[1568603-20190717162353494-1746962175.png]: https://img2018.cnblogs.com/blog/1568603/201907/1568603-20190717162353494-1746962175.png
[1568603-20190717162523502-2084639372.png]: https://img2018.cnblogs.com/blog/1568603/201907/1568603-20190717162523502-2084639372.png
[1568603-20190717162914310-1263231664.png]: https://img2018.cnblogs.com/blog/1568603/201907/1568603-20190717162914310-1263231664.png
[1568603-20190717162947267-1252279078.png]: https://img2018.cnblogs.com/blog/1568603/201907/1568603-20190717162947267-1252279078.png
[1568603-20190717163218687-1766636543.png]: https://img2018.cnblogs.com/blog/1568603/201907/1568603-20190717163218687-1766636543.png
[1568603-20190717163514067-2057663347.png]: https://img2018.cnblogs.com/blog/1568603/201907/1568603-20190717163514067-2057663347.png
[1568603-20190717163627743-1090478176.png]: https://img2018.cnblogs.com/blog/1568603/201907/1568603-20190717163627743-1090478176.png