基于关键字在主流搜索引擎中抓取信息

￡神魔★判官ぃ 2022-05-28 06:17 120阅读 0赞

> 本文首发于我的博客：[http://gongyanli.com][http_gongyanli.com]  
> 代码传送门：[https://github.com/Gladysgong/seCrawler][https_github.com_Gladysgong_seCrawler]  
> 简书: [https://www.jianshu.com/p/4e244563849a][https_www.jianshu.com_p_4e244563849a]  
> CSDN: [https://blog.csdn.net/u012052168/article/details/79762586][https_blog.csdn.net_u012052168_article_details_79762586]

# seCrawler(Search Engine Crawler) #

A scrapy project can crawl search result of Google/Bing/Baidu

## refer ##

copying by [https://github.com/xtt129/seCrawler][https_github.com_xtt129_seCrawler] and rewrite,adding title and abstract.

## prerequisite ##

python 3.5 and scrapy is needed.

## commands ##

run one command to get 50 pages result from search engine with keyword, the result would be kept in the “urls.txt” under the current directory.

#### Bing ####

`scrapy crawl keywordSpider -a keyword=Spider-Man -a se=bing -a pages=50`

#### Baidu ####

`scrapy crawl keywordSpider -a keyword=Spider-Man -a se=baidu -a pages=50`

#### Google ####

`scrapy crawl keywordSpider -a keyword=Spider-Man -a se=google -a pages=50`

## results ##

url,title and abstract will be stored in the urls.txt

## limitation ##

The project doesn’t provide any workaround to the anti-spider measure like CAPTCHA, IP ban list, etc.

But to reduce these measures, we recommand to set `DOWNLOAD_DELAY=10` in settings.py file to add a temporisation (in second) between the crawl of two pages, see details in [Scrapy Setting][].

[http_gongyanli.com]: http://gongyanli.com/%E5%9F%BA%E4%BA%8E%E5%85%B3%E9%94%AE%E5%AD%97%E5%9C%A8%E4%B8%BB%E6%B5%81%E6%90%9C%E7%B4%A2%E5%BC%95%E6%93%8E%E4%B8%AD%E6%8A%93%E5%8F%96%E4%BF%A1%E6%81%AF/
[https_github.com_Gladysgong_seCrawler]: https://github.com/Gladysgong/seCrawler
[https_www.jianshu.com_p_4e244563849a]: https://www.jianshu.com/p/4e244563849a
[https_blog.csdn.net_u012052168_article_details_79762586]: https://blog.csdn.net/u012052168/article/details/79762586
[https_github.com_xtt129_seCrawler]: https://github.com/xtt129/seCrawler
[Scrapy Setting]: https://doc.scrapy.org/en/1.2/topics/settings.html#std:setting-DOWNLOAD_DELAY