scrapy 爬取诗词记录code

我就是我 2022-02-15 07:00 246阅读 0赞

创建项目

scrapy startproject poems

创建爬虫应用

cd poems\poems\spiders
    
    scrapy genspider 名字 域名
    scrapy genspider poem_spider www.gushiwen.org
    
    在poem_spider.py中 修改启始URL
    start_urls = ['https://www.gushiwen.org/default_1.aspx']

在items中定义数据结构

class PoemsItem(scrapy.Item):
        title = scrapy.Field()  # 题目
        dynasty = scrapy.Field()  # 朝代
        author = scrapy.Field()  # 作者
        content = scrapy.Field()  # 内容
        tags = scrapy.Field()  # 标签 tags

settings中设置

USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3648.400 QQBrowser/10.4.3319.400"

[设置一个启动文件main.py][main.py]

from scrapy.cmdline import execute
    
    import sys
    import os
    
    sys.path.append(os.path.dirname(os.path.abspath(__file__)))
    
    
    execute(["scrapy","crawl","poem_spider"])

编写爬虫

shell调试
    scrapy shell https://www.gushiwen.org/default_1.aspx

# -*- coding: utf-8 -*-
    import scrapy
    from poems.items import PoemsItem
    
    class PoemSpiderSpider(scrapy.Spider):
        name = 'poem_spider' # 爬虫名
        allowed_domains = ['www.gushiwen.org'] # 允许的域名
        start_urls = ['https://www.gushiwen.org/default_1.aspx'] # 入口url
    
        def parse(self, response):
    
            docs = response.css(".left .sons")
            for doc in docs:
                poem_tiem = PoemsItem()
                poem_tiem['title'] = doc.css("b::text").extract()[0]
                poem_tiem['dynasty'],poem_tiem['author'] = doc.css(".source a::text").extract()
                poem_tiem['content'] = "".join(doc.css(".contson::text").extract()).strip()
                poem_tiem['tags'] = ",".join(doc.css(".tag a::text").extract())
                yield poem_tiem
            next_link = response.css(".pagesright .amore::attr(href)")
            if next_link:
                next_link = next_link[0].extract()
                yield scrapy.Request("https://www.gushiwen.org" + next_link)

保存到json文件里
    scrapy crawl poem_spider -o test.json
    
    保存到scv里
    scrapy crawl poem_spider -o test.csv

[main.py]: http://xn--main-k84fui24ch8ostdv33d5s9cx44a.py