构建scrapy项目

た入场券 2022-10-07 08:58 233阅读 0赞

构建项目：

scrapy startproject taobao

pycharm打开项目。

在项目根路径创建一个爬虫：

scrapy genspider 爬虫名称 要爬取的限制域

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlbnhpbmdjaGVu_size_16_color_FFFFFF_t_70][]调试工具：

scrapy shell http://www.taobao.com

#选择<title>标签(也可以class),::取标签值，extract提取数据，first指第一个，等价于extract[0]
    response.css('title::text').extract_first()

启动项目：

scrapy crawl quotes

pycharm断点调试：

在根路径下新建文件main.py，内容如下：修改下爬虫名称即可

from scrapy.cmdline import execute
    import os
    import sys
    
    if __name__ == '__main__':
        sys.path.append(os.path.dirname(os.path.abspath(__file__)))
        execute(['scrapy', 'crawl', '爬虫name'])

爬虫如下:

爬虫文件：

import scrapy
    from ..items import MyscrapyItem
    
    
    # 定义爬取逻辑
    class QuotesSpider(scrapy.Spider):
        # 爬虫名称
        name = 'quotes'
        # 允许的域
        allowed_domains = ['lab.scrapyd.cn']
        # 初始url集合
        start_urls = ['http://lab.scrapyd.cn/page/1/']
    
        # 处理response的方法
        def parse(self, response):
            # 用css选择器、获取所有的quote节点
            quotes = response.css('div.quote')
            # 遍历节点集
            for quote in quotes:
                item = MyscrapyItem()
                item["text"] = quote.css('span.text::text').extract_first()
                item["author"] = quote.css('span small.author::text').extract_first()
                item["tags"] = quote.css('div.tags a.tag::text').extract()
                yield item
            # 如果存在下一页则继续递归爬取
            next_page = response.css('.next a::attr(href)').extract_first()
            if next_page is not None:
                yield scrapy.Request(next_page, callback=self.parse)

item文件：

# Define here the models for your scraped items
    #
    # See documentation in:
    # https://docs.scrapy.org/en/latest/topics/items.html
    
    import scrapy
    
    
    class MyscrapyItem(scrapy.Item):
        # define the fields for your item here like:
        # name = scrapy.Field()
        # 定义爬取的数据字段
        text = scrapy.Field()
        author = scrapy.Field()
        tags = scrapy.Field()
        pass

pipelines：

# Define your item pipelines here
    #
    # Don't forget to add your pipeline to the ITEM_PIPELINES setting
    # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
    
    
    # useful for handling different item types with a single interface
    from itemadapter import ItemAdapter
    
    
    class MyscrapyPipeline:
        def process_item(self, item, spider):
            # 处理爬取的结果
            print(item)

要激活pipelines还需要修改一下settings.py

将这一段注释释放掉

ITEM_PIPELINES = {
       'myscrapy.pipelines.MyscrapyPipeline': 300,
    }

注意xpath的使用：

def parse(self, response, **kwargs):
            # 使用绝对路径定位标签
            elements = response.xpath('//div[@class="ui relaxed divided items explore-repo__list"]//div[@class="item"]')
            for element in elements:
                # 注意：再次进行xpath的时候是相对路径在需要//前面加上.。是.//而不是//
                link = self.allow_domains + element.xpath('.//h3/a/@href').get()
                print(link)
            pass

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlbnhpbmdjaGVu_size_16_color_FFFFFF_t_70]: /images/20221005/89f2158a4c26455194a02004f9bb1343.png