scrapy框架-ImagesPipeline图像管道

迈不过友情╰ 2023-10-05 21:39 17阅读 0赞

**1.爬虫文件-高清图片**

from ..items import  GaoqingItem
    import  requests
    
    class GqSpider(scrapy.Spider):
        name = 'gq'
        allowed_domains = ['gaoqing.la']
        start_urls = ['http://gaoqing.la/?s=%E7%94%B5%E5%BD%B1']
    
        def parse(self, response):
            #获取电影
            nodes=response.xpath('//*[@id="post_container"]/li[*]/div/div[2]/h2/a')
            for node in nodes:
               item =GaoqingItem()  #对象item
               # 获取电影名称
               title =node.xpath('./@title').get()
               # 获取电影详情url
               href=node.xpath('./@href').get()
    
               #存储数据
               item['title']=title
               item['href']=href
               # print(item)
    
               #再次发送请求,并传递参数meta={}
               yield scrapy.Request(href,callback=self.parse_data,meta={'item2':item})
    
        #回调数据
        def parse_data(self, response):
            #获取参数的值
            item=response.meta.get('item2')
            #获取电影图片url
            img_url=response.xpath('//*[@id="post_content"]/p[1]/a/img/@src').get()
            # print('图片url>>>',img_url)
    
            # 保存到固定字段
            item['image_urls']=img_url
            # print(item['image_urls'])
    
            # 注意这个必须返回哦到管道！
            yield  item  #返回到管道

**2.存储items文件**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hseDIwMDgwODA4_size_16_color_FFFFFF_t_70][]

**3.管道文件 pipeline.py**

from itemadapter import ItemAdapter
    
    #导入图片管道类
    from scrapy.pipelines.images import ImagesPipeline
    
    import  scrapy
    
    
    # 图片管道类
    class ImgPipeline(ImagesPipeline):
        # 1.根据图片进行请求
        def get_media_requests(self, item, info):
            print('图片管道方法执行！')
            res = scrapy.Request(url=item['image_urls'], meta={'item': item})
            yield res
    
        # 2.指定图片路径/名字
        def file_path(self, request, response=None, info=None, *, item=None):
            # 保存名字
            item = request.meta['item']
            name = item['title'] + '.jpg'
            # print('name-->', name)
            return name
    
        # 3.将item传递给下一个即将被执行的管道类
        def item_completed(self, results, item, info):
            return item

自定义类继承ImagesPipeline无法执行，查看源码

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hseDIwMDgwODA4_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hseDIwMDgwODA4_size_16_color_FFFFFF_t_70 2][]

**4.设置文件settings.py**

#设置使用图片管道保存的路径
    IMAGES_STORE='./imgs'
    
    #开启管道
    ITEM_PIPELINES = {
       # 'gaoqing.pipelines.GaoqingPipeline': 300,
       'gaoqing.pipelines.ImgPipeline': 300,
    }

**5.运行 scrapy crawl gq --nolog**  
![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hseDIwMDgwODA4_size_16_color_FFFFFF_t_70 3][]

总之：使用scrapy保存数据到文件，需要注意以下三点：  
  
　　 1. pipelines文件正确配置  
　　 2. 配置settings.py文件  
　　 3. 爬虫文件parse()函数一定要由yield语句

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hseDIwMDgwODA4_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20210616100516335.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hseDIwMDgwODA4,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hseDIwMDgwODA4_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20210616101219358.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hseDIwMDgwODA4,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hseDIwMDgwODA4_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20210616101236171.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hseDIwMDgwODA4,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hseDIwMDgwODA4_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20210616100830254.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hseDIwMDgwODA4,size_16,color_FFFFFF,t_70