论新手如何使用scrapy框架?

曾经终败给现在 2022-12-06 01:26 185阅读 0赞

### 论新手如何使用scrapy框架? ###

*  区别
 *  基本使用
 *   *  （一）创建项目基本命令操作
 *  案例 --->虎扑篮球新闻
 *   *  (一) 安装模块
     *  (二) 项目建立
     *  (三)目录介绍
     *  (四)修改settings.py
     *  (五)编写item
     *  (六) 保存数据
     *  (七)效果图

# 区别 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center]

# 基本使用 #

## （一）创建项目基本命令操作 ##

（1）安装 Scrapy pip install scrapy  
（2）创建项目 scrapy startproject 项目名称  
（3）第一个爬虫项目

*  步骤一：**scrapy startproject baidu**
 *  步骤二：pycharm 打开项目
 *  步骤三：在 spider 文件夹下，新建 baidu.py 文件或者在spider文件夹下输入**scrapy genspider s\_baidu baidu.com**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 1]
 *  步骤四：cd 项目目录，启动爬虫，命令：scrapy crawl baidu  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 2]  
    **问题：运行后，查看文件，发现并没有保存成文件。**  
    **原因：**  
    Robots 协议（也叫爬虫协议、机器人协议等），全称是“网络爬虫排除标准”（Robots Exclusion Protocol），网站通过 Robots 协议告诉搜索引擎哪些页面可以抓取，哪些页面不能抓取，例如百度  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 3]

为什么之前的爬虫可以爬取呢？是因为之前自己写的爬虫不遵守这个协议，而 scrapy 默认是遵守该协议。  
解决方案：**settings.py—>ROBOTS\_OBEY = False，**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 4]

# 案例 —>虎扑篮球新闻 #

链接:[https://voice.hupu.com/nba][https_voice.hupu.com_nba]

## (一) 安装模块 ##

pip install scrapy

## (二) 项目建立 ##

#在当前目录下
    scrapy startproject hupu

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 5]

#在 spider 文件夹下,输入以下命令
    scrapy genspider hupu_scrapy www

![在这里插入图片描述][2020091119072383.png_pic_center]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 6]

## (三)目录介绍 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 7]

## (四)修改settings.py ##

(1) headings请求头配置  
根据具体情况进行修改和添加  
![在这里插入图片描述][20200911191030616.png_pic_center]

DEFAULT_REQUEST_HEADERS = { 
      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
      'Accept-Language': 'en',
      'user-agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.79 Safari/537.36 Maxthon/5.2.6.1000',
      
    }

(2)robots协议  
全称:网络爬虫排除标准  
作用:网站告诉搜索引擎那些数据可以爬取,那些不可以爬取的协议文件  
scrapy默认遵守协议  
![在这里插入图片描述][20200911191418469.png_pic_center]

## (五)编写item ##

hupu\_spider.py

import scrapy
    #只能用相对路径
    from ..items import HupuItem
    
    
    class HupuSpiderSpider(scrapy.Spider):
        name = 'hupu_spider'
        # allowed_domains = ['www']
        start_urls = ['https://voice.hupu.com/nba']
        for page in range(1,2):
            url=f"https://voice.hupu.com/nba/{str(page)}"
            start_urls.append(url)
    
        def parse(self, response):
            nba_urls=response.xpath('//div[@class="news-list"]/ul/li//h4/a/@href').extract()
            for url in nba_urls:
                yield scrapy.Request(url=url,callback=self.parse_detail,encoding="utf-8")
            print("存储成功")
    
        def parse_detail(self,response):
            #标题
            news_tilte=response.xpath('//h1[@class="headline"]/text()').extract_first().strip()
            #来源
            news_from=response.xpath('//*[@id="source_baidu"]/a/text()').extract_first().strip()
            #公布时间
            news_pub_time=response.xpath('//*[@id="pubtime_baidu"]/text()').extract_first().strip()
            #内容
            news_content=response.xpath('string(//div[@class="artical-content-read"])').extract_first().strip()
            #所有图片
            news_imgs=response.xpath('//div[@class="artical-content-read"]//img/@src').extract()
            items=HupuItem()
            items['news_tilte']=news_tilte
            items['news_from']=news_from
            items['news_pub_time']=news_pub_time
            items['news_content']=news_content
            items['news_imgs']=news_imgs
            items['news_url']=response.url
            yield items

**items.py**

# Define here the models for your scraped items
    #
    # See documentation in:
    # https://docs.scrapy.org/en/latest/topics/items.html
    
    import scrapy
    
    
    class HupuItem(scrapy.Item):
        # define the fields for your item here like:
        news_tilte = scrapy.Field()
        news_from = scrapy.Field()
        news_pub_time = scrapy.Field()
        news_content = scrapy.Field()
        news_imgs = scrapy.Field()
        news_url = scrapy.Field()
        news_id = scrapy.Field()

## (六) 保存数据 ##

1.  **配置pipelines**  
    settings.py  
    ![在这里插入图片描述][20200911194854600.png_pic_center]
2.  **编写存储数据的pipelines.py**

# Define your item pipelines here
    #
    # Don't forget to add your pipeline to the ITEM_PIPELINES setting
    # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
    
    
    # useful for handling different item types with a single interface
    import hashlib
    
    from itemadapter import ItemAdapter
    import pymongo
    
    class HupuPipeline:
        def __init__(self):
            # 1.创建连接
            self.client = pymongo.MongoClient(host='localhost', port=27017)
            # 连接数据库
            self.db = self.client['hupu']
        def get_md5(self,value):
            return hashlib.md5(bytes(value,encoding='utf-8')).hexdigest()
        def process_item(self, item, spider):
            # print(item)
            item['news_id']=self.get_md5(item['news_url'])
            self.db['nba'].update({ 'news_id':item['news_id']},{ '$set':dict(item)},True)
            print("正在存储..........")
            return item

## (七)效果图 ##

![在这里插入图片描述][20200911200926224.png_pic_center]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 8]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center]: /images/20221123/40f1c6d1d83c439f853c81acb0ca603b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221123/4ff69223e37a42d482f414d1cfc41ed7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221123/51fe0b84b8244a14a35aa3e4e1197bee.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 3]: /images/20221123/231bf2bbdd014d2eb82021fe6008eb8e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 4]: /images/20221123/52439f3af03e446cbda3ef0c81ada096.png
[https_voice.hupu.com_nba]: https://voice.hupu.com/nba
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 5]: /images/20221123/ceab04d8346d4109ab5f9d0815925c5b.png
[2020091119072383.png_pic_center]: /images/20221123/daf97f8649a940379cae46f541bf2450.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 6]: /images/20221123/f6ecb06bf25540789fada675e4878025.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 7]: /images/20221123/f772f8e623144250b3d5b132cb428159.png
[20200911191030616.png_pic_center]: /images/20221123/37af78bc4e254de482fafed3025e17fc.png
[20200911191418469.png_pic_center]: /images/20221123/5903ee68e18e459781be276ced279f86.png
[20200911194854600.png_pic_center]: /images/20221123/2705209dcf10455186e48e6bbe7dddbe.png
[20200911200926224.png_pic_center]: /images/20221123/6a19ea35a1b1452985d3760275290c94.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ExMzA5NTI1ODAy_size_16_color_FFFFFF_t_70_pic_center 8]: /images/20221123/395f9135277d4747943a87316551ce15.png