Scrapy第三(②)篇：创建scrapy项目

野性酷女 2024-04-08 08:25 93阅读 0赞

#### 1.创建项目 ####

新建存放项目的文件夹scrapyproject，并黑窗口cd到该文件夹下：

创建项目

scrapy startproject 项目名称

![30c41b4e6c1e4b49ade9cb98dc8bc1f6.png][]

PyCharm打开项目：

![54b9c2b766c045078e5dce229b3643e7.png][]

创建一个爬虫：打开Terminal执行如下【限制域只能是域名，不能含有http前缀】

scrapy genspider 爬虫名称 要爬取的限制域

执行完毕后，在spiders文件夹下会有一个基础爬虫文件

![b12c8f2a90304393b0f083e3f5642589.png][]

#### 2.调试爬虫 ####

根路径下新建文件main.py，内容如下：【修改一下爬虫名称】

from scrapy.cmdline import execute
    import os
    import sys
     
    if __name__ == '__main__':
        sys.path.append(os.path.dirname(os.path.abspath(__file__)))
        execute(['scrapy', 'crawl', '爬虫名称'])

![e7525ee4baf448c999eb68efa6494b36.png][]

#### 3.启动爬虫 ####

启动main.py即可启动爬虫程序。

![c559493a719a4b7c8b51411015a3c57c.png][]

#### 4.默认启动参数 ####

Setting初始化只定义了四个参数，其他都是注释掉的。![4678f7f50a8743e291a4de3225e7d6c1.png][]

#### 5.其他启动参数 ####

# Scrapy settings for testscrapy project
    #
    # For simplicity, this file contains only settings considered important or
    # commonly used. You can find more settings consulting the documentation:
    #
    #     https://docs.scrapy.org/en/latest/topics/settings.html
    #     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
    #     https://docs.scrapy.org/en/latest/topics/spider-middleware.html
    
    BOT_NAME = 'testscrapy'  # 爬虫名称，爬取的时候user-agent默认就是这个名字
    
    SPIDER_MODULES = ['testscrapy.spiders']  # 爬虫模块
    NEWSPIDER_MODULE = 'testscrapy.spiders'  # 爬虫模块
    
    # Crawl responsibly by identifying yourself (and your website) on the user-agent
    # USER_AGENT = 'testscrapy (+http://www.yourdomain.com)'
    
    # Obey robots.txt rules
    ROBOTSTXT_OBEY = True  # 是否遵守网站爬虫协议，当设置为False时，就代表，无论此网站让我不让我爬，我都要爬。
    
    # Configure maximum concurrent requests performed by Scrapy (default: 16)
    # CONCURRENT_REQUESTS = 32  # 并发请求数，如果对方没有做反爬虫机制，就可以开很大的并发，这样就可以一下子返回很多的数据
    
    # Configure a delay for requests for the same website (default: 0)
    # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
    # See also autothrottle settings and docs
    # DOWNLOAD_DELAY = 3  # 下载延迟秒数（下载速度），限制频繁访问，防止被封号，不给别人的网站造成太大的压力
    # The download delay setting will honor only one of:
    # CONCURRENT_REQUESTS_PER_DOMAIN = 16  # 针对每个域名限制 n 个并发，最大为 16
    # CONCURRENT_REQUESTS_PER_IP = 16  # 单IP访问的并发数，如果有值则忽略，对域名并发树的限制，并且下载速度也应用在每个IP，因为一个域名可以有很多个IP，公司可以有很多个服务器，对外部来说只有一个域名
    
    # Disable cookies (enabled by default)
    # COOKIES_ENABLED = False  # 返回的 response 是否解析 cookies
    
    # Disable Telnet Console (enabled by default)
    # TELNETCONSOLE_ENABLED = False  # 通过Telnet 可以监听当前爬虫的状态、信息、操作爬虫等，使用方法是：打开cmd，使用 telent 127.0.0.1 6023 以及est() ，即可进入操作界面
    
    # Override the default request headers:
    # DEFAULT_REQUEST_HEADERS = {
    #   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    #   'Accept-Language': 'en',
    # } # 默认的请求头，每个请求都可以携带，如果针对每个请求头设置，也可以在此界面进行设置
    
    # Enable or disable spider middlewares
    # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
    # SPIDER_MIDDLEWARES = {
    #    'testscrapy.middlewares.TestscrapySpiderMiddleware': 543,
    # } # 爬虫中间件
    
    # Enable or disable downloader middlewares
    # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
    # DOWNLOADER_MIDDLEWARES = {
    #    'testscrapy.middlewares.TestscrapyDownloaderMiddleware': 543,
    # } # 下载中间件
    
    # Enable or disable extensions
    # See https://docs.scrapy.org/en/latest/topics/extensions.html
    # EXTENSIONS = {
    #    'scrapy.extensions.telnet.TelnetConsole': None,
    # } # 其他扩展
    
    # Configure item pipelines
    # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
    # ITEM_PIPELINES = {
    #    'testscrapy.pipelines.TestscrapyPipeline': 300,
    # } # 自定义PIPELINES处理请求，主要为了存储数据使用，每行后面的整型值，确定了他们运行的顺序，item按数字从低到高的顺序，通过pipeline，通常将这些数字定义在0-1000范围内
    
    # Enable and configure the AutoThrottle extension (disabled by default)
    # See https://docs.scrapy.org/en/latest/topics/autothrottle.html
    # AUTOTHROTTLE_ENABLED = True
    # The initial download delay
    # AUTOTHROTTLE_START_DELAY = 5
    # The maximum download delay to be set in case of high latencies
    # AUTOTHROTTLE_MAX_DELAY = 60
    # The average number of requests Scrapy should be sending in parallel to
    # each remote server
    # AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
    # Enable showing throttling stats for every response received:
    # AUTOTHROTTLE_DEBUG = False
    
    # Enable and configure HTTP caching (disabled by default)
    # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
    # HTTPCACHE_ENABLED = True
    # HTTPCACHE_EXPIRATION_SECS = 0
    # HTTPCACHE_DIR = 'httpcache'
    # HTTPCACHE_IGNORE_HTTP_CODES = []
    # HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

#### 6.创建虚拟环境 ####

![a4c77a7951934f5894245f63233306b5.png][]

![90fb2265775f4545b42fe51393891711.png][]这是`setuptools` 版本过高setuptools-61.2.0；而你当前的pycharm版本不支持导致；**删除刚刚虚拟环境；**

如果不想升级pycharm的话，我们可以使用命令来创建：

python  -m venv 虚拟环境名称

![b12e7d546d2448a7a3ae8686aeca9416.png][]

![669f9e38e1e94939a9f38e1f1baf29d9.png][]

7.创建虚拟环境后，需要在虚拟环境中安装一下Scrapy。

pip install scrapy -i https://pypi.tuna.tsinghua.edu.cn/simple

[30c41b4e6c1e4b49ade9cb98dc8bc1f6.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/0ed40f0ed62c49f78e42776f66de7577.png
[54b9c2b766c045078e5dce229b3643e7.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/3d4cc2ad4a2e4df698fad625a7925447.png
[b12c8f2a90304393b0f083e3f5642589.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/cd34b63b258e449b83eeadd619b6a0e0.png
[e7525ee4baf448c999eb68efa6494b36.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/9611db4546ef41c6b8c551d4f287df0b.png
[c559493a719a4b7c8b51411015a3c57c.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/6011d88d60e743ba958f528118950040.png
[4678f7f50a8743e291a4de3225e7d6c1.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/1e6d72f838cf484896827fe4f512059a.png
[a4c77a7951934f5894245f63233306b5.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/79bd311121814e22a98fa77a0cc89b56.png
[90fb2265775f4545b42fe51393891711.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/7cad7777a9f14689a7310051f1628b93.png
[b12e7d546d2448a7a3ae8686aeca9416.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/af8adf59b4274251962ff8e4f6b510ef.png
[669f9e38e1e94939a9f38e1f1baf29d9.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/b5f64d8c0ad34de89ca7656ca71e0759.png