Python 爬虫，scrapy，settings配置文件

怼烎@ 2022-03-25 05:42 115阅读 0赞

**项目名/settings.py（配置文件）：**

# -*- coding: utf-8 -*-
    
    BOT_NAME = '项目名'  # 项目名
    
    
    SPIDER_MODULES = ['项目名.spiders']  # 爬虫模块的位置
    NEWSPIDER_MODULE = '项目名.spiders'
    
    # 定义常量 (数据库地址)。 可以在代码中通过spider对象获取，也可以直接导入使用
    MONGO_HOST = "localhost"
    
    LOG_LEVEL = "WARNING"  # 设置日志等级。 默认debug级别
    LOG_FILE = "./log.log"  # 设置日志保存位置
    
    USER_AGENT = 'xxx'  # 设置请求头，模拟浏览器
    
    # 服从 robots.txt 协议
    ROBOTSTXT_OBEY = False
    
    # 最大并发请求数 (默认16)
    # CONCURRENT_REQUESTS = 32
    
    # 从同一网站下载连续页面之前应等待的时间 (默认0)。 可用于限制爬取速度
    # DOWNLOAD_DELAY = 3
    # 下面两个配置项 配合DOWNLOAD_DELAY使用
    # CONCURRENT_REQUESTS_PER_DOMAIN = 16  # 同一个域名的最大并发请求数
    # CONCURRENT_REQUESTS_PER_IP = 16   # 同一个IP的最大并发请求数
    
    # 禁用cookie (默认开启cookie)
    # COOKIES_ENABLED = False
    # COOKIES_DEBUG = True  # 开启cookie的调试信息
    
    # 禁用 Telnet Console (默认开启)
    # TELNETCONSOLE_ENABLED = False
    
    # 默认请求头 (USER_AGENT需单独设置，请求头中设置无效)
    # DEFAULT_REQUEST_HEADERS = {
    #   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    #   'Accept-Language': 'en',
    # }
    
    # 注册自定义的爬虫中间件
    #SPIDER_MIDDLEWARES = {
    #    '项目名.middlewares.YangguangSpiderMiddleware': 543,
    #}
    
    # 注册自定义的下载中间件
    # DOWNLOADER_MIDDLEWARES = {
    #    '项目名.middlewares.MyCustomDownloaderMiddleware': 543,
    # }
    
    # 注册自定义的扩展插件
    #EXTENSIONS = {
    #    'scrapy.extensions.telnet.TelnetConsole': None,
    #}
    
    # 注册pipelines
    ITEM_PIPELINES = {
       '项目名.pipelines.YangguangPipeline': 300,
    }
    
    # 开启自动限速 (默认未开启)
    #AUTOTHROTTLE_ENABLED = True
    # The initial download delay
    #AUTOTHROTTLE_START_DELAY = 5
    # The maximum download delay to be set in case of high latencies
    #AUTOTHROTTLE_MAX_DELAY = 60
    # The average number of requests Scrapy should be sending in parallel to
    # each remote server
    #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
    # Enable showing throttling stats for every response received:
    #AUTOTHROTTLE_DEBUG = False
    
    # 开启HTTP缓存 (默认未开启)
    #HTTPCACHE_ENABLED = True
    #HTTPCACHE_EXPIRATION_SECS = 0
    #HTTPCACHE_DIR = 'httpcache'
    #HTTPCACHE_IGNORE_HTTP_CODES = []
    #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

**项目名/spiders/爬虫名.py（爬虫，获取settings.py中配置的常量）：**

# -*- coding: utf-8 -*-
    import scrapy
    from 项目名.settings import MONGO_HOST   # 第一种方式：直接导入settings中的常量
    
    class DemoSpider(scrapy.Spider):
        name = '爬虫名'
        allowed_domains = ['baidu.com']
        start_urls = ['http://www.baidu.com']
    
        def parse(self, response):
            # 第二种方式：通过spider对象获取
            self.settings["MONGO_HOST"]   # 没有对应的值会报错
            self.settings.get("MONGO_HOST","默认值")  # 没有对应的值会返回默认值
            pass

**项目名/pipelines.py（管道，通过spider对象获取settings.py中配置的常量）：**

# -*- coding: utf-8 -*-
    
    from 项目名.settings import MONGO_HOST   # 第一种方式：直接导入settings中的常量
    
    
    class DemoPipeline(object):
    
        def process_item(self, item, spider):
            spider.settings.get("MONGO_HOST")  # 通过spider对象获取settings中配置的常量
            pass