scrapy在项目的settings.py文件中进行配置中间件的先后顺序。

ゞ浴缸里的玫瑰 2023-10-08 17:46 37阅读 0赞

在 Scrapy 项目的 settings.py 文件中配置中间件的先后顺序为:

1.  先配置自定义中间件, 设置在 `DOWNLOADER_MIDDLEWARES` 和 `SPIDER_MIDDLEWARES` 中
2.  接着配置内置中间件

例如:

DOWNLOADER_MIDDLEWARES = {
        'myproject.middlewares.MyCustomDownloaderMiddleware': 543,
        'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': None,
        'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware': 350,
        'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 400,
        'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 500,
        'scrapy.downloadermiddlewares.retry.RetryMiddleware': 550,
        'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580,
        'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590,
        'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600,
        'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700,
        'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
        'scrapy.downloadermiddlewares.stats.DownloaderStats': 850,
        'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900,
    }
    
    SPIDER_MIDDLEWARES = {
        'myproject.middlewares.MyCustomSpiderMiddleware': 543,
        'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware': 50,
        'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': 500,
        'scrapy.spidermiddlewares.referer.RefererMiddleware': 700,
        'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware': 800,
        'scrapy.spidermiddlewares.depth.DepthMiddleware': 900,
    }

这里你可以看到在自定义中间件之后是默认的中间件，而且在配置时也会有优先级这么一个概念，数字越小优先级越高。