Python爬虫系列之----Scrapy(四)一个简单的示例

旧城等待， 2022-06-17 03:22 221阅读 0赞

**一、创建一个简单的项目**

**注:以下使用的python3**

在使用Scrapy之前先要创建一个Scrapy项目,可以通过startproject命令来实现,首先在CMD中进入用来储存新建爬虫项目的文件夹,比如我们要在"G:\\Scrapy\_work"目录中创建Scrapy爬虫,可以在CMD中进入该文件夹,然后执行命令:

scrapy startproject myfendo

![20170415173908215][]

![20170415173948746][]

目录结构大概是这样的:

![20170415174927438][]

我们也可以在创建项目的时候加上一些参数进行控制,我们可以通过scrapy startproject -h调出start-project的帮助信息:

![20170415175207876][]

**二、定义Item**

这里以:dmoz.org网站进行测试,Dmoz(The Open Directory Project/ODP)是一个人工编辑管理的目录集合，为 搜索引擎提供结果或数据。

目前这个网站已经关闭。。。

![20170415175728953][]

Item 是保存爬取到的数据的容器；其使用方法和python字典类似， 并且提供了额外保护机制来避免拼写错误导致的未定义字段错误。

首先根据需要从dmoz.org获取到的数据对item进行建模。 我们需要从dmoz.org中获取名字,url,以及网站的描述。 对此，在item中定义相应的字段。编辑 myfendo 目录中的 items.py 文件:

import scrapy
    
    class MyfendoItem(scrapy.Item):
          title=scrapy.Field()
          link=scrapy.Field()
          desc=scrapy.Field()

**三、编写第一个爬虫(Spider)**

Spider是用户编写用于从单个网站(或者一些网站)爬取数据的类。  
  
其包含了一个用于下载的初始URL，如何跟进网页中的链接以及如何分析页面中的内容， 提取生成 item 的方法。  
  
为了创建一个Spider，您必须继承 scrapy.Spider 类， 且定义以下三个属性:  
  
● name: 用于区别Spider。 该名字必须是唯一的，您不可以为不同的Spider设定相同的名字。  
● start\_urls: 包含了Spider在启动时进行爬取的url列表。 因此，第一个被获取到的页面将是其中之一。 后续的URL则从初始的URL获取到的数据中提取。  
● parse() 是spider的一个方法。 被调用时，每个初始URL完成下载后生成的 Response 对象将会作为唯一的参数传递给该函数。 该方法负责解析返回的数据(response data)，提取数据(生成item)以及生成需要进一步处理的URL的 Request 对象。

以下为我们的第一个Spider代码，保存在 myfendo/spiders 目录下的 dmoz\_spider.py 文件中,可以通过genspider命令先查看可以使用的爬虫模板

scrapy genspider -l

![20170415180825733][]

然后才创建一个爬虫模板:scrapy genspider \[-t 模板名称\]  文件名  域名

scrapy genspider -t basic myfendos dmoz.org

![20170415181306676][]

创建成功目录下会多出个文件,文件的内容为

# -*- coding: utf-8 -*-
    import scrapy
    
    
    class MyfendosSpider(scrapy.Spider):
        name = "myfendos"
        allowed_domains = ["dmoz.org"]
        start_urls = (
            'http://www.dmoz.org/',
        )
    
        def parse(self, response):
            pass

修改成如下:

import scrapy
    
    
    class MyfendosSpider(scrapy.Spider):
        name = "myfendos"
        allowed_domains = ["dmoz.org"]
        start_urls = [
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
        ]
    
        def parse(self, response):
            filename = response.url.split("/")[-2]
            with open(filename, 'wb') as f:
                f.write(response.body)

**四、运行爬虫**

进入项目的根目录，执行下列命令启动spider:

G:\Scrapy_work\myfendo>scrapy crawl myfendos
    2017-04-15 18:17:31 [scrapy] INFO: Scrapy 1.1.0rc3 started (bot: myfendo)
    2017-04-15 18:17:31 [scrapy] INFO: Overridden settings: {'BOT_NAME': 'myfendo', 'NEWSPIDER_MODULE': 'myfendo.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['myfendo.spiders']}
    2017-04-15 18:17:31 [scrapy] INFO: Enabled extensions:
    ['scrapy.extensions.corestats.CoreStats',
     'scrapy.extensions.telnet.TelnetConsole',
     'scrapy.extensions.logstats.LogStats']
    Unhandled error in Deferred:
    2017-04-15 18:17:31 [twisted] CRITICAL: Unhandled error in Deferred:
    
    2017-04-15 18:17:31 [twisted] CRITICAL:
    Traceback (most recent call last):
      File "f:\software\python36\lib\site-packages\twisted\internet\defer.py", line 1301, in _inlineCallbacks
        result = g.send(result)
      File "f:\software\python36\lib\site-packages\scrapy\crawler.py", line 72, in crawl
        self.engine = self._create_engine()
      File "f:\software\python36\lib\site-packages\scrapy\crawler.py", line 97, in _create_engine
        return ExecutionEngine(self, lambda _: self.stop())
      File "f:\software\python36\lib\site-packages\scrapy\core\engine.py", line 68, in __init__
        self.downloader = downloader_cls(crawler)
      File "f:\software\python36\lib\site-packages\scrapy\core\downloader\__init__.py", line 88, in __init__
        self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
      File "f:\software\python36\lib\site-packages\scrapy\middleware.py", line 58, in from_crawler
        return cls.from_settings(crawler.settings, crawler)
      File "f:\software\python36\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
        mwcls = load_object(clspath)
      File "f:\software\python36\lib\site-packages\scrapy\utils\misc.py", line 44, in load_object
        mod = import_module(module)
      File "f:\software\python36\lib\importlib\__init__.py", line 126, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 978, in _gcd_import
      File "<frozen importlib._bootstrap>", line 961, in _find_and_load
      File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 678, in exec_module
      File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
      File "f:\software\python36\lib\site-packages\scrapy\downloadermiddlewares\retry.py", line 23, in <module>
        from scrapy.xlib.tx import ResponseFailed
      File "f:\software\python36\lib\site-packages\scrapy\xlib\tx\__init__.py", line 3, in <module>
        from twisted.web import client
      File "f:\software\python36\lib\site-packages\twisted\web\client.py", line 42, in <module>
        from twisted.internet.endpoints import HostnameEndpoint, wrapClientTLS
      File "f:\software\python36\lib\site-packages\twisted\internet\endpoints.py", line 37, in <module>
        from twisted.internet.stdio import StandardIO, PipeAddress
      File "f:\software\python36\lib\site-packages\twisted\internet\stdio.py", line 30, in <module>
        from twisted.internet import _win32stdio
      File "f:\software\python36\lib\site-packages\twisted\internet\_win32stdio.py", line 9, in <module>
        import win32api
    ModuleNotFoundError: No module named 'win32api'

报错提示找不到:"win32api"去这个网站:[http://www.lfd.uci.edu/~gohlke/pythonlibs/\#pywin32][http_www.lfd.uci.edu_gohlke_pythonlibs_pywin32] 下载pywin32-221-cp36-cp36m-win\_amd64.exe

![20170415182402581][]

下载下来之后,进行安装

![20170415183713900][]

安装成功,运行爬虫又报错。。

G:\Scrapy_work\myfendo>scrapy crawl myfendos
    2017-04-15 18:38:24 [scrapy] INFO: Scrapy 1.1.0rc3 started (bot: myfendo)
    2017-04-15 18:38:24 [scrapy] INFO: Overridden settings: {'BOT_NAME': 'myfendo', 'NEWSPIDER_MODULE': 'myfendo.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['myfendo.spiders']}
    2017-04-15 18:38:24 [scrapy] INFO: Enabled extensions:
    ['scrapy.extensions.corestats.CoreStats',
     'scrapy.extensions.telnet.TelnetConsole',
     'scrapy.extensions.logstats.LogStats']
    Unhandled error in Deferred:
    2017-04-15 18:38:24 [twisted] CRITICAL: Unhandled error in Deferred:
    
    2017-04-15 18:38:24 [twisted] CRITICAL:
    Traceback (most recent call last):
      File "f:\software\python36\lib\site-packages\twisted\internet\defer.py", line 1301, in _inlineCallbacks
        result = g.send(result)
      File "f:\software\python36\lib\site-packages\scrapy\crawler.py", line 72, in crawl
        self.engine = self._create_engine()
      File "f:\software\python36\lib\site-packages\scrapy\crawler.py", line 97, in _create_engine
        return ExecutionEngine(self, lambda _: self.stop())
      File "f:\software\python36\lib\site-packages\scrapy\core\engine.py", line 68, in __init__
        self.downloader = downloader_cls(crawler)
      File "f:\software\python36\lib\site-packages\scrapy\core\downloader\__init__.py", line 88, in __init__
        self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
      File "f:\software\python36\lib\site-packages\scrapy\middleware.py", line 58, in from_crawler
        return cls.from_settings(crawler.settings, crawler)
      File "f:\software\python36\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
        mwcls = load_object(clspath)
      File "f:\software\python36\lib\site-packages\scrapy\utils\misc.py", line 44, in load_object
        mod = import_module(module)
      File "f:\software\python36\lib\importlib\__init__.py", line 126, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 978, in _gcd_import
      File "<frozen importlib._bootstrap>", line 961, in _find_and_load
      File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 678, in exec_module
      File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
      File "f:\software\python36\lib\site-packages\scrapy\downloadermiddlewares\retry.py", line 23, in <module>
        from scrapy.xlib.tx import ResponseFailed
      File "f:\software\python36\lib\site-packages\scrapy\xlib\tx\__init__.py", line 3, in <module>
        from twisted.web import client
      File "f:\software\python36\lib\site-packages\twisted\web\client.py", line 42, in <module>
        from twisted.internet.endpoints import HostnameEndpoint, wrapClientTLS
      File "f:\software\python36\lib\site-packages\twisted\internet\endpoints.py", line 37, in <module>
        from twisted.internet.stdio import StandardIO, PipeAddress
      File "f:\software\python36\lib\site-packages\twisted\internet\stdio.py", line 30, in <module>
        from twisted.internet import _win32stdio
      File "f:\software\python36\lib\site-packages\twisted\internet\_win32stdio.py", line 9, in <module>
        import win32api
    ImportError: DLL load failed: 找不到指定的模块。

解决方案：将F:\\Software\\Python36\\Lib\\site-packages\\pywin32\_system32目录创建及以下2个DLL文件复制到system32中：

![20170415192533886][]

然后再次执行命令:

scrapy crawl myfendos

执行成功,不过返回403,由于这个网站目前已经关闭,所以返回的是403!!

G:\Scrapy_work\myfendo>scrapy crawl myfendos
    2017-04-15 19:57:17 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: myfendo)
    2017-04-15 19:57:17 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'myfendo', 'NEWSPIDER_MODULE': 'myfendo.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['myfendo.spiders']}
    2017-04-15 19:57:17 [scrapy.middleware] INFO: Enabled extensions:
    ['scrapy.extensions.corestats.CoreStats',
     'scrapy.extensions.telnet.TelnetConsole',
     'scrapy.extensions.logstats.LogStats']
    2017-04-15 19:57:18 [scrapy.middleware] INFO: Enabled downloader middlewares:
    ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
     'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
     'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
     'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
     'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
     'scrapy.downloadermiddlewares.retry.RetryMiddleware',
     'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
     'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
     'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
     'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
     'scrapy.downloadermiddlewares.stats.DownloaderStats']
    2017-04-15 19:57:18 [scrapy.middleware] INFO: Enabled spider middlewares:
    ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
     'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
     'scrapy.spidermiddlewares.referer.RefererMiddleware',
     'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
     'scrapy.spidermiddlewares.depth.DepthMiddleware']
    2017-04-15 19:57:18 [scrapy.middleware] INFO: Enabled item pipelines:
    []
    2017-04-15 19:57:18 [scrapy.core.engine] INFO: Spider opened
    2017-04-15 19:57:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2017-04-15 19:57:18 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
    2017-04-15 19:57:19 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://www.dmoz.org/robots.txt> (referer: None)
    2017-04-15 19:57:20 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None)
    2017-04-15 19:57:20 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>: HTTP status code is not handled or not allowed
    2017-04-15 19:57:20 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/> (referer: None)
    2017-04-15 19:57:20 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/>: HTTP status code is not handled or not allowed
    2017-04-15 19:57:20 [scrapy.core.engine] INFO: Closing spider (finished)
    2017-04-15 19:57:20 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 734,
     'downloader/request_count': 3,
     'downloader/request_method_count/GET': 3,
     'downloader/response_bytes': 3525,
     'downloader/response_count': 3,
     'downloader/response_status_count/403': 3,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2017, 4, 15, 11, 57, 20, 239129),
     'log_count/DEBUG': 4,
     'log_count/INFO': 9,
     'response_received_count': 3,
     'scheduler/dequeued': 2,
     'scheduler/dequeued/memory': 2,
     'scheduler/enqueued': 2,
     'scheduler/enqueued/memory': 2,
     'start_time': datetime.datetime(2017, 4, 15, 11, 57, 18, 65061)}
    2017-04-15 19:57:20 [scrapy.core.engine] INFO: Spider closed (finished)
    
    G:\Scrapy_work\myfendo>

[20170415173908215]: /images/20220617/3b3176c01bc640c18ede568ec1bd466c.png
[20170415173948746]: /images/20220617/1b9025ef9ee942b2b046f071716a516d.png
[20170415174927438]: /images/20220617/6034569c991c414cbf7c08c3b0706159.png
[20170415175207876]: /images/20220617/5946cf2c3d424258b6492ad82ebb6737.png
[20170415175728953]: /images/20220617/a7faaa9653404cc0be0b38bddd5479fb.png
[20170415180825733]: /images/20220617/25fde88723504c39a9e3b313d4418ba7.png
[20170415181306676]: /images/20220617/3ecaa8a48f044fe7b260f02c81a1b6b6.png
[http_www.lfd.uci.edu_gohlke_pythonlibs_pywin32]: http://www.lfd.uci.edu/~gohlke/pythonlibs/#pywin32
[20170415182402581]: /images/20220617/29334b01672e4697a13bf9272e92c49a.png
[20170415183713900]: /images/20220617/41daddcfe1444910b3f7c2e6d09184a6.png
[20170415192533886]: /images/20220617/9fec99fc91604f678f1601a454044870.png