运行scrapy demo时报错：[twisted] CRITICAL: Unhandled error in Deferred

清疚 2021-09-28 23:34 332阅读 0赞

# 报错+分析 #

原始报错

...
    2019-06-11 01:23:22 [scrapy.core.engine] INFO: Spider opened
    Unhandled error in Deferred:
    2019-06-11 01:23:22 [twisted] CRITICAL: Unhandled error in Deferred:
    
    Traceback (most recent call last):
      File "c:\programdata\anaconda2\lib\site-packages\scrapy\crawler.py", line 172, in crawl
        return self._crawl(crawler, *args, **kwargs)
      File "c:\programdata\anaconda2\lib\site-packages\scrapy\crawler.py", line 176, in _crawl
        d = crawler.crawl(*args, **kwargs)
      File "c:\programdata\anaconda2\lib\site-packages\twisted\internet\defer.py", line 1613, in unwindGenerator
        return _cancellableInlineCallbacks(gen)
      File "c:\programdata\anaconda2\lib\site-packages\twisted\internet\defer.py", line 1529, in _cancellableInlineCallbacks
        _inlineCallbacks(None, g, status)
    --- <exception caught here> ---
      File "c:\programdata\anaconda2\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
        result = g.send(result)
      File "c:\programdata\anaconda2\lib\site-packages\scrapy\crawler.py", line 98, in crawl
        six.reraise(*exc_info)
      File "c:\programdata\anaconda2\lib\site-packages\scrapy\crawler.py", line 82, in crawl
        yield self.engine.open_spider(self.spider, start_requests)
    exceptions.ImportError: DLL load failed: 锟揭诧拷锟斤拷指锟斤拷锟斤拷模锟介。
    ...

一直以为是twisted和python环境不匹配。先查看了python的环境：

-- python环境是win 32 的2.7版本
    D:\qqg\mama\mama\spiders>python
    Python 2.7.16 |Anaconda, Inc.| (default, Mar 14 2019, 15:49:40) [MSC v.1500 32 bit (Intel)] on win32

安装的twisted模块是：Twisted-19.2.1-cp27-cp27m-win32.whl ，19.2.1版本，python2.7环境，win32位系统，怎么都没错。

> 很多人是64位系统，安装了python是32位的，就按照Twisted64位的，这样是错误的，Twisted要和python保持一致。

在git的论坛是看到，发现可能是sqlite的问题，cmd下进入python命令，导入这个包试试：论坛： [https://github.com/scrapy/scrapy/issues/2402][https_github.com_scrapy_scrapy_issues_2402]

D:\qqg\mama\mama\spiders>python
    Python 2.7.16 |Anaconda, Inc.| (default, Mar 14 2019, 15:49:40) [MSC v.1500 32 bit (Intel)] on win32
    
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sqlite
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ImportError: No module named sqlite
    >>> import sqlite3
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\ProgramData\Anaconda2\lib\sqlite3\__init__.py", line 24, in <module>
        from dbapi2 import *
      File "C:\ProgramData\Anaconda2\lib\sqlite3\dbapi2.py", line 28, in <module>
        from _sqlite3 import *
    ImportError: DLL load failed: 找不到指定的模块。

还真是这个包报错，解决办法：导入sqlite3.def和sqlite3.dll两个文件到本地的C:\*\*\\Anaconda2\\DLLs 文件夹里面即可。就是你本地的python环境下的DLLs包。  
sqlite官网： [https://www.sqlite.org/download.html][https_www.sqlite.org_download.html]  
![sqlite3官网][sqlite3]

--------------------

这个问题坑了我一晚上，哎，看视频的时候早就有提示了，但是我看到Anaconda2里面有sqlite的模块，就以为没事了，最后百度了一晚上，看到一个外国网站上有个人提示说 import sqlite 试试，结果还真是这个原因，自己初学爬虫，什么都不懂，见谅了，凌晨2点了，休息了。

[https_github.com_scrapy_scrapy_issues_2402]: https://github.com/scrapy/scrapy/issues/2402
[https_www.sqlite.org_download.html]: https://www.sqlite.org/download.html
[sqlite3]: /images/20210923/34557bfc815241329d92781420081e52.png