在centos7下安装python3.7.9并搭建scrapy2环境

蔚落 2022-12-02 00:49 263阅读 0赞

本人不懂爬虫技术。需要获取页面信息的时候，简单的需求直接用程序扒。复杂的需求己用chrome-mini向下渗透。前两天去参加了一个爬虫技术的聚会，发现这个领域非常有意思，所以回来搭建了个scrapy环境，打算学习学习。  
由于本人是小白，所以搭建过程一步一个坎。这里记录下来，希望能帮助到其他新接触的同学。

一、安装python 3.7.9

centos7自带python2.7.5版本，我尝试在这个版本下直接安装scrapy2，搭建过程没有报错，但是运行程序的时候出问题了，说python2版本太低，所以只好安装python3了。

1、安装一次性开发工具，源码安装软件需要这个  
    yum -y groupinstall "Development tools"

2、安装各种依赖包  
    yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel libffi-devel

3、下载python3.7.9安装包  
    wget "https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tgz"

4、解压缩  
    tar -zxvf Python-3.7.9.tgz

5、进入根目录  
    cd Python-3.7.9

6、编译  
    ./configure --prefix=/usr/local/python3

7、安装  
    make && make install

8、创建软链接  
    ln -s /usr/local/python3/bin/python3 /usr/bin/python3  
    ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3

9、验证  
    \[root@seeker-01 ~\]\# python  
    Python 2.7.5 (default, Aug  7 2019, 00:51:29)   
    \[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)\] on linux2  
    Type "help", "copyright", "credits" or "license" for more information.  
    >>>

\[root@seeker-01 ~\]\# python3  
    Python 3.7.9 (default, Aug 28 2020, 13:28:49)   
    \[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)\] on linux  
    Type "help", "copyright", "credits" or "license" for more information.  
    >>>

二、安装scrapy2

1、安装scrapy2  
    pip3 install scrapy

2、在python3 shell中验证scrapy2  
    \[root@seeker-01 ~\]\# python3  
    Python 3.7.9 (default, Aug 28 2020, 13:28:49)   
    \[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)\] on linux  
    Type "help", "copyright", "credits" or "license" for more information.  
    >>> import scrapy  
    >>> scrapy.version\_info  
    (2, 3, 0)  
    >>>

3、创建软scrapy链接  
    ln -s /usr/local/python3/bin/scrapy  /usr/bin/scrapy

4、验证scrapy2  
    \[root@seeker-01 ~\]\# scrapy  
    Scrapy 2.3.0 - no active project

Usage:  
      scrapy <command> \[options\] \[args\]

Available commands:  
      bench         Run quick benchmark test  
      commands        
      fetch         Fetch a URL using the Scrapy downloader  
      genspider     Generate new spider using pre-defined templates  
      runspider     Run a self-contained spider (without creating a project)  
      settings      Get settings values  
      shell         Interactive scraping console  
      startproject  Create new project  
      version       Print Scrapy version  
      view          Open URL in browser, as seen by Scrapy

\[ more \]      More commands available when run from project directory

Use "scrapy <command> -h" to see more info about a command

三、创建一个scrapy项目

1、创建项目命令  
    scrapy startproject 项目名称(我的项目叫mytest01)

\[root@seeker-01 ~\]\# scrapy startproject mytest01  
    New Scrapy project 'my\_test', using template directory '/usr/local/python3/lib/python3.7/site-packages/scrapy/templates/project', created in:  
        /root/mytest01

You can start your first spider with:  
        cd mytest01  
        scrapy genspider example example.com

执行完成后，我的/root目录下就创建了一个mytest01目录，  
    /root/mytest01/mytest01/spiders就是放置.py文件的位置，我创建的文件名是test01.py

2、写一个scrapy功能  
    在/root/mytest01/mytest01/spiders目录下创建一个test01.py文件，写入下面内容：

import scrapy

class mingyan(scrapy.Spider): \#需要继承scrapy.Spider类

name = "mytest01" \#蜘蛛名  
    def start\_requests(self): \#由此方法通过下面链接爬取页面  
        urls = \[ \#爬取的链接  
            'http://lab.scrapyd.cn/page/1/',  
            'http://lab.scrapyd.cn/page/2/',  
        \]  
        for url in urls:  
            yield scrapy.Request(url=url, callback=self.parse) \#爬取到的页面如何处理？提交给parse方法处理

def parse(self, response):

"""  
        start\_requests已经爬取到页面，那如何提取我们想要的内容呢？那就可以在这个方法里面定义。  
        这里的话，并木有定义，只是简单的把页面做了一个保存，并没有涉及提取我们想要的数据，后面会慢慢说到  
        也就是用xpath、正则、或是css进行相应提取，这个例子就是让你看看scrapy运行的流程：  
        1、定义链接；  
        2、通过链接爬取（下载）页面；  
        3、定义规则，然后提取数据；  
        就是这么个流程，似不似很简单呀？  
        """  
        page = response.url.split("/")\[-2\]     \#根据上面的链接提取分页,如：/page/1/，提取到的就是：1  
        filename = 'mingyan-%s.html' % page    \#拼接文件名，如果是第一页，最终文件名便是：mingyan-1.html  
        with open(filename, 'wb') as f:        \#python文件操作，不多说了；  
            f.write(response.body)             \#刚才下载的页面去哪里了？response.body就代表了刚才下载的页面！  
        self.log('保存文件: %s' % filename)      \# 打个日志

3、执行scrapy文件  
    scrapy crawl mytest01

2020-08-28 14:27:15 \[scrapy.utils.log\] INFO: Scrapy 2.3.0 started (bot: mingyan2)  
2020-08-28 14:27:15 \[scrapy.utils.log\] INFO: Versions: lxml 4.5.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.7.9 (default, Aug 28 2020, 13:28:49) - \[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)\], pyOpenSSL 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020), cryptography 3.1, Platform Linux-3.10.0-1062.12.1.el7.x86\_64-x86\_64-with-centos-7.6.1810-Core  
2020-08-28 14:27:15 \[scrapy.utils.log\] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor  
2020-08-28 14:27:15 \[scrapy.crawler\] INFO: Overridden settings:  
\{'BOT\_NAME': 'MyTest01',  
 'NEWSPIDER\_MODULE': 'MyTest01.spiders',  
 'ROBOTSTXT\_OBEY': True,  
 'SPIDER\_MODULES': \['MyTest01.spiders'\]\}  
2020-08-28 14:27:15 \[scrapy.extensions.telnet\] INFO: Telnet Password: 168ea9bb811b4d38  
2020-08-28 14:27:15 \[scrapy.middleware\] INFO: Enabled extensions:  
\['scrapy.extensions.corestats.CoreStats',  
 'scrapy.extensions.telnet.TelnetConsole',  
 'scrapy.extensions.memusage.MemoryUsage',  
 'scrapy.extensions.logstats.LogStats'\]  
2020-08-28 14:27:15 \[scrapy.middleware\] INFO: Enabled downloader middlewares:  
\['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',  
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',  
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',  
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',  
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',  
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',  
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',  
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',  
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',  
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',  
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',  
 'scrapy.downloadermiddlewares.stats.DownloaderStats'\]  
2020-08-28 14:27:15 \[scrapy.middleware\] INFO: Enabled spider middlewares:  
\['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',  
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',  
 'scrapy.spidermiddlewares.referer.RefererMiddleware',  
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',  
 'scrapy.spidermiddlewares.depth.DepthMiddleware'\]  
2020-08-28 14:27:15 \[scrapy.middleware\] INFO: Enabled item pipelines:  
\[\]  
2020-08-28 14:27:15 \[scrapy.core.engine\] INFO: Spider opened  
2020-08-28 14:27:15 \[scrapy.extensions.logstats\] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)  
2020-08-28 14:27:15 \[scrapy.extensions.telnet\] INFO: Telnet console listening on 127.0.0.1:6023  
2020-08-28 14:27:15 \[scrapy.core.engine\] DEBUG: Crawled (404) <GET http://lab.scrapyd.cn/robots.txt> (referer: None)  
2020-08-28 14:27:15 \[scrapy.core.engine\] DEBUG: Crawled (200) <GET http://lab.scrapyd.cn/page/1/> (referer: None)  
2020-08-28 14:27:15 \[scrapy.core.engine\] DEBUG: Crawled (200) <GET http://lab.scrapyd.cn/page/2/> (referer: None)  
2020-08-28 14:27:15 \[mingyan2\] DEBUG: 保存文件: mingyan-1.html  
2020-08-28 14:27:15 \[mingyan2\] DEBUG: 保存文件: mingyan-2.html  
2020-08-28 14:27:15 \[scrapy.core.engine\] INFO: Closing spider (finished)  
2020-08-28 14:27:15 \[scrapy.statscollectors\] INFO: Dumping Scrapy stats:  
\{'downloader/request\_bytes': 666,  
 'downloader/request\_count': 3,  
 'downloader/request\_method\_count/GET': 3,  
 'downloader/response\_bytes': 6436,  
 'downloader/response\_count': 3,  
 'downloader/response\_status\_count/200': 2,  
 'downloader/response\_status\_count/404': 1,  
 'elapsed\_time\_seconds': 0.344661,  
 'finish\_reason': 'finished',  
 'finish\_time': datetime.datetime(2020, 8, 28, 6, 27, 15, 851756),  
 'log\_count/DEBUG': 5,  
 'log\_count/INFO': 10,  
 'memusage/max': 47456256,  
 'memusage/startup': 47456256,  
 'response\_received\_count': 3,  
 'robotstxt/request\_count': 1,  
 'robotstxt/response\_count': 1,  
 'robotstxt/response\_status\_count/404': 1,  
 'scheduler/dequeued': 2,  
 'scheduler/dequeued/memory': 2,  
 'scheduler/enqueued': 2,  
 'scheduler/enqueued/memory': 2,  
 'start\_time': datetime.datetime(2020, 8, 28, 6, 27, 15, 507095)\}  
2020-08-28 14:27:15 \[scrapy.core.engine\] INFO: Spider closed (finished)

这时会在/root/MyTest01/MyTest01/spiders目录下出现两个文件：  
mytest01-1.html  
mytest01-2.html  
此时说明环境创建成功。