scrapyd+scrapydweb部署和监控分布式爬虫项目（同一台机器）

一时失言乱红尘 2023-06-19 12:29 13阅读 0赞

**1、安装部署scrapyd**  
系统：centos7.6

安装命令：

pip3 install scrapyd（因为本地有2.7+和3.+版本python）

安装成功后新建配置文件：

sudo mkdir /etc/scrapyd

sudo vim /etc/scrapyd/scrapyd.conf

scrapyd.conf 写入如下内容：（给内容在https://scrapyd.readthedocs.io/en/stable/config.html）

\[scrapyd\]  
eggs\_dir = eggs  
logs\_dir = logs  
items\_dir =  
jobs\_to\_keep = 5  
dbs\_dir = dbs  
max\_proc = 0  
max\_proc\_per\_cpu = 10  
finished\_to\_keep = 100  
poll\_interval = 5.0  
bind\_address = 0.0.0.0  
http\_port = 6800  
debug = off  
runner = scrapyd.runner  
application = scrapyd.app.application  
launcher = scrapyd.launcher.Launcher  
webroot = scrapyd.website.Root

\[services\]  
schedule.json = scrapyd.webservice.Schedule  
cancel.json = scrapyd.webservice.Cancel  
addversion.json = scrapyd.webservice.AddVersion  
listprojects.json = scrapyd.webservice.ListProjects  
listversions.json = scrapyd.webservice.ListVersions  
listspiders.json = scrapyd.webservice.ListSpiders  
delproject.json = scrapyd.webservice.DeleteProject  
delversion.json = scrapyd.webservice.DeleteVersion  
listjobs.json = scrapyd.webservice.ListJobs  
daemonstatus.json = scrapyd.webservice.DaemonStatus  
主要修改 bind\_address = 0.0.0.0 (远程访问)  
max\_proc = 0（程序最大运行数，为0时最大执行一个爬虫）

scrapyd 在前台启动，Ctrl+c 退出后scrapyd程序停止  
scrapyd > /root/scrapyd.log & 输出日志到 /root/scrapyd.log，在后台启动。（停止ps -ef|grep scrapyd，kill -9 +进程号）  
需要在青云放开5000端口，可通过API 形式访问主机

> 参考文章地址：https://www.cnblogs.com/ss-py/p/9661928.html

**2、scrapydweb安装部署**  
安装命令：

pip install scrapydweb

启动后自动生成配置文件：

scrapydweb 在前台启动，Ctrl+c 退出后scrapyd程序停止  
scrapyd >web /root/scrapydweb.log & 输出日志到 /root/scrapydweb.log，在后台启动。（停止ps -ef|grep scrapydweb，kill -9 +进程号）  
修改配置文件：/usr/local/lib/python3.7/site-packages/scrapydweb/default\_settings.py （默认安装后配置文件在python程序site-packages/scrapydweb/目录下）  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYXRvbmcxNTE_size_16_color_FFFFFF_t_70]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYXRvbmcxNTE_size_16_color_FFFFFF_t_70 1]

修改配置文件：/usr/local/lib/python3.7/site-packages/scrapydweb/scrapydweb\_settings\_v10.py

![在这里插入图片描述][20191206173907533.png]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYXRvbmcxNTE_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191206173827890.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYXRvbmcxNTE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYXRvbmcxNTE_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191206173841164.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ppYXRvbmcxNTE=,size_16,color_FFFFFF,t_70
[20191206173907533.png]: https://img-blog.csdnimg.cn/20191206173907533.png