nCoV_Crawler2019疫情爬虫笔记

拼搏现实的明天。 2023-07-19 08:27 17阅读 0赞

## nCoV\_Crawler2019疫情爬虫笔记： ##

## GitHub： ##

[Ticsmyc/2019nCoV-Crawler][Ticsmyc_2019nCoV-Crawler]

爬冠状病毒新型肺炎疫情实时数据+数据持久化+邮件通知

## 163邮箱获取授权码： ##

[163邮箱如何开始POP3协议，获取授权码-百度经验][163_POP3_-]

## QQ邮箱获取授权码： ##

[QQ邮箱如何开启POP3协议并获取授权码-百度经验][QQ_POP3_-]

## 报错： ##

Jsoup获取部分页面数据失败 org.jsoup.UnsupportedMimeTypeException: 
    Unhandled content type. Must be text/*, application/xml, or application/xhtml+xml.

## 解决方案： ##

配置.ignoreContentType(true)
    Document doc = Jsoup.connect(url).ignoreContentType(true).
    timeout(3000).header(ProxyHeadKey, ProxyHeadVal).proxy(proxy).get();

## Jsoup如何配置代理访问： ##

参考其他文章

## 报错： ##

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
    PKIX path building failed: sun.security.provider.certpath.SunCertPath
    BuilderException: unable to find valid certification path to requested target

## 解决方案： ##

信任一切SSL证书，信任任何站点，实现https页面的正常访问
    参考其他文章

## 报错： ##

### Error querying database.  Cause: com.mysql.cj.jdbc.exceptions.
    PacketTooBigException: Packet for query is too large (4,739,923 > 65,535).
    You can change this value on the server by setting the 'max_allowed_packet' variable.

## 解决方案： ##

用mybatis批量插入八千多条数据，执行抛出异常，意思是这个语句执行的数据太大了，需要修改mysql配置文件的max\_allowed\_packet属性值。

查看是否是mysql最大允许数据包的容量太小：

SHOW VARIABLES LIKE '%max_allowed_packet%';
    max_allowed_packet	4194304
    slave_max_allowed_packet	1073741824

## 如何修改？ ##

mysql命令行：
    set global max_allowed_packet = 2*1024*1024*10
    然后退出命令行，重启mysql服务，再进入。
    show VARIABLES like '%max_allowed_packet%';

## 安装mysql数据库： ##

[https://downloads.mysql.com/archives/installer/][https_downloads.mysql.com_archives_installer] [【MySQL下载】2020年最新官方正式版MySQL免费下载 - 腾讯软件中心官网][MySQL_2020_MySQL_ -]

## 数据爬取截图： ##

![v2-2b9b8cb1ed8c899962ea817bca27433d\_b.jpg][v2-2b9b8cb1ed8c899962ea817bca27433d_b.jpg]

![v2-ed2b560d0507e543b8692a7b19d78ebf\_b.jpg][v2-ed2b560d0507e543b8692a7b19d78ebf_b.jpg]

![v2-864f8c04277139407a974a0a6510acdd\_b.jpg][v2-864f8c04277139407a974a0a6510acdd_b.jpg]

[Ticsmyc_2019nCoV-Crawler]: https://link.zhihu.com/?target=https%3A//gitee.com/TicsmycL/nCoV_Crawler2019
[163_POP3_-]: https://link.zhihu.com/?target=https%3A//jingyan.baidu.com/article/adc815139f60c2f723bf7385.html
[QQ_POP3_-]: https://link.zhihu.com/?target=https%3A//jingyan.baidu.com/article/90895e0f2af42664ec6b0b14.html
[https_downloads.mysql.com_archives_installer]: https://link.zhihu.com/?target=https%3A//downloads.mysql.com/archives/installer/
[MySQL_2020_MySQL_ -]: https://link.zhihu.com/?target=https%3A//pc.qq.com/detail/3/detail_1303.html
[v2-2b9b8cb1ed8c899962ea817bca27433d_b.jpg]: /images/20230528/5415a46733de48588fc4ca35bec0079d.png
[v2-ed2b560d0507e543b8692a7b19d78ebf_b.jpg]: /images/20230528/15b5573b047d45ada88a008ff944d8b3.png
[v2-864f8c04277139407a974a0a6510acdd_b.jpg]: /images/20230528/d8130f304ec9448f8eefb8d3b82dccd7.png