eight

￡神魔★判官ぃ 2021-12-09 04:51 158阅读 0赞

[eight][]

*  **BeautifulSoup 解析库**

[https://www.cnblogs.com/kermitjam/p/10863925.html][https_www.cnblogs.com_kermitjam_p_10863925.html]

主要功能：从网页爬取数据

但是这个库需要依赖一个解析器 ！！！

主要解析器，及其优缺点：

![1713218-20190621144253879-539848798.png][]

文档树 通过下一个标签查找

遍历文档树

返回一个生成器（generator） 拿到一个对象 从对象里拿到需要的内容 比直接拿到内容节省资源

可以驱动js 内部还是实现异步（并发）请求

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

1 '''
     2 主页:
     3     图标地址、下载次数、大小、详情页地址
     4 
     5 详情页:
     6     游戏名、图标名、好评率、评论数、小编点评、简介、网友评论、1-5张截图链接地址、下载地址
     7 https://www.wandoujia.com/wdjweb/api/category/more?catId=6001&subCatId=0&page=1&ctoken=FRsWKgWBqMBZLdxLaK4iem9B
     8 
     9 https://www.wandoujia.com/wdjweb/api/category/more?catId=6001&subCatId=0&page=2&ctoken=FRsWKgWBqMBZLdxLaK4iem9B
    10 
    11 https://www.wandoujia.com/wdjweb/api/category/more?catId=6001&subCatId=0&page=3&ctoken=FRsWKgWBqMBZLdxLaK4iem9B
    12 
    13 32
    14 '''
    15 import requests
    16 from bs4 import BeautifulSoup
    17 # 1、发送请求
    18 def get_page(url):
    19     response = requests.get(url)
    20     return response
    21 
    22 # 2、开始解析
    23 # 解析主页
    24 def parse_index(data):
    25     soup = BeautifulSoup(data, 'lxml')
    26 
    27     # 获取所有app的li标签
    28     app_list = soup.find_all(name='li', attrs={
            "class": "card"})
    29     for app in app_list:
    30         # print('tank *' * 1000)
    31         # print(app)
    32         # 图标地址
    33         img = app.find(name='img').attrs['data-original']
    34         print(img)
    35 
    36         # 下载次数
    37         down_num = app.find(name='span', attrs={
            "class": "install-count"}).text
    38         print(down_num)
    39 
    40         import re
    41         # 大小
    42         size = soup.find(name='span', text=re.compile("\d+MB")).text
    43         print(size)
    44 
    45         # 详情页地址
    46         detail_url = soup.find(name='a', attrs={
            "class": "detail-check-btn"}).attrs['href']
    47         print(detail_url)
    48 
    49 
    50 def main():
    51     for line in range(1, 33):
    52         url = f"https://www.wandoujia.com/wdjweb/api/category/more?catId=6001&subCatId=0&page={line}&ctoken=FRsWKgWBqMBZLdxLaK4iem9B"
    53 
    54         # 1、往app接口发送请求
    55         response = get_page(url)
    56         # print(response.text)
    57         print('*' * 1000)
    58         # 反序列化为字典
    59         data = response.json()
    60         # 获取接口中app标签数据
    61         app_li = data['data']['content']
    62         # print(app_li)
    63         # 2、解析app标签数据
    64         parse_index(app_li)
    65 
    66 
    67 if __name__ == '__main__':
    68     main()

**爬取豌豆荚app数据（提取游戏主页）**

![1713218-20190621144806535-2009499721.png][]

![1713218-20190621144927465-280600926.png][]

![1713218-20190621145001314-1128211794.png][]

![1713218-20190621145017511-1598353732.png][]

*  **MongoDB 存储库**

https://www.cnblogs.com/kermitjam/p/10863933.html\#\_label1

1.  ** 安装：**

https://www.mongodb.com/download-center\#community （下载地址）

**![1713218-20190621145234307-354512550.png][]**

![1713218-20190621145254864-1264679914.png][]

安装好数据库后 添加环境变量 mongodb bin   将bin目录添加到环境变量中

![1713218-20190621145452741-506967886.png][]

![1713218-20190621145739308-2423620.png][]

![1713218-20190621145749884-679956692.png][]

![1713218-20190621145945661-64917919.png][]

![1713218-20190621145952158-2107330541.png][]

![1713218-20190621145959508-477260962.png][]

![1713218-20190621150005309-333579294.png][]

**2.数据库操作**

**![1713218-20190621152205327-403981590.png][]**

![1713218-20190621152216065-1832102955.png][]

![1713218-20190621152228349-1544268993.png][]

![1713218-20190621152234506-441403512.png][]

![1713218-20190621152244127-681858458.png][]

*  **python链接MongoDB**

**![1713218-20190621152505116-1675178260.png][]**

pip3 install pymongo

![1713218-20190621152554892-42579658.png][]

Mongobd可视化工具 robt3t

posted on 2019-06-21 15:44 [LettersfromSuperz][] 阅读( ...) 评论( ...) [编辑][Link 1] 收藏

转载于:https://www.cnblogs.com/LettersfromSuperz/p/11064995.html

[eight]: https://www.cnblogs.com/LettersfromSuperz/p/11064995.html
[https_www.cnblogs.com_kermitjam_p_10863925.html]: https://www.cnblogs.com/kermitjam/p/10863925.html
[1713218-20190621144253879-539848798.png]: /images/20211209/c3c1ff5ab2f54840999892adf6bfb4b8.png
[ContractedBlock.gif]: https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif
[ExpandedBlockStart.gif]: /images/20211209/aba5926251be41588936cc6322fd5ba5.png
[1713218-20190621144806535-2009499721.png]: /images/20211209/650a3139e11f41c592609ff7ae03316e.png
[1713218-20190621144927465-280600926.png]: /images/20211209/79b5cce1dbf64851810f9e67c3b535bc.png
[1713218-20190621145001314-1128211794.png]: /images/20211209/7a37490c9d0f4503948060c424f83f0c.png
[1713218-20190621145017511-1598353732.png]: /images/20211209/bf03d0943101487686c782ac83d9894c.png
[1713218-20190621145234307-354512550.png]: /images/20211209/14756f0f8a334fc08cbd8aeb9bf197a7.png
[1713218-20190621145254864-1264679914.png]: /images/20211209/f027a04191a94d58ab84127e14ff2b5a.png
[1713218-20190621145452741-506967886.png]: /images/20211209/086143a408b54590ab4fab51b8577158.png
[1713218-20190621145739308-2423620.png]: /images/20211209/33b19a37534647d1b0bc47a18731ddea.png
[1713218-20190621145749884-679956692.png]: /images/20211209/5c3fc3d8f4514486b657c30d45ecd919.png
[1713218-20190621145945661-64917919.png]: /images/20211209/46ee97c574314359b77b1516c66aa93c.png
[1713218-20190621145952158-2107330541.png]: /images/20211209/f6c875b98e3a42fcae63cfb1c9f6bb2d.png
[1713218-20190621145959508-477260962.png]: /images/20211209/aa98a150a59842d5b67394c39948a4eb.png
[1713218-20190621150005309-333579294.png]: /images/20211209/fdcbf2b468c2402e950ad15affdddfba.png
[1713218-20190621152205327-403981590.png]: /images/20211209/d6f9aed4a3cc491183ce02285c598b58.png
[1713218-20190621152216065-1832102955.png]: /images/20211209/70818c3e6cc94190851128fe1bd6e256.png
[1713218-20190621152228349-1544268993.png]: /images/20211209/97966eac067c495ea3cf758a43a874b1.png
[1713218-20190621152234506-441403512.png]: /images/20211209/64f2f47524914d8e9038212f46e26675.png
[1713218-20190621152244127-681858458.png]: /images/20211209/3e2c3be209324331a74a9952e18bed94.png
[1713218-20190621152505116-1675178260.png]: /images/20211209/93094152de5f47f5842bd6ec50e5fd69.png
[1713218-20190621152554892-42579658.png]: /images/20211209/d6351bcb9ad84e6f93b089025ca7d714.png
[LettersfromSuperz]: https://www.cnblogs.com/LettersfromSuperz/
[Link 1]: https://i.cnblogs.com/EditPosts.aspx?postid=11064995