python中request请求库与BeautifulSoup解析库的用法

古城微笑少年丶 2024-03-27 08:55 69阅读 0赞

## python中request请求库与BeautifulSoup解析库的用法 ##

### request ###

#### 安装 ####

打开cmd窗口，检查python环境，需要python3.7版本及以上  
![在这里插入图片描述][e4254d9b35f34a4fa7814ec6c61d8347.png]  
然后输入，下载requests库

pip install requests -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com

![在这里插入图片描述][ee4dfa446f0d455da9bb815c545e1468.png]

#### 创建项目 ####

![在这里插入图片描述][48575faf2e5a41309c081a2e56501a14.png]  
创建python文件，最好不要含有中文字符  
![在这里插入图片描述][1d6bec36cb7a4b1ea592311b6443dc9c.png]

#### 测试代码 ####

![在这里插入图片描述][a5174103644e4eb3bebf48f520e56d43.png]

# 1.导入模块
    # 1.导入模块
    import requests
    
    # 2. 发送请求，获取响应
    response = requests.get("http://www.baidu.com")
    print(response)  # 这里打印的结果是响应码
    
    # 3. 获取响应数据
    # print(response.encoding) # ISO-8859-1
    
    # response.encoding = 'utf-8' # 设置编码格式
    # print(response.text)
    
    # 上面两句话等于下面一句话
    print(response.content.decode())

**运行结果**：  
![在这里插入图片描述][4499387bae3e44a7a87d4e58215a81ac.png]

#### 小案例（请求疫情首页） ####

![在这里插入图片描述][b77260a7773c4ba2855fe03ae03971ea.png]  
**案例代码**：

# 1. 导入模块
    import requests
    
    # 2. 发送请求，获取响应
    response = requests.get("https://ncov.dxy.cn/ncovh5/view/pneumonia")
    
    # 3. 从响应中获取数据
    print(response.content.decode())

**运行结果**：  
![在这里插入图片描述][f05a9d15b8a141b68fe8b9e31b66a9ee.png]

### BeautifulSoup ###

#### 简介 ####

![在这里插入图片描述][51eed760516d45d9ac12671dba72f6dd.png]  
Beautiful Soup 是一个可以从HTML或XML文件中提取数据的Python库.它能够通过你喜欢的转换器实现惯用的文档导航,查找,修改文档的方式.Beautiful Soup会帮你节省数小时甚至数天的工作时间.

#### 安装 ####

运行下面两行命令，或者pycharm可以自动安装。  
`pip install bs4`  
`pip install lxml`

#### 学习代码 ####

# 1. 导入模块
 from bs4 import BeautifulSoup
 
 # 2. 创建BeautifulSoup对象
 soup = BeautifulSoup('<html>data</html>', 'lxml')
 print(soup)

#### 运行结果 ####

![在这里插入图片描述][8468645d612e4673bb5afd082f2dcd49.png]

#### find方法 ####

##### 简介 #####

![在这里插入图片描述][c7ea96d030d04e94b996d9b7724c504d.png]

##### 案例（根据标签名查找） #####

![在这里插入图片描述][ff3b84ea841f4531a5aa9e02fadbae0e.png]  
![在这里插入图片描述][5a79116184844d86b8a256d76cadd1db.png]

**案例代码**：

# 1.导入模块
 from bs4 import BeautifulSoup
 
 # 2.准备文本字符串
 html = '''
 <title>The Dormouse's story</title>
 </head>
 <body>
 
 The Dormouse's story
 
 Once Upon a time three were three little sister;and their names were
 <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>, 
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>and 
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well. 
 
 ...
 </body>
 </html>
 '''
 
 # 3.创建BeautifulSoup对象
 soup = BeautifulSoup(html,'lxml')
 
 # 4.查找title标签
 title = soup.find('title')
 print(title)
 
 # 5.查找a标签
 a = soup.find('a')
 print(a)
 
 #查找所有a标签
 a_s = soup.find_all('a')
 print(a_s)

**运行结果：**  
![在这里插入图片描述][a5906d580e06451fa494cc97618c6318.png]

##### 案例（根据属性查找） #####

![在这里插入图片描述][719800a2655d46369d6e5da05472768b.png]

**案例代码**

# 1.导入模块
 from bs4 import BeautifulSoup
 
 # 2.准备文本字符串
 html = '''
 <title>The Dormouse's story</title>
 </head>
 <body>
 
 The Dormouse's story
 
 Once Upon a time three were three little sister;and their names were
 <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>, 
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>and 
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well. 
 
 ...
 </body>
 </html>
 '''
 
 # 3.创建BeautifulSoup对象
 soup = BeautifulSoup(html,'lxml')
 
 # 二、根据属性查找
 #查找 id 为 link1 的标签
 #方法一：通过命名参数进行查找
 a = soup.find(id = 'link1')
 print(a)
 
 #方法二：使用attrs来指定属性字典，进行查找
 a = soup.find(attrs={
 'id':'link1'})
 print(a)

**运行结果**  
![在这里插入图片描述][caf01f2382a548bebc62de1019289323.png]

##### 案例（根据文本查找） #####

![在这里插入图片描述][497693dd511a4736ab7542cbbb695e68.png]  
**案例代码**

# 1.导入模块
 from bs4 import BeautifulSoup
 
 # 2.准备文本字符串
 html = '''
 <title>The Dormouse's story</title>
 </head>
 <body>
 
 The Dormouse's story
 
 Once Upon a time three were three little sister;and their names were
 <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>, 
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>and 
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well. 
 
 ...
 </body>
 </html>
 '''
 
 # 3.创建BeautifulSoup对象
 soup = BeautifulSoup(html,'lxml')
 
 #三、根据文本查找
 # 获取下面文档中文本为 Elsie 的标签文本
 text = soup.find(text='Elsie')
 print(text)

**运行结果**  
![在这里插入图片描述][c447eb695bc6403aa777a5856bb1c7bd.png]

##### 案例（Tag属性使用） #####

![在这里插入图片描述][b069f407195247ef9f48c28deb6a2b61.png]  
**案例代码**

# 1.导入模块
 from bs4 import BeautifulSoup
 
 # 2.准备文本字符串
 html = '''
 <title>The Dormouse's story</title>
 </head>
 <body>
 
 The Dormouse's story
 
 Once Upon a time three were three little sister;and their names were
 <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>, 
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>and 
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well. 
 
 ...
 </body>
 </html>
 '''
 
 # 3.创建BeautifulSoup对象
 soup = BeautifulSoup(html,'lxml')
 
 a = soup.find(attrs={
 'id':'link1'})
 
 #Tag对象
 print(type(a)) #<class 'bs4.element.Tag'>
 print('标签名：',a.name)
 print('标签所有属性:',a.attrs) #输出的class是一个列表，class 一个属性中可以有多个值
 print('标签文本内容：',a.text)

**运行结果**  
![在这里插入图片描述][28a523b6751545379aa5fa243085fb39.png]

#### 案例（从疫情首页提取各国最新的疫情数据） ####

![在这里插入图片描述][09b2f9b11ec143eaa725ee3abb6d7877.png]  
**ctrl+f**查找某个类型元素的区域，然后，需找到对应标签的id,然后根据id的值来通过find方法获取文本内容。  
![在这里插入图片描述][837ed83acc8540b4a215f118880d791f.png]

**案例代码**：

# 1.导入相关模块
    import requests
    from bs4 import BeautifulSoup
    
    # 2.发送请求，获取疫情首页内容
    response = requests.get('https://ncov.dxy.cn/ncovh5/view/pneumonia')
    home_page = response.content.decode()
    #print(home_page)
    
    # 3.使用 BeautifulSoup 获取疫情数据
    soup = BeautifulSoup(home_page, 'lxml')
    script = soup.find(id='getAreaStat')
    text = script.text
    print(text)

运行结果：  
![在这里插入图片描述][a6e9129d42b94ae2bcf5bd8f880a4f94.png]

[e4254d9b35f34a4fa7814ec6c61d8347.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/846744d6e5af45b1a8b426c6aa05ec04.png
[ee4dfa446f0d455da9bb815c545e1468.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/9ae1c9350dbd4c72b3fbbb36763c4e3c.png
[48575faf2e5a41309c081a2e56501a14.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/8099d2dcf37e45868dc298c451efbf17.png
[1d6bec36cb7a4b1ea592311b6443dc9c.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/fbf55810f57e40c2a5623cfa9d55d737.png
[a5174103644e4eb3bebf48f520e56d43.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/41241e3a1c064c3d931a85ef04c292de.png
[4499387bae3e44a7a87d4e58215a81ac.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/600d3ef910da4ec28e13fbf47cde8cd4.png
[b77260a7773c4ba2855fe03ae03971ea.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/e35aed4625b24521a1ecf2a4fa24da14.png
[f05a9d15b8a141b68fe8b9e31b66a9ee.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/49d3564214c34a46891f8db80bea4303.png
[51eed760516d45d9ac12671dba72f6dd.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/990d633944704d759e983a255bb8f5e7.png
[8468645d612e4673bb5afd082f2dcd49.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/d24cd5b205424af299438011b85b41c3.png
[c7ea96d030d04e94b996d9b7724c504d.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/a52e0415f36449d49024f6dc4461014a.png
[ff3b84ea841f4531a5aa9e02fadbae0e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/5d2743b99edf46ee91fda414fb2787e2.png
[5a79116184844d86b8a256d76cadd1db.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/00dce75e3bd64af3ae125e28252e7d3b.png
[a5906d580e06451fa494cc97618c6318.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/e54d94236e2744df880b8f858a7f7223.png
[719800a2655d46369d6e5da05472768b.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/f7b75f2f6ab4405a976410d304fba856.png
[caf01f2382a548bebc62de1019289323.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/06bf040f84bb4c95bfaea30a541bdc38.png
[497693dd511a4736ab7542cbbb695e68.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/45b06c5a27e64cb3aedafe563d224142.png
[c447eb695bc6403aa777a5856bb1c7bd.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/87ef200075fa4a81b8c39a55e3214c9b.png
[b069f407195247ef9f48c28deb6a2b61.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/38ba550ee55b4d87aeecdaa50f98bcf0.png
[28a523b6751545379aa5fa243085fb39.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/dd76dfa53e3146b48502f36b681586e3.png
[09b2f9b11ec143eaa725ee3abb6d7877.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/3407bac5dbda476baa64cc27f82b1ca1.png
[837ed83acc8540b4a215f118880d791f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/cb8dd570f5134e6ebc50ed33641e0c03.png
[a6e9129d42b94ae2bcf5bd8f880a4f94.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/27/1484ec0e9c0f420d934c0adc61bebf60.png