Beautiful Soup[解析HTML页面、信息标记与提取方法] 》》》实战_中国大学排名爬虫...

系统管理员 2021-11-04 10:14 326阅读 0赞

Beautiful Soup库是解析HTML和XML文件的库

引入方法

![1676033-20190605092108712-4973577.png][]

导入模块使用语句

from bs4 import BeautifulSoup

下方返回的HTML文本就是通过requests获得的网页HTML文本，将它解析成HTML

html.parser是解析html的解析器

![1676033-20190605092541195-2144497759.png][]

print(soup.prettify())

就可以和HTML布局一样打印出来

![1676033-20190605092824548-969121970.png][]

可以看成Beautiful Soup将HTML转换成Beautiful Soup类，然后对这些标签树进行操作

![1676033-20190605093246123-1293811371.png][]  ![1676033-20190605093510434-405885426.png][]

![1676033-20190605093538695-1234669504.png][]

基本元素

![1676033-20190605093656836-459861375.png][]

bs获得元素节点【和HTML操作类似】

获得的是满足的第一个匹配！！！！，这点还是没有lxm下的etree使用xpath好呀

但是有select方法适合etree使用xpath一样而且它使用的是CSS选择器，很方便！，如下访问音乐名

import requests
    from bs4 import BeautifulSoup
    import time
    
    # 请求头
    headers = {
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/'
                      '537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36'
    }
    
    
    # 定义函数get_info()
    def get_info(url):
        # 请求网业
        wb_data = requests.get(url, headers=headers)
        # 解析网业
        soup = BeautifulSoup(wb_data.text, 'lxml')
        # 定位元素并通过selector方法提取
        ranks = soup.select('span.pc_temp_num ')
        titles = soup.select('div.pc_temp_songlist > ul > li > a')
        times = soup.select(' span.pc_temp_num ')
        # 循环遍历
        for rank, title, time, in zip(ranks, titles, times):
            data = {
                'rank': rank.get_text().strip(),
                'singer': title.get_text().split('-')[0],
                'song': title.get_text().split('-')[1],
                'time': time.get_text().strip()
            }
            print(data)
    
    
    if __name__ == '__main__':
        urls = ['https://www.kugou.com/yy/rank/home/{}-8888.html?from=rank'.format(i)
                for i in range(1, 24)]
        # for i in urls:
        #     print(i)
        for url in urls:
            get_info(url)
        time.sleep(1)

![1676033-20190605094514334-2133498190.png][]

获得标签属性信息

![1676033-20190605095250991-1407792860.png][]

获得标签内的文本

.string

![1676033-20190605100251031-1957054896.png][]

注释内容

![1676033-20190605100533423-2141929033.png][]

![1676033-20190605101836854-183567001.png][]

![1676033-20190605103123812-1826798924.png][]

下行遍历

返回列表，除了包括元素节点也包括文本节点

![1676033-20190605103158148-1467535932.png][]         》》》     ![1676033-20190605103510984-439144682.png][]

上行遍历

![1676033-20190605103707015-382138291.png][]       ![1676033-20190605104003227-792105297.png][]

平行遍历

![1676033-20190605104214510-1300465791.png][]       ![1676033-20190605104712767-665168466.png][]

平行遍历可能获得的是string类型

![1676033-20190605104837233-2032535186.png][]

转载于:https://www.cnblogs.com/ningxinjie/p/10978308.html

[1676033-20190605092108712-4973577.png]: /images/20211104/f37eb2336dd74d43a71d4f41faa441b5.png
[1676033-20190605092541195-2144497759.png]: /images/20211104/638b3d1560af46a18f9a59e61503b474.png
[1676033-20190605092824548-969121970.png]: /images/20211104/a1d25f12672543a69f82cb75df2944ec.png
[1676033-20190605093246123-1293811371.png]: https://img2018.cnblogs.com/blog/1676033/201906/1676033-20190605093246123-1293811371.png
[1676033-20190605093510434-405885426.png]: /images/20211104/528bd62e70f7432ab802d776ffa0072e.png
[1676033-20190605093538695-1234669504.png]: /images/20211104/9f9d8d41ff364d44b66293050cb3ddfe.png
[1676033-20190605093656836-459861375.png]: /images/20211104/8f348945cf8949f69a8a82d95b105ced.png
[1676033-20190605094514334-2133498190.png]: /images/20211104/2e41deb161b443de93e55c7f2edb956e.png
[1676033-20190605095250991-1407792860.png]: /images/20211104/eaf3a98019a748aa9981f98830e48c4f.png
[1676033-20190605100251031-1957054896.png]: /images/20211104/1c25b51b542a4242ab3df14e3c6f2899.png
[1676033-20190605100533423-2141929033.png]: /images/20211104/c74d162908a24a9a9beff74054361d80.png
[1676033-20190605101836854-183567001.png]: /images/20211104/c7096b5638d242b0b5f37251b673a964.png
[1676033-20190605103123812-1826798924.png]: /images/20211104/b5ec242d2c3b4f58af67f57af9569225.png
[1676033-20190605103158148-1467535932.png]: https://img2018.cnblogs.com/blog/1676033/201906/1676033-20190605103158148-1467535932.png
[1676033-20190605103510984-439144682.png]: /images/20211104/a27b6a284e474ad4838a3e3fbdc4a8c9.png
[1676033-20190605103707015-382138291.png]: https://img2018.cnblogs.com/blog/1676033/201906/1676033-20190605103707015-382138291.png
[1676033-20190605104003227-792105297.png]: /images/20211104/6b66a6814c2f4922a69f94e57584e64d.png
[1676033-20190605104214510-1300465791.png]: https://img2018.cnblogs.com/blog/1676033/201906/1676033-20190605104214510-1300465791.png
[1676033-20190605104712767-665168466.png]: /images/20211104/4d6cf75ba50f4c4886e81adaf345e78c.png
[1676033-20190605104837233-2032535186.png]: /images/20211104/2883a7661fc94438bac5c3513c192e7b.png