Python笔记-获取某百科页面所有URL（提取某百科所有URL）

╰半橙微兮° 2022-09-04 01:55 240阅读 0赞

程序运行截图如下：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxNzg0NDI3NjE_size_16_color_FFFFFF_t_70][]

这里分析下页面：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxNzg0NDI3NjE_size_16_color_FFFFFF_t_70 1][]

凡是百度百科的都是在此url上

https://baike.baidu.com/item/

xxxxx，所以可以直接提取。

这里我们用个队列，将这个页面的所有有关的url入队，然后出队列，进行访问：

还有个要注意的，要伪造成浏览器，不然会回数据

import requests
    import queue
    import time
    from bs4 import BeautifulSoup
    
    header = {
        'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
        'Accept-Language' : 'zh-CN,zh;q=0.9',
        'Cache-Control' : 'no-cache',
        'Connection' : 'keep-alive',
        'Cookie' : 'xxxxxxx',
        'Host' : 'baike.baidu.com',
        'Pragma' : 'no-cache',
        'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36',
        'sec-ch-ua' : '" Not;A Brand";v="99", "Google Chrome";v="91", "Chromium";v="91"'
    }
    
    baseUrl = "https://baike.baidu.com/item/"
    urlQueue = queue.Queue(10000)
    
    def getRequest(url):
        response = requests.get(url, headers = header)
        return response.text
        pass
    
    if __name__ == '__main__':
    
        urlQueue.put('%E7%BB%9F%E8%AE%A1%E5%AD%A6/1175')
        for i in range(100):
            url = urlQueue.get()
            content = getRequest(baseUrl + url)
            contentSoup = BeautifulSoup(content, "html.parser")
            urlAllList = contentSoup.select("a")
            for urlTmp in urlAllList:
                if urlTmp.attrs.__contains__('href'):
                    urlString = urlTmp['href']
                    if '/item' in urlString:
                        testUrl = urlString.split('/item/')[1]
                        urlQueue.put(testUrl)
                        pass
                    pass
                pass
            print('over')
    
        pass

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxNzg0NDI3NjE_size_16_color_FFFFFF_t_70]: /images/20220829/3c650341ba084b0ea6cdcf6bedf16a5c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxNzg0NDI3NjE_size_16_color_FFFFFF_t_70 1]: /images/20220829/d2b3d39dfc7e460ea9fbe17e454a1305.png