python爬虫（二）----URL管理器和网页解析器

冷不防 2024-03-30 15:08 96阅读 0赞

一、URL管理器  
![在这里插入图片描述][6dc835254211458ab95d681fc5dbc2a6.png]  
注意：数据存储使用python内存时程序运行结束数据不会保留，而保存到redis和MySQL中数据可以长久保存。

代码实现：

class URLManager():
        def __init__(self):
            """
            存取未爬取的url
            """
            self.new_urls = set()
    
    
            """
            存取已爬取的url
            """
            self.old_urls = set()
    
        """
        两个新增url的方法,嵌套使用，分别实现url的单个增加和批量增加
        """
        def add_new_url(self,url):
            if url is None or len(url) == 0:
                return
            if url in self.new_urls or url in self.old_urls:
                return
            self.new_urls.add(url)
        def add_new_urls(self,urls):
            if urls is None or len(urls) == 0:
                return
            for url in urls:
                self.add_new_url(url)
    
        def get_url(self):
            """
            取出url并更改状态
            :return:
            """
            if self.has_new_url():
                url = self.new_urls.pop()
                self.old_urls.add(url)
                return url
            else:
                return None
    
        def has_new_url(self):
            """
            判断容器中是否有待爬取的url
            :return:
            """
            return len(self.new_urls) > 0
    
    if __name__ == "__main__":
        """
        测试
        """
        url_manager = URLManager()
        url_manager.add_new_url("url1")
        url_manager.add_new_urls(["url1" , "url2"])
        print(url_manager.new_urls , url_manager.old_urls)
        """
        打印分割线
        """
        print("#"*30)
        new_url = url_manager.get_url()
        print(url_manager.new_urls, url_manager.old_urls)
    
        print("#" * 30)
        new_url = url_manager.get_url()
        print(url_manager.new_urls, url_manager.old_urls)
    
        print("#"*30)
        print(url_manager.has_new_url)

二、网页解析器Beautiful Soup  
1.  
![在这里插入图片描述][47c269f129244795ac86bd1ff1f8073f.png]  
![在这里插入图片描述][36d06887d10a46c2b1833ec720bcb0dc.png]  
![在这里插入图片描述][9ae9a1011b4644658473ca0d69ae94a4.png]  
![在这里插入图片描述][e1626b8caa0349399ca16e169c5beada.png]  
其实就是根据html标签名称及其他属性来提取指定标签的内容。

[6dc835254211458ab95d681fc5dbc2a6.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/77b461301efa4028b63c223731e9ae3f.png
[47c269f129244795ac86bd1ff1f8073f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/b727ff0403f6464ab7018a3c51cc2fe0.png
[36d06887d10a46c2b1833ec720bcb0dc.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/1b4ebd08858b41eea9d7256df7605a78.png
[9ae9a1011b4644658473ca0d69ae94a4.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/c311894646564a7baa3dc899f18984c3.png
[e1626b8caa0349399ca16e169c5beada.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/668ce987d7f84e66b80e65253bd03ca9.png