普通爬虫 VS 多线程爬虫！Python爬虫运行时间对比

﹏ヽ暗。殇╰゛Y 2022-12-26 10:27 181阅读 0赞

本文的文字及图片来源于网络,仅供学习、交流使用,不具有任何商业用途,如有问题请及时联系我们以作处理。

以下文章来自于爬小虫联盟，作者：Code皮皮虾

![c8774701475ff87f98fbbf17e7a05f72.png][]

# 前言 #

“ 相信各位爬虫小伙伴在工作中，肯定遇到过写了大半天才出来的爬虫，好不容易运行起来，结果跑的贼慢，反正我是遇到过的。如今是大数据的时代，光会写爬虫根本没有什么竞争力，所以要学会对爬虫代码进行优化，优化爬虫的健壮性或者爬取速度等等，这些都能提高自己的竞争力！”

本文爬虫以糗事百科为例，以普通爬虫和多线程爬虫运行时间相比，相信大家都能领略到多线程的厉害之处！

多线程表情包分类爬取实战，话不多说，开干！

**1、普通爬虫**

import requests
    from lxml import etree
    import time
    import sys
    
    headers = {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36"
    }
    
    #爬取
    def Crawl(response):
        e = etree.HTML(response.text)
       #根据class值定位
        span_text = e.xpath("//div[@class='content']/span[1]")
        with open("duanzi.txt", "a", encoding="utf-8") as f:
            for span in span_text:
                info = span.xpath("string(.)")
                f.write(info)
    
    
    #main
    if __name__ == '__main__':
      //记下开始时间
        start = time.time()
        base_url = "https://www.qiushibaike.com/text/page/{}"
        for i in range(1, 14):
          #打印当前爬取的页数
            print("正在爬取第{}页".format(i))
            new_url = base_url.format(i)
            #发送get请求
            response = requests.get(new_url,headers=headers)
            Crawl(response)
      
      #记下结束时间
        end = time.time()
        #相减获取运行时间
        print(end - start)

![9152ed121912632951a3677956379f80.png][]

![b1c79f041954b3acf9da57e104b9446e.png][]

**2、多线程爬虫**

import requests
    from lxml import etree
    #Queue队列，先进先出
    from queue import Queue
    from threading import Thread
    import time
    
    #请求头
    headers = {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36"
    }
    
    #数据获取
    #继承Thread
    class CrawlInfo(Thread):
        #重写init函数
        def __init__(self,url_queue,html_queue):
            Thread.__init__(self)
    
            self.url_queue = url_queue
            self.html_queue = html_queue
    
        # 重写run方法
        def run(self):
            while self.url_queue.empty()== False:
                url = self.url_queue.get()
                response = requests.get(url,headers=headers)
                if response.status_code == 200:
                  #将数据放到队列中
                    self.html_queue.put(response.text)
    
    #数据解析、保存
    #继承Thread
    class ParseInfo(Thread):
        def __init__(self,html_queue):
            Thread.__init__(self)
            self.html_queue = html_queue
    
        def run(self):
          #判断队列是否为空。不为空则继续遍历
            while self.html_queue.empty() == False:
              #从队列中取最后一个数据
                e = etree.HTML(self.html_queue.get())
                span_text = e.xpath("//div[@class='content']/span[1]")
                with open("duanzi.txt", "a", encoding="utf-8") as f:
                    for span in span_text:
                        info = span.xpath("string(.)")
                        f.write(info)
    
    
    #开始
    if __name__ == '__main__':
        start = time.time()
        #实例化
        url_queue = Queue()
        html_queue = Queue()
        
        base_url = "https://www.qiushibaike.com/text/page/{}"
        for i in range(1,14):
            print("正在爬取第{}页".format(i))
            new_url = base_url.format(i)
            url_queue.put(new_url)
    
        crawl_list = []
        for i in range(3):
            Crawl = CrawlInfo(url_queue,html_queue)
            crawl_list.append(Crawl)
            Crawl.start()
    
        for crawl in crawl_list:
          #join()等到队列为空，再执行别的操作
            crawl.join()
    
        parse_list = []
        for i in range(3):
            parse = ParseInfo(html_queue)
            parse_list.append(parse)
            parse.start()
    
        for parse in parse_list:
            parse.join()
    
        end = time.time()
        print(end - start)

![748900c1b6432a8ac92763967dcdc3c8.png][]

![efb16d393e73c6444e6d4addd3d8f3d3.png][]

**3、对比运行**

**普通爬虫**

![26034d33894763e5c0ee149ae756bf5d.png][]

**多线程爬虫**

![af30228c71c7f3cb90dcafc31572c5de.png][]

可能还有些小伙伴觉得这几秒钟的时间没什么大不了的，还是那句话，现在是大数据时代，动不动都是上百万、上千万的数据量，在采集总量中会有很大的区别。 如有不足之处或更多技巧，欢迎指教补充。愿本文的分享对您之后多线程有所帮助。谢谢～

[c8774701475ff87f98fbbf17e7a05f72.png]: /images/20221120/48dc0fd359864f088066196878390ddd.png
[9152ed121912632951a3677956379f80.png]: /images/20221120/abd1edfb41904d69a4fab242244227f5.png
[b1c79f041954b3acf9da57e104b9446e.png]: /images/20221120/c2c111d4360e430ca7f78bfbb3beec24.png
[748900c1b6432a8ac92763967dcdc3c8.png]: /images/20221120/d04bf15be3374308824ebc1241ea7a32.png
[efb16d393e73c6444e6d4addd3d8f3d3.png]: /images/20221120/dac0cc7d653b48aea769aa64f444b7bc.png
[26034d33894763e5c0ee149ae756bf5d.png]: /images/20221120/48ab247506b14182b331ab7574382471.png
[af30228c71c7f3cb90dcafc31572c5de.png]: /images/20221120/32526c1e9f3943b8b21cd4101dc4016b.png