16-python爬虫之Requests库

╰半夏微凉° 2022-12-18 15:55 293阅读 0赞

![null][]

**Requests 是一个 Python 的 HTTP 客户端库。**

Request支持HTTP连接保持和连接池，支持使用cookie保持会话，支持文件上传，支持自动响应内容的编码，支持国际化的URL和POST数据自动编码。

在python内置模块的基础上进行了高度的封装，从而使得python进行网络请求时，变得人性化，使用Requests可以轻而易举的完成浏览器可有的任何操作。现代，国际化，友好。

requests会自动实现持久连接keep-alive

开源地址：https://github.com/kennethreitz/requests

中文文档：http://docs.python-requests.org/zh\_CN/latest/index.html

> 目录
> 
> 一、Requests基础 
> 
> 二、发送请求与接收响应（基本GET请求） 
> 
> 三、发送请求与接收响应（基本POST请求） 
> 
> 四、response属性 
> 
> 五、代理 
> 
> 六、cookie和session 
> 
> 七、案例

## 一、Requests基础 ##

## 1.安装Requests库 ##

pip install  requests

## 2.使用Requests库 ##

import requests

## 二、发送请求与接收响应（基本GET请求） ##

response = requests.get(url)

## 1.传送 parmas参数 ##

•参数包含在url中

response = requests.get("http://httpbin.org/get?name=zhangsan&age=22")
    print(response.text)

![null][null 1]

•通过get方法传送参数

data = {
            "name": "zhangsan",
            "age": 30
        }
    response = requests.get("http://httpbin.org/get", params=data)
    print(response.text)

## 2.模拟发送请求头（传送headers参数） ##

headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36"
    }
    response = requests.get("http://httpbin.org/get", headers=headers)
    print(response.text)

## 三、发送请求与接收响应（基本POST请求） ##

response = requests.post(url, data = data, headers=headers)

## 四、response属性 ##

<table> 
 <thead> 
  <tr> 
   <td>属性</td> 
   <td>描述</td> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>response.text</td> 
   <td>获取str类型（Unicode编码）的响应</td> 
  </tr> 
  <tr> 
   <td>response.content</td> 
   <td>获取bytes类型的响应</td> 
  </tr> 
  <tr> 
   <td>response.status_code</td> 
   <td>获取响应状态码</td> 
  </tr> 
  <tr> 
   <td>response.headers</td> 
   <td>获取响应头</td> 
  </tr> 
  <tr> 
   <td>response.request</td> 
   <td>获取响应对应的请求</td> 
  </tr> 
 </tbody> 
</table>

## 五、代理 ##

proxies = {
        "http": "https://175.44.148.176:9000",
        "https": "https://183.129.207.86:14002"
    }
    response = requests.get("https://www.baidu.com/", proxies=proxies)

## 六、cookie和session ##

•**使用的cookie和session好处：很多网站必须登录之后**(或者获取某种权限之后)才能能够请求到相关数据。•使用的cookie和session的弊端：一套cookie和session往往和一个用户对应.请求太快，请求次数太多，容易被服务器识别为爬虫，从而使账号收到损害。

> 1.不需要cookie的时候尽量不去使用cookie。2.为了获取登录之后的页面，我们必须发送带有cookies的请求，此时为了确保账号安全应该尽量降低数据 采集速度。

## 1.cookie ##

### （1）获取cookie信息 ###

response.cookies

## 2.session ##

### （1）构造session回话对象 ###

session = requests.session()

示例：

def login_renren():
        login_url = 'http://www.renren.com/SysHome.do'
        headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36"
        }
    
    
        session = requests.session()
    
    
        login_data = {
            "email": "账号",
            "password": "密码"
        }
    
    
        response = session.post(login_url, data=login_data, headers=headers)
    
    
        response = session.get("http://www.renren.com/971909762/newsfeed/photo")
        print(response.text)
    
    
    login_renren()

## 七、案例 ##

![null][null 2]

## 案例1：百度贴吧页面爬取（GET请求） ##

import requests
    import sys
    
    
    class BaiduTieBa:
        def __init__(self, name, pn, ):
            self.name = name
            self.url = "http://tieba.baidu.com/f?kw={}&ie=utf-8&pn={}".format(name, pn)
            self.headers = {
                # "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36"
    
    
                # 使用较老版本的请求头，该浏览器不支持js
                "User-Agent": "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"
            }
            self.url_list = [self.url + str(pn*50) for pn in range(pn)]
            print(self.url_list)
    
    
        def get_data(self, url):
            """
            请求数据
            :param url:
            :return:
            """
            response = requests.get(url, headers=self.headers)
            return response.content
    
    
        def save_data(self, data, num):
            """
            保存数据
            :param data:
            :param num:
            :return:
            """
            file_name = "./pages/" + self.name + "_" + str(num) + ".html"
            with open(file_name, "wb") as f:
                f.write(data)
    
    
        def run(self):
            for url in self.url_list:
                data = self.get_data(url)
                num = self.url_list.index(url)
                self.save_data(data, num)
    
    
    if __name__ == "__main__":
        name = sys.argv[1]
        pn = int(sys.argv[2])
        baidu = BaiduTieBa(name, pn)
        baidu.run()

## 案例2：金山词霸翻译（POST请求） ##

import requests
    import sys
    import json
    
    
    class JinshanCiBa:
        def __init__(self, words):
            self.url = "http://fy.iciba.com/ajax.php?a=fy"
            self.headers = {
                "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0",
                "X-Requested-With": "XMLHttpRequest"
            }
            self.post_data = {
                "f": "auto",
                "t": "auto",
                "w": words
            }
    
    
        def get_data(self):
            """
            请求数据
            :param url:
            :return:
            """
            response = requests.post(self.url, data=self.post_data, headers=self.headers)
            return response.text
    
    
        def show_translation(self):
            """
            显示翻译结果
            :param data:
            :param num:
            :return:
            """
            response = self.get_data()
            json_data = json.loads(response, encoding='utf-8')
            if json_data['status'] == 0:
                translation = json_data['content']['word_mean']
            elif json_data['status'] == 1:
                translation = json_data['content']['out']
            else:
                translation = None
            print(translation)
    
    
        def run(self):
            self.show_translation()
    
    
    if __name__ == "__main__":
        words = sys.argv[1]
        ciba = JinshanCiBa(words)
        ciba.run()

## 案例3：百度贴吧图片爬取 ##

### （1）普通版 ###

从已下载页面中提取url来爬取图片（页面下载方法见案例1）

from lxml import etree
    import requests
    
    
    class DownloadPhoto:
        def __init__(self):
            pass
    
    
        def download_img(self, url):
            response = requests.get(url)
            index = url.rfind('/')
            file_name = url[index + 1:]
            print("下载图片：" + file_name)
            save_name = "./photo/" + file_name
            with open(save_name, "wb") as f:
                f.write(response.content)
    
    
        def parse_photo_url(self, page):
            html = etree.parse(page, etree.HTMLParser())
            nodes = html.xpath("//a[contains(@class, 'thumbnail')]/img/@bpic")
            print(nodes)
            print(len(nodes))
            for node in nodes:
                self.download_img(node)
    
    
    if __name__ == "__main__":
        download = DownloadPhoto()
        for i in range(6000):
            download.parse_photo_url("./pages/校花_{}.html".format(i))

### （2）多线程版 ###

main.py

import requests
    from lxml import etree
    
    
    from file_download import DownLoadExecutioner, file_download
    
    
    class XiaoHua:
        def __init__(self, init_url):
            self.init_url = init_url
            self.download_executioner = DownLoadExecutioner()
    
    
        def start(self):
            self.download_executioner.start()
            self.download_img(self.init_url)
    
    
        def download_img(self, url):
            html_text = file_download(url, type='text')
            html = etree.HTML(html_text)
            img_urls = html.xpath("//a[contains(@class,'thumbnail')]/img/@bpic")
            self.download_executioner.put_task(img_urls)
    
    
            # 获取下一页的连接
            next_page = html.xpath("//div[@id='frs_list_pager']/a[contains(@class,'next')]/@href")
            next_page = "http:" + next_page[0]
            self.download_img(next_page)
    
    
    if __name__ == '__main__':
        x = XiaoHua("http://tieba.baidu.com/f?kw=校花&ie=utf-8")
        x.start()

file\_download.py

import requests
    import threading
    from queue import Queue
    
    
    def file_download(url, type='content'):
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko'
        }
        r = requests.get(url, headers=headers)
        if type == 'text':
            return r.text
    
    
        return r.content
    
    
    class DownLoadExecutioner(threading.Thread):
        def __init__(self):
            super().__init__()
            self.q = Queue(maxsize=50)
            # 图片保存目录
            self.save_dir = './img/'
            # 图片计数
            self.index = 0
    
    
        def put_task(self, urls):
            if isinstance(urls, list):
                for url in urls:
                    self.q.put(url)
            else:
                self.q.put(urls)
    
    
        def run(self):
            while True:
                url = self.q.get()
                content = file_download(url)
    
    
                # 截取图片名称
                index = url.rfind('/')
                file_name = url[index+1:]
                save_name = self.save_dir + file_name
                with open(save_name, 'wb+') as f:
                    f.write(content)
                    self.index += 1
                    print(save_name + "下载成功!  当前已下载图片总数：" + str(self.index))

### （3）线程池版 ###

main.py

import requests
    from lxml import etree
    
    
    from file_download_pool import DownLoadExecutionerPool, file_download
    
    
    class XiaoHua:
        def __init__(self, init_url):
            self.init_url = init_url
            self.download_executioner = DownLoadExecutionerPool()
    
    
        def start(self):
            self.download_img(self.init_url)
    
    
        def download_img(self, url):
            html_text = file_download(url, type='text')
            html = etree.HTML(html_text)
            img_urls = html.xpath("//a[contains(@class,'thumbnail')]/img/@bpic")
            self.download_executioner.put_task(img_urls)
    
    
            # 获取下一页的连接
            next_page = html.xpath("//div[@id='frs_list_pager']/a[contains(@class,'next')]/@href")
            next_page = "http:" + next_page[0]
            self.download_img(next_page)
    
    
    if __name__ == '__main__':
        x = XiaoHua("http://tieba.baidu.com/f?kw=校花&ie=utf-8")
        x.start()

file\_download\_pool.py

import requests
    import concurrent.futures as futures
    
    
    def file_download(url, type='content'):
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko'
        }
        r = requests.get(url, headers=headers)
        if type == 'text':
            return r.text
    
    
        return r.content
    
    
    class DownLoadExecutionerPool():
        def __init__(self):
            super().__init__()
            # 图片保存目录
            self.save_dir = './img_pool/'
            # 图片计数
            self.index = 0
            # 线程池
            self.ex = futures.ThreadPoolExecutor(max_workers=30)
    
    
        def put_task(self, urls):
            if isinstance(urls, list):
                for url in urls:
                    self.ex.submit(self.save_img, url)
            else:
                self.ex.submit(self.save_img, urls)
    
    
        def save_img(self, url):
            content = file_download(url)
    
    
            # 截取图片名称
            index = url.rfind('/')
            file_name = url[index+1:]
            save_name = self.save_dir + file_name
            with open(save_name, 'wb+') as f:
                f.write(content)
                self.index += 1
                print(save_name + "下载成功!  当前已下载图片总数：" + str(self.index))

作者：Recalcitrant 链接：https://www.jianshu.com/p/140012f88f8e

《0基础python爬虫系列教程》

<table> 
 <tbody> 
  <tr> 
   <td><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb320125dc458833960e697aa9cf6efd4beda9850e74d05c284f0a9946d98fa93bb12a4324b3&amp;idx=2&amp;mid=2247485516&amp;scene=21&amp;sn=bb0633aa198a9ac55bfa74060155ee74#wechat_redirect" rel="nofollow">01为什么要学习爬虫</a></td> 
   <td><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb320103dc45881592fa51bd62b7d7820b4b506c4a3933a9fad6833d49ce360294f2fe0c6402&amp;idx=2&amp;mid=2247485546&amp;scene=21&amp;sn=9467b8493b4a252d7c39799e58813575#wechat_redirect" rel="nofollow">02-认识python爬虫</a></td> 
  </tr> 
  <tr> 
   <td><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb320103dc4588155293377a2843fc247e3c404f19eb523b875e988e64bc544d98c1b8d7854b&amp;idx=3&amp;mid=2247485546&amp;scene=21&amp;sn=e588e1f9b369323817acb125a7a2fa13#wechat_redirect" rel="nofollow">03-爬虫基本原理</a></td> 
   <td><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb320065dc458973634e1f97683d26b920c3b23dd975286c595065c0e44c3f13a8c74274f7a2&amp;idx=2&amp;mid=2247485708&amp;scene=21&amp;sn=02145cd8d7f0ba50be1ad4f83e60c12b#wechat_redirect" rel="nofollow">04-爬虫利器Fiddler</a></td> 
  </tr> 
  <tr> 
   <td><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb320065dc4589730f0f40ed783f45d1a71974a3037ede904e68fe967752f2def4daf04a8e16&amp;idx=3&amp;mid=2247485708&amp;scene=21&amp;sn=9fe3f40863c13431a80abb6a11bc8f1a#wechat_redirect" rel="nofollow">05-HTTP协议</a></td> 
   <td><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb320047dc458951e91b4f5ac10d8580840e3fd8aa63ac8c217f8df2d9c643cbda75c22def9f&amp;idx=2&amp;mid=2247485742&amp;scene=21&amp;sn=6be1c823ade912410f109a7f42ffffd3#wechat_redirect" rel="nofollow">06-爬虫库urllib</a></td> 
  </tr> 
  <tr> 
   <td><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb320047dc4589512a5d2062893b86474a037111067d05b57c04355b967c22b04ce3395ea96f&amp;idx=3&amp;mid=2247485742&amp;scene=21&amp;sn=7a54ccaa4836d2a7e5128f233e8c9e5d#wechat_redirect" rel="nofollow">07-TCP3</a></td> 
   <td><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb32003bdc45892d17198bb56448b4600aabd63c84667dccd8fc032647300da843d0ea15be88&amp;idx=3&amp;mid=2247485778&amp;scene=21&amp;sn=f086472aa06577ab8c0567a927721994#wechat_redirect" rel="nofollow">08-页面解析之数据提取</a></td> 
  </tr> 
  <tr> 
   <td><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb32003bdc45892dc9d2ae5b11e1ec1357a1d2884ad3d1de51775c02a3045c336068e2ba3b37&amp;idx=2&amp;mid=2247485778&amp;scene=21&amp;sn=adbe45fd796a285d8c54f38d2727c496#wechat_redirect" rel="nofollow">09-XPath 语言</a></td> 
   <td><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb32000edc45891837fbcaa3dc21becf71e86dcd25f599e7d8f59158bf339ddaad9f4c75dd90&amp;idx=2&amp;mid=2247485799&amp;scene=21&amp;sn=6ed8d9212f4bfb4d710523dfb478e9c4#wechat_redirect" rel="nofollow">10-lxml库</a></td> 
  </tr> 
  <tr> 
   <td colspan="1"><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb320010dc458906991372249d605f7ac28b1bf649ce55cf5ba1255b42d9b153b003d6c9ccfd&amp;idx=2&amp;mid=2247485817&amp;scene=21&amp;sn=fc3c41347968b123ed51d0c61508428c#wechat_redirect" rel="nofollow">11-Beautiful Soup</a><br></td> 
   <td colspan="1"><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb3200d6dc4589c005e613206c66915ef25f8740c3947a1cdac2a3f20135dad3b464c64cf5df&amp;idx=2&amp;mid=2247485887&amp;scene=21&amp;sn=9140a4874523e0e2aec3252577ab12a6#wechat_redirect" rel="nofollow">12-正则表达式</a><br></td> 
  </tr> 
  <tr> 
   <td colspan="1"><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb320081dc45899750dfcf9a04789b5a16668d03245d868f182b24b5e8f3914151895e423d25&amp;idx=2&amp;mid=2247485928&amp;scene=21&amp;sn=79e235db280e4faad16141a22c06e37a#wechat_redirect" rel="nofollow">13-python爬虫JSON操作</a><br></td> 
   <td colspan="1"><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb3203bedc458aa8087e8db11b91a93ad354f971089bf1652e260a489b6c33fc39dc1178e401&amp;idx=2&amp;mid=2247486167&amp;scene=21&amp;sn=df18b573a89b0854f1c8b400a81fb61e#wechat_redirect" rel="nofollow">14-Python 读写 Excel</a><br></td> 
  </tr> 
  <tr> 
   <td colspan="1"><a href="http://mp.weixin.qq.com/s?__biz=MzI3MjM5MjE5OA%3D%3D&amp;chksm=eb3203b3dc458aa530bd360b1e1753e719eb918e17abbc817e989be7afee9546edaf68fad65f&amp;idx=2&amp;mid=2247486170&amp;scene=21&amp;sn=46400af957698441417a38c089086cef#wechat_redirect" rel="nofollow">15-python爬取百度贴吧</a><br></td> 
   <td colspan="1"><br></td> 
  </tr> 
 </tbody> 
</table>

<table> 
 <tbody> 
  <tr> 
   <td align="center"><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X2pwZy85eTFUZUhZejFBenluRzJDaWFvTElXMDRZbmNMcmlhdUhIc2NBeWFUaWE2MnNPdHJCZmtzQ3F1ME93ZWliaWFMYkZUaWJCMzk0dXF5MUlJWHYxOGJyY3E5M0pTdy82NDA?x-oss-process=image/format,png"></p><p>IT入门&nbsp;感谢关注</p></td> 
   <td align="left"><p>一学就会&nbsp;Python入门教程 →</p><p>程序员用的单词表→</p><p>练习地址：www.520mg.com/it</p></td> 
  </tr> 
 </tbody> 
</table>

[null]: /images/20221122/c6a6c5007cac459f8dc1c4e4084f477f.png
[null 1]: /images/20221122/9c580f592d1043acae84263beb3773ee.png
[null 2]: /images/20221122/be97f95fc5c84592908175d0b68366d1.png