Python—requests模块详解

- 日理万妓 2023-10-11 19:17 62阅读 0赞

requests是使用Apache2 licensed 许可证的HTTP库。

用python编写。

比urllib2模块更简洁。

Request支持HTTP连接保持和连接池，支持使用cookie保持会话，支持文件上传，支持自动响应内容的编码，支持国际化的URL和POST数据自动编码。

在python内置模块的基础上进行了高度的封装，从而使得python进行网络请求时，变得人性化，使用Requests可以轻而易举的完成浏览器可有的任何操作。

现代，国际化，友好。

requests会自动实现持久连接keep-alive

1)导入模块

import requests

2)发送请求的简洁

示例代码：获取一个网页(个人github)

import requests

r =requests.get('https://github.com/Ranxf') \# 最基本的不带参数的get请求

r1=requests.get(url='http://dict.baidu.com/s', params=\{'wd': 'python'\}) \# 带参数的get请求

我们就可以使用该方式使用以下各种方法

1 requests.get(‘https://github.com/timeline.json’) \# GET请求

2 requests.post(“http://httpbin.org/post”) \# POST请求

3 requests.put(“http://httpbin.org/put”) \# PUT请求

4 requests.delete(“http://httpbin.org/delete”) \# DELETE请求

5 requests.head(“http://httpbin.org/get”) \# HEAD请求

6 requests.options(“http://httpbin.org/get” ) \# OPTIONS请求

3)为url传递参数

>>> url\_params=\{'key':'value'\} \# 字典传递参数，如果值为None的键不会被添加到url中

>>> r=requests.get('your url',params= url\_params)

>>> print(r.url)

your url?key=value

4)响应的内容

r.encoding \#获取当前的编码

r.encoding='utf-8' \#设置编码

r.text \#以encoding解析返回内容。字符串方式的响应体，会自动根据响应头部的字符编码进行解码。

r.content \#以字节形式(二进制)返回。字节方式的响应体，会自动为你解码 gzip 和 deflate 压缩。

r.headers \#以字典对象存储服务器响应头，但是这个字典比较特殊，字典键不区分大小写，若键不存在则返回None

r.status\_code \#响应状态码

r.raw \#返回原始响应体，也就是 urllib 的 response 对象，使用 r.raw.read()

r.ok \# 查看r.ok的布尔值便可以知道是否登陆成功

\#\*特殊方法\*\#

r.json() \#Requests中内置的JSON解码器，以json形式返回,前提返回的内容确保是json格式的，不然解析出错会抛异常

r.raise\_for\_status() \#失败请求(非200响应)抛出异常

post发送json请求：

1 import requests

2 import json

4 r=requests.post('https://api.github.com/some/endpoint', data=json.dumps(\{'some': 'data'\}))

5 print(r.json())

5)定制头和cookie信息

header=\{'user-agent': 'my-app/0.0.1''\}

cookie=\{'key':'value'\}

r =requests.get/post('your url',headers=header,cookies=cookie)

data=\{'some': 'data'\}

headers =\{'content-type': 'application/json',

'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86\_64; rv:22.0) Gecko/20100101 Firefox/22.0'\}

r =requests.post('https://api.github.com/some/endpoint', data=data, headers=headers)

print(r.text)

6)响应状态码

使用requests方法后，会返回一个response对象，其存储了服务器响应的内容，如上实例中已经提到的 r.text、r.status\_code……

获取文本方式的响应体实例：当你访问 r.text 之时，会使用其响应的文本编码进行解码，并且你可以修改其编码让 r.text 使用自定义的编码进行解码。

1 r=requests.get('http://www.itwhy.org')

2 print(r.text, '

\{\}

'.format('\*'\*79), r.encoding)

3 r.encoding='GBK'

4 print(r.text, '

\{\}

'.format('\*'\*79), r.encoding)

示例代码：

1 import requests

3 r=requests.get('https://github.com/Ranxf') \# 最基本的不带参数的get请求

4 print(r.status\_code) \# 获取返回状态

5 r1=requests.get(url='http://dict.baidu.com/s', params=\{'wd': 'python'\}) \# 带参数的get请求

6 print(r1.url)

7 print(r1.text) \# 打印解码后的返回数据

运行结果：

/usr/bin/python3.5 /home/rxf/python3\_1000/1000/python3\_server/python3\_requests/demo1.py

200

http://dict.baidu.com/s?wd=python

…………

Process finished with exit code 0

r.status\_code \#如果不是200，可以使用 r.raise\_for\_status() 抛出异常

7)响应

r.headers \#返回字典类型,头信息

r.requests.headers \#返回发送到服务器的头信息

r.cookies \#返回cookie

r.history \#返回重定向信息,当然可以在请求是加上allow\_redirects=false 阻止重定向

8)超时

r=requests.get('url',timeout=1) \#设置秒数超时，仅对于连接有效

9)会话对象，能够跨请求保持某些参数

s= requests.Session()

s.auth =('auth','passwd')

s.headers =\{'key':'value'\}

r =s.get('url')

r1 =s.get('url1')

10)代理

proxies=\{'http':'ip1','https':'ip2' \}

requests.get('url',proxies=proxies)

汇总：

\# HTTP请求类型

\# get类型

r=requests.get('https://github.com/timeline.json')

\# post类型

r=requests.post("http://m.ctrip.com/post")

\# put类型

r=requests.put("http://m.ctrip.com/put")

\# delete类型

r=requests.delete("http://m.ctrip.com/delete")

\# head类型

r=requests.head("http://m.ctrip.com/head")

\# options类型

r=requests.options("http://m.ctrip.com/get")

\# 获取响应内容

print(r.content) \#以字节的方式去显示，中文显示为字符

print(r.text) \#以文本的方式去显示

\#URL传递参数

payload=\{'keyword': '香港', 'salecityid': '2'\}

r =requests.get("http://m.ctrip.com/webapp/tourvisa/visa\_list", params=payload)

print(r.url) \#示例为http://m.ctrip.com/webapp/tourvisa/visa\_list?salecityid=2&keyword=香港

\#获取/修改网页编码

r=requests.get('https://github.com/timeline.json')

print (r.encoding)

\#json处理

r=requests.get('https://github.com/timeline.json')

print(r.json()) \# 需要先import json

\# 定制请求头

url='http://m.ctrip.com'

headers =\{'User-Agent' : 'Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 4 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19'\}

r =requests.post(url, headers=headers)

print (r.request.headers)

\#复杂post请求

url='http://m.ctrip.com'

payload =\{'some': 'data'\}

r =requests.post(url, data=json.dumps(payload)) \#如果传递的payload是string而不是dict，需要先调用dumps方法格式化一下

\# post多部分编码文件

url='http://m.ctrip.com'

files =\{'file': open('report.xls', 'rb')\}

r =requests.post(url, files=files)

\# 响应状态码

r=requests.get('http://m.ctrip.com')

print(r.status\_code)

\# 响应头

r=requests.get('http://m.ctrip.com')

print (r.headers)

print (r.headers\['Content-Type'\])

print (r.headers.get('content-type')) \#访问响应头部分内容的两种方式

\# Cookies

url='http://example.com/some/cookie/setting/url'

r = requests.get(url)

r.cookies\['example\_cookie\_name'\] \#读取cookies

url ='http://m.ctrip.com/cookies'

cookies =dict(cookies\_are='working')

r =requests.get(url, cookies=cookies) \#发送cookies

\#设置超时时间

r=requests.get('http://m.ctrip.com', timeout=0.001)

\#设置访问代理

proxies= \{

"http": "http://10.10.1.10:3128",

"https": "http://10.10.1.100:4444",

r =requests.get('http://m.ctrip.com', proxies=proxies)

\#如果代理需要用户名和密码，则需要这样：

proxies= \{

"http": "http://user:pass@10.10.1.10:3128/",

\# HTTP请求类型

\# get类型

r=requests.get('https://github.com/timeline.json')

\# post类型

r=requests.post("http://m.ctrip.com/post")

\# put类型

r=requests.put("http://m.ctrip.com/put")

\# delete类型

r=requests.delete("http://m.ctrip.com/delete")

\# head类型

r=requests.head("http://m.ctrip.com/head")

\# options类型

r=requests.options("http://m.ctrip.com/get")

\# 获取响应内容

print(r.content) \#以字节的方式去显示，中文显示为字符

print(r.text) \#以文本的方式去显示

\#URL传递参数

payload=\{'keyword': '香港', 'salecityid': '2'\}

r =requests.get("http://m.ctrip.com/webapp/tourvisa/visa\_list", params=payload)

print(r.url) \#示例为http://m.ctrip.com/webapp/tourvisa/visa\_list?salecityid=2&keyword=香港

\#获取/修改网页编码

r=requests.get('https://github.com/timeline.json')

print (r.encoding)

\#json处理

r=requests.get('https://github.com/timeline.json')

print(r.json()) \# 需要先import json

\# 定制请求头

url='http://m.ctrip.com'

headers =\{'User-Agent' : 'Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 4 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19'\}

r =requests.post(url, headers=headers)

print (r.request.headers)

\#复杂post请求

url='http://m.ctrip.com'

payload =\{'some': 'data'\}

r =requests.post(url, data=json.dumps(payload)) \#如果传递的payload是string而不是dict，需要先调用dumps方法格式化一下

\# post多部分编码文件

url='http://m.ctrip.com'

files =\{'file': open('report.xls', 'rb')\}

r =requests.post(url, files=files)

\# 响应状态码

r=requests.get('http://m.ctrip.com')

print(r.status\_code)

\# 响应头

r=requests.get('http://m.ctrip.com')

print (r.headers)

print (r.headers\['Content-Type'\])

print (r.headers.get('content-type')) \#访问响应头部分内容的两种方式

\# Cookies

url='http://example.com/some/cookie/setting/url'

r = requests.get(url)

r.cookies\['example\_cookie\_name'\] \#读取cookies

url ='http://m.ctrip.com/cookies'

cookies =dict(cookies\_are='working')

r =requests.get(url, cookies=cookies) \#发送cookies

\#设置超时时间

r=requests.get('http://m.ctrip.com', timeout=0.001)

\#设置访问代理

proxies= \{

"http": "http://10.10.1.10:3128",

"https": "http://10.10.1.100:4444",

r =requests.get('http://m.ctrip.com', proxies=proxies)

\#如果代理需要用户名和密码，则需要这样：

proxies= \{

"http": "http://user:pass@10.10.1.10:3128/",

1 \# 1、无参数实例

3 import requests

5 ret=requests.get('https://github.com/timeline.json')

7 print(ret.url)

8 print(ret.text)

12 \# 2、有参数实例

14 import requests

16 payload=\{'key1': 'value1', 'key2': 'value2'\}

17 ret=requests.get("http://httpbin.org/get", params=payload)

19 print(ret.url)

20 print(ret.text)

\# 1、基本POST实例

import requests

payload =\{'key1': 'value1', 'key2': 'value2'\}

ret =requests.post("http://httpbin.org/post", data=payload)

print(ret.text)

\# 2、发送请求头和数据实例

import requests

import json

url ='https://api.github.com/some/endpoint'

payload =\{'some': 'data'\}

headers =\{'content-type': 'application/json'\}

ret =requests.post(url, data=json.dumps(payload), headers=headers)

print(ret.text)

print(ret.cookies)

![Python—requests模块详解\_Python—requests模块详解][Python_requests_Python_requests]![Python—requests模块详解\_Python—requests模块详解\_02][Python_requests_Python_requests_02]

def request(method, url, \*\*kwargs):

"""Constructs and sends a :class:\`Request \`.

:param method: method for the new :class:\`Request\` object.

:param url: URL for the new :class:\`Request\` object.

:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:\`Request\`.

:param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:\`Request\`.

:param json: (optional) json data to send in the body of the :class:\`Request\`.

:param headers: (optional) Dictionary of HTTP Headers to send with the :class:\`Request\`.

:param cookies: (optional) Dict or CookieJar object to send with the :class:\`Request\`.

:param files: (optional) Dictionary of \`\`'name': file-like-objects\`\` (or \`\`\{'name': file-tuple\}\`\`) for multipart encoding upload.

\`\`file-tuple\`\` can be a 2-tuple \`\`('filename', fileobj)\`\`, 3-tuple \`\`('filename', fileobj, 'content\_type')\`\`

or a 4-tuple \`\`('filename', fileobj, 'content\_type', custom\_headers)\`\`, where \`\`'content-type'\`\` is a string

defining the content type of the given file and \`\`custom\_headers\`\` a dict-like object containing additional headers

to add for the file.

:param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.

:param timeout: (optional) How long to wait for the server to send data

before giving up, as a float, or a :ref:\`(connect timeout, read

timeout) \` tuple.

:type timeout: float or tuple

:param allow\_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.

:type allow\_redirects: bool

:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.

:param verify: (optional) whether the SSL cert will be verified. A CA\_BUNDLE path can also be provided. Defaults to \`\`True\`\`.

:param stream: (optional) if \`\`False\`\`, the response content will be immediately downloaded.

:param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.

:return: :class:\`Response \` object

:rtype: requests.Response

Usage::

>>> import requests

>>> req=requests.request('GET', 'http://httpbin.org/get')

"""

参数列表

请求参数

![Python—requests模块详解\_Python—requests模块详解\_03][Python_requests_Python_requests_03]![Python—requests模块详解\_Python—requests模块详解\_04][Python_requests_Python_requests_04]

def param\_method\_url():

\# requests.request(method='get', url='http://127.0.0.1:8000/test/')

\# requests.request(method='post', url='http://127.0.0.1:8000/test/')

pass

def param\_param():

\# - 可以是字典

\# - 可以是字符串

\# - 可以是字节(ascii编码以内)

\# requests.request(method='get',

\# url='http://127.0.0.1:8000/test/',

\# params=\{'k1': 'v1', 'k2': '水电费'\})

\# requests.request(method='get',

\# url='http://127.0.0.1:8000/test/',

\# params="k1=v1&k2=水电费&k3=v3&k3=vv3")

\# requests.request(method='get',

\# url='http://127.0.0.1:8000/test/',

\# params=bytes("k1=v1&k2=k2&k3=v3&k3=vv3", encoding='utf8'))

\# 错误

\# requests.request(method='get',

\# url='http://127.0.0.1:8000/test/',

\# params=bytes("k1=v1&k2=水电费&k3=v3&k3=vv3", encoding='utf8'))

pass

def param\_data():

\# 可以是字典

\# 可以是字符串

\# 可以是字节

\# 可以是文件对象

\# requests.request(method='POST',

\# url='http://127.0.0.1:8000/test/',

\# data=\{'k1': 'v1', 'k2': '水电费'\})

\# requests.request(method='POST',

\# url='http://127.0.0.1:8000/test/',

\# data="k1=v1; k2=v2; k3=v3; k3=v4"

\# )

\# requests.request(method='POST',

\# url='http://127.0.0.1:8000/test/',

\# data="k1=v1;k2=v2;k3=v3;k3=v4",

\# headers=\{'Content-Type': 'application/x-www-form-urlencoded'\}

\# )

\# requests.request(method='POST',

\# url='http://127.0.0.1:8000/test/',

\# data=open('data\_file.py', mode='r', encoding='utf-8'), \# 文件内容是：k1=v1;k2=v2;k3=v3;k3=v4

\# headers=\{'Content-Type': 'application/x-www-form-urlencoded'\}

\# )

pass

def param\_json():

\# 将json中对应的数据进行序列化成一个字符串，json.dumps(...)

\# 然后发送到服务器端的body中，并且Content-Type是 \{'Content-Type': 'application/json'\}

requests.request(method='POST',

url='http://127.0.0.1:8000/test/',

json=\{'k1': 'v1', 'k2': '水电费'\})

def param\_headers():

\# 发送请求头到服务器端

requests.request(method='POST',

url='http://127.0.0.1:8000/test/',

json=\{'k1': 'v1', 'k2': '水电费'\},

headers=\{'Content-Type': 'application/x-www-form-urlencoded'\}

)

def param\_cookies():

\# 发送Cookie到服务器端

requests.request(method='POST',

url='http://127.0.0.1:8000/test/',

data=\{'k1': 'v1', 'k2': 'v2'\},

cookies=\{'cook1': 'value1'\},

)

\# 也可以使用CookieJar(字典形式就是在此基础上封装)

from http.cookiejar import CookieJar

from http.cookiejar import Cookie

obj = CookieJar()

obj.set\_cookie(Cookie(version=0, name='c1', value='v1', port=None, domain='', path='/', secure=False, expires=None,

discard=True, comment=None, comment\_url=None, rest=\{'HttpOnly': None\}, rfc2109=False,

port\_specified=False, domain\_specified=False, domain\_initial\_dot=False, path\_specified=False)

)

requests.request(method='POST',

url='http://127.0.0.1:8000/test/',

data=\{'k1': 'v1', 'k2': 'v2'\},

cookies=obj)

def param\_files():

\# 发送文件

\# file\_dict=\{

\# 'f1': open('readme', 'rb')

\# \}

\# requests.request(method='POST',

\# url='http://127.0.0.1:8000/test/',

\# files=file\_dict)

\# 发送文件，定制文件名

\# file\_dict=\{

\# 'f1': ('test.txt', open('readme', 'rb'))

\# \}

\# requests.request(method='POST',

\# url='http://127.0.0.1:8000/test/',

\# files=file\_dict)

\# 发送文件，定制文件名

\# file\_dict=\{

\# 'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf")

\# \}

\# requests.request(method='POST',

\# url='http://127.0.0.1:8000/test/',

\# files=file\_dict)

\# 发送文件，定制文件名

\# file\_dict=\{

\# 'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf", 'application/text', \{'k1': '0'\})

\# \}

\# requests.request(method='POST',

\# url='http://127.0.0.1:8000/test/',

\# files=file\_dict)

pass

def param\_auth():

from requests.auth import HTTPBasicAuth, HTTPDigestAuth

ret =requests.get('https://api.github.com/user', auth=HTTPBasicAuth('wupeiqi', 'sdfasdfasdf'))

print(ret.text)

\# ret=requests.get('http://192.168.1.1',

\# auth=HTTPBasicAuth('admin', 'admin'))

\# ret.encoding='gbk'

\# print(ret.text)

\# ret=requests.get('http://httpbin.org/digest-auth/auth/user/pass', auth=HTTPDigestAuth('user', 'pass'))

\# print(ret)

def param\_timeout():

\# ret=requests.get('http://google.com/', timeout=1)

\# print(ret)

\# ret=requests.get('http://google.com/', timeout=(5, 1))

\# print(ret)

pass

def param\_allow\_redirects():

ret =requests.get('http://127.0.0.1:8000/test/', allow\_redirects=False)

print(ret.text)

def param\_proxies():

\# proxies=\{

\# "http": "61.172.249.96:80",

\# "https": "http://61.185.219.126:3128",

\# \}

\# proxies=\{'http://10.20.1.128': 'http://10.10.1.10:5323'\}

\# ret=requests.get("http://www.proxy360.cn/Proxy", proxies=proxies)

\# print(ret.headers)

\# from requests.auth import HTTPProxyAuth

\# proxyDict=\{

\# 'http': '77.75.105.165',

\# 'https': '77.75.105.165'

\# \}

\# auth=HTTPProxyAuth('username', 'mypassword')

\# r=requests.get("http://www.google.com", proxies=proxyDict, auth=auth)

\# print(r.text)

pass

def param\_stream():

ret =requests.get('http://127.0.0.1:8000/test/', stream=True)

print(ret.content)

ret.close()

\# from contextlib import closing

\# with closing(requests.get('http://httpbin.org/get', stream=True)) as r:

\# \# 在此处理响应。

\# for i in r.iter\_content():

\# print(i)

def requests\_session():

import requests

session = requests.Session()

\#\#\# 1、首先登陆任何页面，获取cookie

i1 =session.get(url="http://dig.chouti.com/help/service")

\#\#\# 2、用户登陆，携带上一次的cookie，后台对cookie中的 gpsd 进行授权

i2= session.post(

url="http://dig.chouti.com/login",

data=\{

'phone': "8615131255089",

'password': "xxxxxx",

'oneMonth': ""

)

i3 = session.post(

url="http://dig.chouti.com/link/vote?linksId=8589623",

)

print(i3.text)

参数示例代码

\#! /usr/bin/python3

import requests

import json

class url\_request():

def \_\_init\_\_(self):

''' init '''

if \_\_name\_\_=='\_\_main\_\_':

heard =\{'Content-Type': 'application/json'\}

payload =\{'CountryName': '中国',

'ProvinceName': '四川省',

'L1CityName': 'chengdu',

'L2CityName': 'yibing',

'TownName': '',

'Longitude': '107.33393',

'Latitude': '33.157131',

'Language': 'CN'\}

r =requests.post("http://www.xxxxxx.com/CityLocation/json/LBSLocateCity", heards=heard, data=payload)

data = r.json()

if r.status\_code!=200:

print('LBSLocateCity API Error' + str(r.status\_code))

print(data\['CityEntities'\]\[0\]\['CityID'\]) \# 打印返回json中的某个key的value

print(data\['ResponseStatus'\]\['Ack'\])

print(json.dump(data, indent=4, sort\_keys=True, ensure\_ascii=False)) \# 树形打印json，ensure\_ascii必须设为False否则中文会显示为unicode

Xml请求：

\#! /usr/bin/python3

import requests

class url\_request():

def \_\_init\_\_(self):

"""init"""

if \_\_name\_\_=='\_\_main\_\_':

heards =\{'Content-type': 'text/xml'\}

XML =' WeChatJSTicket.JobWS.Job.JobRefreshTicket,WeChatJSTicket.JobWS RUN 1 127.0.0.1 1 false '

url ='http://jobws.push.mobile.xxxxxxxx.com/RefreshWeiXInTokenJob/RefreshService.asmx'

r =requests.post(url=url, heards=heards, data=XML)

data = r.text

print(data)

状态异常处理

import requests

URL ='http://ip.taobao.com/service/getIpInfo.php' \# 淘宝IP地址库API

try:

r =requests.get(URL, params=\{'ip': '8.8.8.8'\}, timeout=1)

r.raise\_for\_status() \# 如果响应状态码不是 200，就主动抛出异常

except requests.RequestException as e:

print(e)

else:

result = r.json()

print(type(result), result, sep='

上传文件

使用request模块，也可以上传文件，文件的类型会自动进行处理：

import requests

url ='http://127.0.0.1:8080/upload'

files =\{'file': open('/home/rxf/test.jpg', 'rb')\}

\#files=\{'file': ('report.jpg', open('/home/lyb/sjzl.mpg', 'rb'))\} \#显式的设置文件名

r =requests.post(url, files=files)

print(r.text)

request更加方便的是，可以把字符串当作文件进行上传：

import requests

url ='http://127.0.0.1:8080/upload'

files =\{'file': ('test.txt', b'Hello Requests.')\} \#必需显式的设置文件名

r =requests.post(url, files=files)

print(r.text)

6) 身份验证

基本身份认证(HTTP Basic Auth)

import requests

from requests.auth import HTTPBasicAuth

r =requests.get('https://httpbin.org/hidden-basic-auth/user/passwd', auth=HTTPBasicAuth('user', 'passwd'))

\# r=requests.get('https://httpbin.org/hidden-basic-auth/user/passwd', auth=('user', 'passwd')) \# 简写

print(r.json())

另一种非常流行的HTTP身份认证形式是摘要式身份认证，Requests对它的支持也是开箱即可用的:

requests.get(URL, auth=HTTPDigestAuth('user', 'pass')

Cookies与会话对象

如果某个响应中包含一些Cookie，你可以快速访问它们：

import requests

r =requests.get('http://www.google.com.hk/')

print(r.cookies\['NID'\])

print(tuple(r.cookies))

要想发送你的cookies到服务器，可以使用 cookies 参数：

import requests

url ='http://httpbin.org/cookies'

cookies =\{'testCookies\_1': 'Hello\_Python3', 'testCookies\_2': 'Hello\_Requests'\}

\# 在Cookie Version 0中规定空格、方括号、圆括号、等于号、逗号、双引号、斜杠、问号、@，冒号，分号等特殊符号都不能作为Cookie的内容。

r=requests.get(url, cookies=cookies)

print(r.json())

会话对象让你能够跨请求保持某些参数，最方便的是在同一个Session实例发出的所有请求之间保持cookies，且这些都是自动处理的，甚是方便。

下面就来一个真正的实例，如下是快盘签到脚本：

import requests

headers =\{'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,\*/\*;q=0.8',

'Accept-Encoding': 'gzip, deflate, compress',

'Accept-Language': 'en-us;q=0.5,en;q=0.3',

'Cache-Control': 'max-age=0',

'Connection': 'keep-alive',

'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86\_64; rv:22.0) Gecko/20100101 Firefox/22.0'\}

s = requests.Session()

s.headers.update(headers)

\# s.auth=('superuser', '123')

s.get('https://www.kuaipan.cn/account\_login.htm')

\_URL ='http://www.kuaipan.cn/index.php'

s.post(\_URL, params=\{'ac':'account', 'op':'login'\},

data=\{'username':'\*\*\*\*@foxmail.com', 'userpwd':'\*\*\*\*\*\*\*\*', 'isajax':'yes'\})

r =s.get(\_URL, params=\{'ac':'zone', 'op':'taskdetail'\})

print(r.json())

s.get(\_URL, params=\{'ac':'common', 'op':'usersign'\})

这是一个基本的文件保存操作，但这里有几个值得注意的问题：

1.安装requests包，命令行输入pip install requests即可自动安装。很多人推荐使用requests，自带的urllib.request也可以抓取网页源码

2.open方法encoding参数设为utf-8，否则保存的文件会出现乱码。

3.如果直接在cmd中输出抓取的内容，会提示各种编码错误，所以保存到文件查看。

4.with open方法是更好的写法，可以自动操作完毕后释放资源

\#! /urs/bin/python3

import requests

'''requests模块抓取网页源码并保存到文件示例'''

html =requests.get("http://www.baidu.com")

with open('test.txt', 'w', encoding='utf-8') as f:

f.write(html.text)

'''读取一个txt文件，每次读取一行，并保存到另一个txt文件中的示例'''

ff =open('testt.txt', 'w', encoding='utf-8')

with open('test.txt', encoding="utf-8") as f:

for line in f:

ff.write(line)

ff.close()

因为在命令行中打印每次读取一行的数据，中文会出现编码错误，所以每次读取一行并保存到另一个文件，这样来测试读取是否正常。(注意open的时候制定encoding编码方式)

![Python—requests模块详解\_Python—requests模块详解\_05][Python_requests_Python_requests_05]![Python—requests模块详解\_Python—requests模块详解\_06][Python_requests_Python_requests_06]

\#!/usr/bin/env python

\# -\*- coding:utf-8 -\*-

import requests

\# \#\#\#\#\#\#\#\#\#\#\#\#\#\# 方式一 \#\#\#\#\#\#\#\#\#\#\#\#\#\#

"""

\# \#\# 1、首先登陆任何页面，获取cookie

i1=requests.get(url="http://dig.chouti.com/help/service")

i1\_cookies=i1.cookies.get\_dict()

\# \#\# 2、用户登陆，携带上一次的cookie，后台对cookie中的 gpsd 进行授权

i2=requests.post(

url="http://dig.chouti.com/login",

data=\{

'phone': "8615131255089",

'password': "xxooxxoo",

'oneMonth': ""

\},

cookies=i1\_cookies

)

\# \#\# 3、点赞(只需要携带已经被授权的gpsd即可)

gpsd=i1\_cookies\['gpsd'\]

i3=requests.post(

url="http://dig.chouti.com/link/vote?linksId=8589523",

cookies=\{'gpsd': gpsd\}

)

print(i3.text)

"""

\# \#\#\#\#\#\#\#\#\#\#\#\#\#\# 方式二 \#\#\#\#\#\#\#\#\#\#\#\#\#\#

"""

import requests

session=requests.Session()

i1=session.get(url="http://dig.chouti.com/help/service")

i2=session.post(

url="http://dig.chouti.com/login",

data=\{

'phone': "8615131255089",

'password': "xxooxxoo",

'oneMonth': ""

)

i3=session.post(

url="http://dig.chouti.com/link/vote?linksId=8589523"

)

print(i3.text)

"""

抽屉新热榜

![Python—requests模块详解\_Python—requests模块详解\_07][Python_requests_Python_requests_07]![Python—requests模块详解\_Python—requests模块详解\_08][Python_requests_Python_requests_08]

\#!/usr/bin/env python

\# -\*- coding:utf-8 -\*-

import requests

from bs4 import BeautifulSoup

\# \#\#\#\#\#\#\#\#\#\#\#\#\#\# 方式一 \#\#\#\#\#\#\#\#\#\#\#\#\#\#

\# \# 1. 访问登陆页面，获取 authenticity\_token

\# i1=requests.get('https://github.com/login')

\# soup1=BeautifulSoup(i1.text, features='lxml')

\# tag=soup1.find(name='input', attrs=\{'name': 'authenticity\_token'\})

\# authenticity\_token=tag.get('value')

\# c1=i1.cookies.get\_dict()

\# i1.close()

\# \# 1. 携带authenticity\_token和用户名密码等信息，发送用户验证

\# form\_data=\{

\# "authenticity\_token": authenticity\_token,

\# "utf8": "",

\# "commit": "Sign in",

\# "login": "wupeiqi@live.com",

\# 'password': 'xxoo'

\# \}

\# i2=requests.post('https://github.com/session', data=form\_data, cookies=c1)

\# c2=i2.cookies.get\_dict()

\# c1.update(c2)

\# i3=requests.get('https://github.com/settings/repositories', cookies=c1)

\# soup3=BeautifulSoup(i3.text, features='lxml')

\# list\_group=soup3.find(name='div', class\_='listgroup')

\# from bs4.element import Tag

\# for child in list\_group.children:

\# if isinstance(child, Tag):

\# project\_tag=child.find(name='a', class\_='mr-1')

\# size\_tag=child.find(name='small')

\# temp="项目:%s(%s); 项目路径:%s" % (project\_tag.get('href'), size\_tag.string, project\_tag.string, )

\# print(temp)

\# \#\#\#\#\#\#\#\#\#\#\#\#\#\# 方式二 \#\#\#\#\#\#\#\#\#\#\#\#\#\#

\# session=requests.Session()

\# \# 1. 访问登陆页面，获取 authenticity\_token

\# i1=session.get('https://github.com/login')

\# soup1=BeautifulSoup(i1.text, features='lxml')

\# tag=soup1.find(name='input', attrs=\{'name': 'authenticity\_token'\})

\# authenticity\_token=tag.get('value')

\# c1=i1.cookies.get\_dict()

\# i1.close()

\# \# 1. 携带authenticity\_token和用户名密码等信息，发送用户验证

\# form\_data=\{

\# "authenticity\_token": authenticity\_token,

\# "utf8": "",

\# "commit": "Sign in",

\# "login": "wupeiqi@live.com",

\# 'password': 'xxoo'

\# \}

\# i2=session.post('https://github.com/session', data=form\_data)

\# c2=i2.cookies.get\_dict()

\# c1.update(c2)

\# i3=session.get('https://github.com/settings/repositories')

\# soup3=BeautifulSoup(i3.text, features='lxml')

\# list\_group=soup3.find(name='div', class\_='listgroup')

\# from bs4.element import Tag

\# for child in list\_group.children:

\# if isinstance(child, Tag):

\# project\_tag=child.find(name='a', class\_='mr-1')

\# size\_tag=child.find(name='small')

\# temp="项目:%s(%s); 项目路径:%s" % (project\_tag.get('href'), size\_tag.string, project\_tag.string, )

\# print(temp)

github

![Python—requests模块详解\_Python—requests模块详解\_09][Python_requests_Python_requests_09]![Python—requests模块详解\_Python—requests模块详解\_10][Python_requests_Python_requests_10]

\#!/usr/bin/env python

\# -\*- coding:utf-8 -\*-

import time

import requests

from bs4 import BeautifulSoup

session = requests.Session()

i1 = session.get(

url='https://www.zhihu.com/\#signin',

headers=\{

'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_10\_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',

)

soup1 =BeautifulSoup(i1.text, 'lxml')

xsrf\_tag =soup1.find(name='input', attrs=\{'name': '\_xsrf'\})

xsrf =xsrf\_tag.get('value')

current\_time = time.time()

i2 = session.get(

url='https://www.zhihu.com/captcha.gif',

params=\{'r': current\_time, 'type': 'login'\},

headers=\{

'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_10\_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',

\})

with open('zhihu.gif', 'wb') as f:

f.write(i2.content)

captcha =input('请打开zhihu.gif文件，查看并输入验证码：')

form\_data = \{

"\_xsrf": xsrf,

'password': 'xxooxxoo',

"captcha": 'captcha',

'email': '424662508@qq.com'

i3 = session.post(

url='https://www.zhihu.com/login/email',

data=form\_data,

headers=\{

'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_10\_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',

)

i4 = session.get(

url='https://www.zhihu.com/settings/profile',

headers=\{

'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_10\_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',

)

soup4 =BeautifulSoup(i4.text, 'lxml')

tag =soup4.find(id='rename-section')

nick\_name =tag.find('span',class\_='name').string

print(nick\_name)

知乎

![Python—requests模块详解\_Python—requests模块详解\_11][Python_requests_Python_requests_11]![Python—requests模块详解\_Python—requests模块详解\_12][Python_requests_Python_requests_12]

\#!/usr/bin/env python

\# -\*- coding:utf-8 -\*-

import re

import json

import base64

import rsa

import requests

def js\_encrypt(text):

b64der ='MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCp0wHYbg/NOPO3nzMD3dndwS0MccuMeXCHgVlGOoYyFwLdS24Im2e7YyhB0wrUsyYf0/nhzCzBK8ZC9eCWqd0aHbdgOQT6CuFQBMjbyGYvlVYU2ZP7kG9Ft6YV6oc9ambuO7nPZh+bvXH0zDKfi02prknrScAKC0XhadTHT3Al0QIDAQAB'

der = base64.standard\_b64decode(b64der)

pk = rsa.PublicKey.load\_pkcs1\_openssl\_der(der)

v1 =rsa.encrypt(bytes(text, 'utf8'), pk)

value =base64.encodebytes(v1).replace(b'

', b'')

value =value.decode('utf8')

return value

session = requests.Session()

i1 =session.get('https://passport.cnblogs.com/user/signin')

rep =re.compile("'VerificationToken': '(.\*)'")

v = re.search(rep, i1.text)

verification\_token =v.group(1)

form\_data = \{

'input1': js\_encrypt('wptawy'),

'input2': js\_encrypt('asdfasdf'),

'remember': False

i2 =session.post(url='https://passport.cnblogs.com/user/signin',

data=json.dumps(form\_data),

headers=\{

'Content-Type': 'application/json; charset=UTF-8',

'X-Requested-With': 'XMLHttpRequest',

'VerificationToken': verification\_token\}

)

i3 =session.get(url='https://i.cnblogs.com/EditDiary.aspx')

print(i3.text)

博客园

![Python—requests模块详解\_Python—requests模块详解\_13][Python_requests_Python_requests_13]![Python—requests模块详解\_Python—requests模块详解\_14][Python_requests_Python_requests_14]

\#!/usr/bin/env python

\# -\*- coding:utf-8 -\*-

import requests

\# 第一步：访问登陆页,拿到X\_Anti\_Forge\_Token，X\_Anti\_Forge\_Code

\# 1、请求url:https://passport.lagou.com/login/login.html

\# 2、请求方法:GET

\# 3、请求头:

\# User-agent

r1=requests.get('https://passport.lagou.com/login/login.html',

headers=\{

'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36',

\},

)

X\_Anti\_Forge\_Token =re.findall("X\_Anti\_Forge\_Token='(.\*?)'", r1.text, re.S)\[0\]

X\_Anti\_Forge\_Code =re.findall("X\_Anti\_Forge\_Code='(.\*?)'", r1.text, re.S)\[0\]

print(X\_Anti\_Forge\_Token, X\_Anti\_Forge\_Code)

\# print(r1.cookies.get\_dict())

\# 第二步：登陆

\# 1、请求url:https://passport.lagou.com/login/login.json

\# 2、请求方法:POST

\# 3、请求头:

\# cookie

\# User-agent

\# Referer:https://passport.lagou.com/login/login.html

\# X-Anit-Forge-Code:53165984

\# X-Anit-Forge-Token:3b6a2f62-80f0-428b-8efb-ef72fc100d78

\# X-Requested-With:XMLHttpRequest

\# 4、请求体：

\# isValidate:true

\# username:15131252215

\# password:ab18d270d7126ea65915c50288c22c0d

\# request\_form\_verifyCode:''

\# submit:''

r2= requests.post(

'https://passport.lagou.com/login/login.json',

headers=\{

'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36',

'Referer': 'https://passport.lagou.com/login/login.html',

'X-Anit-Forge-Code': X\_Anti\_Forge\_Code,

'X-Anit-Forge-Token': X\_Anti\_Forge\_Token,

'X-Requested-With': 'XMLHttpRequest'

\},

data=\{

"isValidate": True,

'username': '15131255089',

'password': 'ab18d270d7126ea65915c50288c22c0d',

'request\_form\_verifyCode': '',

'submit': ''

\},

cookies=r1.cookies.get\_dict()

)

print(r2.text)

拉勾网

参考：

http://cn.python-requests.org/zh\_CN/latest/user/quickstart.html\#id4

http://www.python-requests.org/en/master/

http://docs.python-requests.org/en/latest/user/quickstart/

https://www.cnblogs.com/tangdongchu/p/4229049.html\#t0

http://www.cnblogs.com/wupeiqi/articles/6283017.html

[Python_requests_Python_requests]: https://img-blog.csdnimg.cn/img_convert/c962540251da351ff9a1762b1261f493.gif
[Python_requests_Python_requests_02]: https://img-blog.csdnimg.cn/img_convert/13503da1b6c853ea8301bbcba37328a7.gif
[Python_requests_Python_requests_03]: https://img-blog.csdnimg.cn/img_convert/3f5d9c6792d176e1536c57a6acb30132.gif
[Python_requests_Python_requests_04]: https://img-blog.csdnimg.cn/img_convert/f6342a493d36b3a86d511b08639f92ba.gif
[Python_requests_Python_requests_05]: https://img-blog.csdnimg.cn/img_convert/cd977cbfad44263bf5ba6fd5d2a149c0.gif
[Python_requests_Python_requests_06]: https://img-blog.csdnimg.cn/img_convert/f2884985d2dcee57bf3f26e54fbae40e.gif
[Python_requests_Python_requests_07]: https://img-blog.csdnimg.cn/img_convert/068f12a3dfeeaa6c9b00e81b86392fec.gif
[Python_requests_Python_requests_08]: https://img-blog.csdnimg.cn/img_convert/39612c21ea0025a787a809e5ba4bbcc4.gif
[Python_requests_Python_requests_09]: https://img-blog.csdnimg.cn/img_convert/1fa05f1bc4d9ffdbc12f5c21d1546e07.gif
[Python_requests_Python_requests_10]: https://img-blog.csdnimg.cn/img_convert/f8c08f7032f99acac5d173de2f3ee555.gif
[Python_requests_Python_requests_11]: https://img-blog.csdnimg.cn/img_convert/0afb7f9c66657ba3eccd520b380d25c8.gif
[Python_requests_Python_requests_12]: https://img-blog.csdnimg.cn/img_convert/b60a718a92241d7b6fb3382d4efda38d.gif
[Python_requests_Python_requests_13]: https://img-blog.csdnimg.cn/img_convert/58c332d9b4ed72d7e795bab3b244b259.gif
[Python_requests_Python_requests_14]: https://img-blog.csdnimg.cn/img_convert/548db8727bd38567a7e6f9666b81acbe.gif