爬取网页的通用代码框架

短命女 2023-07-18 09:56 15阅读 0赞

### 爬取网页的通用代码框架 ###

import requests
    def getHTMLText(url):
    	try:
    		r=requests.get(url,timeout=30)
    		r.raise_for_status()   #若状态不是200，引发HTTPError异常
    		r.encoding=r.apparent_encoding
    		return r.text
    	except:
    		return "产生异常"
    
    if __name__=="__main__":
    	url="http://www.baidu.com"
    	print(getHTMLText(url))

### 一：京东商品页面的爬取 ###

import requests
    url="https://pro.jd.com/mall/active/28bw6fLCSoxGTKKBiQZLmUrVoSKV/index.html"
    try:
    	r=requests.get(url)
    	r.raise_for_status()
    	r.encoding=r.apparent_encoding
    	print(r.text[:1000])
    except :
    	print("爬取失败")

r.status\_code返回的是200，可以直接爬取

### 二：亚马逊商品页面的爬取 ###

import requests
    url="https://www.amazon.cn/dp/B07JMQTHWR/ref=s9_acsd_hps_bw_c2_x_0_i?pf_rd_m=A1U5RCOVU0NYF2&pf_rd_s=merchandised-search-2&pf_rd_r=PF3XEDEN44FP3A9EN2RM&pf_rd_t=101&pf_rd_p=3f149c06-7221-4924-a124-ebbe0310b38c&pf_rd_i=116169071"
    try:
    	kv={'user-agent':'Mozilla/5.0'}
    	r=requests.get(url,headers=kv)
    	r.raise_for_status()
    	r.encoding=r.apparent_encoding
    	print(r.text[1000:2000])
    except :
    	print("爬取失败")

r.status\_code返回的是503，所以需要修改headers字段，以隐藏自己爬虫的身份

### 三：百度搜索关键词提交 ###

import requests
    keyword="Python"
    try:
    	kv={'wd':keyword}
    	r=requests.get("http://www.baidu.com/s",params=kv)
    	print(r.request.url)
    	r.raise_for_status()
    	print(len(r.text))
    except :
    	print("爬取失败")

疑似被重定向

![20200326211000949.PNG][]

### 四：360搜索关键词提交 ###

import requests
    keyword="Python"
    try:
    	kv={'q':keyword}
    	r=requests.get("http://www.so.com/s",params=kv)
    	print(r.request.url)
    	r.raise_for_status()
    	print(len(r.text))
    except :
    	print("爬取失败")

成功

![20200326211241566.PNG][]

### 五：网络图片的爬取和存储 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xZXzYyNA_size_16_color_FFFFFF_t_70][]

爬取成功，图片保存到了D盘，名字为abc.jpg

import requests
    import os
    url="https://timgsa.baidu.com/timg?image&quality=80&size=b9999_10000&sec=1585238676005&di=806d8fd3824fb2380f9c1f0d2d4ab76a&imgtype=0&src=http%3A%2F%2Fimg.article.pchome.net%2F00%2F38%2F35%2F30%2Fpic_lib%2Fwm%2FChuntian11.jpg"
    path="D:/abc.jpg"
    try:
    	if not os.path.exists(path):  #判断图片是否存在
    		r=requests.get(url)
    		with open(path,'wb') as f:
    			f.write(r.content)
    			f.close()
    			print("文件保存成功")
    	else:
    		print("文件已存在")
    except :
    	print("爬取失败")

[20200326211000949.PNG]: /images/20230528/4b094f923ac04feba67b4f2e689cd65a.png
[20200326211241566.PNG]: /images/20230528/ab8d90e43e964b20be81642eb872c1cf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xZXzYyNA_size_16_color_FFFFFF_t_70]: /images/20230528/b1ee7c48874845ab8f497efa01bf8672.png