Python在线爬取网页视频系列(二)

小咪咪 2023-01-10 10:29 64阅读 0赞

**you-get是什么**

1.  任意打开打开[网址][Link 1]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw_size_16_color_FFFFFF_t_70_pic_center]
2.  点击右键-检查-选择Network-点击XHR-Ctrl+R刷新界面  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw_size_16_color_FFFFFF_t_70]
3.  相关的爬取代码为：

import requests
    import json
    # 1.确定爬取的url路径及headers参数
    base_url = "https://www.ku6.com/video/feed?pageNo=0&pageSize=40"
    headers = {
        "User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"}
    
    # 2.发送请求 --- requests模拟浏览器数据发送请求，获取响应数据
    response = requests.get(base_url, headers=headers)
    data = response.text
    print(data)
    
    # 3. 解析数据： json模块—把json字符串解析成python可交互的数据类型
    # 3.1 数据转换
    json_data = json.loads(data)
    # 3.2 解析数据
    data_list = json_data['data']
    
    for a in data_list:
        video_title = a['title'] + '.mp4'
        video_url = a['playUrl']
    
        # 再次发送请求，请求视频数据
        print("正在下载：", video_title)
        video_data = requests.get(video_url, headers=headers).content
    
        # 保存数据
        with open("video\\" + video_title, 'wb') as f:
            f.write(video_data)
            print("下载完成")

解析数据时，字典的键title和playUrl根据下图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw_size_16_color_FFFFFF_t_70 1]

[Link 1]: https://www.ku6.com/index
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw_size_16_color_FFFFFF_t_70_pic_center]: /images/20221119/1daac09e492e44a4b8646ec0d5dd2527.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw_size_16_color_FFFFFF_t_70]: /images/20221119/a04aafffcca94107a6ed3849aeb9f8a3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI4MzM5Nw_size_16_color_FFFFFF_t_70 1]: /images/20221119/b95bcbce001c45dca70cc1a8c69d3c32.png