Python爬取淘数据平台商品数据，发现假发的市场原来那么火

约定不等于承诺〃 2022-12-27 07:40 88阅读 0赞

# 前言 #

本文的文字及图片来源于网络,仅供学习、交流使用,不具有任何商业用途,如有问题请及时联系我们以作处理。

最近发现了一个不错的数据网站，叫“淘数据”。里面的数据都是淘宝的商家数据，包含了店名、类目、标价、成交均价、销量、销售金额等

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzQ4NDA1Nzgx_size_16_color_FFFFFF_t_70][]

这个网站还是一位同学和我说才知道的，既然这样，那就开始爬

![d09c646a8cc1ed8f42fcc6e52d8ba9d4.png][]

# 项目目标 #

爬取淘宝假发专业数据，假发是我当时随便选的，再想选别的，要收费了

![1841f57529e9129067decdc225b41c9c.gif][]

可能是命运的安排吧，知道程序员需要什么

![33552a3d93913b95de464711962f4715.png][]

受害者地址

https://www.taosj.com/industry/index.html#/data/hotitems/?cid=50023283&brand=&type=&pcid=

# 环境 #

Python3.6

pycharm

# 爬虫代码 #

**导入所需工具**

import requests
    import csv

**分析网页，先F12打开开发者工具，复制你所需要的数据，找数据所在的标签**

![3e84b6d0eae4680c718e1d3b16dce579.png][]

![4392870efddef286277e42f9369d5d72.png][]

**找到所需要的URL和headers中的参数**

![6fcf79cc4b7a0621288640f4bdbdbdb8.png][]

![1c6765766e632cb7f0f38a11c74909ba.png][]

url = 'https://www.taosj.com/data/industry/hotitems/list?cid=50023283&brand=&type=ALL&date=1596211200000&pageNo=1&pageSize=10&orderType=desc&orderField='.format(page)
    
    headers = {
        'Host':'www.taosj.com',
        'Referer':'https://www.taosj.com/industry/index.html',
        'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36',
    }
    
    response = requests.get(url=url, headers=headers)
    html_data = response.json()

**在json数据中提取相关数据**

lis = html_data['data']['list']
    for li in lis:
        tb_url = 'https://detail.tmall.com/item.htm?id={}'.format(li['id'])
        dit = {
            '标题': li['title'],
            '店铺名称': li['shop'],
            '类目': li['nextCatName'],
            '标价': li['oriPrice'],
            '成交均价': li['price'],
            '销售量': li['offer30'],
            '销售金额': li['price30'],
            '淘宝地址': tb_url,
        }

**保存数据**

f = open('淘数据.csv', mode='a', encoding='utf-8-sig', newline='')
    csv_writer = csv.DictWriter(f, fieldnames=['标题', '品牌', '店铺名称', '类目', '标价', '成交均价', '销售量', '销售金额', '淘宝地址'])
    csv_writer.writeheader()
    print(dit)

# 效果图 #

![ce8a9aa525173a5adde939c2b7d5840c.png][]

![ab82616f4ae7387af9a9b117cf435dba.png][]

**完整代码**

import requests
    import csv
    
    f = open('淘数据.csv', mode='a', encoding='utf-8-sig', newline='')
    csv_writer = csv.DictWriter(f, fieldnames=['标题', '品牌', '店铺名称', '类目', '标价', '成交均价', '销售量', '销售金额', '淘宝地址'])
    csv_writer.writeheader()
    
    for page in range(1, 51):
        url = 'https://www.taosj.com/data/industry/hotitems/list?cid=50023282&brand=&type=ALL&date=1596211200000&pageNo={}&pageSize=10&orderType=desc&orderField=amount&searchKey='.format(page)
        """
        复制开发者工具中的requests headers 中的参数 记得加cookie
        """
        headers = {
    
            'Host': 'www.taosj.com',
            'Referer': 'https://www.taosj.com/industry/index.html',
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36',
        }
    
        response = requests.get(url=url, headers=headers)
        html_data = response.json()
    
        lis = html_data['data']['list']
        for li in lis:
            tb_url = 'https://detail.tmall.com/item.htm?id={}'.format(li['id'])
            dit = {
                '标题': li['title'],
                '品牌': li['brand'],
                '店铺名称': li['shop'],
                '类目': li['nextCatName'],
                '标价': li['oriPrice'],
                '成交均价': li['price'],
                '销售量': li['offer30'],
                '销售金额': li['price30'],
                '淘宝地址': tb_url,
            }
            csv_writer.writerow(dit)
            print(dit)

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzQ4NDA1Nzgx_size_16_color_FFFFFF_t_70]: /images/20221120/516d10566d984cc780a9536fb08d9e29.png
[d09c646a8cc1ed8f42fcc6e52d8ba9d4.png]: /images/20221120/e8d1af3084284de09b22471743757311.png
[1841f57529e9129067decdc225b41c9c.gif]: /images/20221120/10d042e5a4564662b9829927bdcd4f1f.png
[33552a3d93913b95de464711962f4715.png]: /images/20221120/569321a5375848f883d3834dbd6c5064.png
[3e84b6d0eae4680c718e1d3b16dce579.png]: /images/20221120/307c0c6eb7484201994b34bdc885b836.png
[4392870efddef286277e42f9369d5d72.png]: /images/20221120/be3b52b786774efc870d165c0b52e93b.png
[6fcf79cc4b7a0621288640f4bdbdbdb8.png]: /images/20221120/0711a38c7ca44f289502e2198062e35f.png
[1c6765766e632cb7f0f38a11c74909ba.png]: /images/20221120/90dff63ac28d4f219fc127b8999e4462.png
[ce8a9aa525173a5adde939c2b7d5840c.png]: /images/20221120/1e13e9a6da584f83a9011472dfb651fe.png
[ab82616f4ae7387af9a9b117cf435dba.png]: /images/20221120/5bf4b68cd68545b3a73030c92a93eb52.png