【阿里天池-医学影像报告异常检测】1 数据分析和探索

女爷i 2022-11-01 05:24 174阅读 0赞

### 目录 ###

*  1 赛题
 *  2 数据分析
 *   *  2.1 读取数据
     *  2.2 统计词数
     *  2.3 统计词频
     *  2.4 统计句子长度
     *  2.5 label类别分布
     *  2.6 正负样本分布
 *  2.7 缺失值
 *  2.8 统计句子最后一个字符词频

# 1 赛题 #

[全球人工智能技术创新大赛【赛道一】-官方赛道地址][-]  
[2021年赛道一：医学影像报告异常检测][2021]

# 2 数据分析 #

## 2.1 读取数据 ##

import numpy as np
    import pandas as pd
    import tensorflow as tf
    import matplotlib.pyplot as plt
    plt.rcParams['font.sans-serif'] = ['SimHei'] # 步骤一（替换sans-serif字体）
    plt.rcParams['axes.unicode_minus'] = False   # 步骤二（解决坐标轴负数的负号显示问题）
    train_data_file = './raw_data/track1_round1_train_20210222.csv'
    test_data_file = './raw_data/track1_round1_testA_20210222.csv'
    picture_file = './documentation/picture/'

## 2.2 统计词数 ##

with open(train_data_file, 'r') as f:
        lines = f.readlines()
    train_texts, train_labels = [], []
    for id, line in enumerate(lines):
        line = line.strip().replace('|', '').split(',')
        text = line[1].strip().split(' ')
        text = [int(word) for word in text]
        train_texts.append(text)
        train_labels.append(line[2])
        
    with open(test_data_file, 'r') as f:
        lines = f.readlines()
    test_texts = []
    for id, line in enumerate(lines):
        line = line.strip().replace('|', '').split(',')
        text = line[1].strip().split(' ')
        text = [int(word) for word in text]
        test_texts.append(text)

（1）训练集的词数

vocab_size = max(max(text) for text in train_texts) - min(min(text) for text in train_texts) + 1
    print("训练集的vpcab_size: {}".format(vocab_size))

> 训练集的vpcab\_size: 858

（2）测试集的词数

vocab_size = max(max(text) for text in train_texts) - min(min(text) for text in test_texts) + 1
    print("测试集的vocab_size: {}".format(vocab_size))

> 测试集的vpcab\_size: 858

测试集中所用字和训练集相同

## 2.3 统计词频 ##

word_count = np.zeros(vocab_size, dtype='int32')
    for text in train_texts:
        for word in text:
            word_count[word] += 1
    sorted_index = word_count.argsort()

（1）统计词频最低的top20

# 最小的词频
    for i in range(20):
        word = sorted_index[i]
        print(str(word) + "|" + str(word_count[word]))

> 236|18  
> 466|18  
> 246|19  
> 82|19  
> 29|19  
> 263|20  
> 451|21  
> 849|22  
> 804|22  
> 384|22  
> 684|23  
> 360|23  
> 714|23  
> 210|23  
> 275|24  
> 157|24  
> 678|24  
> 186|24  
> 696|24  
> 467|24

（2）统计词频最高的top20

# 最大的词频
    for i in range(-1,-21,-1):
        word = sorted_index[i]
        print(str(word) + "|" + str(word_count[word]))

> 693|5477  
> 328|4009  
> 698|3592  
> 380|3198  
> 177|2844  
> 415|2785  
> 381|2699  
> 809|2111  
> 266|1501  
> 623|1467  
> 14|1434  
> 852|1387  
> 256|1344  
> 842|1267  
> 832|1245  
> 172|1243  
> 399|1155  
> 204|1142  
> 382|1135  
> 582|1130

## 2.4 统计句子长度 ##

df_train = pd.DataFrame()
    df_train['train_text'] = train_texts
    df_train['train_label'] = train_label
    df_train['text_length'] = df.train_text.apply(len)
    df_train['disease_num'] = df.train_label.apply(lambda x:len(x.strip().split()))
    df_test = pd.DataFrame()
    df_test['test_text'] = test_texts
    df_test['text_length'] = df_test.test_text.apply(len)
    text_length = [len(text) for text in train_texts]
    _ = plt.hist(text_length, bins=50, label='句长分布')
    _ = plt.legend()

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNTY5Ng_size_16_color_FFFFFF_t_70_pic_center]

（1）训练集句子长度统计

df_train.text_length.describe()

> count 10000.000000  
> mean 41.564800  
> std 18.349127  
> min 4.000000  
> 25% 29.000000  
> 50% 38.000000  
> 75% 52.000000  
> max 104.000000  
> Name: text\_length, dtype: float64

（2）测试集句子长度统计

df_test.text_length.describe()
    count    3000.000000
    mean       40.409667
    std        17.695561
    min         4.000000
    25%        28.000000
    50%        37.000000
    75%        50.000000
    max       102.000000
    Name: text_length, dtype: float64

## 2.5 label类别分布 ##

label_counts = np.zeros(17, dtype='int32')
    for labels in train_labels:
        labels = labels.strip().split(' ')
        for label in labels:
            if label != '':
                label_counts[int(label)]+=1
    _ = plt.bar(range(17), label_counts, label='各标签数量统计')
    for x, count in enumerate(label_counts):
        plt.text(x-0.5, count+20, "{}".format(count))
    _ = plt.xlabel("label")
    _ = plt.xticks(range(0,17))
    _ = plt.ylabel("频数")
    _ = plt.title("各个标签出现频数")
    plt.savefig(picture_file+'label_freq.jpg')

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNTY5Ng_size_16_color_FFFFFF_t_70_pic_center 1]

## 2.6 正负样本分布 ##

total_num = df_train.shape[0]
    for id, count in enumerate(label_counts):
        print("{}|{}|{}|{:.2f}%".format(id, count, total_num-count, 100*count/total_num))

> 0|1107|8893|11.07%  
> 1|1150|8850|11.50%  
> 2|930|9070|9.30%  
> 3|445|9555|4.45%  
> 4|1231|8769|12.31%  
> 5|386|9614|3.86%  
> 6|209|9791|2.09%  
> 7|1080|8920|10.80%  
> 8|1077|8923|10.77%  
> 9|845|9155|8.45%  
> 10|377|9623|3.77%  
> 11|1033|8967|10.33%  
> 12|417|9583|4.17%  
> 13|269|9731|2.69%  
> 14|549|9451|5.49%  
> 15|1846|8154|18.46%  
> 16|777|9223|7.77%

# 2.7 缺失值 #

（１）text  
无  
（2）label  
有，暂未统计

# 2.8 统计句子最后一个字符词频 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNTY5Ng_size_16_color_FFFFFF_t_70_pic_center 2]

[-]: https://tianchi.aliyun.com/competition/entrance/531852/introduction?spm=5176.12281957.1004.6.38b03eafApg5Vq
[2021]: https://betterbench.blog.csdn.net/article/details/112742659
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNTY5Ng_size_16_color_FFFFFF_t_70_pic_center]: /images/20221024/b342234d531b45c1918d1f605341cef7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNTY5Ng_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221024/509436d50e3c4bf7a80d49b9154cee02.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzNTY5Ng_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221024/f0c48c57f3d54f7a9c514ef7fd6bb973.png