2021广东工业智造创新大赛—智能算法赛总结（一）

我会带着你远行 2022-10-25 01:30 186阅读 0赞

**目录**

赛题

一、赛题背景

二、赛题数据说明

初赛数据示例

初赛数据提供

训练数据文件结构

标注说明

类别说明

三、提交说明

评测结果提交

python示例代码

四、评估指标

初赛

数据分析

查看数据

处理数据

--------------------

# **赛题** #

## **一、赛题背景** ##

佛山作为国内最大的瓷砖生产制造基地之一，拥有众多瓷砖厂家和品牌。经前期调研，瓷砖生产环节一般（不同类型砖工艺不一样，这里以抛釉砖为例）经过原材料混合研磨、脱水、压胚、喷墨印花、淋釉、烧制、抛光，最后进行质量检测和包装。得益于产业自动化的发展，目前生产环节已基本实现无人化。而质量检测环节仍大量依赖人工完成。一般来说，一条产线需要配2～6名质检工，长时间在高光下观察瓷砖表面寻找瑕疵。这样导致质检效率低下、质检质量层次不齐且成本居高不下。瓷砖表检是瓷砖行业生产和质量管理的重要环节，也是困扰行业多年的技术瓶颈。

本赛场聚焦瓷砖表面瑕疵智能检测，要求选手开发出高效可靠的计算机视觉算法，提升瓷砖表面瑕疵质检的效果和效率，降低对大量人工的依赖。要求算法尽可能快与准确的给出瓷砖疵点具体的位置和类别，主要考察疵点的定位和分类能力。

## **二、赛题数据说明** ##

大赛深入到佛山瓷砖知名企业，在产线上架设专业拍摄设备，实地采集生产过程真实数据，解决企业真实的痛点需求。大赛数据覆盖到了瓷砖产线所有常见瑕疵，包括粉团、角裂、滴釉、断墨、滴墨、B孔、落脏、边裂、缺角、砖渣、白边等。实拍图示例如下：

![20210130160719969.png][]

针对某些缺陷在特定视角下的才能拍摄到，每块砖拍摄了三张图，包括低角度光照黑白图、高角度光照黑白图、彩色图，示例如下：

![20210130160731318.png][]

数据主要分为两种：

1.  白板瓷砖。花色简单，数量总共约12000张，包含训练集和测试集，用于初赛。
2.  复杂瓷砖。花色相对复杂，并提供相应的模板图片（同花色且无瑕疵图片），数量总共约12000张，包含训练集和测试集，用于复赛。

### **初赛数据示例** ###

白板数据包含有瑕疵图片、无瑕疵图片和标注数据。标注数据标注瑕疵位置和类别信息。图片示例如下：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70][]

### **初赛数据提供** ###

1.  初赛训练集于1月4日提供。
2.  初赛一阶段测试集于1月4日提供下载，初赛二阶段测试集1月28号提供下载。

## ##

### **训练数据文件结构** ###

└── dataset

├── Readme.md

├── train_annos.json

├── train_imgs

train\_imgs：训练图片数据，jpg格式  
train\_annos.json：训练标注数据，json格式  
[Readme.md][]：说明文件

### **标注说明** ###

训练标注是`train_annos.json`，内容如下：

[
        {
            "name": "226_46_t20201125133518273_CAM1.jpg",
            "image_height": 6000,
            "image_width": 8192,
            "category": 4,
            "bbox": [
                1587,
                4900,
                1594,
                4909
            ]
        },
        '''
        '''
    ]

具体说明如下：  
`name`是图片名，"226\_46"代表砖的唯一id，"CAM1"代表相机1拍照所得，一般来说每块砖会有三张样本，分别是CAM1，CAM2，CAM3。`image_height`和`image_width`是图片高宽，`category"`是类别id，`bbox`是目标框信息xyrb格式，分别指`[``左上角``x``坐标，左上角``y``坐标，右下角``x``坐标，右下角``y``坐标``]`

### **类别说明** ###

id和瑕疵名的对应关系如下：

\{ "0": "背景", "1": "边异常", "2": "角异常", "3": "白色点瑕疵", "4": "浅色块瑕疵", "5": "深色点块瑕疵", "6": "光圈瑕疵", "7": "记号笔", "8": "划伤" \}

其中，`记号笔`和`划伤`是复赛新增。

## **三、提交说明** ##

### **评测结果提交** ###

参赛者需要提供一份json文件包含所有预测结果，文件内容如下:

[
        {
            "name": "226_46_t20201125133518273_CAM1.jpg",
            "category": 4,
            "bbox": [
                5662,
                2489,
                5671,
                2497
            ],
            "score": 0.130577
        },
        {
            "name": "226_46_t20201125133518273_CAM1.jpg",
            "category": 2,
            "bbox": [
                6643,
                5416,
                6713,
                5444
            ],
            "score": 0.120612
        },
        ...
        ...
        {
            "name": "230_118_t20201126144204721_CAM2.jpg",
            "category": 5,
            "bbox": [
                3543,
                3875,
                3554,
                3889
            ],
            "score": 0.160216
        }
    ]

具体说明如下：  
1.提交的`json`文件中包含多个疵点样本，每个疵点样本都包含`name`、`category`、`bbox`、`score`四个字段。  
2.`name`字段为图片名称；`category`字段为类别标签；`bbox`为`xyrb`格式坐标框，分别指`[``左上角``x``坐标，左上角``y``坐标，右下角``x``坐标，右下角``y``坐标``]`，小数点后保留2位；`score`为置信度概率，范围0-1；  
3.对于单张图片存在多个疵点样本时，依次列出即可；对于不存在疵点的无疵点图片，不能出现在`json`列表中。  
4.`name`字段不允许出现非测试集中的图片名。

### **python示例代码** ###

result=[]
    result.append({'name': image_name(str),'category': defect_label(int),'bbox':bbox(xyxy,float),'score': score(float)})
    import json
    with open('result.json', 'w') as fp:
         json.dump(result, fp, indent=4, ensure_ascii=False)

## **四、评估指标** ##

### **初赛** ###

赛题分数计算方式: `0.2ACC+0.8mAP`

`ACC`：是有瑕疵或无瑕疵的分类指标，考察瑕疵检出能力。  
其中提交结果`name`字段中出现过的测试图片均认为有瑕疵，未出现的测试图片认为是无瑕疵。

`mAP`：参照`PASCALVOC`的评估标准计算瑕疵的mAP值。  
参考链接：[https://github.com/rafaelpadilla/Object-Detection-Metrics][https_github.com_rafaelpadilla_Object-Detection-Metrics] 具体逻辑见evaluator文件。  
  
需要指出，本次大赛评分计算过程中，分别在检测框和真实框的交并比(`IoU`)在阈值`0.1`，`0.3`，`0.5`下计算`mAP`，最终`mAP`取三个值的平均值。

## ##

# **数据分析** #

## **查看数据** ##

首先查看下载来的数据，随意打开一张图片，查看图片的属性。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 1][]

从上图可以看出图像是8192×6000大小的彩色图片，图片尺寸和COCO数据的图片相比大了很多。然后在数据集转为Labelme的标注数据。转化代码如下：

import sys
    import os.path as osp
    import io
    from labelme.logger import logger
    from labelme import PY2
    from labelme import QT4
    import PIL.Image
    import base64
    from labelme import utils
    import os
    import cv2
    import xml.etree.ElementTree as ET
    from PIL import ImageFile
    ImageFile.LOAD_TRUNCATED_IMAGES = True
    module_path = os.path.abspath(os.path.join('..'))
    if module_path not in sys.path:
        sys.path.append(module_path)
    import json
    from PIL import Image
    
    Image.MAX_IMAGE_PIXELS = None
    
    class_name_dic = {
        "0": "背景",
        "1": "边异常",
        "2": "角异常",
        "3": "白色点瑕疵",
        "4": "浅色块瑕疵",
        "5": "深色点块瑕疵",
        "6": "光圈瑕疵"
    }
    rawImgDir = '../tile_round1_train_20201231/train_imgs/'
    rawLabelDir = '../tile_round1_train_20201231/train_annos.json'
    with open(rawLabelDir) as f:
        annos = json.load(f)
    
    #
    image_ann = {}
    for i in range(len(annos)):
        anno = annos[i]
        name = anno['name']
        if name not in image_ann:
            image_ann[name] = []
        image_ann[name].append(i)
    
    
    def load_image_file(filename):
        try:
            image_pil = PIL.Image.open(filename)
        except IOError:
            logger.error('Failed opening image file: {}'.format(filename))
            return
    
        # apply orientation to image according to exif
        image_pil = utils.apply_exif_orientation(image_pil)
    
        with io.BytesIO() as f:
            ext = osp.splitext(filename)[1].lower()
            if PY2 and QT4:
                format = 'PNG'
            elif ext in ['.jpg', '.jpeg']:
                format = 'JPEG'
            else:
                format = 'PNG'
            image_pil.save(f, format=format)
            f.seek(0)
            return f.read()
    
    
    def dict_json(flags, imageData, shapes, imagePath, fillColor=None, lineColor=None, imageHeight=100, imageWidth=100):
        '''
        :param imageData: str
        :param shapes: list
        :param imagePath: str
        :param fillColor: list
        :param lineColor: list
        :return: dict
        '''
        return {"version": "3.16.4", "flags": flags, "shapes": shapes, 'lineColor': lineColor, "fillColor": fillColor,
                'imagePath': imagePath.split('/')[-1], "imageData": imageData, 'imageHeight': imageHeight,
                'imageWidth': imageWidth}
    
    
    data = json.load(open('1.json'))
    for name in image_ann.keys():
        indexs = image_ann[name]
        height, width = annos[indexs[0]]["image_height"], annos[indexs[0]]["image_width"]
        imagePath=rawImgDir+name
        print(imagePath)
        if not os.path.exists(imagePath): # True/False
            continue
        image = cv2.imread(imagePath)
        shapes = data["shapes"]
        version = data["version"]
        flags = data["flags"]
        lineColor = data["lineColor"]
        fillColor = data['fillColor']
        newshapes = []
        for inx in indexs:
            obj = annos[inx]
            assert name == obj['name']
            bbox = obj['bbox']
            category = obj['category']
            xmin, ymin, xmax, ymax = bbox
            class_name = class_name_dic[str(category)]
            newPoints = [[float(xmin), float(ymin)], [float(xmax), float(ymax)]]
            line_color = None
            fill_color = None
            shape_type = 'rectangle'
            flags = flags
            newshapes.append(
                {"label": class_name, "line_color": line_color, "fill_color": fill_color, "points": newPoints,
                 "shape_type": shape_type, "flags": flags})
        imageData_90 = load_image_file(imagePath)
        imageData_90 = base64.b64encode(imageData_90).decode('utf-8')
        imageHeight = image.shape[0]
        imageWidth = image.shape[1]
        data_90 = dict_json(flags, imageData_90, newshapes, imagePath, fillColor, lineColor, imageHeight, imageWidth)
        json_file = imagePath[:-4] + '.json'
        print(json_file)
        json.dump(data_90, open(json_file, 'w'))

运行结果：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 2][]

运行完成后，用Labelme查看检测的目标。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 3][]

## 处理数据  ##

待检测目标和图片大小相比小了很多，所以不能采用直接将图片Resize大小放到模型。我在本次比赛采用的策略是：小图训练，大图测试。

1、把训练集的大图切分成固定大小的图片，然后放入模型训练。

2、对测试集采用大图测试。

制作训练集

自作训练集，我在这里分了两个步骤，首先将Labelme数据转为txt数据集。再将图片和txt数据集按照滑窗的方式切成800×800的图片。然后再转为Labelme数据（其实不用这么麻烦，主要是不想再写代码了，直接复用以前写的）

将Labelme数据转为txt数据集代码：

import json
    import os
    from glob import glob
    import shutil
    
    # convert labelme json to DOTA txt format
    
    def custombasename(fullname):
        return os.path.basename(os.path.splitext(fullname)[0])
    IN_PATH = '../tile_round1_train_20201231/train_imgs/'
    OUT_PATH = '../labeltxt'
    if not os.path.exists(OUT_PATH):
        os.makedirs(OUT_PATH)
    file_list = glob(IN_PATH + '/*.json')
    for i in range(len(file_list)):
        with open(file_list[i]) as f:
            label_str = f.read()
            label_dict = json.loads(label_str)  # json文件读入dict
            imgepath=file_list[i][:-5]+'.jpg'
            print(imgepath)
            # 输出 txt 文件的路径
            out_file = OUT_PATH + '/' + custombasename(file_list[i]) + '.txt'
            print(out_file)
            shutil.copy(imgepath, OUT_PATH)
            # 写入 poly 四点坐标 和 label
            fout = open(out_file, 'w')
            out_str = ''
            for shape_dict in label_dict['shapes']:
                out_str += shape_dict['label'] + ' '
                points = shape_dict['points']
                for p in points:
                    out_str += (str(p[0]) + ' ' + str(p[1]) + ' ')
                out_str +='\n'
            fout.write(out_str)
            fout.close()
        print('%d/%d' % (i + 1, len(file_list)))

将大图切成小图

import os
    import math
    from glob import glob
    from PIL import ImageFile
    ImageFile.LOAD_TRUNCATED_IMAGES = True
    size = 800
    step = 700
    labelme_path = '../labeltxt/'
    if not os.path.exists('../CutResult'):
        os.makedirs('../CutResult')
    from PIL import Image
    
    Image.MAX_IMAGE_PIXELS = None
    
    def custombasename(fullname):
        return os.path.basename(os.path.splitext(fullname)[0])
    
    
    image_files = glob(labelme_path + "*.jpg")
    for fileImag in image_files:
        img = Image.open(fileImag)
        objectList = []
        with open(fileImag[:-4] + ".txt") as f:
            for line in f.readlines():
                for aa in line.split(' '):
                    if aa != '\n':
                        objectList.append(aa)
        name = custombasename(fileImag)
        print(name)
        print(objectList)
        width, height = img.size
        for i in range(math.ceil(width / step)):
            for j in range(math.ceil(height / step)):
                x1 = i * step
                y1 = j * step
                x2 = size + i * step
                y2 = size + j * step
                if x2 > width:
                    x2 = width
                    x1 = width - size
                if y2 > height:
                    y2 = height
                    y1 = height - size
                listLable = []
                for k in range(int(len(objectList) / 5)):
                    object_label = objectList[k * 5]
                    object_x1 = objectList[k * 5 + 1]
                    object_y1 = objectList[k * 5 + 2]
                    object_x2 = objectList[k * 5 + 3]
                    object_y2 = objectList[k * 5 + 4]
                    if float(object_x1) >= float(x1) and float(object_y1) >= float(y1) and float(object_x2) <= float(
                            x2) and float(
                            object_y2) <= float(y2):
                        listLable.append(object_label)
                        listLable.append(str(float(object_x1) - float(x1)))
                        listLable.append(str(float(object_y1) - float(y1)))
                        listLable.append(str(float(object_x2) - float(x1)))
                        listLable.append(str(float(object_y2) - float(y1)))
                if len(listLable) > 0:
                    cropped = img.crop((x1, y1, x2, y2))  # (left, upper, right, lower)
                    cropped.save("../CutResult/%s-%s_%s.jpg" % (name, x1, y1))
                    test_str = " ".join(listLable)
                    file = open('../CutResult/%s-%s_%s.txt' % (name, x1, y1), 'w')
                    file.write(test_str);
                    file.close()

将txt数据集转为json

import os
    import sys
    import os.path as osp
    import io
    from labelme.logger import logger
    from labelme import PY2
    from labelme import QT4
    import PIL.Image
    from scipy import ndimage
    import base64
    from labelme import utils
    
    module_path = os.path.abspath(os.path.join('..'))
    if module_path not in sys.path:
        sys.path.append(module_path)
    import json
    from PIL import Image
    
    Image.MAX_IMAGE_PIXELS = None
    imagepath = 'CutResult//'
    
    
    def load_image_file(filename):
        try:
            image_pil = PIL.Image.open(filename)
        except IOError:
            logger.error('Failed opening image file: {}'.format(filename))
            return
    
        # apply orientation to image according to exif
        image_pil = utils.apply_exif_orientation(image_pil)
    
        with io.BytesIO() as f:
            ext = osp.splitext(filename)[1].lower()
            if PY2 and QT4:
                format = 'PNG'
            elif ext in ['.jpg', '.jpeg']:
                format = 'JPEG'
            else:
                format = 'PNG'
            image_pil.save(f, format=format)
            f.seek(0)
            return f.read()
    
    
    def dict_json(flags, imageData, shapes, imagePath, fillColor=None, lineColor=None, imageHeight=100, imageWidth=100):
        '''
        :param imageData: str
        :param shapes: list
        :param imagePath: str
        :param fillColor: list
        :param lineColor: list
        :return: dict
        '''
        return {"version": "3.16.4", "flags": flags, "shapes": shapes, 'lineColor': lineColor, "fillColor": fillColor,
                'imagePath': imagePath.split('/')[-1], "imageData": imageData, 'imageHeight': imageHeight,
                'imageWidth': imageWidth}
    
    
    import cv2
    from glob import glob
    
    txtFiles_Path = "CutResult"
    txtFiles = glob(txtFiles_Path + "/*.txt")
    print(txtFiles)
    for file in txtFiles:
        print(file)
        txtFile = file
        imagePH = file.replace(".txt", ".jpg")
        image = cv2.imread(imagePH)
        listResult = []
        with open(txtFile) as f:
            for line in f.readlines():
                for aa in line.split(' '):
                    if aa != '\n':
                        listResult.append(aa)
        data = json.load(open('1.json'))
        shapes = data["shapes"]
        version = data["version"]
        flags = data["flags"]
        lineColor = data["lineColor"]
        fillColor = data['fillColor']
        newshapes = []
        for elem in range(int(len(listResult) / 5)):
            labelName = listResult[elem * 5]
            xmin = listResult[elem * 5 + 1]
            ymin = listResult[elem * 5 + 2]
            xmax = listResult[elem * 5 + 3]
            ymax = listResult[elem * 5 + 4]
            line_color = None
            fill_color = None
            newPoints = [[float(xmin), float(ymin)],
                         [float(xmax), float(ymax)]]
            shape_type = 'rectangle'
            flags = flags
            newshapes.append({"label": labelName, "line_color": line_color, "fill_color": fill_color, "points": newPoints,
                              "shape_type": shape_type, "flags": flags})
        imageData_90 = load_image_file(imagePH)
        imageData_90 = base64.b64encode(imageData_90).decode('utf-8')
        print(image.shape)
        imageHeight = image.shape[0]
        imageWidth = image.shape[1]
        data_90 = dict_json(flags, imageData_90, newshapes, imagePH, fillColor, lineColor, imageHeight, imageWidth)
        json_file = imagePH[:-4] + '.json'
        json.dump(data_90, open(json_file, 'w'))

查看转化后的数据集

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 4][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 5][]

经过处理后，终于看着舒服点了。

[20210130160719969.png]: /images/20221024/9d11973e4f7d44dda4b013a2c120337a.png
[20210130160731318.png]: /images/20221024/54ca789a80024670a6f06f9b5f3f22e9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70]: /images/20221024/1c7937b75db545c5a627dcf7e29e3510.png
[Readme.md]: http://readme.md/
[https_github.com_rafaelpadilla_Object-Detection-Metrics]: https://github.com/rafaelpadilla/Object-Detection-Metrics
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 1]: /images/20221024/16542f9a8dad4e4a9bae0a5ba8f358cf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 2]: /images/20221024/e1d1c678a13840708ffb41beb98ef325.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 3]: /images/20221024/505725e744e144e1b9083e6d9c9101ed.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20210130210744509.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hoaGhoaGhoaGh3d3d3d3d3d3d3_size_16_color_FFFFFF_t_70 5]: /images/20221024/3fbfae43f7ca48a581094115d2e05a5c.png