Quantization for Rapid Deployment of Deep Neural Networks

叁歲伎倆 2023-02-22 10:46 15阅读 0赞

# **Quantization for Rapid Deployment of Deep Neural Networks** #

Jun Haeng Lee, Sangwon Ha, Saerom Choi, Won-Jo Lee, Seungwon Lee

quantization [ˌkwɒntɪ'zeɪʃən]：n. 量子化，分层，量化
    Korea [kə'rɪə]：n. 韩国，朝鲜
    South Korea，Republic of Korea：大韩民国，韩国
    Gyeonggi-do，province surrounding Seoul：京畿道
    Suwon，Suwon-si [suwʌn]：水原市
    Yeongtong-gu：灵通区
    130, Samsung-ro：三星路 130 号
    Samsung Advanced Institute of Technology，SAIT：三星综合技术院
    130, Samsung-ro, Yeongtong-gu, Suwon-si, Gyeonggi-do, 16678, Korea.

## Abstract ##

This paper aims at rapid deployment of the state-of-the-art deep neural networks (DNNs) to energy efficient accelerators without time-consuming fine tuning or the availability of the full datasets. Converting DNNs in full precision to limited precision is essential in taking advantage of the accelerators with reduced memory footprint and computation power. However, such a task is not trivial since it often requires the full training and validation datasets for profiling the network statistics and fine tuning the networks to recover the accuracy lost after quantization. **To address these issues, we propose a simple method recognizing channel-level distribution to reduce the quantization-induced accuracy loss and minimize the required image samples for profiling.** We evaluated our method on eleven networks trained on the ImageNet classification benchmark and a network trained on the Pascal VOC object detection benchmark. **The results prove that the networks can be quantized into 8-bit integer precision without fine tuning.**  
本文旨在将最先进的深度神经网络 (DNN) 快速部署到高能效的加速器上，而无需进行费时的 fine tuning 或提供完整的数据集。将 DNN 以全精度转换为有限精度减少内存占用量和计算能力对于利用加速器至关重要。但是，这样的任务并非易事，因为它通常需要完整的训练和验证数据集来分析网络统计信息并 fine tuning 网络以恢复量化后丢失的精度。为了解决这些问题，我们提出了一种识别 channel-level distribution 的简单方法，以减少量化引起的精度损失，并最大程度地减少剖析所需的图像样本。我们在 ImageNet 分类基准上训练的 11 个网络和 Pascal VOC 目标检测基准上训练的 1 个网络上评估了我们的方法。结果证明，无需进行微调，就可以将网络量化为 8 位整数精度。

footprint ['fʊt.prɪnt]：n. 足迹，脚印，面积，(通信卫星) 覆盖区

## 1 Introduction ##

Deploying state-of-the-art deep neural networks (DNNs) to embedded systems is a challenging task due to the inherent nature of huge number of computations and large memory requirements. **These impediments are partly caused by the considerable amount of the redundancies found in the network parameters intended for ease of training.** Thus, the parameters encompasses abundant opportunities for trimming strategies, namely **pruning and quantizing to low precision \[1, 4, 7\]**. However, running DNN inference on accelerators equipped with fixed-point arithmetic units in an energy efficient manner also requires limiting the precision of the feature maps \[15, 17, 18\].  
由于大量计算和大量内存需求的内在本质，将最新的深度神经网络 (DNN) 部署到嵌入式系统是一项具有挑战性的任务。这些障碍部分是由于在网络参数中存在大量的冗余 (为了便于训练导致的)。因此，这些参数包含大量微调策略的机会，即剪枝和量化到低精度 \[1, 4, 7\]。但是，以节能的方式在配备定点算术单元的加速器上运行 DNN 推理还需要限制特征图的精度 \[15, 17, 18\]。

Previous works \[4, 6, 15, 17\] have exhibited converting pretrained DNNs to 8-bit precision does not induce any accuracy loss. **The feature maps and the network parameters were quantized at the granularity of layers to accommodate for large diversities in the dynamic range across the layers.** Even though they showed good results for a few popular DNNs like AlexNet, VGG-Net, or GoogLeNet, it is not clear whether it would still work for many other recent DNNs with **compact architectures**.  
先前的工作 \[4, 6, 15, 17\] 展示了将预训练的 DNN 转换为 8 位精度不会引起任何精度损失。在 layer 的粒度上对特征图和网络参数进行了量化，以适应跨图层动态范围内的较大差异。即使它们对于 AlexNet, VGG-Net, or GoogLeNet 等一些流行的 DNN 都表现出了不错的效果，但尚不清楚它是否仍适用于其它许多具有紧凑架构的 DNN。

From our experiments, we were able to observe that this was not the case for some of the recent state-of-the-art DNNs. For example, applying 8-bit quantization to the individual layers of the MobileNet series as done in the previous works showed large accuracy degradations. The excessive accuracy loss could be mitigated by fine tuning the quantized networks \[6, 15\]. **However, in order to reach a competitive level of accuracy for each network with fine tuning, full-size training and validation datasets were needed to be incorporated along with painstakingly long periods of optimization.** As many DNN developers only provide the pretrained networks in full precision without the training or the validation datasets from reasons such as **privacy or the outright massiveness of the data size**, such obstacles hinder rapid and easy deployment of DNNs in full precision to embedded accelerators designed for low precision.  
从我们的实验中，我们可以发现，对于某些最新的 DNN 而言并非如此。例如，如先前工作中所述，对 MobileNet 系列的各个 layer 应用 8 位量化会显示出较大的精度下降。可以通过对量化网络进行微调来减轻过度的准确性损失 \[6, 15\]。但是，为了通过微调达到每个网络的竞争性精度水平，需要结合全尺寸的训练和验证数据集以及艰苦的长时间优化。由于许多 DNN 开发人员仅提供全精度的预训练网络，而不提供训练或验证数据集有隐私或数据量的庞大的原因，这些障碍阻碍了将 DNN 快速，轻松地以全精度部署到专为低精度设计的嵌入式加速器中。

impediment [ɪm'pedɪmənt]：n. 障碍，阻碍，妨碍，口吃
    ease [iːz]：v. 缓解，减轻，放松，降低 n. 容易，安逸，自在，舒适
    encompass [ɪn'kʌmpəs]：v. 包含，包围，围绕，围住
    granularity [grænjʊ'lærɪtɪ]：n. 颗粒性
    individual [.ɪndɪ'vɪdʒuəl]：n. 个人，与众不同的人，有个性的人，某种类型的人 adj. 单独的，个别的，一个人的，供一人用的
    obstacle ['ɒbstək(ə)l]：n. 障碍，障碍物，阻碍，绊脚石
    hinder ['hɪndə(r)]：v. 阻碍，妨碍，阻挡 adj. 后面的
    painstakingly ['peɪnzteɪkɪŋlɪ]：adj. 极力的
    outright [.aʊt'raɪt]：adj. 完全的，彻底的，绝对的，公开的 adv. 公开地，直率地，毫无保留地，完全彻底
    massiveness ['mæsɪvnɪs]：n. 普遍，块状，无层理

Instead of converting a full precision pretrained network to lower precision suitable for embedded accelerators, it is also possible to train one from scratch by constraining the weights and the activations \[10, 18, 28, 29\]. **However, achieving state-of-the-art accuracy on large benchmarks such as ImageNet classification, increase in the network size in terms of connections \[18\] or integrating complicated training process \[29\] is required.** Nevertheless, these methods have not been proven for various types of network architectures. Considering the fact that GPUs are the most popular devices for training, it is more practical to convert DNNs into lower precision after utilizing the GPUs’ full precision data path for training.  
代替将全精度的预训练网络转换为适用于嵌入式加速器的较低精度，还可以通过限制权重和激活来从头训练一个模型 \[10, 18, 28, 29\]。但是，在大型基准 (例如 ImageNet 分类) 上达到最高水平的准确率，需要在连接方面增加网络规模 \[18\] 或集成复杂的训练过程 \[29\]。但是，尚未针对各种类型的网络体系结构证明这些方法。考虑到 GPU 是最流行的训练设备，在利用 GPU 的全精度数据方式进行训练后，将 DNN 转换为较低精度更为实际。

In this paper, **we introduce a novel technique in which fine tuning is not necessary for 8-bit linear quantization which quantizes the feature maps and the parameters for individual channels instead of layers to accommodate for the inter-channel variations in the dynamic range.** Our method significantly reduces the accuracy loss caused by quantizing to lower precision without increasing the inference computation cost. The results show that various state-of-the-art DNNs trained on the ImageNet dataset can readily be converted for 8-bit fixed-point accelerators without fine tuning by using a few training samples for profiling.  
在本文中，我们介绍了一种新颖的技术，其中对于 8 位线性量化不需要微调，该技术可以量化特征图和单个通道的参数而不是层，以适应动态范围内通道间的变化。我们的方法在不增加推理计算成本的情况下，极大地降低了量化到低精度带来的精度损失。结果表明，通过使用一些训练样本进行剖析，可以轻松地对 ImageNet 数据集上训练的各种最新 DNN 进行 8 位定点加速器的转换，而无需进行微调。

readily ['redɪli]：adv. 欣然地，乐意地，快捷地，轻而易举地
    accommodate [ə'kɒmədeɪt]：v. 容纳，顺应，提供住宿 (或膳宿、座位等)，提供空间

## 2 Low precision quantization ##

**It is common practice to quantize the activations and the network parameters for each layer to account for the differences in the dynamic range across the layers \[6, 15, 17\].** Previous implementations such as Ristretto\[6\], a fixed-point quantization simulator based on Caffe, reserves three placeholders for the **fractional length**s (defined as the number of required bits for the fractional part of a fixed-point number) per layer, one each for the input and output feature maps (**IFM** and **OFM** respectively) and for the layer parameters (weights and biases). At every layer, IFM, OFM, and the weights are polled separately for max values and the fractional lengths are calculated accordingly. During run-time, the MSBs and LSBs of the parameters and the activations of each layer are clipped to be containable within the given bit-width and the fractional lengths in order to emulate a generic fixed-point H/W implementation. We will use the term **layer-wise quantization** hereafter to describe this scheme in contrast with **channel-wise quantization** proposed in this paper.  
通常的做法是量化每一层的激活和网络参数，以解决各层之间动态范围的差异 \[6, 15, 17\]。以前的实现，例如 Ristretto\[6\] (一种基于 Caffe 的定点量化模拟器)，为每层 fractional length (定义为定点数小数部分所需的位数) 保留了三个占位符，每个分别用于输入和输出特征图 (分别为 IFM 和 OFM) 和层参数 (权重和偏差)。在每一层，分别对 IFM，OFM 和权重进行轮询以获取最大值，并相应地计算 fractional length。在运行时，参数的 MSB 和 LSB 以及每一层的激活都被裁剪为可包含在给定的位宽和小数长度内，以模拟通用的定点硬件实现。与本文提出的逐通道量化相比，下文中将使用术语逐层量化来描述此方案。

**A major down-side of the layer-wise quantization is that the inter-channel variations of the feature maps and the weights are not fully accounted for. Since the fractional length is usually selected to cover the maximum value in a layer, the layer-wise quantization tends to cause excessive information loss in channels with a smaller dynamic range.** Therefore, accuracy may be degraded significantly and sometimes never recovered even after exhaustive retraining.  
分层量化的主要缺点是特征图和权重的通道间变化并未得到充分考虑。由于通常选择小数长度以覆盖一层中的最大值，因此逐层量化往往会在动态范围较小的通道中导致过多的信息丢失。因此，准确性可能会大大降低，有时甚至在详尽的再训练之后也永远无法恢复。

poll [pəʊl]：n. 民意测验，民意调查，选举投票，计票 v. 获得 (票数)，对 … 进行民意调查 adj. 当事人一方作成的
    exhaustive [ɪɡ'zɔːstɪv]：adj. 详尽的，彻底的，全面的
    down-side：n. 底侧，下降趋势，阴暗面，缺点

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NoZW5neXExMTY_size_16_color_FFFFFF_t_70_pic_center]

## 3 Benchmark results ##

### 3.1 ImageNet classification task ###

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NoZW5neXExMTY_size_16_color_FFFFFF_t_70_pic_center 1]  
Figure 4: Effect of profiling dataset size on accuracy with quantization for Inception-v3. **MAX method requires large number of samples for profiling to reach a stable accuracy. Laplace method stabilizes quickly with a few samples.**  
Figure 4: 分析数据集大小对 Inception-v3 量化准确性的影响。MAX 方法需要大量样本才能进行分析，以达到稳定的精度。拉普拉斯 (Laplace) 方法只需少量样品即可快速稳定下来。

### 3.2 Object detection ###

We performed network quantization on YOLO-v2\[20\], a state-of-the-art object detection network. The network was trained and tested on the Pascal VOC dataset \[5\].  
我们在最先进的物体检测网络 YOLO-v2 \[20\] 上进行了网络量化。该网络在 Pascal VOC 数据集上进行了训练和测试 \[5\]。

Table 2 shows the loss in mean AP after quantization using our method in comparison with the layer-wise quantization. The layer-wise quantization caused 2.5% point drop in mean AP after quantization. **However, our method did not suffer from such a problem by selecting the fractional lengths adapted to the individual channels.**  
表 2 显示了与 layer-wise 量化相比，使用我们的方法进行量化后 mean AP 的损失。layer-wise 量化导致量化后 mean AP 下降 2.5％。但是，通过选择适合各个通道的 fractional length，我们的方法不会遇到这样的问题。

Table 2: Loss in mean AP after 8-bit quantization in YOLO-v2. No retraining performed. `Reference (Float32)` lists baseline accuracy while all other figures are accuracy losses. Loss above 1.0% point is in **bold face**.  
Table 2: YOLO-v2 中 8 位量化后 mean AP 的损失。不执行再训练。`Reference (Float32)` 列出了基线精度，而所有其他数字均为精度损失。精度损失超过 1.0％ 点是**粗体**。  
![在这里插入图片描述][20201116232858325.png_pic_center]

## 5 Conclusion ##

In this paper, we proposed a set of methods for rapid deployment of DNNs trained in full precision to fixed point accelerators with limited precision computation units. **The channel-wise quantization recognizes the inter-channel diversities in the dynamic range of the feature maps. HW cost for implementation is minimized by adjusting the fractional lengths of the kernel parameters.** We evaluated our method on eleven state-of-the-art DNNs trained on the ImageNet dataset and an object detection network trained on Pascal VOC dataset. In comparison to the previous method (i.e the layer-wise quantization), the channel-wise quantization can reduce the accuracy loss caused by quantization substantially.  
在本文中，我们提出了一套方法，用于将以全精度训练的 DNN 快速部署到包含有限精度计算单元的定点加速器。逐通道量化识别特征图的动态范围内的通道间多样性。通过调整内核参数的小数长度，可将实现的硬件成本降至最低。我们在 ImageNet 数据集上训练的 11 个最新 DNN 和 Pascal VOC 数据集上训练的目标检测网络上评估了我们的方法。与先前的方法 (即，逐层量化) 相比，逐信道量化可大大地减少由量化引起的精度损失。

**We also showed that quantization requires just a few image samples if we utilize the n-th moments of the activations instead of the maximum value.** In this way, deployment is possible even when just a few training samples are available for the trained network model. **We further improved our method by considering the variations in distribution across the channels. A simple classifier was used for selecting the best-fit PDF for each channel from the statistical features.** We were able to accomplish negligible accuracy loss (less than 1% point in eleven networks out of twelve) after quantization without fine tuning.  
我们还发现，如果我们利用激活的第 n 个时刻而不是最大值，则量化仅需要少量图像样本。这样，即使只有几个训练样本可用于训练的网络模型，也可以进行部署。通过考虑跨通道分布的变化，我们进一步改进了我们的方法。一个简单的分类器用于从统计特征中为每个通道选择最适合的 PDF。在不进行微调的情况下进行量化后，我们能够实现可忽略不计的精度损失 (在十二个网络中，十一个网络的精度下降不足 1％)。

substantially [səb'stænʃ(ə)li]：adv. 基本上，大体上，非常，大大地

## References ##

[https://arxiv.org/abs/1810.05488][https_arxiv.org_abs_1810.05488]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NoZW5neXExMTY_size_16_color_FFFFFF_t_70_pic_center]: https://img-blog.csdnimg.cn/20201118233440767.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NoZW5neXExMTY=,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NoZW5neXExMTY_size_16_color_FFFFFF_t_70_pic_center 1]: https://img-blog.csdnimg.cn/20201117233853546.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NoZW5neXExMTY=,size_16,color_FFFFFF,t_70#pic_center
[20201116232858325.png_pic_center]: https://img-blog.csdnimg.cn/20201116232858325.png#pic_center
[https_arxiv.org_abs_1810.05488]: https://arxiv.org/abs/1810.05488