卷积核_华为诺亚方舟实验室开源即插即用的多用卷积核(NeurIPS 2018)

女爷i 2023-01-07 03:47 154阅读 0赞

华为诺亚方舟实验室开源了一种即插即用的卷积核，该项工作发表在NeurIPS 2018上，论文题目为《Learning Versatile Filters for Efficient Convolutional Neural Networks》，合作单位还有北京大学和悉尼大学。 这种卷积核可以被直接用来替换深度神经网络中的传统卷积核，在降低存储和计算量后，精度甚至还可以超越原来的模型，并且可以用在多种计算机视觉任务上，例如目标检测和图像超分辨率。   论文地址： http://papers.nips.cc/paper/7433-learning-versatile-filters-for-efficient-convolutional-neural-networks.pdf 开源地址： https://github.com/huawei-noah/Versatile-Filters   **研究背景** 得益于GPU的快速发展，和越来越多可用的训练数据，深度卷积神经网络已经成为计算机视觉领域中不可或缺的一项技术。 深度卷积神经网络模型在移动设备(例如，手机、摄像头、智能家居)上，同样具有非常多的应用需求。 但是，由于它们并不具备有良好计算性能和较大内存的GPU计算卡，在这些设备上使用深度神经网络模型存在许多困难，例如硬件限制和传输限制等。 **多用卷积核** 本文提出了一种用来构建高效卷积神经网络的多用卷积核。 考虑到高效深度学习技术在低成本硬件上运行的需求，一系列学习紧致神经网络的相关工作在近年来被相继地提出。 大多数现有工作旨在利用不同的方式来缩小卷积核，例如，开发小的、稀疏的或二值化的卷积核。 相反，我们从增加的角度去处理这些卷积核。 一些列的辅卷积核将会从主卷积核中导出。 这些辅卷积核的参数都是继承于主卷积核，所以并不占用额外的存储。 但是，在展开进行计算的过程中，它们可以通过整合从不同感受野提取到的信息来显著地提升卷积核的性能。 除了空间多用卷积核，我们进一步地从通道的角度来开发多用卷积核。 本文所提出的新方法可以被直接地用来替换现有卷积神经网络中的卷积核。 在标准数据集和神经网络模型上的实验结果表明，利用本文所提出的多用卷积核构建的卷积神经网络能够实现与原始卷积核相当的精度，但是需要更少的内存和计算开销。 **空间多用卷积核：** 对于输入数 ![5121d4d3f84684c36f7d026ae9c9b946.png][] ![7fd5d8ebba6234829b26db71ef37027b.png][] 和一个常规的卷积核(主卷积核 ![dca1aeed95a70162a842ba3c209093d2.png][] ，记 ![f568258847d9b7f3a5b736f82bf675d9.png][] 是为它的尺寸。 空间多用卷积核可以将其在空间上进行采样，生成一系列的空间辅卷 ![88525bab7debc27ea9be32a7899d790c.png][] ![4cf9e9ca25ecc2c361a6a6ebc1c04ebb.png][] ，其中 ![ec9a3fe5a42872317f0d3270e4246156.png][] ![bc6aeaf83d14397ac6922422ab305452.png][] 。 其中每一个空间辅卷积核都是通过一个模板在主卷积核上进行采样得到的，第 ![9616549905906b3dc9f35505516bdd03.png][] 个模板 ![191e63835c9d4e53f66c21b27ed24738.png][] 的计算方式为： ![4fc61ed15dc0a5f48ac0db5beeb8c3e7.png][]

然后，![4123f78a2fc59e6d5d72230cbc25b814.png][]就可以通过![f70edc7084549c51f8a287b46d93821e.png][]![8bc447674dc062c276336de3b8e09d18.png][]来计算，![0515e916baa3a83ef12cce2469479894.png][]矩阵是元素之间的乘积。更具体地，![6b6bb1b134eac5953b3e670735e5633e.png][]是主卷积核本身，![d708c3d6075b995834cf83cb31ae603f.png][]去掉了主卷积核中最外圈的参数，![bd71730182852ce520477462ddb5c8c5.png][]是主卷积核中最内圈的参数(例如，![5b4a57abcd9cef4282868ddf04129c79.png][]是一个![aff71652528bc5ef91f54a750bc9546c.png][]卷积核，当![cf63d9716c09f966fad6b8eb187a6ce6.png][]是一个奇数的时候)。图1展示了一个![6bf7d771c630dad50845ecd24681b2f9.png][]的空间多用卷积核的计算方式。

![78f40ea85622580314acf21244c6d148.png][] 图1 空间多用卷积核示意图 **通道多用卷积核：** 如图2所示，原始的卷积核的通道个数是等于输入数据的通道个数的。 利用本文所用的通道多用卷积核，我们可以在主卷积核 ![dca1aeed95a70162a842ba3c209093d2.png][]的通道上进行采样，生成一系列的通道多用卷积核(图2中的 ![da8ad8bb954e686bc2fd9a5642dc0d14.png][] ![3a5673d4dda86f30bf7e5cee28d56780.png][] )。 然后与输入数据 ![495a2415d9178ad2088f8d4c357217fc.png][] 相结合生成特征图。 通过这种方式，我们可以利用同样的参数获得更多的特征图，进一步地降低深度神经网络模型的内存和计算开销。 ![82d4bd2b0a9162aa10bbd44211a9a071.png][] 图2 通道多用卷积核示意图 **实验结果** **图像分类：** 首先，我们将本文所提出的算法在ImageNet数据集上进行图像分类的测试实验。 表1展示了几个标准模型(AlexNet，ResNet-50和ResNeXt-50)利用本文所提出的多用卷积核之后的实验结果。 我们并不改变这些神经网络模型的结构(层数、每层的卷积核尺寸、参数等)，仅仅根据本文所提出的多用卷积核的计算方式来对每一层的卷积核个数进行减少。 表1 本文所提出的多用卷积核在ImageNet 2012数据集上的结果统计

![3c239b118a2b73a0b3cfd82e5da67e1b.png][]

**图像超分辨率：** 除了利用本文所提出的多用卷积核来进行图像分类实验之外，我们将其应用在图像处理的问题中。 以图像超分辨率为例，我们替换了VDSR网络中原始卷积核为本文中所述的多用卷积核，具体的实验结果如表2所示。 表2本文所提出的卷积核与在图像超分辨率模型上的结果统计 ![ccca387328dbdc173874589af9a005a2.png][] 图3展示了利用本文所提出的多用卷积核之后的图像超分辨率实验结果，我们使用轻型高效的深度卷积神经网络得到了具有更高清晰度的图像。 ![acb6612f7623cb40551aee49ae8e9142.png][] 图3 本文所提出的卷积核与轻型深度卷积神经网络模型的比较结果 Huawei Noah’Ark Lab presents a plug-and-play versatile convolution filter for building efficient CNNs.   **Abstract** Huawei Noah’Ark Lab presents a plug-and-play versatile convolution filter for building efficient CNNs, cooperating with Peking University and the University of Sydney. This work was accepted by NeurIPS 2018, namely, Learning Versatile Filters for Efficient Convolutional Neural Networks. The new filter can be directly utilized for replacing conventional filters. After reducing the storage and computational complexity, the performance of the resulting network using versatile filters can even surpass that of the original model. The new method can also be applied in a wide variety of computer applications such as object detection and image super-resolution.   Paper URL:  http://papers.nips.cc/paper/7433-learning-versatile-filters-for-efficient-convolutional-neural-networks.pdf Github URL: https://github.com/huawei-noah/Versatile-Filters   **Background** Thanks to the large amount of accessible training data and computational power of GPUs, considerable computer vision applications have received remarkable progress with the help of convolutional neural networks (CNNs) in last decade. Most of deep networks are widely used in the scenario of abundant computation and storage resources, but they cannot easily adapt to mobile platforms, such as smartphones and cameras. Taking ResNet-50 with 54 convolutional layers as an example, about 97 MB memory is required to store all filters and over 4 billions times of floating number multiplications have to be operated for an image. **Versatile Filters** This paper introduces versatile filters to construct efficient convolutional neural network. Considering the demands of efficient deep learning techniques running on cost-effective hardware, a number of methods have been developed to learn compact neural networks. Most of these works aim to slim down filters in different ways, e.g. investigating small,sparse or binarized filters. In contrast, we treat filters from an additive perspective. A series of secondary filters can be derived from a primary filter. These secondary filters all inherit in the primary filter without occupying more storage, but once been unfolded in computation they could significantly enhance the capability of the filter by integrating information extracted from different receptive fields. Besides spatial versatile filters,we additionally investigate versatile filters from the channel perspective. The new techniques are general to upgrade filters in existing CNNs. Experimental results on benchmark datasets and neural networks demonstrate that CNNs constructed with our versatile filters are able to achieve comparable accuracy as that oforiginal filters, but require less memory and FLOPs. **Spatial Versatile Filters:** Consider the input data  ![5121d4d3f84684c36f7d026ae9c9b946.png][]   ![7fd5d8ebba6234829b26db71ef37027b.png][]and a primary convolution  ![dca1aeed95a70162a842ba3c209093d2.png][] ，filter, where ![f568258847d9b7f3a5b736f82bf675d9.png][]  is the size of the convolution filter. We propose to derive a series of secondary filters  ![88525bab7debc27ea9be32a7899d790c.png][] ![4cf9e9ca25ecc2c361a6a6ebc1c04ebb.png][], where ![ec9a3fe5a42872317f0d3270e4246156.png][] ![bc6aeaf83d14397ac6922422ab305452.png][] . To maximally explore the potential of primary filter each secondary filters is directly inherited from with a mask  ![191e63835c9d4e53f66c21b27ed24738.png][] : ![4fc61ed15dc0a5f48ac0db5beeb8c3e7.png][]

and ![4123f78a2fc59e6d5d72230cbc25b814.png][] is calculated as ![f70edc7084549c51f8a287b46d93821e.png][]![8bc447674dc062c276336de3b8e09d18.png][] , where ![0515e916baa3a83ef12cce2469479894.png][] is the element-wise multiplication. More specifically,  ![6b6bb1b134eac5953b3e670735e5633e.png][] is the filter itself,  ![d708c3d6075b995834cf83cb31ae603f.png][] discards the outermost circle of parameters in ![dca1aeed95a70162a842ba3c209093d2.png][] , and ![bd71730182852ce520477462ddb5c8c5.png][] is the innermost circle of parameters in ![dca1aeed95a70162a842ba3c209093d2.png][] , (i. e. ![bd71730182852ce520477462ddb5c8c5.png][] is a ![aff71652528bc5ef91f54a750bc9546c.png][] filter given an odd ![cf63d9716c09f966fad6b8eb187a6ce6.png][] ). Example secondary filters for a ![6bf7d771c630dad50845ecd24681b2f9.png][]  filter can be seen in Figure 1.

![bf1dba34c837eba69125a52301a1feac.png][] Figure 1 An illustration of the proposed spatial versatile convolution filters. **Channel Versatile Filters:**  As illustrated in Figure 2, the channel number of the original filter is equal to that of the input data. By exploiting the proposed channel versatile filter, we can also generate a series of channel versatile secondary filters and corresponding feature maps.In addition, the proposes channel versatile filters can also significantly reduce the memory usage and computational complexity of CNNs. ![d2d92b1a4eb32122fcd9f755561c80b4.png][] Figure 2 The diagram of the proposed channel versatile convolution filters. **Experimental Results** **Image Classification:** We next employed the proposed method on an extremely large image dataset, namely ImageNet ILSVRC 2012 dataset, which contains over 1.2M training images and 50k validation images Three baseline architectures, AlexNet,ResNet-50 and ResNeXt-50, were selected for conducting the following experiments. Note that, all training settings such as weight decay and learning rate used the default setting to ensure fair comparisons. Table 1 Statistics for versatile filters on the ImageNet 2012 dataset.

![3c239b118a2b73a0b3cfd82e5da67e1b.png][]

**Image Super-Resolution:**  In addition, to investigate the generalization ability of the proposed versatile convolution filter, we further employed it on the single image super-resolution experiment as shown in Table 2. Table 2 Statistics for versatile filters on VDSR. ![ccca387328dbdc173874589af9a005a2.png][] Figure 3 illustrates some visualization results of the baseline model and the proposed method. Results generated by networks using the proposed versatile convolution filters are better than those of the baseline model since we can provide the same amount feature maps with multi-scale information, which is able to make the estimation smooth in every scale. ![acb6612f7623cb40551aee49ae8e9142.png][] Figure 3 Image super-resolution results of the baseline VDSR model and the proposed versatile Filters.

[5121d4d3f84684c36f7d026ae9c9b946.png]: https://img-blog.csdnimg.cn/img_convert/5121d4d3f84684c36f7d026ae9c9b946.png
[7fd5d8ebba6234829b26db71ef37027b.png]: https://img-blog.csdnimg.cn/img_convert/7fd5d8ebba6234829b26db71ef37027b.png
[dca1aeed95a70162a842ba3c209093d2.png]: https://img-blog.csdnimg.cn/img_convert/dca1aeed95a70162a842ba3c209093d2.png
[f568258847d9b7f3a5b736f82bf675d9.png]: https://img-blog.csdnimg.cn/img_convert/f568258847d9b7f3a5b736f82bf675d9.png
[88525bab7debc27ea9be32a7899d790c.png]: https://img-blog.csdnimg.cn/img_convert/88525bab7debc27ea9be32a7899d790c.png
[4cf9e9ca25ecc2c361a6a6ebc1c04ebb.png]: https://img-blog.csdnimg.cn/img_convert/4cf9e9ca25ecc2c361a6a6ebc1c04ebb.png
[ec9a3fe5a42872317f0d3270e4246156.png]: https://img-blog.csdnimg.cn/img_convert/ec9a3fe5a42872317f0d3270e4246156.png
[bc6aeaf83d14397ac6922422ab305452.png]: https://img-blog.csdnimg.cn/img_convert/bc6aeaf83d14397ac6922422ab305452.png
[9616549905906b3dc9f35505516bdd03.png]: https://img-blog.csdnimg.cn/img_convert/9616549905906b3dc9f35505516bdd03.png
[191e63835c9d4e53f66c21b27ed24738.png]: https://img-blog.csdnimg.cn/img_convert/191e63835c9d4e53f66c21b27ed24738.png
[4fc61ed15dc0a5f48ac0db5beeb8c3e7.png]: /images/20221119/7747d8d55c324b968fbc44272944602d.png
[4123f78a2fc59e6d5d72230cbc25b814.png]: https://img-blog.csdnimg.cn/img_convert/4123f78a2fc59e6d5d72230cbc25b814.png
[f70edc7084549c51f8a287b46d93821e.png]: https://img-blog.csdnimg.cn/img_convert/f70edc7084549c51f8a287b46d93821e.png
[8bc447674dc062c276336de3b8e09d18.png]: https://img-blog.csdnimg.cn/img_convert/8bc447674dc062c276336de3b8e09d18.png
[0515e916baa3a83ef12cce2469479894.png]: https://img-blog.csdnimg.cn/img_convert/0515e916baa3a83ef12cce2469479894.png
[6b6bb1b134eac5953b3e670735e5633e.png]: https://img-blog.csdnimg.cn/img_convert/6b6bb1b134eac5953b3e670735e5633e.png
[d708c3d6075b995834cf83cb31ae603f.png]: https://img-blog.csdnimg.cn/img_convert/d708c3d6075b995834cf83cb31ae603f.png
[bd71730182852ce520477462ddb5c8c5.png]: https://img-blog.csdnimg.cn/img_convert/bd71730182852ce520477462ddb5c8c5.png
[5b4a57abcd9cef4282868ddf04129c79.png]: https://img-blog.csdnimg.cn/img_convert/5b4a57abcd9cef4282868ddf04129c79.png
[aff71652528bc5ef91f54a750bc9546c.png]: https://img-blog.csdnimg.cn/img_convert/aff71652528bc5ef91f54a750bc9546c.png
[cf63d9716c09f966fad6b8eb187a6ce6.png]: https://img-blog.csdnimg.cn/img_convert/cf63d9716c09f966fad6b8eb187a6ce6.png
[6bf7d771c630dad50845ecd24681b2f9.png]: https://img-blog.csdnimg.cn/img_convert/6bf7d771c630dad50845ecd24681b2f9.png
[78f40ea85622580314acf21244c6d148.png]: https://img-blog.csdnimg.cn/img_convert/78f40ea85622580314acf21244c6d148.png
[da8ad8bb954e686bc2fd9a5642dc0d14.png]: https://img-blog.csdnimg.cn/img_convert/da8ad8bb954e686bc2fd9a5642dc0d14.png
[3a5673d4dda86f30bf7e5cee28d56780.png]: https://img-blog.csdnimg.cn/img_convert/3a5673d4dda86f30bf7e5cee28d56780.png
[495a2415d9178ad2088f8d4c357217fc.png]: https://img-blog.csdnimg.cn/img_convert/495a2415d9178ad2088f8d4c357217fc.png
[82d4bd2b0a9162aa10bbd44211a9a071.png]: https://img-blog.csdnimg.cn/img_convert/82d4bd2b0a9162aa10bbd44211a9a071.png
[3c239b118a2b73a0b3cfd82e5da67e1b.png]: https://img-blog.csdnimg.cn/img_convert/3c239b118a2b73a0b3cfd82e5da67e1b.png
[ccca387328dbdc173874589af9a005a2.png]: https://img-blog.csdnimg.cn/img_convert/ccca387328dbdc173874589af9a005a2.png
[acb6612f7623cb40551aee49ae8e9142.png]: https://img-blog.csdnimg.cn/img_convert/acb6612f7623cb40551aee49ae8e9142.png
[bf1dba34c837eba69125a52301a1feac.png]: https://img-blog.csdnimg.cn/img_convert/bf1dba34c837eba69125a52301a1feac.png
[d2d92b1a4eb32122fcd9f755561c80b4.png]: https://img-blog.csdnimg.cn/img_convert/d2d92b1a4eb32122fcd9f755561c80b4.png