复旦与华为出品SOFT：一种去除softmax的线性复杂度Transformer！

不念不忘少年蓝@ 2022-08-28 01:51 158阅读 0赞

点上方**计算机视觉联盟**获取更多干货

仅作学术分享，不代表本公众号立场，侵权联系删除

转载于：Sophia知乎

https://zhuanlan.zhihu.com/p/425944505

**985人工智能博士笔记推荐**

**[周志华《机器学习》手推笔记正式开源！附pdf下载链接，Github2500星！][pdf_Github2500]**

今天推荐的文章是复旦大学与华为诺亚方舟实验室共同关于Transformer自注意力机制复杂度的思考和改善，提出了一种全新的softmax-free 的Transformer。SOFT显著改善了现有ViT方案的计算效率。在ImageNet分类任务上取得了比已有CNN、Transformer更佳的精度-复杂度均衡。

![b499cbf0b8c730d067f385713971560f.png][]

论文地址：https://arxiv.org/pdf/2110.11945.pdf

代码地址：https://github.com/fudan-zvg/SOFT

项目地址：https://fudan-zvg.github.io/SOFT/

## 摘要： ##

![5ef192897471c957446bf73f602c9708.png][]

## 参数对比 ##

![3646dcbf8405f87e203f9c41cd888dd2.png][]

## SOFT核心 ##

![e49efad030758f4d1a393636c52ea156.png][] ![94e27b15505dac1243ac5b1565acbf69.png][]

## 效果对比： ##

![30b7d18abfe132364262af898c5cbe90.png][] ![f3b81d5996bc378d7a02206bc717a575.png][] ![3ebbdc766232fea3e0f4601ba36d550a.png][]

兼顾精度和算力的平衡，值得学习！

**往期论文推荐：**

[更小、更轻、更通用视觉Transformer！苹果提出MobileViT模型！][Transformer_MobileViT]

[高效！Anchor DETR：旷视提出一种基于Transformer的目标检测神器！][Anchor DETR_Transformer]

[小目标神器！TPH-YOLOv5：将Transformer预测加载Yolov5！][TPH-YOLOv5_Transformer_Yolov5]

[全新思路！阿里达摩院将Transformer引入在线行为检测！ICCV2021][Transformer_ICCV2021]

[高效新思路！实例分割算法！腾讯、华科出品！入选ICCV 2021!][ICCV 2021]

[超强总结！456篇参考文献！《计算机视觉对抗攻击与防御》综述！][TPH-YOLOv5_Transformer_Yolov5]

[最新变体Transformer！∞-former！DeepMind 出品！][Transformer_-former_DeepMind]

[最新综述！深度神经网络视觉识别！329篇文献！西安交大、清华等出品][329]

[超强YOLOP！CVPR 2022领域主席团队开源！全景感知多任务SOTA！][TPH-YOLOv5_Transformer_Yolov5]

[Fastformer：史上最强最快Transformer！清华、MSRA出品！][Fastformer_Transformer_MSRA]

[脑洞大开！油画渲染的新算法 Paint Transformer！ICCV2021 Oral！][Paint Transformer_ICCV2021 Oral]

[首次！阿里达摩院将Pure Transformer 应用于目标重识别ReID！][Pure Transformer _ReID]

[ResNet超强变体CoTNet！一种新的Transformer计算机视觉模块！][ResNet_CoTNet_Transformer]

[新模型！Conformer！Transformer与CNN的超强融合！][Conformer_Transformer_CNN]

[ImageNet又被Long-Short Transformer 霸榜！][ImageNet_Long-Short Transformer]

[新想法！华科、清华、康奈尔联合提出“圆形卷积”！聚合方形和圆形优势][ImageNet_Long-Short Transformer]

[表现SOTA！DetCo算法：目标检测无监督对比学习][SOTA_DetCo]

[新模型SkipNet在ImageNet分类任务大放光彩！优化损失函数！][TPH-YOLOv5_Transformer_Yolov5]

[高效神器！小目标检测迎来曙光！SSPNet金字塔网络！川大出品][SSPNet]

[超强！MDETR：基于Transformer的端到端目标检测神器！开源！][TPH-YOLOv5_Transformer_Yolov5]

[优秀！一种巧妙组合算法YOLOX！旷视重磅开源！][YOLOX]

[新突破！Swin-UNet：基于纯 Transformer 结构的医学图像分割网络][Swin-UNet_ Transformer]

**-------------------**

***END***

***--------------------***

我是[王博Kings][Kings]，985AI博士，华为云专家、CSDN博客专家（人工智能领域优质作者）。单个AI开源项目现在已经获得了2100+标星。现在在做AI相关内容，欢迎一起交流学习、生活各方面的问题，一起加油进步！

我们微信交流群涵盖以下方向（但并不局限于以下内容）：人工智能，计算机视觉，自然语言处理，目标检测，语义分割，自动驾驶，GAN，强化学习，SLAM，人脸检测，最新算法，最新论文，OpenCV，TensorFlow，PyTorch，开源框架，学习方法...

这是我的私人微信，位置有限，一起进步！

**![a61b201d3a1c0e33c8257b0a8d2684a7.png][]**

**王博的公众号，欢迎关注，干货多多**

**手推笔记：**

**增长见识：**

**其他学习笔记：**

![f1234e83113afc6817f9446aaa7d5f08.gif][]

**点分享**

![b632411cd5b9cca4755328ec254fed17.gif][]

**点收藏**

![0baec84139ced50ad9edfae164c9f3db.gif][]

**点点赞**

![79a64c8f0df77c75eb4d549b930925c8.gif][]

**点在看**

[pdf_Github2500]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247486763&idx=1&sn=da878ccafba79932be8eeca955697477&chksm=fcb71194cbc0988259f32b0b7a77fe7830c477c9c9a4fcb00f2e366c97d776574c2bc949e993&scene=21#wechat_redirect
[b499cbf0b8c730d067f385713971560f.png]: /images/20220828/e85ec423c7eb4d02a7d1486c361c676d.png
[5ef192897471c957446bf73f602c9708.png]: /images/20220828/f3fabddf0ff7464b8fafaecba9b157a1.png
[3646dcbf8405f87e203f9c41cd888dd2.png]: /images/20220828/2239491a8ed24084a912bbe5a01befbd.png
[e49efad030758f4d1a393636c52ea156.png]: https://img-blog.csdnimg.cn/img_convert/e49efad030758f4d1a393636c52ea156.png
[94e27b15505dac1243ac5b1565acbf69.png]: /images/20220828/30fd19b83bbb4944a10cdee309efad8e.png
[30b7d18abfe132364262af898c5cbe90.png]: https://img-blog.csdnimg.cn/img_convert/30b7d18abfe132364262af898c5cbe90.png
[f3b81d5996bc378d7a02206bc717a575.png]: https://img-blog.csdnimg.cn/img_convert/f3b81d5996bc378d7a02206bc717a575.png
[3ebbdc766232fea3e0f4601ba36d550a.png]: /images/20220828/c0678a1d7afb415e88ecaeb7bfe428f2.png
[Transformer_MobileViT]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247522707&idx=2&sn=e2c79b6e113bfd477d0dc8feb1314ed4&chksm=fcb4852ccbc30c3ab8824ef21ac36040760aa032cfc57a1f7322214ebc5993d975f9ede082be&scene=21#wechat_redirect
[Anchor DETR_Transformer]: https://blog.csdn.net/Sophia_11/article/details/120465460
[TPH-YOLOv5_Transformer_Yolov5]: https://blog.csdn.net/Sophia_11/article/details/120446147
[Transformer_ICCV2021]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247520701&idx=2&sn=ce909619d50417c6353b740550126ec3&chksm=fcb48d02cbc30414f71967d9d83496db71344e0a393b80806186d732092a0c3ebfaa3d207b40&scene=21#wechat_redirect
[ICCV 2021]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247520601&idx=2&sn=f06f7deb121d774b8683ab5043bbdd53&chksm=fcb48de6cbc304f0c5a699798315ccf14ff3898308df8bb637852c8c7ab2461fb86cd27314ce&scene=21#wechat_redirect
[Transformer_-former_DeepMind]: https://blog.csdn.net/Sophia_11/article/details/120944445
[329]: https://blog.csdn.net/Sophia_11/article/details/120212914
[Fastformer_Transformer_MSRA]: https://blog.csdn.net/weixin_47196664/article/details/120031052
[Paint Transformer_ICCV2021 Oral]: https://blog.csdn.net/Sophia_11/article/details/120386973
[Pure Transformer _ReID]: https://blog.csdn.net/gzq0723/article/details/119813919
[ResNet_CoTNet_Transformer]: https://blog.csdn.net/amusi1994/article/details/119363684
[Conformer_Transformer_CNN]: https://blog.csdn.net/Sophia_11/article/details/119860627
[ImageNet_Long-Short Transformer]: https://blog.csdn.net/Sophia_11/article/details/120375839
[SOTA_DetCo]: https://blog.csdn.net/qq_29462849/article/details/119123163
[SSPNet]: https://blog.csdn.net/Sophia_11/article/details/120124261
[YOLOX]: https://blog.csdn.net/Sophia_11/article/details/119012008
[Swin-UNet_ Transformer]: https://blog.csdn.net/amusi1994/article/details/116957208
[Kings]: https://blog.csdn.net/Sophia_11/article/details/114274846
[a61b201d3a1c0e33c8257b0a8d2684a7.png]: /images/20220828/4767b80a4e424339a9b7d4d79f64331e.png
[Link 1]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247486235&idx=1&sn=60774f6e71af3c1df5fcdaf37407c1ce&chksm=fcb717a4cbc09eb2915aa8504be858f681e5cc4ea95cc091335f021b30b566f2f16e03ae0739&scene=21#wechat_redirect
[Link 2]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247486809&idx=2&sn=1db1d3e29268334b78faf1b981030974&chksm=fcb711e6cbc098f0db247fdf9739a4e82aaada8231d5b3c6c79b1c19e79d81e3216a7e7476d3&scene=21#wechat_redirect
[Link 3]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247486859&idx=1&sn=9ee1c93b8ca69d71ea1db16e022f2691&chksm=fcb71134cbc09822e382eaeb834f9ee66f78ea78636494cd22bcb0493b4e2ae7ef57fe99fe58&scene=21#wechat_redirect
[Link 4]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247486891&idx=1&sn=48f35f7bcaa7dd559288231c3ac07e30&chksm=fcb71114cbc09802e6d99fe3cd5a41d6da03b1e9eb5442b01b84d9c9edcd49cad0bb50a1309c&scene=21#wechat_redirect
[Link 5]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247487004&idx=1&sn=b5f13ba410cee5730d1e3784b7fc4c9b&chksm=fcb712a3cbc09bb5aa3e4f16254cd52a72a3e5393370bc5c1ca78682161c74ad81443e01135d&scene=21#wechat_redirect
[Link 6]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247487301&idx=1&sn=a5220ac4604bfdb15ee2eb52357a19c1&chksm=fcb713facbc09aecde8ec575fe96d7ef2b406b4bb377c63e197cdeaf04cba14e76b8f75c30e4&scene=21#wechat_redirect
[Link 7]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247487435&idx=2&sn=d3d9a9ecd93f6defe865da36aede1777&chksm=fcb71374cbc09a62446ef8f03e22dcd2b1092169ee0a472a7d3f0e1d6209300943b03d389e76&scene=21#wechat_redirect
[Link 8]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247488226&idx=1&sn=6694a44c5bc8d8207e8348cfbb141297&chksm=fcb70e5dcbc0874b355226a7ef07f0d2e315b13df95c3df2812a70b4eb6ab2d68184e1c3cc86&scene=21#wechat_redirect
[Link 9]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247488404&idx=2&sn=36b6c46669b1c5f1c0922f3c476b2d85&chksm=fcb70f2bcbc0863d86fdf807a48cd919d75556299c4b911602a052cd92d31cc57fbae1600fe2&scene=21#wechat_redirect
[Link 10]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247488468&idx=2&sn=7071a2a562afa337e7b46b490aa44090&chksm=fcb70f6bcbc0867dcd4cd5a16a7b5ce3b88bb10e9af6165797ffe10f7cca8b9d3a0eb365774f&scene=21#wechat_redirect
[Link 11]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247488759&idx=2&sn=91be2e59ada4107851289c68a64bffb3&chksm=fcb70848cbc0815e03947731557b16897112f37c9b67cd436013a7cf70738cada21925d37ab4&scene=21#wechat_redirect
[Link 12]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247490386&idx=2&sn=321179886eb4f6a2c6bd55aed3578097&chksm=fcb707edcbc08efb406cec43da6540df1e7c99ed88f02ebd2c313ad3549d082679a10cdf6165&scene=21#wechat_redirect
[Link 13]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247494603&idx=3&sn=1d0ca7f2eef4ace9941ce2f3ff130dbb&chksm=fcb4f774cbc37e62a1425aca8d97e593e30bd09a7b3a8f4b3d0d9ac32877979944af03030717&scene=21#wechat_redirect
[Link 14]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247495199&idx=3&sn=4e56ec540c4fa222d9e57804b554ea70&chksm=fcb4f2a0cbc37bb60193b8a653016a0d5ea933f1b9c69e39d1b06861813c78fc7c574b527454&scene=21#wechat_redirect
[Link 15]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247503912&idx=1&sn=7cd6fee3a4fd042be5ae8f54d750052b&chksm=fcb4cc97cbc345818ec45020b51297150f8b2f1f211f8d74523529f02e14a47af812a5827443&scene=21#wechat_redirect
[Link 16]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247508471&idx=1&sn=9d6a781d8015ca651f231543977b99f0&chksm=fcb4bd48cbc3345e5169a622d44d6cb1ac4ff9a31970b4cf7a132e7d766f5ae90164a1d2b662&scene=21#wechat_redirect
[Link 17]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247512691&idx=1&sn=5a9a03b656da97558fd537df2161a07e&chksm=fcb4aecccbc327da843ff574076da8701708dbd39a957ad41d6b2b9d4ac2d6e64584af2c7426&scene=21#wechat_redirect
[Link 18]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247511199&idx=1&sn=dcc96018356c7aa1f07f594ea2db53ab&chksm=fcb4b020cbc3393667267919d45479df44e03b8fccaf5ea1ecffadc4fd664e8505dc0db723ec&scene=21#wechat_redirect
[Link 19]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247510579&idx=1&sn=8b7f6facff81a9e55f653bad1c6b016f&chksm=fcb4b68ccbc33f9a2ffcfe67d87fafdd025566c76d9ad4caf9951d9ac084dae803f52ab70048&scene=21#wechat_redirect
[Link 20]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247510383&idx=1&sn=f659ab4c9b0973276be95c9bab89d5b1&chksm=fcb4b5d0cbc33cc6a9771ce2d6cd88e57b3def32c9d0e3ca8b09e413855a71f422794971fe4f&scene=21#wechat_redirect
[Link 21]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247510244&idx=1&sn=5714c36390b4a7307252c4d02cfaecf0&chksm=fcb4b45bcbc33d4de651e9b095c49a760eb4b909a1a7c0553343bd2bb3bee93b34285bbfacd3&scene=21#wechat_redirect
[Top2_2021_ _]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247505138&idx=1&sn=ef6102d36f7aa7191982c6b6eddb9872&chksm=fcb4c84dcbc3415be45e48c522029fb90f1c395278708f38c6d9b68d2aadaa0afff68b3ff607&scene=21#wechat_redirect
[Link 22]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247498233&idx=2&sn=7663c29445a7f07f839424656ef28cb5&chksm=fcb4e546cbc36c50b7711c59b77d01048cc81302e4a4a9f4fdbf83715360b14c5110f873b485&scene=21#wechat_redirect
[Link 23]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247497362&idx=2&sn=5b3436738550a7e4d62c2bb98befe3df&chksm=fcb4ea2dcbc3633b228e5712ecfc9d302ff56b7d96e83232f1c7ffb0a8bd2ffc908b5003b9d9&scene=21#wechat_redirect
[_ _ _]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247498631&idx=2&sn=7a91e5850ec4885a4d7028b925e6f2d2&chksm=fcb4e738cbc36e2e8ed92fcc6acbf5024ee73d4f5b2e05bcd2c31333fad86d6d4638b06fef6f&scene=21#wechat_redirect
[Link 24]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247489104&idx=1&sn=189ff572dc14ee137e83e7e4e7a4cc9f&chksm=fcb70aefcbc083f9b351b1b29c56d70cc1748181583490c3769b77a495e74747b8f7491ee39a&scene=21#wechat_redirect
[Link 25]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247487741&idx=2&sn=40bbb83c4ecda43fcc11fc3eb3a7130f&chksm=fcb70c42cbc085545c4ccfd51563cc59e38134147803c0c59e0aa3827385f060ccc75b3df888&scene=21#wechat_redirect
[PyTorch_Tensor]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247504061&idx=1&sn=207d2bd4895e0b4e303c3d9d63511c72&chksm=fcb4cc02cbc3451449159333c651688e7d71c7a744a4e4e0d914fb14adb5ad4f7e122eff7e18&scene=21#wechat_redirect
[CNN_ _]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247503173&idx=1&sn=e92e9adb903e7a87d0b64ceff982020c&chksm=fcb4d1facbc358ec09986cd8b32ce222c3394c46e2f0bd60c282d63ffeea897e7211984e6fca&scene=21#wechat_redirect
[Link 26]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247502893&idx=1&sn=e5720b57b0804dca182552ba9d6c1eb3&chksm=fcb4d092cbc359841d6f5a9e3b838e3894d34b35d74a1627fa9bf37e27c019ca9d9af84c362d&scene=21#wechat_redirect
[Transformer _]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247502134&idx=1&sn=ed4a262db0be219feb017c08f2185779&chksm=fcb4d589cbc35c9f0156ac454dccb82a501f4b8129adbbcf94a3ca83d3af129f4ebadf173e75&scene=21#wechat_redirect
[Scaled-YOLOv4_ _]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247500056&idx=2&sn=95a04371dc0b49e8c862b03a88edbb8c&chksm=fcb4dda7cbc354b14c4cd4f63eadf02680112ee21015fe372cf3055f9b577b7dd989843eda1f&scene=21#wechat_redirect
[PyTorch_ _]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247497349&idx=3&sn=b4b1ea0af5d42c817bc776cab0a8ac28&chksm=fcb4ea3acbc3632cd61c21d40df0c8cd9e84603fcbe551ceea721e4d8c43959a3c74e929a0fe&scene=21#wechat_redirect
[PyTorch_ _ 1]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247497036&idx=2&sn=552069978cf317b9581b4c9a62a003e2&chksm=fcb4e9f3cbc360e51de7a7ceb3a6adde123e488d98a3baf4204270311ef609fa5510682ff29b&scene=21#wechat_redirect
[Numpy]: https://blog.csdn.net/dafeidouzi/article/details/100520906
[10]: https://blog.csdn.net/lvsehaiyang1993/article/details/80342473
[CVPR 2020]: https://blog.csdn.net/u013685264/article/details/104984809
[Link 27]: https://blog.csdn.net/fengdu78/article/details/107308594
[YOLOv4]: https://blog.csdn.net/fengdu78/article/details/106450074
[f1234e83113afc6817f9446aaa7d5f08.gif]: /images/20220828/9bd27b4b3024438aacc75d187ddf3824.png
[b632411cd5b9cca4755328ec254fed17.gif]: /images/20220828/357376abff334249a97bd8f14f225ed8.png
[0baec84139ced50ad9edfae164c9f3db.gif]: /images/20220828/160b0e54d89045b486c64a0703293193.png
[79a64c8f0df77c75eb4d549b930925c8.gif]: /images/20220828/72ad3ca7d09c428c8fa26d5871b14aa9.png