NLP-2015：Luong NMT模型【Attention类型：Global Attention、Local Attention（实践中很少用）】

深碍√TFBOYSˉ_ 2023-10-05 20:16 13阅读 0赞

《原始论文：Effective Approaches to Attention-based Neural Machine Translation》

在实际应用中，全局注意力机制比局部注意力机制有着更广泛的应用，因为局部注意力机制需要预测一个位置向量  p t p\_t pt，而这个位置向量的预测并不是非常准确的，会影响对齐向量的准确率。同时，在处理不是很长的源端句子时，相比于全局注意力并没有减少很多计算量。

## 一、全局注意力机制【Global Attention】 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70_pic_center]  
Global Attention原理：使用源语言全部信息做注意力机制；  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70]

## 二、局部注意力机制【Local Attention】 ##

全局注意力机制有一个缺点，就是它必须关注每个目标单词的来源句子的所有单词，这一点代价高昂，并且可能使得翻译更长的序列（如段落或文档）变得非常困难。为了解决这个不足，该论文提出了一个局部注意力机制，该机制选择性地只关注每个目标词源位置的一小部分。

局部注意力是一种介于软注意力机制和硬注意力机制之间的注意力方式，其结构如下图所示：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70_pic_center 1]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70 1]

--------------------

参考资料：  
[Effective Approaches to Attention-based Neural Machine Translation 中英文对照翻译][Effective Approaches to Attention-based Neural Machine Translation]  
[【论文笔记】Effective Approaches to Attention-based Neural Machine Translation][Effective Approaches to Attention-based Neural Machine Translation 1]  
[博客：Effective Approaches to Attention-based Neural Machine Translation][Effective Approaches to Attention-based Neural Machine Translation 2]  
[博客：Effective Approaches to Attention-based Neural Machine Translation(Global和Local attention)][Effective Approaches to Attention-based Neural Machine Translation_Global_Local attention]  
[《Effective Approaches to Attention-based Neural Machine Translation》—— 基于注意力机制的有效神经机器翻译方法][Effective Approaches to Attention-based Neural Machine Translation_]  
[【NLP论文笔记】Effective Approaches to Attention-based Neural Machine Translation][NLP_Effective Approaches to Attention-based Neural Machine Translation]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70_pic_center]: https://img-blog.csdnimg.cn/e37c8eba72e4459aa6843b21e5e2cc52.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE=,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/e056869f5a7142099152dbf3ba968075.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70_pic_center 1]: https://img-blog.csdnimg.cn/c1149a58c2844b10beffa43c284ded6b.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE=,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/db7c16ba8976493dad04505ca2f09649.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTMyNTA4NjE=,size_16,color_FFFFFF,t_70
[Effective Approaches to Attention-based Neural Machine Translation]: https://zhuanlan.zhihu.com/p/38205832
[Effective Approaches to Attention-based Neural Machine Translation 1]: https://blog.csdn.net/weixin_39837402/article/details/80523090
[Effective Approaches to Attention-based Neural Machine Translation 2]: https://blog.csdn.net/sinat_33741547/article/details/85293872
[Effective Approaches to Attention-based Neural Machine Translation_Global_Local attention]: https://www.cnblogs.com/lzida9223/p/10533583.html
[Effective Approaches to Attention-based Neural Machine Translation_]: https://blog.csdn.net/qq_37388085/article/details/109108483
[NLP_Effective Approaches to Attention-based Neural Machine Translation]: https://www.jianshu.com/p/1c24eba3ba9c