LSTM缓解梯度消失的原因

蔚落 2021-11-05 07:06 284阅读 0赞

![776760-20190724190019380-1236973429.png][]

\$c\_\{t\}=c\_\{t-1\} \\otimes \\sigma\\left(W\_\{f\} \\cdot\\left\[H\_\{t-1\}, X\_\{t\}\\right\]\\right) \\oplus \\tanh \\left(W\_\{c\} \\cdot\\left\[H\_\{t-1\}, X\_\{t\}\\right\]\\right) \\otimes \\sigma\\left(W\_\{i\} \\cdot\\left\[H\_\{t-1\}, X\_\{t\}\\right\]\\right)\$

反向传播公式：

\$\\begin\{aligned\} \\frac\{\\partial E\_\{k\}\}\{\\partial W\}=& \\frac\{\\partial E\_\{k\}\}\{\\partial H\_\{k\}\} \\frac\{\\partial H\_\{k\}\}\{\\partial C\_\{k\}\} \\frac\{\\partial C\_\{k\}\}\{\\partial C\_\{k-1\}\} \\ldots \\frac\{\\partial C\_\{2\}\}\{\\partial C\_\{1\}\} \\frac\{\\partial C\_\{1\}\}\{\\partial W\}=\\\\ & \\frac\{\\partial E\_\{k\}\}\{\\partial H\_\{k\}\} \\frac\{\\partial H\_\{k\}\}\{\\partial C\_\{k\}\}\\left(\\prod\_\{t=2\}^\{k\} \\frac\{\\partial C\_\{t\}\}\{\\partial C\_\{t-1\}\}\\right) \\frac\{\\partial C\_\{1\}\}\{\\partial W\} \\end\{aligned\}\$

括号中的部分是累乘项：

$\\frac\{\\partial c\_\{t\}\}\{\\partial c\_\{t-1\}\}=\\sigma\\left(W\_\{f\} \\cdot\\left\[H\_\{t-1\}, X\_\{t\}\\right\]\\right) + $  
\$\\frac\{d\}\{d \\mathcal\{C\}\_\{t-1\}\}\\left(\\tanh \\left(W\_\{c\} \\cdot\\left\[H\_\{t-1\}, X\_\{t\}\\right\]\\right) \\otimes \\sigma\\left(W\_\{i\} \\cdot\\left\[H\_\{t-1\}, X\_\{t\}\\right\]\\right)\\right)\$

也就是说，这里的累乘单元是两项和形式，其中前部分是遗忘门的值。遗忘门决定了上一个细胞状态的保留比例，其取值可以接近于1，也就是说可以把遗忘门看成：\$\\sigma\\left(W\_\{f\} \\cdot\\left\[H\_\{t-1\}, X\_\{t\}\\right\]\\right) \\approx \\overrightarrow\{1\}\$，所以LSTM中：

\$\\frac\{\\partial E\_\{k\}\}\{\\partial W\} \\approx \\frac\{\\partial E\_\{k\}\}\{\\partial H\_\{k\}\} \\frac\{\\partial H\_\{k\}\}\{\\partial c\_\{k\}\}\\left(\\Pi\_\{t=2\}^\{k\} \\sigma\\left(W\_\{f\} \\cdot\\left\[H\_\{t-1\}, X\_\{t\}\\right\]\\right)\\right) \\frac\{\\partial C\_\{1\}\}\{\\partial w\} \\nrightarrow 0\$

所以，LSTM能缓解梯度消失。

转载于:https://www.cnblogs.com/Elaine-DWL/p/11240213.html

[776760-20190724190019380-1236973429.png]: /images/20211104/bf6db224d464448ea9a3318632a1cce6.png