为什么使用负采样技术(附详细求导过程)

﹏ヽ暗。殇╰゛Y 2022-12-30 14:00 149阅读 0赞

word2vec的初心是什么，为什么使用负采样技术。后面看论文过程经常遇到负采样，如mepath2vec等等，对这个知识点不了解清楚，论文很多精华部分理解就会有偏差了。

看了很多博客和B站视频讲解，还是没有讲解清楚。

最后还是知乎大佬说清楚了。[详见][Link 1]

# **1 用负采样带来的问题** #

*  ## 参数更新慢 ##

其一 我们每次只对窗口中出现的几个单词参数进行更新，但是在计算梯度的过程中，是对整个参数矩阵进行运算，这样参数矩阵中的大部分值都是0。

![3fea874798109e21a4019d35921dcb6c.png][]

*  ## 计算开销大 ##

其二 是 导致计算效率低下  
softmax函数激活，会导致计算全部的单词‘得分’（滑动窗口在滑动过程中，需要计算滑动窗口里面单词然后求和，效率低）

![2590dd830cab8459e08ca0c8d181c390.png][]

#       **2. 使用负采样** #

负采样的核心思想是：

计算中心词和窗口中上下文词的真实“得分”，再加一些“噪声”，即词表中的随机词（滑动窗口以外）和中心词的“得分”。  
  
**目标方程**：

![d1b2d35bf38ba345b92b85ba68b99a09.png][]

![03f5831b9073429c66a8add0c7e274c2.png][]

（**最大化 真实单词对“得分”+    “噪声”作为目标方程**。）  
采用上述公式解决了之前说的两个问题：

*  仅对K个参数进行采样
 *  放弃softmax函数，采用sigmoid函数，这样就不存在先求一遍窗口中所有单词的‘“得分”的情况

#    **3. 计算梯度** #

把最大化问题变为最小化问题：

![\\mathbf\{J\_\{t \}(\\theta )= -log\\delta (u\_o^\{T\}u\_c) - \\sum\_\{i=1\}^\{k\}\\mathbb\{E\}\_\{jP(w)\}\[log\\delta (-u\_j^\{T\}u\_c\]\}][mathbf_J_t _theta _ -log_delta _u_o_T_u_c_ - _sum_i_1_k_mathbb_E_jP_w_log_delta _-u_j_T_u_c]

里面涉及到的变量是![u\_0][u_0] , ![u\_j][u_j]和![u\_c][u_c].分别进行求偏导即可。然后进行更新

## 对![u\_c][u_c]求偏导 ##

其他参数为不管，看为常量。

其中里面的sigmoid函数(除log外)求导后为：![\\frac\{\\partial \\delta (u\_o^\{T\}u\_c)\} \{\\partial u\_c\} =\\delta (u\_o^\{T\}u\_c)\\cdot (1-\\delta (u\_o^\{T\}u\_c))u\_o][frac_partial _delta _u_o_T_u_c_ _partial u_c_ _delta _u_o_T_u_c_cdot _1-_delta _u_o_T_u_c_u_o]

![\\frac\{\\partial J(\\theta ) \}\{\\partial u\_c\} =\\frac\{\\partial -log\\delta (u\_o^\{T\}u\_c) \}\{\\partial u\_c\} - \\frac\{\\partial \\sum\_\{i=1\}^\{k\}\\mathbb\{E\}\_\{jP(w)\}\[log\\delta (-u\_j^\{T\}u\_c)\]\} \{\\partial u\_c\}][frac_partial J_theta _ _partial u_c_ _frac_partial -log_delta _u_o_T_u_c_ _partial u_c_ - _frac_partial _sum_i_1_k_mathbb_E_jP_w_log_delta _-u_j_T_u_c_ _partial u_c]

![= -\\frac\{1\}\{\\delta (u\_o^\{T\}u\_c)\}\[\\delta (u\_o^\{T\}u\_c)\\cdot (1-\\delta (u\_o^\{T\}u\_c))u\_o\] - \\sum\_\{i=1\}^\{k\}\\frac\{1\}\{\\delta (-u\_i^\{T\}u\_c)\}\[\\delta (-u\_i^\{T\}u\_c)\\cdot (1-\\delta (-u\_i^\{T\}u\_c))\\cdot (-u\_i)\]][-_frac_1_delta _u_o_T_u_c_delta _u_o_T_u_c_cdot _1-_delta _u_o_T_u_c_u_o_ - _sum_i_1_k_frac_1_delta _-u_i_T_u_c_delta _-u_i_T_u_c_cdot _1-_delta _-u_i_T_u_c_cdot _-u_i]

注意这里的log看着以e为底，T这个求和与![u\_c][u_c]无关，k这个求和与![u\_c][u_c]有关.

化简得：

![\\frac\{\\partial J(\\theta ) \}\{\\partial u\_c\} = (\\delta (u\_o^\{T\}u\_c)-1)u\_o - \\sum\_\{i=1\}^\{k\} (\\delta (-u\_i^\{T\}u\_c)-1)u\_i][frac_partial J_theta _ _partial u_c_ _ _delta _u_o_T_u_c_-1_u_o - _sum_i_1_k_ _delta _-u_i_T_u_c_-1_u_i],前面知乎大佬没有进行最大化变最小化，直接就给出了答案，是错的。应该先最小化，在求导，结果才对。

## 对![u\_o][u_o]求偏导 ##

其他参数为不管，看为常量。

按照![u\_c][u_c]求导思路：

![\\frac\{\\partial J(\\theta ) \}\{\\partial u\_o\} =\\frac\{\\partial -log\\delta (u\_o^\{T\}u\_c) \}\{\\partial u\_o\} = -\\frac\{1\}\{\\delta (u\_o^\{T\}u\_c)\}\[\\delta (u\_o^\{T\}u\_c)\\cdot (1-\\delta (u\_o^\{T\}u\_c))u\_0\]][frac_partial J_theta _ _partial u_o_ _frac_partial -log_delta _u_o_T_u_c_ _partial u_o_ _ -_frac_1_delta _u_o_T_u_c_delta _u_o_T_u_c_cdot _1-_delta _u_o_T_u_c_u_0]

化简得：

![\\frac\{\\partial J(\\theta ) \}\{\\partial u\_o\} = (\\delta (u\_o^\{T\}u\_c)-1)u\_o][frac_partial J_theta _ _partial u_o_ _ _delta _u_o_T_u_c_-1_u_o]

## 对![u\_i][u_i]求偏导 ##

![\\frac\{\\partial J(\\theta ) \}\{\\partial u\_i\} =0 - \\frac\{\\partial \\sum\_\{i=1\}^\{k\}\\mathbb\{E\}\_\{jP(w)\}\[log\\delta (-u\_i^\{T\}u\_c\]\} \{\\partial u\_i\}][frac_partial J_theta _ _partial u_i_ _0 - _frac_partial _sum_i_1_k_mathbb_E_jP_w_log_delta _-u_i_T_u_c_ _partial u_i]

![= 0- \\sum\_\{i=1\}^\{k\}\\frac\{1\}\{\\delta (-u\_i^\{T\}u\_c)\}\[\\delta (-u\_i^\{T\}u\_c)\\cdot (1-\\delta (-u\_i^\{T\}u\_c))-u\_c\]][0- _sum_i_1_k_frac_1_delta _-u_i_T_u_c_delta _-u_i_T_u_c_cdot _1-_delta _-u_i_T_u_c_-u_c]

化简得：

![\\frac\{\\partial J(\\theta ) \}\{\\partial u\_i\} = - \\sum\_\{i=1\}^\{k\} (\\delta (-u\_i^\{T\}u\_c)-1)u\_c][frac_partial J_theta _ _partial u_i_ _ - _sum_i_1_k_ _delta _-u_i_T_u_c_-1_u_c]

[Link 1]: https://zhuanlan.zhihu.com/p/66088781
[3fea874798109e21a4019d35921dcb6c.png]: /images/20221120/0179a9c3d5a94cf88c5254ef3762156f.png
[2590dd830cab8459e08ca0c8d181c390.png]: /images/20221120/46bb77d484b34718bd8ab3b02182a924.png
[d1b2d35bf38ba345b92b85ba68b99a09.png]: /images/20221120/31d3fd7376ed4c979060da0357b9806f.png
[03f5831b9073429c66a8add0c7e274c2.png]: /images/20221120/7b804811335d49ac86e44f4d8f30e854.png
[mathbf_J_t _theta _ -log_delta _u_o_T_u_c_ - _sum_i_1_k_mathbb_E_jP_w_log_delta _-u_j_T_u_c]: https://private.codecogs.com/gif.latex?%5Cmathbf%7BJ_%7Bt%20%7D%28%5Ctheta%20%29%3D%20-log%5Cdelta%20%28u_o%5E%7BT%7Du_c%29%20-%20%5Csum_%7Bi%3D1%7D%5E%7Bk%7D%5Cmathbb%7BE%7D_%7BjP%28w%29%7D%5Blog%5Cdelta%20%28-u_j%5E%7BT%7Du_c%5D%7D
[u_0]: https://private.codecogs.com/gif.latex?u_0
[u_j]: https://private.codecogs.com/gif.latex?u_j
[u_c]: https://private.codecogs.com/gif.latex?u_c
[frac_partial _delta _u_o_T_u_c_ _partial u_c_ _delta _u_o_T_u_c_cdot _1-_delta _u_o_T_u_c_u_o]: https://private.codecogs.com/gif.latex?%5Cfrac%7B%5Cpartial%20%5Cdelta%20%28u_o%5E%7BT%7Du_c%29%7D%20%7B%5Cpartial%20u_c%7D%20%3D%5Cdelta%20%28u_o%5E%7BT%7Du_c%29%5Ccdot%20%281-%5Cdelta%20%28u_o%5E%7BT%7Du_c%29%29u_o
[frac_partial J_theta _ _partial u_c_ _frac_partial -log_delta _u_o_T_u_c_ _partial u_c_ - _frac_partial _sum_i_1_k_mathbb_E_jP_w_log_delta _-u_j_T_u_c_ _partial u_c]: https://private.codecogs.com/gif.latex?%5Cfrac%7B%5Cpartial%20J%28%5Ctheta%20%29%20%7D%7B%5Cpartial%20u_c%7D%20%3D%5Cfrac%7B%5Cpartial%20-log%5Cdelta%20%28u_o%5E%7BT%7Du_c%29%20%7D%7B%5Cpartial%20u_c%7D%20-%20%5Cfrac%7B%5Cpartial%20%5Csum_%7Bi%3D1%7D%5E%7Bk%7D%5Cmathbb%7BE%7D_%7BjP%28w%29%7D%5Blog%5Cdelta%20%28-u_j%5E%7BT%7Du_c%29%5D%7D%20%7B%5Cpartial%20u_c%7D
[-_frac_1_delta _u_o_T_u_c_delta _u_o_T_u_c_cdot _1-_delta _u_o_T_u_c_u_o_ - _sum_i_1_k_frac_1_delta _-u_i_T_u_c_delta _-u_i_T_u_c_cdot _1-_delta _-u_i_T_u_c_cdot _-u_i]: https://private.codecogs.com/gif.latex?%3D%20-%5Cfrac%7B1%7D%7B%5Cdelta%20%28u_o%5E%7BT%7Du_c%29%7D%5B%5Cdelta%20%28u_o%5E%7BT%7Du_c%29%5Ccdot%20%281-%5Cdelta%20%28u_o%5E%7BT%7Du_c%29%29u_o%5D%20-%20%5Csum_%7Bi%3D1%7D%5E%7Bk%7D%5Cfrac%7B1%7D%7B%5Cdelta%20%28-u_i%5E%7BT%7Du_c%29%7D%5B%5Cdelta%20%28-u_i%5E%7BT%7Du_c%29%5Ccdot%20%281-%5Cdelta%20%28-u_i%5E%7BT%7Du_c%29%29%5Ccdot%20%28-u_i%29%5D
[frac_partial J_theta _ _partial u_c_ _ _delta _u_o_T_u_c_-1_u_o - _sum_i_1_k_ _delta _-u_i_T_u_c_-1_u_i]: https://private.codecogs.com/gif.latex?%5Cfrac%7B%5Cpartial%20J%28%5Ctheta%20%29%20%7D%7B%5Cpartial%20u_c%7D%20%3D%20%28%5Cdelta%20%28u_o%5E%7BT%7Du_c%29-1%29u_o%20-%20%5Csum_%7Bi%3D1%7D%5E%7Bk%7D%20%28%5Cdelta%20%28-u_i%5E%7BT%7Du_c%29-1%29u_i
[u_o]: https://private.codecogs.com/gif.latex?u_o
[frac_partial J_theta _ _partial u_o_ _frac_partial -log_delta _u_o_T_u_c_ _partial u_o_ _ -_frac_1_delta _u_o_T_u_c_delta _u_o_T_u_c_cdot _1-_delta _u_o_T_u_c_u_0]: https://private.codecogs.com/gif.latex?%5Cfrac%7B%5Cpartial%20J%28%5Ctheta%20%29%20%7D%7B%5Cpartial%20u_o%7D%20%3D%5Cfrac%7B%5Cpartial%20-log%5Cdelta%20%28u_o%5E%7BT%7Du_c%29%20%7D%7B%5Cpartial%20u_o%7D%20%3D%20-%5Cfrac%7B1%7D%7B%5Cdelta%20%28u_o%5E%7BT%7Du_c%29%7D%5B%5Cdelta%20%28u_o%5E%7BT%7Du_c%29%5Ccdot%20%281-%5Cdelta%20%28u_o%5E%7BT%7Du_c%29%29u_0%5D
[frac_partial J_theta _ _partial u_o_ _ _delta _u_o_T_u_c_-1_u_o]: https://private.codecogs.com/gif.latex?%5Cfrac%7B%5Cpartial%20J%28%5Ctheta%20%29%20%7D%7B%5Cpartial%20u_o%7D%20%3D%20%28%5Cdelta%20%28u_o%5E%7BT%7Du_c%29-1%29u_o
[u_i]: https://private.codecogs.com/gif.latex?u_i
[frac_partial J_theta _ _partial u_i_ _0 - _frac_partial _sum_i_1_k_mathbb_E_jP_w_log_delta _-u_i_T_u_c_ _partial u_i]: https://private.codecogs.com/gif.latex?%5Cfrac%7B%5Cpartial%20J%28%5Ctheta%20%29%20%7D%7B%5Cpartial%20u_i%7D%20%3D0%20-%20%5Cfrac%7B%5Cpartial%20%5Csum_%7Bi%3D1%7D%5E%7Bk%7D%5Cmathbb%7BE%7D_%7BjP%28w%29%7D%5Blog%5Cdelta%20%28-u_i%5E%7BT%7Du_c%5D%7D%20%7B%5Cpartial%20u_i%7D
[0- _sum_i_1_k_frac_1_delta _-u_i_T_u_c_delta _-u_i_T_u_c_cdot _1-_delta _-u_i_T_u_c_-u_c]: https://private.codecogs.com/gif.latex?%3D%200-%20%5Csum_%7Bi%3D1%7D%5E%7Bk%7D%5Cfrac%7B1%7D%7B%5Cdelta%20%28-u_i%5E%7BT%7Du_c%29%7D%5B%5Cdelta%20%28-u_i%5E%7BT%7Du_c%29%5Ccdot%20%281-%5Cdelta%20%28-u_i%5E%7BT%7Du_c%29%29-u_c%5D
[frac_partial J_theta _ _partial u_i_ _ - _sum_i_1_k_ _delta _-u_i_T_u_c_-1_u_c]: https://private.codecogs.com/gif.latex?%5Cfrac%7B%5Cpartial%20J%28%5Ctheta%20%29%20%7D%7B%5Cpartial%20u_i%7D%20%3D%20-%20%5Csum_%7Bi%3D1%7D%5E%7Bk%7D%20%28%5Cdelta%20%28-u_i%5E%7BT%7Du_c%29-1%29u_c