Bi-LSTM-CRF for Sequence Labeling

布满荆棘的人生 2021-10-29 10:22 201阅读 0赞

做了一段时间的Sequence Labeling的工作，发现在NER任务上面，很多论文都采用LSTM-CRFs的结构。CRF在最后一层应用进来可以考虑到概率最大的最优label路径，可以提高指标。

一般的深度学习框架是没有CRF layer的，需要手动实现。最近在学习[PyTorch][]，里面有一个[Bi-LSTM-CRF][]的tutorial实现。不得不说PyTorch的tutorial真是太良心了，基本涵盖了NLP领域各个流行的model实现。在这里从头梳理一遍，也记录下学习过程中的一些问题。

![v2-b8204995a72a0fbeebaec0203a627f2e_hd.png][]Bi-LSTM-CRF的结构一般如上，最后一层利用CRF来学习一个最优路径。Bi-LSTM layer的输出维度是tag size，这就相当于是每个词![\[公式\]][equation_tex_w_i]映射到tag的发射概率值，设Bi-LSTM的输出矩阵为![\[公式\]][equation_tex_P]，其中![\[公式\]][equation_tex_P_7Bi_2Cj_7D]代表词![\[公式\]][equation_tex_w_i]映射到![\[公式\]][equation_tex_tag_j]的非归一化概率。对于CRF来说，我们假定存在一个转移矩阵![\[公式\]][equation_tex_A]，则![\[公式\]][equation_tex_A_7Bi_2Cj_7D]代表![\[公式\]][equation_tex_tag_i]转移到![\[公式\]][equation_tex_tag_j]的转移概率。

对于输入序列![\[公式\]][equation_tex_X]对应的输出tag序列![\[公式\]][equation_tex_y]，定义分数为

![v2-e037384808ffa4608351e7546d798830_hd.png][]利用Softmax函数，我们为每一个正确的tag序列![\[公式\]][equation_tex_y]定义一个概率值（![\[公式\]][equation_tex_Y_X]代表所有的tag序列，包括不可能出现的）

![v2-ea14c0d53ac1d346852bc7dd4c7de3a3_hd.png][]因而在训练中，我们只需要最大化似然概率![\[公式\]][equation_tex_p_28y_7CX_29]即可，这里我们利用对数似然

![v2-ddc500ac489c7c5198fe1ef4a33716af_hd.png][]

所以我们将损失函数定义为![\[公式\]][equation_tex_-log_28p_28y_7CX_29_29]，就可以利用梯度下降法来进行网络的学习了。

在对损失函数进行计算的时候，![\[公式\]][equation_tex_S_28X_2Cy_29]的计算很简单，而![\[公式\]][equation_tex_log_28_5Csum_7B_5Cwidetilde_7By_7D_5Cin_Y_X_7D_e_5E_7BS_28X_2C_5Cwidetilde_7By_7D_29_7D_29]（下面记作logsumexp）的计算稍微复杂一些，因为需要计算每一条可能路径的分数。这里用一种简便的方法，对于到词![\[公式\]][equation_tex_w_7Bi_2B1_7D]的路径，可以先把到词![\[公式\]][equation_tex_w_i]的logsumexp计算出来，因为

![v2-f12fce12419c2f242ebc60f01ca311be_hd.png][]因此先计算每一步的路径分数和直接计算全局分数相同，但这样可以大大减少计算的时间。下面是PyTorch中的代码

def _forward_alg(self, feats): # Do the forward algorithm to compute the partition function init_alphas = torch.Tensor(1, self.tagset_size).fill_(-10000.) # START_TAG has all of the score. init_alphas[0][self.tag_to_ix[START_TAG]] = 0. # Wrap in a variable so that we will get automatic backprop forward_var = autograd.Variable(init_alphas) # Iterate through the sentence for feat in feats: alphas_t = [] # The forward variables at this timestep for next_tag in range(self.tagset_size): # broadcast the emission score: it is the same regardless of # the previous tag emit_score = feat[next_tag].view( 1, -1).expand(1, self.tagset_size) # the ith entry of trans_score is the score of transitioning to # next_tag from i trans_score = self.transitions[next_tag].view(1, -1) # The ith entry of next_tag_var is the value for the # edge (i -> next_tag) before we do log-sum-exp next_tag_var = forward_var + trans_score + emit_score # The forward variable for this tag is log-sum-exp of all the # scores. alphas_t.append(log_sum_exp(next_tag_var)) forward_var = torch.cat(alphas_t).view(1, -1) terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]] alpha = log_sum_exp(terminal_var) return alpha

在解码时，采用Viterbi算法

def _viterbi_decode(self, feats): backpointers = [] # Initialize the viterbi variables in log space init_vvars = torch.Tensor(1, self.tagset_size).fill_(-10000.) init_vvars[0][self.tag_to_ix[START_TAG]] = 0 # forward_var at step i holds the viterbi variables for step i-1 forward_var = autograd.Variable(init_vvars) for feat in feats: bptrs_t = [] # holds the backpointers for this step viterbivars_t = [] # holds the viterbi variables for this step for next_tag in range(self.tagset_size): # next_tag_var[i] holds the viterbi variable for tag i at the # previous step, plus the score of transitioning # from tag i to next_tag. # We don't include the emission scores here because the max # does not depend on them (we add them in below) next_tag_var = forward_var + self.transitions[next_tag] best_tag_id = argmax(next_tag_var) bptrs_t.append(best_tag_id) viterbivars_t.append(next_tag_var[0][best_tag_id]) # Now add in the emission scores, and assign forward_var to the set # of viterbi variables we just computed forward_var = (torch.cat(viterbivars_t) + feat).view(1, -1) backpointers.append(bptrs_t) # Transition to STOP_TAG terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]] best_tag_id = argmax(terminal_var) path_score = terminal_var[0][best_tag_id] # Follow the back pointers to decode the best path. best_path = [best_tag_id] for bptrs_t in reversed(backpointers): best_tag_id = bptrs_t[best_tag_id] best_path.append(best_tag_id) # Pop off the start tag (we dont want to return that to the caller) start = best_path.pop() assert start == self.tag_to_ix[START_TAG] # Sanity check best_path.reverse() return path_score, best_path

全部代码实现可以移步[Bi-LSTM-CRF][]。

## 参考 ##

[Bidirectional LSTM-CRF Models for Sequence Tagging][]  
[Neural Architectures for Named Entity Recognition][]  
[Advanced: Making Dynamic Decisions and the Bi-LSTM CRF][Bi-LSTM-CRF]

转载于:https://www.cnblogs.com/jfdwd/p/11185644.html

[PyTorch]: https://link.zhihu.com/?target=http%3A//pytorch.org/tutorials/
[Bi-LSTM-CRF]: https://link.zhihu.com/?target=http%3A//pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html
[v2-b8204995a72a0fbeebaec0203a627f2e_hd.png]: https://pic3.zhimg.com/80/v2-b8204995a72a0fbeebaec0203a627f2e_hd.png
[equation_tex_w_i]: /images/20211029/3754c5e3f8e34fdb95d28db410058ee4.png
[equation_tex_P]: https://www.zhihu.com/equation?tex=P
[equation_tex_P_7Bi_2Cj_7D]: https://www.zhihu.com/equation?tex=P_%7Bi%2Cj%7D
[equation_tex_tag_j]: https://www.zhihu.com/equation?tex=tag_j
[equation_tex_A]: https://www.zhihu.com/equation?tex=A
[equation_tex_A_7Bi_2Cj_7D]: https://www.zhihu.com/equation?tex=A_%7Bi%2Cj%7D
[equation_tex_tag_i]: https://www.zhihu.com/equation?tex=tag_i
[equation_tex_X]: https://www.zhihu.com/equation?tex=X
[equation_tex_y]: /images/20211029/4b05502aa7354ed2bcbd461957f9b1b1.png
[v2-e037384808ffa4608351e7546d798830_hd.png]: https://pic1.zhimg.com/80/v2-e037384808ffa4608351e7546d798830_hd.png
[equation_tex_Y_X]: https://www.zhihu.com/equation?tex=Y_X
[v2-ea14c0d53ac1d346852bc7dd4c7de3a3_hd.png]: https://pic4.zhimg.com/80/v2-ea14c0d53ac1d346852bc7dd4c7de3a3_hd.png
[equation_tex_p_28y_7CX_29]: https://www.zhihu.com/equation?tex=p%28y%7CX%29
[v2-ddc500ac489c7c5198fe1ef4a33716af_hd.png]: /images/20211029/ecc6de35a269462eb75f268ddc432259.png
[equation_tex_-log_28p_28y_7CX_29_29]: /images/20211029/c5c41d222ff6468aa7089cce4cf5da2e.png
[equation_tex_S_28X_2Cy_29]: https://www.zhihu.com/equation?tex=S%28X%2Cy%29
[equation_tex_log_28_5Csum_7B_5Cwidetilde_7By_7D_5Cin_Y_X_7D_e_5E_7BS_28X_2C_5Cwidetilde_7By_7D_29_7D_29]: https://www.zhihu.com/equation?tex=log%28%5Csum_%7B%5Cwidetilde%7By%7D%5Cin+Y_X%7D+e%5E%7BS%28X%2C+%5Cwidetilde%7By%7D%29%7D%29
[equation_tex_w_7Bi_2B1_7D]: https://www.zhihu.com/equation?tex=w_%7Bi%2B1%7D
[v2-f12fce12419c2f242ebc60f01ca311be_hd.png]: /images/20211029/7862bb0626084da7914ae85b24d785f5.png
[Bidirectional LSTM-CRF Models for Sequence Tagging]: https://link.zhihu.com/?target=https%3A//arxiv.org/abs/1508.01991
[Neural Architectures for Named Entity Recognition]: https://link.zhihu.com/?target=https%3A//arxiv.org/abs/1603.01360