Conformer（运用在WeNet中的理解与分析）

电玩女神 2024-04-06 13:58 58阅读 0赞

#### Conformer（在ASR框架WeNet中的运用你有看过吗？） ####

*  1、Conformer论文
 *  2、Conformer结构
 *  3、Conformer在WeNet中代码理解
 *   *  3.1、进ConformerBlock前的处理
     *   *  3.1.1、SpecAug
         *  3.1.2、Convolution Subsampling
     *  3.2、ConformerBlock结构与公式
     *  3.3、Feed Forward Module Code
     *  3.4、Multi-Head Self Attention Module Code
     *  3.5、Convolution Module Code
 *  4、总结

本文主要结合WeNet理解Conformer，具体细节可以看原文哦  
关于WeNet代码详解可以查看知乎 ‘迷途小书僮’

## 1、Conformer论文 ##

conformer原文：《Conformer: Convolution-augmented Transformer for Speech Recognition》

提一下大家熟知的transformer原文《attention is all you need》，conformer即在transformer结构上多加了一个convolution而来的。

## 2、Conformer结构 ##

Conformer整体结构：SpecAug、ConvolutionSubsampling、Linear、Dropout、ConformerBlocks×N。

ConformerBlock结构（N个该结构）：Feed Forward Module、Multi-Head Self Attention Module、Convolution Module、Feed Forward Module、Layernorm。其中每个Module都是前接一个Layernorm后接一个Dropout，且都有残差链连接，残差数据为输入数据本身。

马卡龙结构：可以看到ConformerBlock神似马卡龙结构，即两个一样的Feed Forward Module中间夹了Multi-Head Self Attention Module和Convolution Module。

![在这里插入图片描述][b88ec25747b54b9287a94310a2bf61cc.png]

## 3、Conformer在WeNet中代码理解 ##

语音识别框架WeNet中encoder结构可选用Conformer，且通过官方文件可以看到，以conformer作为encoder结构比transformer有更好的效果。所以在做wenet训练时候更多的是用conformer作为encoder结构。

主要代码包括encoder.py、subsampling.py、embedding.py、encoder\_layer.py、positionwise\_feed\_forward.py、attention.py、convolution.py。

### 3.1、进ConformerBlock前的处理 ###

WeNet中我们提取语音特征之后，会将其送入到encoder中。

#### 3.1.1、SpecAug ####

在处理数据时进行

#### 3.1.2、Convolution Subsampling ####

代码区：encoder.py–>subsampling.py&embedding.py

**a**、对输入数据进行下采样，WeNet中有多项选择，即subsampling.py中提到的Conv2dSubsampling4、Conv2dSubsampling6、Conv2dSubsampling8.  
**b**、实验中采用Conv2dSubsampling6、Conv2dSubsampling8训练1wh数据会出现loss为nan现象，因此最终选择Conv2dSubsampling4作为Convolution Subsampling。  
**c**、使用Conv2dSubsampling4对输入数据进行下采样，代码中体现为对输入数据做两次卷积核为3×3，步长为2的Conv2d，做两次以后即对输入数据进行了四倍下采样。  
**d**、Convolution Subsampling部分代码，实验中设置输出维度即odim为512，batch\_size为14。

self.conv = torch.nn.Sequential(
                torch.nn.Conv2d(1, odim, 3, 2),
                torch.nn.ReLU(),
                torch.nn.Conv2d(odim, odim, 3, 2),
                torch.nn.ReLU(),
            )

**e**、实验中数据的shape体现  
Conv2dSubsampling4之前

(Pdb) p xs.shape
    torch.Size([14, 975, 80])

Conv2dSubsampling4之后

(Pdb) p xs.shape
    torch.Size([14, 243, 512])

可以看到经过Conv2dSubsampling4之后维度1从975变换到243，即进行了四倍下采样。

### 3.2、ConformerBlock结构与公式 ###

代码区：conformerblock: encoder.py–>encoder\_layer.py

**a**、经过前面的处理后正式进入到conformerblock中，实验中使用了N=12层的conformerblock结构。

**b**、conformerblock公式如下，这里就不赘述啦，就是根据conformerblock结构而来的公式。

![在这里插入图片描述][175e893146d543e59ebaa0c0ced8a589.png]

### 3.3、Feed Forward Module Code ###

代码区：Feed Forward Module: encoder.py–>encoder\_layer.py–>positionwise\_feed\_forward.py

**a**、结构  
如下所示，主要为两个线性层，其中激活函数为swish。conformerblock中包括两个Feed Forward Module，这俩结构是一样的。  
![在这里插入图片描述][a03cc96b70404c0cab73e73d7ce1d223.png]  
**b**、代码

self.w_1 = torch.nn.Linear(idim, hidden_units)
            self.activation = activation
            self.dropout = torch.nn.Dropout(dropout_rate)
            self.w_2 = torch.nn.Linear(hidden_units, idim)

Feed Forward Module前

(Pdb) p x.shape
    torch.Size([14, 243, 512])
    (Pdb) p x
    tensor([[[ -0.9448,  -1.6156,   0.8714,  ...,  -0.6092,   8.3108,  -4.7808],
             [ -1.9032,  -0.7550,   0.0000,  ...,   0.0000,   8.4346,  -5.1248],
             [  0.0000,   0.1946,   1.7132,  ...,   0.1049,   0.0000,   0.0000],
             ...,
             [-12.1167,  -2.8028,  -9.4789,  ...,  -7.2485,   8.0067,  -7.4645],
             [ -0.5737,  -3.4279,  -2.4972,  ...,  -3.8395,   7.2618,  -7.5300],
             [ -0.0902,  -1.9239,  -0.6181,  ...,   1.2788,   5.7042,   0.0000]],
    
            [[  0.0000,  -3.5419,   0.0000,  ...,  -4.7302,   0.7809,  -0.7692],
             [ -4.9884,  -1.4948,   0.4247,  ...,  -3.1005,   0.0594,   0.2251],
             [ -5.3625,   0.0000,  -1.1701,  ...,  -2.4318,  -1.0293,   1.1975],
             ...,
             [ -9.5743,  -7.7568, -13.6071,  ...,  -5.8194,  12.7505,   0.0000],
             [ -9.5743,  -7.7568, -13.6071,  ...,  -5.8194,  12.7505,  -7.8113],
             [  0.0000,  -7.7568, -13.6071,  ...,  -5.8194,  12.7505,  -7.8113]],
    
            [[  0.7055,   2.6856,  -5.1106,  ...,   0.6471,  -5.7998,   7.7951],
             [  4.6700,   3.8121,  -4.1586,  ...,   0.9645,  -5.9060,   7.7761],
             [  0.0000,   1.8728,  -3.5463,  ...,   1.7811,  -5.1110,   8.0430],
             ...,
             [ -9.5743,   0.0000, -13.6071,  ...,  -5.8194,  12.7505,  -7.8113],
             [ -9.5743,  -7.7568, -13.6071,  ...,   0.0000,  12.7505,  -7.8113],
             [ -9.5743,  -7.7568, -13.6071,  ...,  -5.8194,  12.7505,  -7.8113]],
    
            ...,
    
            [[ -5.9051,   0.0000,   0.0000,  ...,  -1.7656,   1.2541,  -2.9259],
             [ -7.2387,   0.0000,  -1.9941,  ...,  -1.9837,   3.7928,  -3.0524],
             [-11.5061,   4.6892,  -2.9696,  ...,  -3.9930,   0.2395,  -1.5693],
             ...,
             [ -9.5743,  -7.7568, -13.6071,  ...,  -5.8194,  12.7505,  -7.8113],
             [ -9.5743,  -7.7568, -13.6071,  ...,   0.0000,  12.7505,  -7.8113],
             [ -9.5743,   0.0000, -13.6071,  ...,  -5.8194,  12.7505,  -7.8113]],
    
            [[ -4.0918,   1.2283,  -3.7218,  ...,  -4.6928,   4.6370,  -4.9392],
             [  0.0000,   2.6892,  -2.0451,  ...,  -0.4186,   3.7887,  -2.1638],
             [  0.0000,   3.7681,  -0.6485,  ...,  -1.5094,   1.3679,  -2.6088],
             ...,
             [ -9.5743,  -7.7568, -13.6071,  ...,  -5.8194,  12.7505,  -7.8113],
             [ -9.5743,  -7.7568, -13.6071,  ...,  -5.8194,  12.7505,  -7.8113],
             [ -9.5743,  -7.7568, -13.6071,  ...,  -5.8194,  12.7505,  -7.8113]],
    
            [[ -5.7208,   0.5752,  -3.4696,  ...,  -3.7705,  11.7755,   1.7834],
             [ -7.1437,   0.8723,  -4.2169,  ...,  -3.9519,  11.1012,   1.8506],
             [ -7.0842,   0.5833,  -3.3386,  ...,  -3.5101,  11.4416,   1.6952],
             ...,
             [ -9.5743,  -7.7568, -13.6071,  ...,   0.0000,   0.0000,  -7.8113],
             [ -9.5743,  -7.7568, -13.6071,  ...,  -5.8194,  12.7505,  -7.8113],
             [ -9.5743,  -7.7568, -13.6071,  ...,  -5.8194,  12.7505,   0.0000]]],
           device='cuda:0', grad_fn=<FusedDropoutBackward0>)

Feed Forward Module后

(Pdb) p x.shape
    torch.Size([14, 243, 512])
    (Pdb) p x
    
               7.2815e+00, -7.6011e+00],
             [ 3.1489e-02, -1.8858e+00, -4.5769e-01,  ...,  1.3952e+00,
               5.6306e+00, -3.0690e-02]],
    
            [[-6.3229e-03, -3.4414e+00, -4.1966e-02,  ..., -4.7639e+00,
               6.3168e-01, -8.2709e-01],
             [-4.9884e+00, -1.4948e+00,  4.0035e-01,  ..., -2.9774e+00,
              -6.4502e-03,  1.0855e-01],
             [-5.3848e+00,  2.1896e-02, -1.2273e+00,  ..., -2.4318e+00,
              -1.0881e+00,  1.1193e+00],
             ...,
             [-9.5743e+00, -7.6100e+00, -1.3619e+01,  ..., -5.6579e+00,
               1.2705e+01, -1.1259e-01],
             [-9.4011e+00, -7.7079e+00, -1.3600e+01,  ..., -5.7637e+00,
               1.2693e+01, -7.8329e+00],
             [ 1.6623e-01, -7.7410e+00, -1.3594e+01,  ..., -5.7195e+00,
               1.2776e+01, -7.9591e+00]],
    
            [[ 7.0553e-01,  2.7021e+00, -5.1429e+00,  ...,  6.9559e-01,
              -5.9345e+00,  7.7479e+00],
             [ 4.8407e+00,  3.8736e+00, -4.2463e+00,  ...,  1.0054e+00,
              -5.9870e+00,  7.6246e+00],
             [ 1.0486e-01,  1.9964e+00, -3.6064e+00,  ...,  1.7598e+00,
              -5.1110e+00,  7.9280e+00],
             ...,
             [-9.3324e+00,  1.0171e-01, -1.3625e+01,  ..., -5.7424e+00,
               1.2724e+01, -7.9598e+00],
             [-9.3546e+00, -7.6776e+00, -1.3659e+01,  ...,  0.0000e+00,
               1.2747e+01, -7.9354e+00],
             [-9.3427e+00, -7.7067e+00, -1.3743e+01,  ..., -5.8194e+00,
               1.2808e+01, -7.8516e+00]],
    
            [[-3.8864e+00,  1.2016e+00, -3.7942e+00,  ..., -4.6340e+00,
               4.6085e+00, -5.0171e+00],
             [ 1.6116e-01,  2.6320e+00, -2.1391e+00,  ..., -4.4683e-01,
               3.7133e+00, -2.2118e+00],
             [ 1.1952e-01,  3.7163e+00, -7.2823e-01,  ..., -1.5372e+00,
               1.3950e+00, -2.6756e+00],
             ...,
             [-9.3344e+00, -7.7682e+00, -1.3643e+01,  ..., -5.6883e+00,
               1.2767e+01, -7.9511e+00],
             [-9.4534e+00, -7.7568e+00, -1.3671e+01,  ..., -5.6858e+00,
               1.2750e+01, -7.9653e+00],
             [-9.3937e+00, -7.6900e+00, -1.3694e+01,  ..., -5.7628e+00,
               1.2702e+01, -7.8852e+00]],
    
            [[-5.4615e+00,  5.6451e-01, -3.4696e+00,  ..., -3.7742e+00,
               1.1608e+01,  1.5476e+00],
             [-7.1437e+00,  8.0560e-01, -4.2403e+00,  ..., -3.9945e+00,
               1.0934e+01,  1.6021e+00],
             [-6.8293e+00,  5.6961e-01, -3.4408e+00,  ..., -3.4578e+00,
               1.1293e+01,  1.3934e+00],
             ...,
             [-9.3903e+00, -7.7132e+00, -1.3642e+01,  ...,  1.5418e-01,
              -6.0617e-02, -7.9958e+00],
             [-9.5743e+00, -7.7289e+00, -1.3659e+01,  ..., -5.7391e+00,
               1.2684e+01, -7.9137e+00],
             [-9.3177e+00, -7.7448e+00, -1.3654e+01,  ..., -5.8194e+00,
               1.2788e+01, -1.5587e-01]]], device='cuda:0', grad_fn=<AddBackward0>)

可以看到经过Feed Forward Module之后数据维度并不发生改变，但经过layer norm等处理后，数据的大小发生了改变。同理，在另一个Feed Forward Module中也是同样的操作。

### 3.4、Multi-Head Self Attention Module Code ###

代码区：Multi-Head Self Attention Module: encoder.py–>encoder\_layer.py–>attention.py

**这一部分会开一篇新的来仔细说说**

**a**、结构  
Multi-Head Self Attention Module主要由带有相对位置编码多头自注意力（Multi-Head Self Attention with relative positional embedding）组成。实验中用到了头为8的自注意力模块。  
![在这里插入图片描述][2ff3ab6cd573421a9a4b76e4641ffce6.png]

**b**、代码  
代码区域为attention.py这里，细节比较多，会开一篇新的说说。

Multi-Head Self Attention Module前

(Pdb) p x.shape
    torch.Size([14, 243, 512])
    (Pdb) p x
    
               1.2808e+01, -7.8516e+00]],
    
            [[-3.8864e+00,  1.2016e+00, -3.7942e+00,  ..., -4.6340e+00,
               4.6085e+00, -5.0171e+00],
             [ 1.6116e-01,  2.6320e+00, -2.1391e+00,  ..., -4.4683e-01,
               3.7133e+00, -2.2118e+00],
             [ 1.1952e-01,  3.7163e+00, -7.2823e-01,  ..., -1.5372e+00,
               1.3950e+00, -2.6756e+00],
             ...,
             [-9.3344e+00, -7.7682e+00, -1.3643e+01,  ..., -5.6883e+00,
               1.2767e+01, -7.9511e+00],
             [-9.4534e+00, -7.7568e+00, -1.3671e+01,  ..., -5.6858e+00,
               1.2750e+01, -7.9653e+00],
             [-9.3937e+00, -7.6900e+00, -1.3694e+01,  ..., -5.7628e+00,
               1.2702e+01, -7.8852e+00]],
    
            [[-5.4615e+00,  5.6451e-01, -3.4696e+00,  ..., -3.7742e+00,
               1.1608e+01,  1.5476e+00],
             [-7.1437e+00,  8.0560e-01, -4.2403e+00,  ..., -3.9945e+00,
               1.0934e+01,  1.6021e+00],
             [-6.8293e+00,  5.6961e-01, -3.4408e+00,  ..., -3.4578e+00,
               1.1293e+01,  1.3934e+00],
             ...,
             [-9.3903e+00, -7.7132e+00, -1.3642e+01,  ...,  1.5418e-01,
              -6.0617e-02, -7.9958e+00],
             [-9.5743e+00, -7.7289e+00, -1.3659e+01,  ..., -5.7391e+00,
               1.2684e+01, -7.9137e+00],
             [-9.3177e+00, -7.7448e+00, -1.3654e+01,  ..., -5.8194e+00,
               1.2788e+01, -1.5587e-01]]], device='cuda:0', grad_fn=<AddBackward0>)

Multi-Head Self Attention Module后

(Pdb) p x.shape
    torch.Size([14, 243, 512])
    (Pdb) p x
    tensor([[[-1.0271e+00, -1.9463e+00,  1.7051e+00,  ..., -6.3574e-01,
               8.7831e+00, -4.8254e+00],
             [-1.7611e+00, -7.7109e-01, -9.2721e-02,  ..., -6.5186e-02,
               8.9391e+00, -5.1968e+00],
             [-1.4328e-01, -1.5383e-01,  2.4761e+00,  ..., -4.7934e-02,
               5.2762e-01, -4.3704e-02],
             ...,
             [-1.2330e+01, -3.0957e+00, -8.8320e+00,  ..., -7.2978e+00,
               8.4464e+00, -7.7051e+00],
             [-5.8125e-01, -3.7171e+00, -1.6540e+00,  ..., -3.8881e+00,
               7.7412e+00, -7.6324e+00],
             [ 3.1489e-02, -1.8858e+00,  3.3470e-01,  ...,  1.2515e+00,
               5.6306e+00, -4.4427e-02]],
    
            [[-6.3229e-03, -3.3921e+00,  4.8096e-01,  ..., -4.8776e+00,
               7.2262e-01, -1.3703e+00],
             [-5.3446e+00, -1.4451e+00,  9.1622e-01,  ..., -3.1011e+00,
               7.9614e-02, -4.2886e-01],
             [-5.7261e+00,  6.9976e-02, -7.1110e-01,  ..., -2.5597e+00,
              -9.9130e-01,  5.8256e-01],
             ...,
             [-9.9296e+00, -7.5594e+00, -1.3619e+01,  ..., -5.6579e+00,
               1.2793e+01, -6.5060e-01],
             [-9.7484e+00, -7.6608e+00, -1.3076e+01,  ..., -5.8727e+00,
               1.2796e+01, -8.3650e+00],
               1.2876e+01, -8.1519e+00]],
    
            [[-3.8864e+00,  8.1121e-01, -3.2379e+00,  ..., -4.6340e+00,
               4.5953e+00, -5.4232e+00],
             [-1.0173e-01,  2.2622e+00, -1.6064e+00,  ..., -7.8868e-01,
               3.7027e+00, -2.6075e+00],
             [-1.6345e-01,  3.3597e+00, -1.9494e-01,  ..., -1.8532e+00,
               1.3767e+00, -3.0669e+00],
             ...,
             [-9.6426e+00, -8.1642e+00, -1.3643e+01,  ..., -6.0016e+00,
               1.2767e+01, -8.3592e+00],
             [-9.6984e+00, -8.1595e+00, -1.3115e+01,  ..., -6.0114e+00,
               1.2719e+01, -8.3549e+00],
             [-9.6625e+00, -8.0812e+00, -1.3120e+01,  ..., -6.0365e+00,
               1.2702e+01, -8.2902e+00]],
    
            [[-5.4615e+00,  7.9916e-01, -3.4321e+00,  ..., -3.8729e+00,
               1.1705e+01,  1.5476e+00],
             [-6.9851e+00,  1.0249e+00, -4.2206e+00,  ..., -4.0995e+00,
               1.1052e+01,  1.2745e+00],
             [-6.6968e+00,  8.0111e-01, -3.4151e+00,  ..., -3.5240e+00,
               1.1393e+01,  1.0780e+00],
             ...,
             [-9.2554e+00, -7.4362e+00, -1.3618e+01,  ...,  1.5418e-01,
               5.9763e-02, -8.3539e+00],
             [-9.4399e+00, -7.4798e+00, -1.3644e+01,  ..., -5.8081e+00,
               1.2792e+01, -8.1860e+00],
             [-9.1668e+00, -7.5268e+00, -1.3658e+01,  ..., -5.9279e+00,
               1.2788e+01, -4.8667e-01]]], device='cuda:0', grad_fn=<AddBackward0>)

可以看到经过Multi-Head Self Attention Module数据维度不发生改变，只是做了一系列的数学计算。

### 3.5、Convolution Module Code ###

代码区：Convolution Module: encoder.py–>encoder\_layer.py–>convolution.py

**a**、结构  
从图中可以看到，Convolution Module包括两个逐点卷积和一个深度卷积，第一个激活函数使用Glu，第二个激活函数使用Swish，中间进行了一次batch norm。逐点卷积为卷积核为1×1，步长为1的Conv1d，深度卷积为卷积核为31×31，步长为1的Conv1d。

![在这里插入图片描述][2a786e7174ba41ccb5530d5af1bc7e72.png]  
**b**、代码

x = self.pointwise_conv1(x)  
             x = nn.functional.glu(x, dim=1)  
             # 1D Depthwise Conv
             x = self.depthwise_conv(x)
             if self.use_layer_norm:
                 x = x.transpose(1, 2)
             x = self.activation(self.norm(x))
             if self.use_layer_norm:
                 x = x.transpose(1, 2)
             x = self.pointwise_conv2(x)
             # mask batch padding
             if mask_pad.size(2) > 0:  
                 x.masked_fill_(~mask_pad, 0.0)

Convolution Module前

(Pdb) p x.shape
    torch.Size([14, 243, 512])
    (Pdb) p x
    
               1.2876e+01, -8.1519e+00]],
    
            [[-3.8864e+00,  8.1121e-01, -3.2379e+00,  ..., -4.6340e+00,
               4.5953e+00, -5.4232e+00],
             [-1.0173e-01,  2.2622e+00, -1.6064e+00,  ..., -7.8868e-01,
               3.7027e+00, -2.6075e+00],
             [-1.6345e-01,  3.3597e+00, -1.9494e-01,  ..., -1.8532e+00,
               1.3767e+00, -3.0669e+00],
             ...,
             [-9.6426e+00, -8.1642e+00, -1.3643e+01,  ..., -6.0016e+00,
               1.2767e+01, -8.3592e+00],
             [-9.6984e+00, -8.1595e+00, -1.3115e+01,  ..., -6.0114e+00,
               1.2719e+01, -8.3549e+00],
             [-9.6625e+00, -8.0812e+00, -1.3120e+01,  ..., -6.0365e+00,
               1.2702e+01, -8.2902e+00]],
    
            [[-5.4615e+00,  7.9916e-01, -3.4321e+00,  ..., -3.8729e+00,
               1.1705e+01,  1.5476e+00],
             [-6.9851e+00,  1.0249e+00, -4.2206e+00,  ..., -4.0995e+00,
               1.1052e+01,  1.2745e+00],
             [-6.6968e+00,  8.0111e-01, -3.4151e+00,  ..., -3.5240e+00,
               1.1393e+01,  1.0780e+00],
             ...,
             [-9.2554e+00, -7.4362e+00, -1.3618e+01,  ...,  1.5418e-01,
               5.9763e-02, -8.3539e+00],
             [-9.4399e+00, -7.4798e+00, -1.3644e+01,  ..., -5.8081e+00,
               1.2792e+01, -8.1860e+00],
             [-9.1668e+00, -7.5268e+00, -1.3658e+01,  ..., -5.9279e+00,
               1.2788e+01, -4.8667e-01]]], device='cuda:0', grad_fn=<AddBackward0>)

Convolution Module后

(Pdb) p x.shape
    torch.Size([14, 243, 512])
    (Pdb) p x
    
             [ -9.7045,  -8.1045, -13.0602,  ...,  -0.5921,  12.8330,  -8.4895],
             [ -9.6829,  -8.1501, -13.1456,  ...,  -6.4132,  12.8319,  -8.3690]],
    
            ...,
    
            [[ -6.7102,   0.1239,   0.1389,  ...,  -1.9504,   1.5525,  -3.0805],
             [ -7.8718,  -0.4553,  -1.8495,  ...,  -1.8900,   4.1916,  -3.3994],
             [-11.9641,   4.3714,  -2.5728,  ...,  -4.5918,   0.2353,  -1.9963],
             ...,
             [ -9.9326,  -8.3047, -12.9756,  ...,  -5.7134,  12.7981,  -8.2065],
             [ -9.8994,  -8.2996, -13.6337,  ...,   0.1135,  12.7336,  -8.2202],
             [ -9.7145,  -0.6672, -12.9155,  ...,  -5.7244,  12.8760,  -8.1519]],
    
            [[ -3.7198,   1.2760,  -3.5933,  ...,  -4.8444,   4.7165,  -5.4366],
             [ -0.0161,   2.6363,  -1.8160,  ...,  -1.1275,   3.7269,  -2.8670],
             [  0.0800,   3.8032,  -0.1034,  ...,  -1.9010,   1.3917,  -3.0669],
             ...,
             [ -9.6426,  -8.1642, -13.6431,  ...,  -6.0016,  12.7671,  -8.3592],
             [ -9.6984,  -8.1595, -13.1149,  ...,  -6.0114,  12.7185,  -8.3549],
             [ -9.6625,  -8.0812, -13.1198,  ...,  -6.0365,  12.7020,  -8.2902]],
    
            [[ -5.7886,   1.3010,  -3.4402,  ...,  -3.9976,  12.0589,   1.7659],
             [ -6.8290,   1.5354,  -4.4869,  ...,  -4.4033,  11.0707,   1.5978],
             [ -6.7460,   0.8011,  -3.4777,  ...,  -3.5240,  11.3933,   1.4383],
             ...,
             [ -9.2554,  -7.4362, -13.6181,  ...,   0.1542,   0.0598,  -8.3539],
             [ -9.4399,  -7.4798, -13.6437,  ...,  -5.8081,  12.7918,  -8.1860],
             [ -9.1668,  -7.5268, -13.6582,  ...,  -5.9279,  12.7876,  -0.4867]]],
           device='cuda:0', grad_fn=<AddBackward0>)

可以看到经过convolution之后数据维度未发生改变，数据大小在经过三次卷积与其他操作后发生了改变

## 4、总结 ##

以上为Conformer作为encoder结构在WeNet中的使用，wenet中总体结构与conformer论文一致，但是由于输入的语音数据长度不等的问题，因此在处理数据时会引入mask的操作，mask也是很重要的，这里可以结合代码进行查看。

减小参数量：Conformer结构参数量很大，未来可以考虑对conformer中的Multi-Head Self Attention Module与Convolution Module部分进行优化，减少参数量以减小模型大小。

优化模型效果：confomerblock中结构都为前接Layernorm后接Dropout，且带有残差连接，未来可考虑对残差连接进行修改，能不能提高识别准确率？或者对于loss而言，是否有别的改进？

针对wenet中的Conformer还有很多可以改进的地方，期待大家评论一起讨论。

[b88ec25747b54b9287a94310a2bf61cc.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/06/8ab61c763ed1466cb0e5e8107b6e9f5d.png
[175e893146d543e59ebaa0c0ced8a589.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/06/51ac6ec4c5af4cdd95a50f2ee60d7ff6.png
[a03cc96b70404c0cab73e73d7ce1d223.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/06/5dab359f05b143f7b9ac1e648f9b1d61.png
[2ff3ab6cd573421a9a4b76e4641ffce6.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/06/d6b6f2b14bd04b8892c840d33d320f8b.png
[2a786e7174ba41ccb5530d5af1bc7e72.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/06/97f7440b2a7b4acea89e77527b8cd710.png