torch构建常见网络层

本是古典何须时尚 2023-06-15 07:23 15阅读 0赞

### 文章目录 ###

*   *  conv层：
     *  pooling 层：
     *  BatchNorm层
     *  矩阵运算
     *  dropout层
     *  relu层
     *  interpolate实现插值和上采样
     *  自定义函数为nn.module
     *  mobilenet v1- depthwise层
     *  residual模块
     *  fire\_moudle模块
     *  mobilenet v2 InvertedResidual 模块：
     *  depthwise conv实现
     *  初始化权重方法一：
     *  初始化权重方法二：
     *  mobilenet v3 block模块
     *  FPN模块
     *  ChannelShuffle模块

博主汇总一下经常用到的网络层，方便记忆哈。参数的意思很好理解，就不注释了，有问题留言哈，希望对您有用！

## conv层： ##

nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True)

## pooling 层： ##

nn.MaxPool2d(kernel_size, stride=None, padding=0, dilation=1, return_indices=False, ceil_mode=False)

## BatchNorm层 ##

对于需要预训练模型的网络可以去掉pretrain模型，使用batchnorm修正数据流  
参考[ScratchDet: Training Single-Shot Object Detectors from Scratch][ScratchDet_ Training Single-Shot Object Detectors from Scratch]

nn.BatchNorm2d(out)

## 矩阵运算 ##

torch.mul(a, b)是矩阵a和b对应位相乘，a和b的维度必须相等，比如a的维度是(1, 2)，b的维度是(1, 2)，返回的仍是(1, 2)的矩阵
    torch.mm(a, b)是矩阵a和b矩阵相乘，比如a的维度是(1, 2)，b的维度是(2, 3)，返回的就是(1, 3)

## dropout层 ##

dropout = nn.Dropout2d(p)   # p信道随机稀疏率 0.03

## relu层 ##

batchnorm 后接relu

nn.ReLU(inplace=True)

## interpolate实现插值和上采样 ##

torch.nn.functional.interpolate(input, size=None, scale_factor=None, mode='nearest', align_corners=None)

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zODYzMjI0Ng_size_16_color_FFFFFF_t_70]  
**input (Tensor)** – 输入张量  
**size (int or Tuple\[int\] or Tuple\[int, int\] or Tuple\[int, int, int\])** –输出大小.  
**scale\_factor (float or Tuple\[float\])** – 指定输出为输入的多少倍数。如果输入为tuple，其也要制定为tuple类型

**mode (str)** – 可使用的上采样算法，有’nearest’, ‘linear’, ‘bilinear’, ‘bicubic’ , ‘trilinear’和’area’. 默认使用’nearest’

**align\_corners (bool, optional)** –几何上，我们认为输入和输出的像素是正方形，而不是点。如果设置为True，则输入和输出张量由其角像素的中心点对齐，从而保留角像素处的值。如果设置为False，则输入和输出张量由它们的角像素的角点对齐，插值使用边界外值的边值填充;当scale\_factor保持不变时，使该操作独立于输入大小。仅当使用的算法为’linear’, ‘bilinear’, 'bilinear’or 'trilinear’时可以使用。默认设置为False

## 自定义函数为nn.module ##

class hsigmoid(nn.Module):
        def forward(self, x):
            out = F.relu6(x + 3, inplace=True) / 6
            return out
    #moudlelist引用直接加 hsigmoid()

## mobilenet v1- depthwise层 ##

（group=sr，表示分组卷积，将输入channel分成  
 c h a n n e l / s r channel/sr channel/sr组输出） 这里要求inp，和oup都能整除sr

def conv_dw(inp, oup, stride):
                return nn.Sequential(
                    nn.Conv2d(inp, inp, 3, stride, 1, groups=inp, bias=False),
                    nn.BatchNorm2d(inp),
                    nn.ReLU(inplace=True),
        
                    nn.Conv2d(inp, oup, 1, 1, 0, bias=False),
                    nn.BatchNorm2d(oup),
                    nn.ReLU(inplace=True),
                )

## residual模块 ##

class residual(nn.Module):
        def __init__(self, inp_dim, out_dim, k=3, stride=1):
            super(residual, self).__init__()
            p = (k - 1) // 2
    
            self.conv1 = nn.Conv2d(inp_dim, out_dim, (k, k), padding=(p, p), stride=(stride, stride), bias=False)
            self.bn1   = nn.BatchNorm2d(out_dim)
            self.relu1 = nn.ReLU(inplace=True)
    
            self.conv2 = nn.Conv2d(out_dim, out_dim, (k, k), padding=(p, p), bias=False)        
            self.bn2   = nn.BatchNorm2d(out_dim)
            
            self.skip  = nn.Sequential(
                nn.Conv2d(inp_dim, out_dim, (1, 1), stride=(stride, stride), bias=False),
                nn.BatchNorm2d(out_dim)
            ) if stride != 1 or inp_dim != out_dim else nn.Sequential()
            self.relu  = nn.ReLU(inplace=True)
    
        def forward(self, x):
            conv1 = self.conv1(x)
            bn1   = self.bn1(conv1)
            relu1 = self.relu1(bn1)
    
            conv2 = self.conv2(relu1)
            bn2   = self.bn2(conv2)
    
            skip  = self.skip(x)
            return self.relu(bn2 + skip)

## fire\_moudle模块 ##

一种使用mobilenet group机制改进residual模块，shrink网络参数量  
group=sr机制将维度缩小 1 / s r 1/sr 1/sr倍，然后torch.cat机制融合拼接，回到原来大小

class fire_module(nn.Module):
        def __init__(self, inp_dim, out_dim, sr=2, stride=1):
            super(fire_module, self).__init__()
            self.conv1    = nn.Conv2d(inp_dim, out_dim // sr, kernel_size=1, stride=1, bias=False)
            self.bn1      = nn.BatchNorm2d(out_dim // sr)
            self.conv_1x1 = nn.Conv2d(out_dim // sr, out_dim // 2, kernel_size=1, stride=stride, bias=False)
            self.conv_3x3 = nn.Conv2d(out_dim // sr, out_dim // 2, kernel_size=3, padding=1, 
                                      stride=stride, groups=out_dim // sr, bias=False)
            self.bn2      = nn.BatchNorm2d(out_dim)
            self.skip     = (stride == 1 and inp_dim == out_dim)
            self.relu     = nn.ReLU(inplace=True)
    
        def forward(self, x):
            conv1 = self.conv1(x)
            bn1   = self.bn1(conv1)
            conv2 = torch.cat((self.conv_1x1(bn1), self.conv_3x3(bn1)), 1)
            bn2   = self.bn2(conv2)
            if self.skip:
                return self.relu(bn2 + x)
            else:
                return self.relu(bn2)

## mobilenet v2 InvertedResidual 模块： ##

这个模块与mobilenet v1 和residual差别请看 [这里][Link 1]  
该模块先1\*1扩张--------》然后depthwise  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zODYzMjI0Ng_size_16_color_FFFFFF_t_70 1]

class InvertedResidual(nn.Module):
        def __init__(self, inp, oup, stride, expand_ratio):
            super(InvertedResidual, self).__init__()
            self.stride = stride
            assert stride in [1, 2]
    
            hidden_dim = int(inp * expand_ratio)
            self.use_res_connect = self.stride == 1 and inp == oup
    
            if expand_ratio == 1:
                self.conv = nn.Sequential(
                    # dw
                    nn.Conv2d(hidden_dim, hidden_dim, 3, stride, 1, groups=hidden_dim, bias=False),
                    nn.BatchNorm2d(hidden_dim),
                    nn.ReLU6(inplace=True),
                    # pw-linear
                    nn.Conv2d(hidden_dim, oup, 1, 1, 0, bias=False),
                    nn.BatchNorm2d(oup),
                )
            else:
                self.conv = nn.Sequential(
                    # pw
                    nn.Conv2d(inp, hidden_dim, 1, 1, 0, bias=False),
                    nn.BatchNorm2d(hidden_dim),
                    nn.ReLU6(inplace=True),
                    # dw
                    nn.Conv2d(hidden_dim, hidden_dim, 3, stride, 1, groups=hidden_dim, bias=False),
                    nn.BatchNorm2d(hidden_dim),
                    nn.ReLU6(inplace=True),
                    # pw-linear
                    nn.Conv2d(hidden_dim, oup, 1, 1, 0, bias=False),
                    nn.BatchNorm2d(oup),
                )
    
        def forward(self, x):
            if self.use_res_connect:
                return x + self.conv(x)
            else:
                return self.conv(x)

## depthwise conv实现 ##

Depthwise Convolution的一个卷积核负责一个通道，一个通道只被一个卷积核卷积。上面所提到的常规卷积每个卷积核是同时操作输入图片的每个通道。  
同样是对于一张5×5像素、三通道彩色输入图片（shape为5×5×3），Depthwise Convolution首先经过第一次卷积运算，不同于上面的常规卷积，DW完全是在二维平面内进行。卷积核的数量与上一层的通道数相同（通道和卷积核一一对应）。所以一个三通道的图像经过运算后生成了3个Feature map(如果有same padding则尺寸与输入层相同为5×5)，如下图所示。

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zODYzMjI0Ng_size_16_color_FFFFFF_t_70 2]

torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1,bias=True)

其中，输入输出，核的size，步长，补零都不用说了，dilation是关于卷积核的，不讲了，groups就是实现depthwise conv的关键，默认为1，意思是将输入分为一组，此时是常规卷积，当将其设为in\_channels时，意思是将输入的每一个通道作为一组，然后分别对其卷积，输出通道数为k，最后再将每组的输出串联，最后通道数为in\_channels\*K。

## 初始化权重方法一： ##

def _initialize_weights(self):
            for m in self.modules():
                if isinstance(m, nn.Conv2d):
                    n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                    m.weight.data.normal_(0, math.sqrt(2. / n))
                    if m.bias is not None:
                        m.bias.data.zero_()
                elif isinstance(m, nn.BatchNorm2d):
                    m.weight.data.fill_(1)
                    m.bias.data.zero_()
                elif isinstance(m, nn.Linear):
                    n = m.weight.size(1)
                    m.weight.data.normal_(0, 0.01)
                    m.bias.data.zero_()

## 初始化权重方法二： ##

在这里插入代码片

## mobilenet v3 block模块 ##

block模块 借鉴mobilenet v2, 先1X1扩张------->depthwise, 还使用了一种Squeeze-and-Excite 技术来实现跳跃连接。代码自定义relu6的nn.moudle ，然后semodule里面的nn.adaptiveavgpool2d 将特征图拉伸成1\*1  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zODYzMjI0Ng_size_16_color_FFFFFF_t_70 3]

class hswish(nn.Module):
        def forward(self, x):
            out = x * F.relu6(x + 3, inplace=True) / 6
            return out
    
    
    class hsigmoid(nn.Module):
        def forward(self, x):
            out = F.relu6(x + 3, inplace=True) / 6
            return out
    
    
    class SeModule(nn.Module):
        def __init__(self, in_size, reduction=4):
            super(SeModule, self).__init__()
            self.se = nn.Sequential(
                nn.AdaptiveAvgPool2d(1),
                nn.Conv2d(in_size, in_size // reduction, kernel_size=1, stride=1, padding=0, bias=False),
                nn.BatchNorm2d(in_size // reduction),
                nn.ReLU(inplace=True),
                nn.Conv2d(in_size // reduction, in_size, kernel_size=1, stride=1, padding=0, bias=False),
                nn.BatchNorm2d(in_size),
                hsigmoid()
            )
    
        def forward(self, x):
            return x * self.se(x)
    
    
    class Block(nn.Module):
        '''expand + depthwise + pointwise'''
        def __init__(self, kernel_size, in_size, expand_size, out_size, nolinear, semodule, stride):
            super(Block, self).__init__()
            self.stride = stride
            self.se = semodule
    
            self.conv1 = nn.Conv2d(in_size, expand_size, kernel_size=1, stride=1, padding=0, bias=False)
            self.bn1 = nn.BatchNorm2d(expand_size)
            self.nolinear1 = nolinear
            self.conv2 = nn.Conv2d(expand_size, expand_size, kernel_size=kernel_size, stride=stride, padding=kernel_size//2, groups=expand_size, bias=False)
            self.bn2 = nn.BatchNorm2d(expand_size)
            self.nolinear2 = nolinear
            self.conv3 = nn.Conv2d(expand_size, out_size, kernel_size=1, stride=1, padding=0, bias=False)
            self.bn3 = nn.BatchNorm2d(out_size)
    
            self.shortcut = nn.Sequential()
            if stride == 1 and in_size != out_size:
                self.shortcut = nn.Sequential(
                    nn.Conv2d(in_size, out_size, kernel_size=1, stride=1, padding=0, bias=False),
                    nn.BatchNorm2d(out_size),
                )
    
        def forward(self, x):
            out = self.nolinear1(self.bn1(self.conv1(x)))
            out = self.nolinear2(self.bn2(self.conv2(out)))
            out = self.bn3(self.conv3(out))
            if self.se != None:
                out = self.se(out)
            out = out + self.shortcut(x) if self.stride==1 else out
            return out

## FPN模块 ##

class FPN(nn.Module):
        """ Module that adds FPN on top of a list of feature maps. The feature maps are currently supposed to be in increasing depth order, and must be consecutive """
    
        def __init__(  #in_channels_list 输入的是P3-P5的feature map
            self, in_channels_list, out_channels, conv_block, top_blocks=None
        ):
            """ Arguments: in_channels_list (list[int]): number of channels for each feature map that will be fed out_channels (int): number of channels of the FPN representation top_blocks (nn.Module or None): if provided, an extra operation will be performed on the output of the last (smallest resolution) FPN output, and the result will extend the result list """
            super(FPN, self).__init__()
            self.inner_blocks = []
            self.layer_blocks = []
            for idx, in_channels in enumerate(in_channels_list, 1):
                inner_block = "fpn_inner{}".format(idx)
                layer_block = "fpn_layer{}".format(idx)
    
                if in_channels == 0:
                    continue
                inner_block_module = conv_block(in_channels, out_channels, 1)
                layer_block_module = conv_block(out_channels, out_channels, 3, 1)
                self.add_module(inner_block, inner_block_module)
                self.add_module(layer_block, layer_block_module)
                self.inner_blocks.append(inner_block)
                self.layer_blocks.append(layer_block)
            self.top_blocks = top_blocks
    
        def forward(self, x):
            """ Arguments: x (list[Tensor]): feature maps for each feature level. Returns: results (tuple[Tensor]): feature maps after FPN layers. They are ordered from highest resolution first. """
            last_inner = getattr(self, self.inner_blocks[-1])(x[-1])
            results = []
            results.append(getattr(self, self.layer_blocks[-1])(last_inner))
            for feature, inner_block, layer_block in zip(
                x[:-1][::-1], self.inner_blocks[:-1][::-1], self.layer_blocks[:-1][::-1]
            ):
                if not inner_block:
                    continue
                # inner_top_down = F.interpolate(last_inner, scale_factor=2, mode="nearest")
                inner_lateral = getattr(self, inner_block)(feature)
                inner_top_down = F.interpolate(
                    last_inner, size=(int(inner_lateral.shape[-2]), int(inner_lateral.shape[-1])),
                    mode='nearest'
                )
                last_inner = inner_lateral + inner_top_down
                results.insert(0, getattr(self, layer_block)(last_inner))
    
            if isinstance(self.top_blocks, LastLevelP6P7):
                last_results = self.top_blocks(x[-1], results[-1])
                results.extend(last_results)
            elif isinstance(self.top_blocks, LastLevelMaxPool):
                last_results = self.top_blocks(results[-1])
                results.extend(last_results)
    
            return tuple(results)  
    
    
    class LastLevelMaxPool(nn.Module):
        def forward(self, x):
            return [F.max_pool2d(x, 1, 2, 0)]
    
    
    class LastLevelP6P7(nn.Module):
        """ This module is used in RetinaNet to generate extra layers, P6 and P7. """
        def __init__(self, in_channels, out_channels):
            super(LastLevelP6P7, self).__init__()
            self.p6 = nn.Conv2d(in_channels, out_channels, 3, 2, 1)
            self.p7 = nn.Conv2d(out_channels, out_channels, 3, 2, 1)
            for module in [self.p6, self.p7]:
                nn.init.kaiming_uniform_(module.weight, a=1)
                nn.init.constant_(module.bias, 0)
            self.use_P5 = in_channels == out_channels
    
        def forward(self, c5, p5):
            x = p5 if self.use_P5 else c5
            p6 = self.p6(x)
            p7 = self.p7(F.relu(p6))
            return [p6, p7]

## ChannelShuffle模块 ##

class ChannelShuffle(nn.Module):
        def __init__(self, groups):
            super(ChannelShuffle, self).__init__()
            self.groups = groups
    
        def forward(self, x):
            """Channel shuffle: [N,C,H,W] -> [N,g,C/g,H,W] -> [N,C/g,g,H,w] -> [N,C,H,W]"""
            N, C, H, W = x.size()
            g = self.groups
            assert C % g == 0, "Incompatible group size {} for input channel {}".format(
                g, C
            )
            return (
                x.view(N, g, int(C / g), H, W)
                .permute(0, 2, 1, 3, 4)
                .contiguous()
                .view(N, C, H, W)
            )

[ScratchDet_ Training Single-Shot Object Detectors from Scratch]: https://arxiv.org/abs/1810.08425
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zODYzMjI0Ng_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200113140029217.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zODYzMjI0Ng==,size_16,color_FFFFFF,t_70
[Link 1]: https://blog.csdn.net/u011995719/article/details/79135818
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zODYzMjI0Ng_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191122143206497.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zODYzMjI0Ng==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zODYzMjI0Ng_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200113110426430.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zODYzMjI0Ng==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zODYzMjI0Ng_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/201911221429490.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zODYzMjI0Ng==,size_16,color_FFFFFF,t_70