tensorflow 2.0 深度学习 (第三部分卷积神经网络 part2)

电玩女神 2023-07-02 05:22 22阅读 0赞

# ![20191009191333910.png][] #

# [日萌社][Link 1] #

[人工智能AI：Keras PyTorch MXNet TensorFlow PaddlePaddle 深度学习实战（不定时更新）][AI_Keras PyTorch MXNet TensorFlow PaddlePaddle]

--------------------

[tensorflow 2.0 深度学习（第一部分 part1）][tensorflow 2.0 _ part1]

[tensorflow 2.0 深度学习（第一部分 part2）][tensorflow 2.0 _ part2]

[tensorflow 2.0 深度学习（第一部分 part3）][tensorflow 2.0 _ part3]

[tensorflow 2.0 深度学习（第二部分 part1）][tensorflow 2.0 _ part1 1]

[tensorflow 2.0 深度学习（第二部分 part2）][tensorflow 2.0 _ part2 1]

[tensorflow 2.0 深度学习（第二部分 part3）][tensorflow 2.0 _ part3 1]

[tensorflow 2.0 深度学习 (第三部分 卷积神经网络 part1)][tensorflow 2.0 _ _ _ part1]

[tensorflow 2.0 深度学习 (第三部分 卷积神经网络 part2)][tensorflow 2.0 _ _ _ part2]

[tensorflow 2.0 深度学习（第四部分 循环神经网络）][tensorflow 2.0 _]

[tensorflow 2.0 深度学习（第五部分 GAN生成神经网络 part1）][tensorflow 2.0 _ GAN_ part1]

[tensorflow 2.0 深度学习（第五部分 GAN生成神经网络 part2）][tensorflow 2.0 _ GAN_ part2]

[tensorflow 2.0 深度学习（第六部分 强化学习）][tensorflow 2.0 _ 1]

--------------------

# **各种经典卷积网络** #

--------------------

# **LeNet-5**![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70][] #

import tensorflow as tf
    help(tf.losses.categorical_crossentropy) 
    	查看categorical_crossentropy函数的默认参数列表和使用方法介绍
     	其中形参默认为from_logits=False，网络预测值y_pred 表示必须为经过了 Softmax函数的输出值。
    	当 from_logits 设置为 True 时，网络预测值y_pred 表示必须为还没经过 Softmax 函数的变量 z。
    	from_logits=True 标志位将softmax激活函数实现在损失函数中，便不需要手动添加softmax损失函数，提升数值计算稳定性。
    	from_logits 指的就是是否有经过Logistic函数，常见的Logistic函数包括Sigmoid、Softmax函数。
    
    Help on function categorical_crossentropy in module tensorflow.python.keras.losses:
    categorical_crossentropy(y_true, y_pred, from_logits=False, label_smoothing=0) 
        Computes the categorical crossentropy loss.
        Args:
          y_true: tensor of true targets.
          y_pred: tensor of predicted targets.
          from_logits: Whether `y_pred` is expected to be a logits tensor. By default,
            we assume that `y_pred` encodes a probability distribution.

1.BCE(Binary Cross-Entropy 实际即Sigmoid激活+CE)
    	1.import tensorflow as tf
    	  help(tf.losses.binary_crossentropy) 
    	  打印信息如下 binary_crossentropy(y_true, y_pred, from_logits=False, label_smoothing=0)
     
    	2.from tensorflow import keras
    	  help(keras.losses.BinaryCrossentropy()) 
    	  打印信息如下 class BinaryCrossentropy(LossFunctionWrapper)
    			BinaryCrossentropy(from_logits=False, label_smoothing=0, reduction='auto', name='binary_crossentropy')
     
    2.CCE(Categorical Cross-Entropy 实际即Softmax激活+CE) 
    	1.import tensorflow as tf
    	help(tf.losses.categorical_crossentropy) 
    	打印信息如下 categorical_crossentropy(y_true, y_pred, from_logits=False, label_smoothing=0)
    
    	2.from tensorflow import keras
    	  help(keras.losses.CategoricalCrossentropy()) 
    	  打印信息如下 class CategoricalCrossentropy(LossFunctionWrapper)
    			CategoricalCrossentropy(from_logits=False, label_smoothing=0, reduction='auto', name='categorical_crossentropy')
    
    3.不论是BCE(Binary Cross-Entropy)还是CCE(Categorical Cross-Entropy)，其中的from_logits参数都默认为False。
    	from_logits=False：BCE(Binary Cross-Entropy)损失函数不会内建Sigmoid激活函数运算，CCE(Categorical Cross-Entropy)损失函数不会内建Softmax激活函数运算。
    	from_logits=True：BCE(Binary Cross-Entropy)损失函数内建了Sigmoid激活函数运算，CCE(Categorical Cross-Entropy)损失函数内建了Softmax激活函数运算。
      	如果使用默认的from_logits=False，则网络输出层需要加上Sigmoid/Softmax激活函数运算。
    	如果使用 from_logits=True，则网络输出层无需加上Sigmoid/Softmax激活函数运算，因为要避免输出连续做两次Sigmoid/Softmax激活函数运算。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 1][]

--------------------

# **AlexNet**![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 2][] #

--------------------

# **VGG** #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 3][]

--------------------

# **GoogLeNet**![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 4][] #

--------------------

# **CIFAR10 与 VGG13 实战** #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 5][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 6][]

def preprocess(x, y):
        # 标准化：将数据映射到-1~1之间
        x = 2*tf.cast(x, dtype=tf.float32) / 255.-1
        y = tf.cast(y, dtype=tf.int32)
        return x,y
        
    # 在线下载，加载CIFAR10数据集
    (x,y), (x_test, y_test) = datasets.cifar100.load_data()
    # 删除y的一个维度，[b,1] => [b]
    y = tf.squeeze(y, axis=1)
    y_test = tf.squeeze(y_test, axis=1)
    # 打印训练集和测试集的形状
    print(x.shape, y.shape, x_test.shape, y_test.shape)
    # 构建训练集对象
    train_db = tf.data.Dataset.from_tensor_slices((x,y))
    train_db = train_db.shuffle(1000).map(preprocess).batch(128)
    # 构建测试集对象
    test_db = tf.data.Dataset.from_tensor_slices((x_test,y_test))
    test_db = test_db.map(preprocess).batch(128)
    # 从训练集中采样一个Batch，观察
    sample = next(iter(train_db))
    print('sample:', sample[0].shape, sample[1].shape, tf.reduce_min(sample[0]), tf.reduce_max(sample[0]))
    
    # 先创建包含多层的列表
    conv_layers = [ 
        # Conv-Conv-Pooling单元1
        # 64个3x3卷积核, 输入输出同大小
        layers.Conv2D(64, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(64, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        # 高宽减半
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same'),
    
        # Conv-Conv-Pooling单元2,输出通道提升至128，高宽大小减半
        layers.Conv2D(128, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(128, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        # 高宽减半
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same'),
    
        # Conv-Conv-Pooling单元3,输出通道提升至256，高宽大小减半
        layers.Conv2D(256, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(256, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        # 高宽减半
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same'),
    
        # Conv-Conv-Pooling单元4,输出通道提升至512，高宽大小减半
        layers.Conv2D(512, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(512, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        # 高宽减半
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same'),
    
        # Conv-Conv-Pooling单元5,输出通道提升至512，高宽大小减半
        layers.Conv2D(512, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(512, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        # 高宽减半
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same')
    ]
    
    # 利用前面创建的层列表构建网络容器
    conv_net = Sequential(conv_layers)
     
    # 创建3层全连接层子网络
    fc_net = Sequential([
        layers.Dense(256, activation=tf.nn.relu),
        layers.Dense(128, activation=tf.nn.relu),
        layers.Dense(100, activation=None),
    ])
     
    # build2个子网络，并打印网络参数信息
    conv_net.build(input_shape=[4, 32, 32, 3])
    fc_net.build(input_shape=[4, 512])
    conv_net.summary()
    fc_net.summary()
     
    # 列表合并，合并2个子网络的参数
    variables = conv_net.trainable_variables + fc_net.trainable_variables
    # 对所有参数求梯度
    grads = tape.gradient(loss, variables)
    # 自动更新
    optimizer.apply_gradients(zip(grads, variables))

import  tensorflow as tf
    from    tensorflow.keras import layers, optimizers, datasets, Sequential
    import  os
    os.environ['TF_CPP_MIN_LOG_LEVEL']='2'
    tf.random.set_seed(2345)
    
    # 5 units of conv + max pooling
    conv_layers = [ 
        # unit 1
        layers.Conv2D(64, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(64, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same'),
        # unit 2
        layers.Conv2D(128, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(128, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same'),
        # unit 3
        layers.Conv2D(256, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(256, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same'),
        # unit 4
        layers.Conv2D(512, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(512, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same'),
        # unit 5
        layers.Conv2D(512, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(512, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same')
    ]
    
    def preprocess(x, y):
        # 标准化：数据映射到[-1,1]之间
        x = 2*tf.cast(x, dtype=tf.float32) / 255.-1
        y = tf.cast(y, dtype=tf.int32)
        return x,y
    
    (x,y), (x_test, y_test) = datasets.cifar10.load_data()
    y = tf.squeeze(y, axis=1)
    y_test = tf.squeeze(y_test, axis=1)
    print(x.shape, y.shape, x_test.shape, y_test.shape)
    train_db = tf.data.Dataset.from_tensor_slices((x,y))
    train_db = train_db.shuffle(1000).map(preprocess).batch(128)
    test_db = tf.data.Dataset.from_tensor_slices((x_test,y_test))
    test_db = test_db.map(preprocess).batch(64)
    sample = next(iter(train_db))
    print('sample:', sample[0].shape, sample[1].shape,
          tf.reduce_min(sample[0]), tf.reduce_max(sample[0]))
    
    def main():
        # [b, 32, 32, 3] => [b, 1, 1, 512]
        conv_net = Sequential(conv_layers)
        fc_net = Sequential([
            layers.Dense(256, activation=tf.nn.relu),
            layers.Dense(128, activation=tf.nn.relu),
            layers.Dense(10, activation=None),
        ])
        conv_net.build(input_shape=[None, 32, 32, 3])
        fc_net.build(input_shape=[None, 512])
        conv_net.summary()
        fc_net.summary()
    
        optimizer = optimizers.Adam(lr=1e-4)
        # [1, 2] + [3, 4] => [1, 2, 3, 4]
        variables = conv_net.trainable_variables + fc_net.trainable_variables
    
        for epoch in range(50):
            for step, (x,y) in enumerate(train_db):
                with tf.GradientTape() as tape:
                    # [b, 32, 32, 3] => [b, 1, 1, 512]
                    out = conv_net(x)
                    # flatten, => [b, 512]
                    out = tf.reshape(out, [-1, 512])
                    # [b, 512] => [b, 10]
                    logits = fc_net(out)
                    # [b] => [b, 10]
                    y_onehot = tf.one_hot(y, depth=10)
                    # compute loss
                    loss = tf.losses.categorical_crossentropy(y_onehot, logits, from_logits=True)
                    loss = tf.reduce_mean(loss)
                grads = tape.gradient(loss, variables)
                optimizer.apply_gradients(zip(grads, variables))
                if step %100 == 0:
                    print(epoch, step, 'loss:', float(loss))
    
            total_num = 0
            total_correct = 0
            for x,y in test_db:
                out = conv_net(x)
                out = tf.reshape(out, [-1, 512])
                logits = fc_net(out)
                prob = tf.nn.softmax(logits, axis=1) #把输出转换为概率值
                pred = tf.argmax(prob, axis=1) #获取最大概率值对应的索引值作为类别号
                pred = tf.cast(pred, dtype=tf.int32)
                correct = tf.cast(tf.equal(pred, y), dtype=tf.int32) #比对最大概率值对应的索引值(类别号)和真实标签类别号是否一致
                correct = tf.reduce_sum(correct)
                total_num += x.shape[0]
                total_correct += int(correct)
            acc = total_correct / total_num
            print(epoch, 'acc:', acc)
    
    if __name__ == '__main__':
        main()

#  #

--------------------

# **深度残差网络 ResNet、DenseNet** #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 7][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 8][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 9][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 10][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 11][]

import  os
    import  tensorflow as tf
    import  numpy as np
    from    tensorflow import keras
     
    tf.random.set_seed(22)
    np.random.seed(22)
    os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
    assert tf.__version__.startswith('2.')
     
    (x_train, y_train), (x_test, y_test) = keras.datasets.fashion_mnist.load_data()
    x_train, x_test = x_train.astype(np.float32)/255., x_test.astype(np.float32)/255.
    # [b, 28, 28] => [b, 28, 28, 1]
    x_train, x_test = np.expand_dims(x_train, axis=3), np.expand_dims(x_test, axis=3)
    # one hot encode the labels. convert back to numpy as we cannot use a combination of numpy
    # and tensors as input to keras
    y_train_ohe = tf.one_hot(y_train, depth=10).numpy()
    y_test_ohe = tf.one_hot(y_test, depth=10).numpy()
    print(x_train.shape, y_train.shape) #(60000, 28, 28, 1) (60000,)
    print(x_test.shape, y_test.shape) #(10000, 28, 28, 1) (10000,)
    
    # 3x3 convolution
    def conv3x3(channels, stride=1, kernel=(3, 3)):
        return keras.layers.Conv2D(channels, kernel, strides=stride, padding='same', use_bias=False, kernel_initializer=tf.random_normal_initializer())
    
    class ResnetBlock(keras.Model):
        def __init__(self, channels, strides=1, residual_path=False):
            super(ResnetBlock, self).__init__()
            self.channels = channels
            self.strides = strides
            self.residual_path = residual_path
            self.conv1 = conv3x3(channels, strides)
            self.bn1 = keras.layers.BatchNormalization()
            self.conv2 = conv3x3(channels)
            self.bn2 = keras.layers.BatchNormalization()
            if residual_path:
                #表示使用strides=2的1x1的逐点卷积，实际即把上一个block块中最后一个layer层的输出作为残差连接，	
    	   #和当前block块中当前layer层的输出进行两者元素相加
                self.down_conv = conv3x3(channels, strides, kernel=(1, 1))
                self.down_bn = tf.keras.layers.BatchNormalization()
    
        def call(self, inputs, training=None):
            residual = inputs
            x = self.bn1(inputs, training=training)
            x = tf.nn.relu(x)
            x = self.conv1(x)
            x = self.bn2(x, training=training)
            x = tf.nn.relu(x)
            x = self.conv2(x)
            # this module can be added into self.
            # however, module in for can not be added.
            if self.residual_path:
                residual = self.down_bn(inputs, training=training)
                residual = tf.nn.relu(residual)
                residual = self.down_conv(residual)
            x = x + residual
            return x
    
    class ResNet(keras.Model):
        def __init__(self, block_list, num_classes, initial_filters=16, **kwargs):
            super(ResNet, self).__init__(**kwargs)
            self.num_blocks = len(block_list)
            self.block_list = block_list
            self.in_channels = initial_filters
            self.out_channels = initial_filters
            self.conv_initial = conv3x3(self.out_channels)
            self.blocks = keras.models.Sequential(name='dynamic-blocks')
    
            # build all the blocks
            #遍历每个block块
            for block_id in range(len(block_list)):
                #遍历每个block块中的layer层
                for layer_id in range(block_list[block_id]):
                    #只有当不为第一个block块的同时，为某个block块第一个layer层时，即只要添加新一个block块的第一个layer层时便会执行如下判断
     	       #当前程序只有在block_id为1同时layer_id为0，或者block_id为2同时layer_id为0时，即只要添加新一个block块的第一个layer层时便会执行如下判断
                    if block_id != 0 and layer_id == 0:
                        #因为此时out_channels*=2输出通道数翻倍了，所以设置strides=2把输出的高宽减半，同时设置了residual_path=True，
    		  #表示使用strides=2的1x1的逐点卷积，实际即把上一个block块中最后一个layer层的输出作为残差连接，
    		  #和当前block块中当前layer层的输出进行两者元素相加
                        block = ResnetBlock(self.out_channels, strides=2, residual_path=True)
                    #只要为第一个block块中的任意一个layer层时，或者说为非第一个block块中的非第一个layer层时，都会执行如下判断
                    #当前程序只有在block_id为0同时layer_id为0或1，或者block_id为1或2同时layer_id为1时，才会执行如下判断
                    else:
    		  #当前程序不会执行如下判断代码
                        if self.in_channels != self.out_channels:
                            residual_path = True
    		  #当前程序只有在block_id为0同时layer_id为0或1，或者block_id为1或2同时layer_id为1时，才会执行如下判断
                        else:
                            residual_path = False
    	 	  #当前程序的这句代码的参数residual_path只会为False
                        block = ResnetBlock(self.out_channels, residual_path=residual_path)
    
                    self.in_channels = self.out_channels
                    self.blocks.add(block)
                self.out_channels *= 2 #只要当添加新一个block块时，新block块的输出通道数便会翻倍
    
            self.final_bn = keras.layers.BatchNormalization()
            self.avg_pool = keras.layers.GlobalAveragePooling2D()
            self.fc = keras.layers.Dense(num_classes)
    
        def call(self, inputs, training=None):
            out = self.conv_initial(inputs)
            out = self.blocks(out, training=training)
            out = self.final_bn(out, training=training)
            out = tf.nn.relu(out)
            out = self.avg_pool(out)
            out = self.fc(out)
            return out
    
    def main():
        num_classes = 10 #类别数
        batch_size = 32
        epochs = 1
        # build model and optimizer
        #构建6个block块，并且每个block块有2个layer层
        model = ResNet([2, 2, 2], num_classes)
        model.compile(optimizer=keras.optimizers.Adam(0.001),
                      loss=keras.losses.CategoricalCrossentropy(from_logits=True),
                      metrics=['accuracy'])
        model.build(input_shape=(None, 28, 28, 1))
        print("Number of variables in the model :", len(model.variables))
        model.summary()
        # train
        model.fit(x_train, y_train_ohe, batch_size=batch_size, epochs=epochs, validation_data=(x_test, y_test_ohe), verbose=1)
        # evaluate on test set
        #evaluate评价验证时输出loss和accuracy
        scores = model.evaluate(x_test, y_test_ohe, batch_size, verbose=1)
        print("Final test loss and accuracy :", scores)
    
    if __name__ == '__main__':
        main()

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 12][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 13][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 14][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 15][]

import  tensorflow as tf
    from    tensorflow import keras
    from    tensorflow.keras import layers, Sequential
     
    class BasicBlock(layers.Layer):
        def __init__(self, filter_num, stride=1):
            super(BasicBlock, self).__init__()
            self.conv1 = layers.Conv2D(filter_num, (3, 3), strides=stride, padding='same')
            self.bn1 = layers.BatchNormalization()
            self.relu = layers.Activation('relu')
            self.conv2 = layers.Conv2D(filter_num, (3, 3), strides=1, padding='same')
            self.bn2 = layers.BatchNormalization()
    
            if stride != 1:
                self.downsample = Sequential()
                self.downsample.add(layers.Conv2D(filter_num, (1, 1), strides=stride))
            else:
                self.downsample = lambda x:x
    
        def call(self, inputs, training=None):
            # [b, h, w, c]
            out = self.conv1(inputs)
            out = self.bn1(out,training=training)
            out = self.relu(out)
            out = self.conv2(out)
            out = self.bn2(out,training=training)
            identity = self.downsample(inputs)
            output = layers.add([out, identity])
            output = tf.nn.relu(output)
            return output
     
    class ResNet(keras.Model):
        def __init__(self, layer_dims, num_classes=100): # [2, 2, 2, 2]
            super(ResNet, self).__init__()
            self.stem = Sequential([layers.Conv2D(64, (3, 3), strides=(1, 1)),
                                    layers.BatchNormalization(),
                                    layers.Activation('relu'),
                                    layers.MaxPool2D(pool_size=(2, 2), strides=(1, 1), padding='same')
                                    ])
            self.layer1 = self.build_resblock(64,  layer_dims[0])
            self.layer2 = self.build_resblock(128, layer_dims[1], stride=2)
            self.layer3 = self.build_resblock(256, layer_dims[2], stride=2)
            self.layer4 = self.build_resblock(512, layer_dims[3], stride=2)
            # output: [b, 512, h, w],
            self.avgpool = layers.GlobalAveragePooling2D()
            self.fc = layers.Dense(num_classes)
    
        def call(self, inputs, training=None):
            x = self.stem(inputs,training=training)
            x = self.layer1(x,training=training)
            x = self.layer2(x,training=training)
            x = self.layer3(x,training=training)
            x = self.layer4(x,training=training)
            # [b, c]
            x = self.avgpool(x)
            # [b, 100]
            x = self.fc(x)
            return x
    
        def build_resblock(self, filter_num, blocks, stride=1):
            res_blocks = Sequential()
            # may down sample
            res_blocks.add(BasicBlock(filter_num, stride))
            for _ in range(1, blocks):
                res_blocks.add(BasicBlock(filter_num, stride=1))
            return res_blocks
    
    def resnet18():
        return ResNet([2, 2, 2, 2])
    
    def resnet34():
        return ResNet([3, 4, 6, 3])

import  os
    os.environ['TF_CPP_MIN_LOG_LEVEL']='2'
    import  tensorflow as tf
    from    tensorflow.keras import layers, optimizers, datasets, Sequential 
    from    resnet import resnet18 
    tf.random.set_seed(2345)
    
    def preprocess(x, y):
        # [-1~1]
        x = tf.cast(x, dtype=tf.float32) / 255. - 0.5
        y = tf.cast(y, dtype=tf.int32)
        return x,y
    
    (x,y), (x_test, y_test) = datasets.cifar100.load_data()
    y = tf.squeeze(y, axis=1)
    y_test = tf.squeeze(y_test, axis=1)
    print(x.shape, y.shape, x_test.shape, y_test.shape)
    train_db = tf.data.Dataset.from_tensor_slices((x,y))
    train_db = train_db.shuffle(1000).map(preprocess).batch(512)
    test_db = tf.data.Dataset.from_tensor_slices((x_test,y_test))
    test_db = test_db.map(preprocess).batch(512)
    sample = next(iter(train_db))
    print('sample:', sample[0].shape, sample[1].shape, tf.reduce_min(sample[0]), tf.reduce_max(sample[0]))
    
    def main():
        # [b, 32, 32, 3] => [b, 1, 1, 512]
        model = resnet18()
        model.build(input_shape=(None, 32, 32, 3))
        model.summary()
        optimizer = optimizers.Adam(lr=1e-3)
    
        for epoch in range(500):
            for step, (x,y) in enumerate(train_db):
                with tf.GradientTape() as tape:
                    # [b, 32, 32, 3] => [b, 100]
                    logits = model(x,training=True)
                    # [b] => [b, 100]
                    y_onehot = tf.one_hot(y, depth=100)
                    # compute loss
                    loss = tf.losses.categorical_crossentropy(y_onehot, logits, from_logits=True)
                    loss = tf.reduce_mean(loss)
    
                grads = tape.gradient(loss, model.trainable_variables)
                optimizer.apply_gradients(zip(grads, model.trainable_variables))
                if step %50 == 0:
                    print(epoch, step, 'loss:', float(loss))
    
            total_num = 0
            total_correct = 0
            for x,y in test_db:
                logits = model(x,training=False)
                prob = tf.nn.softmax(logits, axis=1) #把输出转换为概率值
                pred = tf.argmax(prob, axis=1)
                pred = tf.cast(pred, dtype=tf.int32)
                correct = tf.cast(tf.equal(pred, y), dtype=tf.int32)
                correct = tf.reduce_sum(correct)
                total_num += x.shape[0]
                total_correct += int(correct)
            acc = total_correct / total_num
            print(epoch, 'acc:', acc)
    
    if __name__ == '__main__':
        main()

--------------------

# **CIFAR10 与 ResNet18 实战** #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 16][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 17][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 18][]

class BasicBlock(layers.Layer):
        # 残差模块类
        def __init__(self, filter_num, stride=1):
            super(BasicBlock, self).__init__()
            # f(x)包含了2个普通卷积层，创建卷积层1
            self.conv1 = layers.Conv2D(filter_num, (3, 3), strides=stride, padding='same')
            self.bn1 = layers.BatchNormalization()
            self.relu = layers.Activation('relu')
            # 创建卷积层2
            self.conv2 = layers.Conv2D(filter_num, (3, 3), strides=1, padding='same')
            self.bn2 = layers.BatchNormalization()
            # 插入identity层
            if stride != 1: 
                self.downsample = Sequential()
                self.downsample.add(layers.Conv2D(filter_num, (1, 1), strides=stride))
            else: 
                # 否则，直接连接
                self.downsample = lambda x:x
     
        def call(self, inputs, training=None):
            # 前向传播函数
            out = self.conv1(inputs) # 通过第一个卷积层
            out = self.bn1(out)
            out = self.relu(out)
            out = self.conv2(out) # 通过第二个卷积层
            out = self.bn2(out)
            # 输入通过identity()转换
            identity = self.downsample(inputs)
            # f(x)+x运算
            output = layers.add([out, identity])
            # 再通过激活函数并返回
            output = tf.nn.relu(output)
            return output

import  tensorflow as tf
    from    tensorflow import keras
    from    tensorflow.keras import layers, Sequential
     
    class BasicBlock(layers.Layer):
        # 残差模块
        def __init__(self, filter_num, stride=1):
            super(BasicBlock, self).__init__()
            # 第一个卷积单元
            self.conv1 = layers.Conv2D(filter_num, (3, 3), strides=stride, padding='same')
            self.bn1 = layers.BatchNormalization()
            self.relu = layers.Activation('relu')
            # 第二个卷积单元
            self.conv2 = layers.Conv2D(filter_num, (3, 3), strides=1, padding='same')
            self.bn2 = layers.BatchNormalization()
            # 通过1x1卷积完成shape匹配
            if stride != 1:
                self.downsample = Sequential()
                self.downsample.add(layers.Conv2D(filter_num, (1, 1), strides=stride))
            else:
                # shape匹配，直接短接
                self.downsample = lambda x:x
    
        def call(self, inputs, training=None):
            # [b, h, w, c]，通过第一个卷积单元
            out = self.conv1(inputs)
            out = self.bn1(out)
            out = self.relu(out)
            # 通过第二个卷积单元
            out = self.conv2(out)
            out = self.bn2(out)
            # 通过identity模块
            identity = self.downsample(inputs)
            # 2条路径输出直接相加
            output = layers.add([out, identity])
            output = tf.nn.relu(output) # 激活函数
            return output
    
    class ResNet(keras.Model):
        # 通用的ResNet实现类
        def __init__(self, layer_dims, num_classes=10): # [2, 2, 2, 2]
            super(ResNet, self).__init__()
            # 根网络，预处理
            self.stem = Sequential([layers.Conv2D(64, (3, 3), strides=(1, 1)),
                                    layers.BatchNormalization(),
                                    layers.Activation('relu'),
                                    layers.MaxPool2D(pool_size=(2, 2), strides=(1, 1), padding='same')
                                    ])
            # 堆叠4个Block，每个block包含了多个BasicBlock,设置步长不一样
            self.layer1 = self.build_resblock(64,  layer_dims[0])
            self.layer2 = self.build_resblock(128, layer_dims[1], stride=2)
            self.layer3 = self.build_resblock(256, layer_dims[2], stride=2)
            self.layer4 = self.build_resblock(512, layer_dims[3], stride=2)
            # 通过Pooling层将高宽降低为1x1
            self.avgpool = layers.GlobalAveragePooling2D()
            # 最后连接一个全连接层分类
            self.fc = layers.Dense(num_classes)
    
        def call(self, inputs, training=None):
            # 通过根网络
            x = self.stem(inputs)
            # 一次通过4个模块
            x = self.layer1(x)
            x = self.layer2(x)
            x = self.layer3(x)
            x = self.layer4(x)
            # 通过池化层
            x = self.avgpool(x)
            # 通过全连接层
            x = self.fc(x)
            return x
    
        def build_resblock(self, filter_num, blocks, stride=1):
            # 辅助函数，堆叠filter_num个BasicBlock
            res_blocks = Sequential()
            # 只有第一个BasicBlock的步长可能不为1，实现下采样
            res_blocks.add(BasicBlock(filter_num, stride))
            for _ in range(1, blocks):
                #其他BasicBlock步长都为1
                res_blocks.add(BasicBlock(filter_num, stride=1))
            return res_blocks
     
    def resnet18():
        # 通过调整模块内部BasicBlock的数量和配置实现不同的ResNet
        return ResNet([2, 2, 2, 2])
     
    def resnet34():
         # 通过调整模块内部BasicBlock的数量和配置实现不同的ResNet
        return ResNet([3, 4, 6, 3])

import  tensorflow as tf
    from    tensorflow.keras import layers, optimizers, datasets, Sequential
    import  os
    from    resnet import resnet18
    os.environ['TF_CPP_MIN_LOG_LEVEL']='2'
    tf.random.set_seed(2345)
     
    def preprocess(x, y):
        # 将数据映射到-1~1
        x = 2*tf.cast(x, dtype=tf.float32) / 255. - 1
        y = tf.cast(y, dtype=tf.int32) # 类型转换
        return x,y
    
    (x,y), (x_test, y_test) = datasets.cifar10.load_data() # 加载数据集
    y = tf.squeeze(y, axis=1) # 删除不必要的维度
    y_test = tf.squeeze(y_test, axis=1) # 删除不必要的维度
    print(x.shape, y.shape, x_test.shape, y_test.shape)
    train_db = tf.data.Dataset.from_tensor_slices((x,y)) # 构建训练集
    # 随机打散，预处理，批量化
    train_db = train_db.shuffle(1000).map(preprocess).batch(512)
    test_db = tf.data.Dataset.from_tensor_slices((x_test,y_test)) #构建测试集
    # 随机打散，预处理，批量化
    test_db = test_db.map(preprocess).batch(512)
    # 采样一个样本
    sample = next(iter(train_db))
    print('sample:', sample[0].shape, sample[1].shape, tf.reduce_min(sample[0]), tf.reduce_max(sample[0]))
    
    def main():
        # [b, 32, 32, 3] => [b, 1, 1, 512]
        model = resnet18() # ResNet18网络
        model.build(input_shape=(None, 32, 32, 3))
        model.summary() # 统计网络参数
        optimizer = optimizers.Adam(lr=1e-4) # 构建优化器
    
        for epoch in range(100): # 训练epoch
            for step, (x,y) in enumerate(train_db):
                with tf.GradientTape() as tape:
                    # [b, 32, 32, 3] => [b, 10],前向传播
                    logits = model(x)
                    # [b] => [b, 10],one-hot编码
                    y_onehot = tf.one_hot(y, depth=10)
                    # 计算交叉熵
                    loss = tf.losses.categorical_crossentropy(y_onehot, logits, from_logits=True)
                    loss = tf.reduce_mean(loss)
                # 计算梯度信息
                grads = tape.gradient(loss, model.trainable_variables)
                # 更新网络参数
                optimizer.apply_gradients(zip(grads, model.trainable_variables))
                if step %50 == 0:
                    print(epoch, step, 'loss:', float(loss))
    
            total_num = 0
            total_correct = 0
            for x,y in test_db:
                logits = model(x)
                prob = tf.nn.softmax(logits, axis=1)  #把输出转换为概率值
                pred = tf.argmax(prob, axis=1)  #获取最大概率值对应的索引值作为类别号
                pred = tf.cast(pred, dtype=tf.int32)
                #比对最大概率值对应的索引值(类别号)和真实标签类别号是否一致
                correct = tf.cast(tf.equal(pred, y), dtype=tf.int32)
                correct = tf.reduce_sum(correct)
                total_num += x.shape[0]
                total_correct += int(correct)
            acc = total_correct / total_num
            print(epoch, 'acc:', acc)
    
    if __name__ == '__main__':
        main()

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 19][]

import  os
    os.environ['TF_CPP_MIN_LOG_LEVEL']='2'
    import  tensorflow as tf
    from    tensorflow.keras import layers, optimizers, datasets, Sequential
    tf.random.set_seed(2345)
    
    # 5 units of conv + max pooling
    conv_layers = [ 
        # unit 1
        layers.Conv2D(64, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(64, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same'),
        # unit 2
        layers.Conv2D(128, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(128, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same'),
        # unit 3
        layers.Conv2D(256, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(256, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same'),
        # unit 4
        layers.Conv2D(512, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(512, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same'),
        # unit 5
        layers.Conv2D(512, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.Conv2D(512, kernel_size=[3, 3], padding="same", activation=tf.nn.relu),
        layers.MaxPool2D(pool_size=[2, 2], strides=2, padding='same')
    ]
    
    def preprocess(x, y):
        # 标准化映射数据到[0~1]
        x = tf.cast(x, dtype=tf.float32) / 255.
        y = tf.cast(y, dtype=tf.int32)
        return x,y
     
    (x,y), (x_test, y_test) = datasets.cifar100.load_data()
    y = tf.squeeze(y, axis=1)
    y_test = tf.squeeze(y_test, axis=1)
    print(x.shape, y.shape, x_test.shape, y_test.shape)
    train_db = tf.data.Dataset.from_tensor_slices((x,y))
    train_db = train_db.shuffle(1000).map(preprocess).batch(128)
    test_db = tf.data.Dataset.from_tensor_slices((x_test,y_test))
    test_db = test_db.map(preprocess).batch(64)
    sample = next(iter(train_db))
    print('sample:', sample[0].shape, sample[1].shape, tf.reduce_min(sample[0]), tf.reduce_max(sample[0]))
    
    def main():
        # [b, 32, 32, 3] => [b, 1, 1, 512]
        conv_net = Sequential(conv_layers)
        fc_net = Sequential([
            layers.Dense(256, activation=tf.nn.relu),
            layers.Dense(128, activation=tf.nn.relu),
            layers.Dense(100, activation=None),
        ])
        conv_net.build(input_shape=[None, 32, 32, 3])
        fc_net.build(input_shape=[None, 512])
        optimizer = optimizers.Adam(lr=1e-4)
        # [1, 2] + [3, 4] => [1, 2, 3, 4]
        variables = conv_net.trainable_variables + fc_net.trainable_variables
    
        for epoch in range(50):
            for step, (x,y) in enumerate(train_db):
                with tf.GradientTape() as tape:
                    # [b, 32, 32, 3] => [b, 1, 1, 512]
                    out = conv_net(x)
                    # flatten, => [b, 512]
                    out = tf.reshape(out, [-1, 512])
                    # [b, 512] => [b, 100]
                    logits = fc_net(out)
                    # [b] => [b, 100]
                    y_onehot = tf.one_hot(y, depth=100)
                    # compute loss
                    loss = tf.losses.categorical_crossentropy(y_onehot, logits, from_logits=True)
                    loss = tf.reduce_mean(loss)
    
                grads = tape.gradient(loss, variables)
                optimizer.apply_gradients(zip(grads, variables))
                if step %100 == 0:
                    print(epoch, step, 'loss:', float(loss))
    
            total_num = 0
            total_correct = 0
            for x,y in test_db:
                out = conv_net(x)
                out = tf.reshape(out, [-1, 512])
                logits = fc_net(out)
                prob = tf.nn.softmax(logits, axis=1) #softmax把输出值转换为概率值
                pred = tf.argmax(prob, axis=1)
                pred = tf.cast(pred, dtype=tf.int32)
                correct = tf.cast(tf.equal(pred, y), dtype=tf.int32)
                correct = tf.reduce_sum(correct)
                total_num += x.shape[0]
                total_correct += int(correct)
            acc = total_correct / total_num
            print(epoch, 'acc:', acc)
    
    if __name__ == '__main__':
        main()

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 20][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 21][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 22][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 23][]

--------------------

# **各种不同用途的卷积层** #

--------------------

# **空洞卷积**![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 24][] #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 25][]

# 模拟输入
    x = tf.random.normal([1,7,7,1]) 
    # 空洞卷积设置dilation_rate参数，1个3x3的卷积核。
    layer = layers.Conv2D(1,kernel_size=3,strides=1,dilation_rate=2)
    out = layer(x) # 前向计算
    out.shape

--------------------

# **转置卷积conv2d\_transpose（向上采样）** #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 26][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 27][]

转置卷积运算原理1：o+2p-k为s倍数
    	1.o+2p-k为s倍数：o指转置卷积层输出的宽/高，p指转置卷积核的填充p，k指转置卷积核的宽w/高h，s指转置卷积核的步长s。
    	2.比如一个宽w/高h为2x2的单通道特征图(即转置卷积输入)，转置卷积核的宽w/高h为3x3(即k)，步长s为2，填充p为0，那么转置卷积计算流程如下：
    		1.第一步：在输入的单通道特征图中的数据点之间均匀插入转置卷积核步长s-1个空白数据点，那么从2x2变成3x3的矩阵。
    		2.第二步：3x3矩阵的周围需要填充转置卷积核相应的k-p-1，得出3x3矩阵周围填充3-0-1=2行2列，那么从3x3变成7x7的矩阵。
    		3.第三步：对7x7的矩阵进行普通卷积运算，该卷积核的宽w/高h为3x3(即k，和转置卷积核的宽w/高h相同)，
    			 此阶段的该卷积核的步长s永远为1(与转置卷积的步长s不同)，填充p为0(和转置卷积核的填充p相同)，
    			 根据普通卷积运算公式o=⌊(i+2*p-k)/s⌋+1（向上取整⌈⌉:比自己大的最小整数、向下取整⌊⌋:比自己小的最大整数），
    			 得出o=⌊(7+2*0-3)/1⌋+1=5。
    	3.结论：最终第三步普通卷积运算输出的5x5即为这个转置卷积运算的输出，那么转置卷积运算输入(2x2)与输出(5x5)的关系即为
    	        当o+2p-k为s倍数时得出o=(i-1)*s+k-2*p，o指转置卷积层输出的宽/高，p指转置卷积核的填充p，k指转置卷积核的宽w/高h，s指转置卷积核的步长s，
    	        i指转置卷积运算输入，那么根据上式得出当5+2*0-3=2为2倍数时即有(2-1)*2+3-2*0=5。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 28][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 29][]

#下面基于 TensorFlow 实现上述例子的转置卷积运算：
    import tensorflow as tf
    # 创建 X 矩阵，高宽为 5x5
    x = tf.range(25)+1 
    #<tf.Tensor: id=5, shape=(25,), dtype=int32, numpy=
    #array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
    #       18, 19, 20, 21, 22, 23, 24, 25])>
    # Reshape 为合法维度的张量
    x = tf.reshape(x,[1,5,5,1])
    x.shape #TensorShape([1, 5, 5, 1])
    x = tf.cast(x, tf.float32)
    # 创建固定内容的卷积核矩阵，shape为[3, 3, 1, 1]即[k,k,输入通道数,输出通道数]
    w = tf.constant([[-1,2,-3.],[4,-5,6],[-7,8,-9]])
    w.shape #TensorShape([3, 3])
    # 调整为合法维度的张量
    w = tf.expand_dims(w,axis=2) 
    w.shape #TensorShape([3, 3, 1])	
    w = tf.expand_dims(w,axis=3) 
    w.shape #TensorShape([3, 3, 1, 1]) 即 [k,k,输入通道数,输出通道数]
    #通过 padding 参数来设置填充，这个参数有两个取值："valid" 表示不使用填充（只使用有效的窗口位置），padding 参数的默认值为 "valid"；
    #"same" 表示“填充后输出的宽度和高度与输入相同”。
    # 进行普通卷积运算
    out = tf.nn.conv2d(x,w,strides=2,padding='VALID')
    # 输出的高宽为 2x2
    out 
    #<tf.Tensor: id=14, shape=(1, 2, 2, 1), dtype=float32, numpy=
    #array([[[[ -67.],
    #         [ -77.]],
    #        [[-117.],
    #         [-127.]]]], dtype=float32)>
    
    
    # 将普通卷积的输出2x2（[1, 2, 2, 1]）作为转置卷积的输入，验证转置卷积的输出是否为5x5（[1,5,5,1]）
    # 普通卷积的输出作为转置卷积的输入，进行转置卷积运算
    xx = tf.nn.conv2d_transpose(out, w, strides=2, padding='VALID', output_shape=[1,5,5,1])
    # 输出的高宽为 5x5（[1,5,5,1]）。转置卷积能够把普通卷积的输出恢复为普通卷积的输入，但转置卷积的输出并不等同于普通卷积的输入
    xx.shape #TensorShape([1, 5, 5, 1])

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 30][]

#当普通卷积运算的输入高宽为5x5或6x6时，输出均为一样的2x2
    import tensorflow as tf
    # 创建固定内容的卷积核矩阵，shape为[3, 3, 1, 1]即[k,k,输入通道数,输出通道数]
    w = tf.constant([[-1,2,-3.],[4,-5,6],[-7,8,-9]])
    # 调整为合法维度的张量
    w = tf.expand_dims(w,axis=2) 
    w.shape #TensorShape([3, 3, 1])	
    w = tf.expand_dims(w,axis=3) 
    w.shape #TensorShape([3, 3, 1, 1]) 即 [k,k,输入通道数,输出通道数]
    x = tf.random.normal([1,6,6,1])
    #通过 padding 参数来设置填充，这个参数有两个取值："valid" 表示不使用填充（只使用有效的窗口位置），padding 参数的默认值为 "valid"；
    #"same" 表示“填充后输出的宽度和高度与输入相同”。padding='SAME'只有在strides=1时才是输出、输入同大小。
    # 6x6 的输入经过普通卷积
    out = tf.nn.conv2d(x,w, strides=2, padding='VALID')
    out.shape #TensorShape([1, 2, 2, 1]) #当输入的高宽为5x5或6x6时，输出均为一样的2x2
    x = tf.random.normal([1,5,5,1])
    # 6x6 的输入经过普通卷积
    out = tf.nn.conv2d(x,w, strides=2, padding='VALID')
    out.shape #TensorShape([1, 2, 2, 1]) #当输入的高宽为5x5或6x6时，输出均为一样的2x2

1.普通卷积运算公式o=⌊(i+2*p-k)/s⌋+1（向上取整⌈⌉:比自己大的最小整数、向下取整⌊⌋:比自己小的最大整数）
      比如一个宽w/高h为5x5或6x6的单通道特征图(即普通卷积运算的输入i)，卷积核的宽w/高h为3x3(即k)，步长s为2(步长必须大于1)，填充p为0，普通卷积运算输出o均为2x2，
      表示当步长s大于1的情况下，通过最后的向下取整后，会使得出现 多种不同输入尺寸大小i 能对应到 相同的卷积输出尺寸大小o。
      比如普通卷积运算的输入i为5，那么⌊(5+2*0-3)/2⌋+1=2；普通卷积运算的输入i为6，那么⌊(6+2*0-3)/2⌋+1=2。
    
    2.转置卷积运算原理2：o+2p-k不为s倍数
    	1.o+2p-k不为s倍数：o指转置卷积层输出的宽/高，p指转置卷积核的填充p，k指转置卷积核的宽w/高h，s指转置卷积核的步长s。
    	2.在普通卷积运算中，只要当步长s大于1的情况下，向下取整后都能出现 多种不同输入尺寸大小i 会对应到 相同的卷积输出尺寸大小o，
    	  因此在转置卷积运算中，自然希望使用相同的输入尺寸大小i 经过转置卷积运算后，能得到不同的转置卷积输出尺寸大小o，
    	  因此便需要填充a行a列实现不同尺寸大小的转置卷积输出o，从而恢复普通卷积中不同尺寸大小的输入i能对应输出到同一个尺寸大小的输出o。
    	3.比如现在的普通卷积运算宽w/高h为5x5或6x6的单通道特征图(即普通卷积运算的输入i)，卷积核的宽w/高h为3x3(即k)，步长s为2(步长必须大于1)，填充p为0，
     	  普通卷积运算输出o均为2x2，现在希望通过2x2的转置卷积输入，实现自定义转置卷积输出为5x5或6x6，转置卷积核填充p为0，转置卷积核的宽w/高h为3x3，
    	  转置卷积核的步长s为2，需要以下两步实现如上目的：
    		1.第一步：a=(o+2p-k)%s，需要填充a行a列实现不同尺寸大小的转置卷积输出o，从而恢复普通卷积中不同尺寸大小的输入i能对应输出到同一个尺寸大小的输出o。
    		 	 比如希望通过转置卷积输出为5x5(o为5)时，(5+2*0-3)%2=0，即填充0行0列实现转置卷积输出尺寸大小o为5x5。
    			 比如希望通过转置卷积输出为6x6(o为6)时，(6+2*0-3)%2=1，即填充1行1列实现转置卷积输出尺寸大小o为6x6。
    		2.第二步：o=(i-1)*s+k-2*p+a，根据第一步所计算好的需要填充的a行a列，转置卷积输出希望的尺寸大小o。
    			 比如希望通过转置卷积输出为5x5(o为5)时，而计算好的需要填充的0行0列：(2-1)*2+3-2*0+0=5。
    			 比如希望通过转置卷积输出为6x6(o为6)时，而计算好的需要填充的1行1列：(2-1)*2+3-2*0+1=6。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 31][]

import tensorflow as tf
    
    # 创建固定内容的卷积核矩阵，shape为[3, 3, 1, 1]即[k,k,输入通道数,输出通道数]
    w = tf.constant([[-1,2,-3.],[4,-5,6],[-7,8,-9]])
    # 调整为合法维度的张量
    w = tf.expand_dims(w,axis=2) 
    w.shape #TensorShape([3, 3, 1])	
    w = tf.expand_dims(w,axis=3) 
    w.shape #TensorShape([3, 3, 1, 1]) 即 [k,k,输入通道数,输出通道数]
    x = tf.random.normal([1,6,6,1])
    #通过 padding 参数来设置填充，这个参数有两个取值："valid" 表示不使用填充（只使用有效的窗口位置），padding 参数的默认值为 "valid"；
    #"same" 表示“填充后输出的宽度和高度与输入相同”。padding='SAME'只有在strides=1时才是输出、输入同大小。
    # 6x6 的输入经过普通卷积
    out = tf.nn.conv2d(x,w, strides=2, padding='VALID')
    out.shape #TensorShape([1, 2, 2, 1]) #当输入的高宽为5x5或6x6时，输出均为一样的2x2
    x = tf.random.normal([1,5,5,1])
    # 6x6 的输入经过普通卷积
    out = tf.nn.conv2d(x,w, strides=2, padding='VALID') 
    out.shape #TensorShape([1, 2, 2, 1]) #当输入的高宽为5x5或6x6时，输出均为一样的2x2
     
    #只需要指定输出尺寸即可，TensorFlow会自动推导需要填充的行列数
    xx = tf.nn.conv2d_transpose(out, w, strides=2, padding='VALID', output_shape=[1,6,6,1])
    xx.shape #TensorShape([1, 6, 6, 1])
    xx = tf.nn.conv2d_transpose(out, w, strides=2, padding='VALID', output_shape=[1,5,5,1])
    xx.shape #TensorShape([1, 5, 5, 1])

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 32][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 33][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 34][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 35][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 36][]

# 创建X矩阵
    x = tf.range(25)+1
    # Reshape为合法维度的张量
    x = tf.reshape(x,[1,5,5,1])
    x = tf.cast(x, tf.float32)
    # 创建固定内容的卷积核矩阵
    w = tf.constant([[-1,2,-3.],[4,-5,6],[-7,8,-9]])
    # 调整为合法维度的张量
    w = tf.expand_dims(w,axis=2)
    w = tf.expand_dims(w,axis=3)
    # 进行普通卷积运算
    out = tf.nn.conv2d(x,w,strides=2,padding='VALID')
    out
     
    # 普通卷积的输出作为转置卷积的输入，进行转置卷积运算
    xx = tf.nn.conv2d_transpose(out, w, strides=2, padding='VALID', output_shape=[1,5,5,1])
    #<tf.Tensor: id=117, shape=(5, 5), dtype=float32, numpy=
    #array([[   67.,  -134.,   278.,  -154.,   231.],
    #       [ -268.,   335.,  -710.,   385.,  -462.],
    #       [  586.,  -770.,  1620.,  -870.,  1074.],
    #       [ -468.,   585., -1210.,   635.,  -762.],
    #       [  819.,  -936.,  1942., -1016.,  1143.]], dtype=float32)>
    
    
    x = tf.random.normal([1,6,6,1])
    # 6x6的输入经过普通卷积
    out = tf.nn.conv2d(x,w,strides=2,padding='VALID')
    out
    #<tf.Tensor: id=21, shape=(1, 2, 2, 1), dtype=float32, numpy=
    #array([[[[ 20.438847 ],
    #         [ 19.160788 ]],
    #        [[  0.8098897],
    #         [-28.30303  ]]]], dtype=float32)>
    
    # 恢复出6x6大小
    xx = tf.nn.conv2d_transpose(out, w, strides=2, padding='VALID', output_shape=[1,6,6,1])
    xx
    
    # 创建转置卷积类
    layer = layers.Conv2DTranspose(1,kernel_size=3,strides=2,padding='VALID')
    xx2 = layer(out)
    xx2

--------------------

# **Keras 深度可分离卷积** #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 37][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 38][]

--------------------

# **深度可分离卷积** #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 39][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 40][]

--------------------

# **Inception**![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 41][] #

import  os
    import  tensorflow as tf
    import  numpy as np
    from    tensorflow import keras
     
    tf.random.set_seed(22)
    np.random.seed(22)
    os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
    assert tf.__version__.startswith('2.')
     
    (x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
    x_train, x_test = x_train.astype(np.float32)/255., x_test.astype(np.float32)/255.
    # [b, 28, 28] => [b, 28, 28, 1]
    x_train, x_test = np.expand_dims(x_train, axis=3), np.expand_dims(x_test, axis=3)
    db_train = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(256)
    db_test = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(256)
    print(x_train.shape, y_train.shape) #(60000, 28, 28, 1) (60000,)
    print(x_test.shape, y_test.shape) #(10000, 28, 28, 1) (10000,)
     
    class ConvBNRelu(keras.Model):
        def __init__(self, ch, kernelsz=3, strides=1, padding='same'):
            super(ConvBNRelu, self).__init__()
            self.model = keras.models.Sequential([
                keras.layers.Conv2D(ch, kernelsz, strides=strides, padding=padding),
                keras.layers.BatchNormalization(),
                keras.layers.ReLU()
            ])
    
        def call(self, x, training=None):
            x = self.model(x, training=training)
            return x 
    
    class InceptionBlk(keras.Model):
        def __init__(self, ch, strides=1):
            super(InceptionBlk, self).__init__()
            self.ch = ch
            self.strides = strides
            self.conv1 = ConvBNRelu(ch, strides=strides)
            self.conv2 = ConvBNRelu(ch, kernelsz=3, strides=strides)
            self.conv3_1 = ConvBNRelu(ch, kernelsz=3, strides=strides)
            self.conv3_2 = ConvBNRelu(ch, kernelsz=3, strides=1)
            self.pool = keras.layers.MaxPooling2D(3, strides=1, padding='same')
            self.pool_conv = ConvBNRelu(ch, strides=strides)
    
        def call(self, x, training=None):
            x1 = self.conv1(x, training=training)
            x2 = self.conv2(x, training=training)
            x3_1 = self.conv3_1(x, training=training)
            x3_2 = self.conv3_2(x3_1, training=training)
            x4 = self.pool(x)
            x4 = self.pool_conv(x4, training=training)
            # concat along axis=channel 只在通道维上连接
            x = tf.concat([x1, x2, x3_2, x4], axis=3)
            return x
    
    class Inception(keras.Model):
        def __init__(self, num_layers, num_classes, init_ch=16, **kwargs):
            super(Inception, self).__init__(**kwargs)
            self.in_channels = init_ch
            self.out_channels = init_ch #经过每个num_layers层之后，out_channels输出通道数都要翻倍，输出的高宽同时也要减半
            self.num_layers = num_layers
            self.init_ch = init_ch
            self.conv1 = ConvBNRelu(init_ch)
            self.blocks = keras.models.Sequential(name='dynamic-blocks')
    
            for block_id in range(num_layers):
                for layer_id in range(2):
                    if layer_id == 0:
                        #经过每个num_layers层之后的第一层都要输出的高宽减半
                        block = InceptionBlk(self.out_channels, strides=2) #输出的高宽减半
                    else:
                        block = InceptionBlk(self.out_channels, strides=1)
                    self.blocks.add(block)
                # enlarger out_channels per block    
                self.out_channels *= 2 #经过每个num_layers层之后，out_channels输出通道数都要翻倍
    
            self.avg_pool = keras.layers.GlobalAveragePooling2D()
            self.fc = keras.layers.Dense(num_classes)
    
        def call(self, x, training=None):
            out = self.conv1(x, training=training)
            out = self.blocks(out, training=training)
            out = self.avg_pool(out)
            out = self.fc(out)
            return out    
    
    # build model and optimizer
    batch_size = 32
    epochs = 100
    model = Inception(2, 10)
    # derive input shape for every layers.
    model.build(input_shape=(None, 28, 28, 1))
    model.summary()
    #Model: "inception"
    #_________________________________________________________________
    #Layer (type)                 Output Shape              Param #
    #=================================================================
    #conv_bn_relu (ConvBNRelu)    multiple                  224
    #_________________________________________________________________
    #dynamic-blocks (Sequential)  multiple                  292704
    #_________________________________________________________________
    #global_average_pooling2d (Gl multiple                  0
    #_________________________________________________________________
    #dense (Dense)                multiple                  1290
    #=================================================================
    #Total params: 294,218
    #Trainable params: 293,226
    #Non-trainable params: 992
    #_________________________________________________________________
    
    optimizer = keras.optimizers.Adam(learning_rate=1e-3)
    criteon = keras.losses.CategoricalCrossentropy(from_logits=True)
    acc_meter = keras.metrics.Accuracy()
    
    for epoch in range(100):
        for step, (x, y) in enumerate(db_train):
            with tf.GradientTape() as tape:
                # print(x.shape, y.shape)
                # [b, 10]
                logits = model(x)
                # [b] vs [b, 10]
                loss = criteon(tf.one_hot(y, depth=10), logits)
            grads = tape.gradient(loss, model.trainable_variables)
            optimizer.apply_gradients(zip(grads, model.trainable_variables))
            if step % 10 == 0:
                print(epoch, step, 'loss:', loss.numpy())
        acc_meter.reset_states() #清空准确率指标
        for x, y in db_test:
            # [b, 10]
            logits = model(x, training=False)
            # [b, 10] => [b]
            pred = tf.argmax(logits, axis=1)
            print(x.shape, y.shape, pred.shape) #(256, 28, 28, 1) (256,) (256,)
            # y的批量[b] 和 pred的批量[b] 对比计算
            acc_meter.update_state(y, pred) #计算准确率指标
        print(epoch, 'evaluation acc:', acc_meter.result().numpy()) #获取准确率指标

[20191009191333910.png]: https://img-blog.csdnimg.cn/20191009191333910.png
[Link 1]: http://www.rimengshe.com/
[AI_Keras PyTorch MXNet TensorFlow PaddlePaddle]: https://blog.csdn.net/zimiao552147572/article/details/88867161
[tensorflow 2.0 _ part1]: https://azusa.blog.csdn.net/article/details/104086445
[tensorflow 2.0 _ part2]: https://azusa.blog.csdn.net/article/details/104086557
[tensorflow 2.0 _ part3]: https://azusa.blog.csdn.net/article/details/104086663
[tensorflow 2.0 _ part1 1]: https://azusa.blog.csdn.net/article/details/104086741
[tensorflow 2.0 _ part2 1]: https://azusa.blog.csdn.net/article/details/104086773
[tensorflow 2.0 _ part3 1]: https://azusa.blog.csdn.net/article/details/104087912
[tensorflow 2.0 _ _ _ part1]: https://azusa.blog.csdn.net/article/details/104088133
[tensorflow 2.0 _ _ _ part2]: https://azusa.blog.csdn.net/article/details/104088230
[tensorflow 2.0 _]: https://azusa.blog.csdn.net/article/details/104088366
[tensorflow 2.0 _ GAN_ part1]: https://azusa.blog.csdn.net/article/details/104088512
[tensorflow 2.0 _ GAN_ part2]: https://azusa.blog.csdn.net/article/details/104088559
[tensorflow 2.0 _ 1]: https://azusa.blog.csdn.net/article/details/104089741
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200126170448427.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200126170518694.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200126170640298.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200126170833723.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200126170853516.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200126170959693.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200126171006995.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200126171044873.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200126171048298.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200126171112414.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/2020012617111653.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20200126171119615.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20200126171146183.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20200126171154193.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 14]: https://img-blog.csdnimg.cn/20200126171202163.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 15]: https://img-blog.csdnimg.cn/2020012617120024.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 16]: https://img-blog.csdnimg.cn/20200126171251811.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 17]: https://img-blog.csdnimg.cn/20200126171258853.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 18]: https://img-blog.csdnimg.cn/20200126171308668.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 19]: https://img-blog.csdnimg.cn/20200126171332824.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 20]: https://img-blog.csdnimg.cn/20200126171345568.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 21]: https://img-blog.csdnimg.cn/20200126171351831.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 22]: https://img-blog.csdnimg.cn/20200126171357130.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 23]: https://img-blog.csdnimg.cn/20200126171402275.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 24]: https://img-blog.csdnimg.cn/20200126171502794.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 25]: https://img-blog.csdnimg.cn/20200126171508679.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 26]: https://img-blog.csdnimg.cn/20200126171531906.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 27]: https://img-blog.csdnimg.cn/20200126171528857.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 28]: https://img-blog.csdnimg.cn/20200126171615953.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 29]: https://img-blog.csdnimg.cn/20200126171621961.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 30]: https://img-blog.csdnimg.cn/20200126171632985.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 31]: https://img-blog.csdnimg.cn/20200126171706869.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 32]: https://img-blog.csdnimg.cn/20200126171718594.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 33]: https://img-blog.csdnimg.cn/2020012617172544.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 34]: https://img-blog.csdnimg.cn/20200126171731102.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 35]: https://img-blog.csdnimg.cn/20200126171737463.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 36]: https://img-blog.csdnimg.cn/20200126171744354.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 37]: https://img-blog.csdnimg.cn/20200126171809701.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 38]: https://img-blog.csdnimg.cn/20200126171814137.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 39]: https://img-blog.csdnimg.cn/202001261718320.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 40]: https://img-blog.csdnimg.cn/20200126171835719.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 41]: https://img-blog.csdnimg.cn/20200126171909316.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9henVzYS5ibG9nLmNzZG4ubmV0,size_16,color_FFFFFF,t_70