【Tensorflow】动态更改学习率

﹏ヽ暗。殇╰゛Y 2022-05-11 00:14 246阅读 0赞

首先定义学习率衰减方式

LEARNING_RATE_BASE = 0.1  # 最初学习率
    LEARNING_RATE_DECAY = 0.99  # 学习率的衰减率
    LEARNING_RATE_STEP = 1  # 喂入多少轮BATCH-SIZE以后，更新一次学习率。一般为总样本数量/BATCH_SIZE
    gloabl_steps = tf.Variable(0, trainable=False)  # 计数器，用来记录运行了几轮的BATCH_SIZE，初始为0，设置为不可训练
    learning_rate = tf.train.exponential_decay(LEARNING_RATE_BASE
                                               , gloabl_steps,
                                               LEARNING_RATE_STEP,
                                               LEARNING_RATE_DECAY,
                                               staircase=True)
    # 定义指数下降学习率。
    # 如果staircase=True，那就表明每decay_steps次计算学习速率变化，更新原始学习速率，如果是False，那就是每一步都更新学习速率。红色表示False，绿色表示True。

然后放入梯度优化 (反向传播)

train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=gloabl_steps)

测试程序（跑二次函数极值点）

# coding:utf-8
    import os
    
    os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
    
    import tensorflow as tf
    
    LEARNING_RATE_BASE = 0.1  # 最初学习率
    LEARNING_RATE_DECAY = 0.99  # 学习率的衰减率
    LEARNING_RATE_STEP = 1  # 喂入多少轮BATCH-SIZE以后，更新一次学习率。一般为总样本数量/BATCH_SIZE
    gloabl_steps = tf.Variable(0, trainable=False)  # 计数器，用来记录运行了几轮的BATCH_SIZE，初始为0，设置为不可训练
    learning_rate = tf.train.exponential_decay(LEARNING_RATE_BASE
                                               , gloabl_steps,
                                               LEARNING_RATE_STEP,
                                               LEARNING_RATE_DECAY,
                                               staircase=True)
    # 定义指数下降学习率。
    # 如果staircase=True，那就表明每decay_steps次计算学习速率变化，更新原始学习速率，如果是False，那就是每一步都更新学习速率。红色表示False，绿色表示True。
    
    
    w = tf.Variable(tf.constant(5, tf.float32))
    
    loss = tf.square(w + 1)
    
    train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=gloabl_steps)
    print('start')
    with tf.Session() as sess:
        init_op = tf.global_variables_initializer()
        sess.run(init_op)
        for i in range(100):
            sess.run(train_step)
            w_val = sess.run(w)
            loss_val = sess.run(loss)
            learning_rate_val = sess.run(learning_rate)
            print('After %s steps:learning_rate is %f ,w is %f ,loss is %f ' % (i, learning_rate_val, w_val, loss_val))

运行结果

After 0 steps:learning_rate is 0.099000 ,w is 3.800000 ,loss is 23.040001 
    After 1 steps:learning_rate is 0.098010 ,w is 2.849600 ,loss is 14.819419 
    After 2 steps:learning_rate is 0.097030 ,w is 2.095001 ,loss is 9.579033 
    After 3 steps:learning_rate is 0.096060 ,w is 1.494386 ,loss is 6.221960 
    After 4 steps:learning_rate is 0.095099 ,w is 1.015166 ,loss is 4.060895 
    After 5 steps:learning_rate is 0.094148 ,w is 0.631886 ,loss is 2.663051 
    After 6 steps:learning_rate is 0.093207 ,w is 0.324608 ,loss is 1.754587 
    After 7 steps:learning_rate is 0.092274 ,w is 0.077684 ,loss is 1.161402 
    After 8 steps:learning_rate is 0.091352 ,w is -0.121202 ,loss is 0.772287 
    After 9 steps:learning_rate is 0.090438 ,w is -0.281761 ,loss is 0.515867 
    After 10 steps:learning_rate is 0.089534 ,w is -0.411674 ,loss is 0.346128 
    After 11 steps:learning_rate is 0.088638 ,w is -0.517024 ,loss is 0.233266 
    After 12 steps:learning_rate is 0.087752 ,w is -0.602644 ,loss is 0.157891 
    After 13 steps:learning_rate is 0.086875 ,w is -0.672382 ,loss is 0.107334 
    After 14 steps:learning_rate is 0.086006 ,w is -0.729305 ,loss is 0.073276 
    After 15 steps:learning_rate is 0.085146 ,w is -0.775868 ,loss is 0.050235 
    After 16 steps:learning_rate is 0.084294 ,w is -0.814036 ,loss is 0.034583 
    After 17 steps:learning_rate is 0.083451 ,w is -0.845387 ,loss is 0.023905 
    After 18 steps:learning_rate is 0.082617 ,w is -0.871193 ,loss is 0.016591 
    After 19 steps:learning_rate is 0.081791 ,w is -0.892476 ,loss is 0.011561 
    After 20 steps:learning_rate is 0.080973 ,w is -0.910065 ,loss is 0.008088 
    After 21 steps:learning_rate is 0.080163 ,w is -0.924629 ,loss is 0.005681 
    After 22 steps:learning_rate is 0.079361 ,w is -0.936713 ,loss is 0.004005 
    After 23 steps:learning_rate is 0.078568 ,w is -0.946758 ,loss is 0.002835 
    After 24 steps:learning_rate is 0.077782 ,w is -0.955125 ,loss is 0.002014 
    After 25 steps:learning_rate is 0.077004 ,w is -0.962106 ,loss is 0.001436 
    After 26 steps:learning_rate is 0.076234 ,w is -0.967942 ,loss is 0.001028 
    After 27 steps:learning_rate is 0.075472 ,w is -0.972830 ,loss is 0.000738 
    After 28 steps:learning_rate is 0.074717 ,w is -0.976931 ,loss is 0.000532 
    After 29 steps:learning_rate is 0.073970 ,w is -0.980378 ,loss is 0.000385 
    After 30 steps:learning_rate is 0.073230 ,w is -0.983281 ,loss is 0.000280 
    After 31 steps:learning_rate is 0.072498 ,w is -0.985730 ,loss is 0.000204 
    After 32 steps:learning_rate is 0.071773 ,w is -0.987799 ,loss is 0.000149 
    After 33 steps:learning_rate is 0.071055 ,w is -0.989550 ,loss is 0.000109 
    After 34 steps:learning_rate is 0.070345 ,w is -0.991035 ,loss is 0.000080 
    After 35 steps:learning_rate is 0.069641 ,w is -0.992297 ,loss is 0.000059 
    After 36 steps:learning_rate is 0.068945 ,w is -0.993369 ,loss is 0.000044 
    After 37 steps:learning_rate is 0.068255 ,w is -0.994284 ,loss is 0.000033 
    After 38 steps:learning_rate is 0.067573 ,w is -0.995064 ,loss is 0.000024 
    After 39 steps:learning_rate is 0.066897 ,w is -0.995731 ,loss is 0.000018 
    After 40 steps:learning_rate is 0.066228 ,w is -0.996302 ,loss is 0.000014 
    After 41 steps:learning_rate is 0.065566 ,w is -0.996792 ,loss is 0.000010 
    After 42 steps:learning_rate is 0.064910 ,w is -0.997213 ,loss is 0.000008 
    After 43 steps:learning_rate is 0.064261 ,w is -0.997575 ,loss is 0.000006 
    After 44 steps:learning_rate is 0.063619 ,w is -0.997886 ,loss is 0.000004 
    After 45 steps:learning_rate is 0.062982 ,w is -0.998155 ,loss is 0.000003 
    After 46 steps:learning_rate is 0.062353 ,w is -0.998388 ,loss is 0.000003 
    After 47 steps:learning_rate is 0.061729 ,w is -0.998589 ,loss is 0.000002 
    After 48 steps:learning_rate is 0.061112 ,w is -0.998763 ,loss is 0.000002 
    After 49 steps:learning_rate is 0.060501 ,w is -0.998914 ,loss is 0.000001 
    After 50 steps:learning_rate is 0.059896 ,w is -0.999045 ,loss is 0.000001 
    After 51 steps:learning_rate is 0.059297 ,w is -0.999160 ,loss is 0.000001 
    After 52 steps:learning_rate is 0.058704 ,w is -0.999259 ,loss is 0.000001 
    After 53 steps:learning_rate is 0.058117 ,w is -0.999346 ,loss is 0.000000 
    After 54 steps:learning_rate is 0.057536 ,w is -0.999422 ,loss is 0.000000 
    After 55 steps:learning_rate is 0.056960 ,w is -0.999489 ,loss is 0.000000 
    After 56 steps:learning_rate is 0.056391 ,w is -0.999547 ,loss is 0.000000 
    After 57 steps:learning_rate is 0.055827 ,w is -0.999598 ,loss is 0.000000 
    After 58 steps:learning_rate is 0.055268 ,w is -0.999643 ,loss is 0.000000 
    After 59 steps:learning_rate is 0.054716 ,w is -0.999682 ,loss is 0.000000 
    After 60 steps:learning_rate is 0.054169 ,w is -0.999717 ,loss is 0.000000 
    After 61 steps:learning_rate is 0.053627 ,w is -0.999748 ,loss is 0.000000 
    After 62 steps:learning_rate is 0.053091 ,w is -0.999775 ,loss is 0.000000 
    After 63 steps:learning_rate is 0.052560 ,w is -0.999799 ,loss is 0.000000 
    After 64 steps:learning_rate is 0.052034 ,w is -0.999820 ,loss is 0.000000 
    After 65 steps:learning_rate is 0.051514 ,w is -0.999839 ,loss is 0.000000 
    After 66 steps:learning_rate is 0.050999 ,w is -0.999855 ,loss is 0.000000 
    After 67 steps:learning_rate is 0.050489 ,w is -0.999870 ,loss is 0.000000 
    After 68 steps:learning_rate is 0.049984 ,w is -0.999883 ,loss is 0.000000 
    After 69 steps:learning_rate is 0.049484 ,w is -0.999895 ,loss is 0.000000 
    After 70 steps:learning_rate is 0.048989 ,w is -0.999905 ,loss is 0.000000 
    After 71 steps:learning_rate is 0.048499 ,w is -0.999915 ,loss is 0.000000 
    After 72 steps:learning_rate is 0.048014 ,w is -0.999923 ,loss is 0.000000 
    After 73 steps:learning_rate is 0.047534 ,w is -0.999930 ,loss is 0.000000 
    After 74 steps:learning_rate is 0.047059 ,w is -0.999937 ,loss is 0.000000 
    After 75 steps:learning_rate is 0.046588 ,w is -0.999943 ,loss is 0.000000 
    After 76 steps:learning_rate is 0.046122 ,w is -0.999948 ,loss is 0.000000 
    After 77 steps:learning_rate is 0.045661 ,w is -0.999953 ,loss is 0.000000 
    After 78 steps:learning_rate is 0.045204 ,w is -0.999957 ,loss is 0.000000 
    After 79 steps:learning_rate is 0.044752 ,w is -0.999961 ,loss is 0.000000 
    After 80 steps:learning_rate is 0.044305 ,w is -0.999965 ,loss is 0.000000 
    After 81 steps:learning_rate is 0.043862 ,w is -0.999968 ,loss is 0.000000 
    After 82 steps:learning_rate is 0.043423 ,w is -0.999971 ,loss is 0.000000 
    After 83 steps:learning_rate is 0.042989 ,w is -0.999973 ,loss is 0.000000 
    After 84 steps:learning_rate is 0.042559 ,w is -0.999975 ,loss is 0.000000 
    After 85 steps:learning_rate is 0.042133 ,w is -0.999978 ,loss is 0.000000 
    After 86 steps:learning_rate is 0.041712 ,w is -0.999979 ,loss is 0.000000 
    After 87 steps:learning_rate is 0.041295 ,w is -0.999981 ,loss is 0.000000 
    After 88 steps:learning_rate is 0.040882 ,w is -0.999983 ,loss is 0.000000 
    After 89 steps:learning_rate is 0.040473 ,w is -0.999984 ,loss is 0.000000 
    After 90 steps:learning_rate is 0.040068 ,w is -0.999985 ,loss is 0.000000 
    After 91 steps:learning_rate is 0.039668 ,w is -0.999987 ,loss is 0.000000 
    After 92 steps:learning_rate is 0.039271 ,w is -0.999988 ,loss is 0.000000 
    After 93 steps:learning_rate is 0.038878 ,w is -0.999989 ,loss is 0.000000 
    After 94 steps:learning_rate is 0.038490 ,w is -0.999990 ,loss is 0.000000 
    After 95 steps:learning_rate is 0.038105 ,w is -0.999990 ,loss is 0.000000 
    After 96 steps:learning_rate is 0.037724 ,w is -0.999991 ,loss is 0.000000 
    After 97 steps:learning_rate is 0.037346 ,w is -0.999992 ,loss is 0.000000 
    After 98 steps:learning_rate is 0.036973 ,w is -0.999992 ,loss is 0.000000 
    After 99 steps:learning_rate is 0.036603 ,w is -0.999993 ,loss is 0.000000