PyTorch项目应用实例(四)设置learning_rate的decay

落日映苍穹つ 2022-02-02 06:13 483阅读 0赞

背景：训练的过程之中，需要对learning rate设置decay。我们现在设置learning rate，然后将它进行衰减，从而获得更好的效果。

**目录**

一、标准做法

二、learning rate用于训练

2.1 我们的程序的嵌套

2.2 adjust learning rate

2.3 我们的方案

--------------------

# 一、标准做法 #

很容易看懂，直接用lr\_scheduler.StepLR(optimizer\_ft, step\_size=5, gamma=0.2)函数进行learning rate的decay

# define cost function
        criterion = nn.CrossEntropyLoss()
    
        # Observe that all parameters are being optimized
        optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.005, momentum=0.9)
    
        # Decay LR by a factor of 0.2 every 5 epochs
        exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=5, gamma=0.2)
    
        # multi-GPU
        model_ft = torch.nn.DataParallel(model_ft, device_ids=[0])
    
        # train model
        print("start train_model......")
        model_ft = train_model(model=model_ft,
                               criterion=criterion,
                               optimizer=optimizer_ft,
                               scheduler=exp_lr_scheduler,
                               num_epochs=15,
                               use_gpu=use_gpu)

# 二、learning rate用于训练 #

## 2.1 我们的程序的嵌套 ##

在general\_train.py之中

engine = GCNMultiLabelMAPEngine(state)
        engine.learning(model, criterion, train_dataset, val_dataset, optimizer)

engine.py之中，learning的class

for epoch in range(self.state['start_epoch'], self.state['max_epochs']):
                self.state['epoch'] = epoch
                # lr = self.adjust_learning_rate(optimizer) #fixme
                # print('lr:{:.5f}'.format(lr)) # fixme
                self.adjust_learning_rate(optimizer)  # fixme: not return lr for printing
    
                # train for one epoch
                self.train(train_loader, model, criterion, optimizer, epoch)
                # evaluate on validation set
                prec1 = self.validate(val_loader, model, criterion)

learning rate应该是在optimzer之中定义的，用adjust\_learning\_rate函数直接对optimizer进行操作，然后对lr进行decay，训练时候直接调用optimizer函数即可。

## 2.2 adjust learning rate ##

原函数中，adjust learning rate中，

def adjust_learning_rate(self, optimizer):
            """Sets the learning rate to the initial LR decayed by 10 every 30 epochs"""
            # lr = args.lr * (0.1 ** (epoch // 30))
            # decay = 0.1 ** (sum(self.state['epoch'] >= np.array(self.state['epoch_step'])))
            # decay = 0.1 ** (self.state['epoch'] // self.state['epoch_step'])
            # lr = self.state['lr'] * decay # fixme
            for i, param_group in enumerate(optimizer.param_groups):
                if i == 0:
                    # print(param_group.)
                    decay = 0.1 ** (self.state['epoch'] // self.state['epoch_step'])
                    # print(self.state['lrp'])
                    param_group['lr'] = decay * self.state['lrp']  # fixme
                    # param_group['lr'] = lr
    
                    print('backbone learning rate', param_group['lr'])
                if i == 1:
                    decay = 0.1 ** (self.state['epoch'] // self.state['epoch_step'])
                    # print(self.state['lr'])
                    param_group['lr'] = decay * self.state['lr']  # fixme
                    # param_group['lr'] = lr
                    print('head learning rate', param_group['lr'])
    
            # return lr

即，由当前epoch得到学习率。//表示除以之后向下取整。

## 2.3 我们的方案 ##

因为是从第43次加载模型的，所以运行为0.001 \* 0.9^epoch

for i, param_group in enumerate(optimizer.param_groups):
                if i == 0:
                    # print(param_group.)
                    up=self.state['epoch']-43
                    decay = 0.9 ** up
                    # print(self.state['lrp'])
                    param_group['lr'] = decay * 0.001  # fixme
                    # param_group['lr'] = lr
    
                    print('backbone learning rate', param_group['lr'])
                if i == 1:
                    up = self.state['epoch'] - 43
                    decay = 0.9 ** up
                    # print(self.state['lr'])
                    param_group['lr'] = decay * 0.001  # fixme
                    # param_group['lr'] = lr
                    print('head learning rate', param_group['lr'])