Pytorch 根据纪元数更改学习率

Question

当我设置学习率并发现训练几个 epoch 后准确性无法提高时

optimizer = optim.Adam(model.parameters(), lr = 1e-4)
n_epochs = 10
for i in range(n_epochs):
// some training here

如果我想使用阶跃衰减：每 5 个 epoch 将学习率降低 10 倍，我该怎么做？

Answer 1

您可以使用学习率调度程序torch.optim.lr_scheduler.StepLR

import torch.optim.lr_scheduler.StepLR
scheduler = StepLR(optimizer, step_size=5, gamma=0.1)

通过每step_size个时期gamma衰减每个参数组的学习率，请在此处查看文档 文档中的示例

# Assuming optimizer uses lr = 0.05 for all groups
# lr = 0.05     if epoch < 30
# lr = 0.005    if 30 <= epoch < 60
# lr = 0.0005   if 60 <= epoch < 90
# ...
scheduler = StepLR(optimizer, step_size=30, gamma=0.1)
for epoch in range(100):
train(...)
validate(...)
scheduler.step()

例：

import torch
import torch.optim as optim
optimizer = optim.SGD([torch.rand((2,2), requires_grad=True)], lr=0.1)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)

for epoch in range(1, 21):
scheduler.step()
print('Epoch-{0} lr: {1}'.format(epoch, optimizer.param_groups[0]['lr']))
if epoch % 5 == 0:print()

Epoch-1 lr: 0.1
Epoch-2 lr: 0.1
Epoch-3 lr: 0.1
Epoch-4 lr: 0.1
Epoch-5 lr: 0.1
Epoch-6 lr: 0.010000000000000002
Epoch-7 lr: 0.010000000000000002
Epoch-8 lr: 0.010000000000000002
Epoch-9 lr: 0.010000000000000002
Epoch-10 lr: 0.010000000000000002
Epoch-11 lr: 0.0010000000000000002
Epoch-12 lr: 0.0010000000000000002
Epoch-13 lr: 0.0010000000000000002
Epoch-14 lr: 0.0010000000000000002
Epoch-15 lr: 0.0010000000000000002
Epoch-16 lr: 0.00010000000000000003
Epoch-17 lr: 0.00010000000000000003
Epoch-18 lr: 0.00010000000000000003
Epoch-19 lr: 0.00010000000000000003
Epoch-20 lr: 0.00010000000000000003

更多关于如何调整学习率 -torch.optim.lr_scheduler提供了几种根据周期数调整学习率的方法。

Answer 2

我更喜欢这个而不是从torch模块中获得

def adjust_learning_rate_poly(optimizer, initial_lr, iteration, max_iter):
"""Sets the learning rate
# Adapted from PyTorch Imagenet example:
# https://github.com/pytorch/examples/blob/master/imagenet/main.py
"""
lr = initial_lr * ( 1 - (iteration / max_iter)) * ( 1 - (iteration / max_iter))
if ( lr < 1.0e-7 ):
lr = 1.0e-7
for param_group in optimizer.param_groups:
param_group['lr'] = lr
return lr
for epoch in range(0, max_epoch):
lr = adjust_learning_rate_poly(optimizer, lr_rate, epoch, max_epoch)

lr_rate是我的起点learning_rate