如何获得PyTorch的实际学习率?

Question

我正在尝试使用PyTorch为我的神经网络找到适当的学习率。我已经实现了torch.optim.lr_scheduler.CyclicLR来获得学习率。但是我不知道应该选择的实际学习率是多少。数据集为MNIST_TINY。

代码:

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=1e-7, max_lr=0.1, step_size_up=5., mode='triangular')
lrs= []
for epoch in range(5):
model.train()
for data, label in train_dls:
optimizer.zero_grad()
target = model(data)
train_step_loss = loss_fn(target, label) #CrossEntropyLoss
train_step_loss.backward()
optimizer.step()
print(f'Epoch:{epoch+1} | Optim:{optimizer.param_groups[0]["lr"]:.4f} | Loss: {train_step_loss:.2f}')
lrs.append(optimizer.param_groups[0]["lr"])
scheduler.step()

Epoch:1 | Optim:0.0000 | Loss: 0.70
Epoch:1 | Optim:0.0000 | Loss: 0.70
Epoch:1 | Optim:0.0000 | Loss: 0.70
Epoch:1 | Optim:0.0000 | Loss: 0.70
Epoch:1 | Optim:0.0000 | Loss: 0.70
Epoch:1 | Optim:0.0000 | Loss: 0.69
Epoch:1 | Optim:0.0000 | Loss: 0.69
Epoch:1 | Optim:0.0000 | Loss: 0.68
Epoch:1 | Optim:0.0000 | Loss: 0.69
Epoch:1 | Optim:0.0000 | Loss: 0.69
Epoch:1 | Optim:0.0000 | Loss: 0.69
Epoch:1 | Optim:0.0000 | Loss: 0.72
Epoch:2 | Optim:0.0200 | Loss: 0.70
Epoch:2 | Optim:0.0200 | Loss: 0.70
Epoch:2 | Optim:0.0200 | Loss: 0.70
Epoch:2 | Optim:0.0200 | Loss: 0.70
Epoch:2 | Optim:0.0200 | Loss: 0.69
Epoch:2 | Optim:0.0200 | Loss: 0.69
Epoch:2 | Optim:0.0200 | Loss: 0.69
Epoch:2 | Optim:0.0200 | Loss: 0.69
Epoch:2 | Optim:0.0200 | Loss: 0.69
Epoch:2 | Optim:0.0200 | Loss: 0.69
Epoch:2 | Optim:0.0200 | Loss: 0.70
Epoch:2 | Optim:0.0200 | Loss: 0.68
Epoch:3 | Optim:0.0400 | Loss: 0.70
Epoch:3 | Optim:0.0400 | Loss: 0.70
Epoch:3 | Optim:0.0400 | Loss: 0.70
Epoch:3 | Optim:0.0400 | Loss: 0.68
Epoch:3 | Optim:0.0400 | Loss: 0.68
Epoch:3 | Optim:0.0400 | Loss: 0.69
Epoch:3 | Optim:0.0400 | Loss: 0.70
Epoch:3 | Optim:0.0400 | Loss: 0.68
Epoch:3 | Optim:0.0400 | Loss: 0.70
Epoch:3 | Optim:0.0400 | Loss: 0.69
Epoch:3 | Optim:0.0400 | Loss: 0.70
Epoch:3 | Optim:0.0400 | Loss: 0.65
Epoch:4 | Optim:0.0600 | Loss: 0.69
Epoch:4 | Optim:0.0600 | Loss: 0.68
Epoch:4 | Optim:0.0600 | Loss: 0.68
Epoch:4 | Optim:0.0600 | Loss: 0.73
Epoch:4 | Optim:0.0600 | Loss: 0.70
Epoch:4 | Optim:0.0600 | Loss: 0.71
Epoch:4 | Optim:0.0600 | Loss: 0.71
Epoch:4 | Optim:0.0600 | Loss: 0.68
Epoch:4 | Optim:0.0600 | Loss: 0.71
Epoch:4 | Optim:0.0600 | Loss: 0.69
Epoch:4 | Optim:0.0600 | Loss: 0.69
Epoch:4 | Optim:0.0600 | Loss: 0.72
Epoch:5 | Optim:0.0800 | Loss: 0.69
Epoch:5 | Optim:0.0800 | Loss: 0.69
Epoch:5 | Optim:0.0800 | Loss: 0.70
Epoch:5 | Optim:0.0800 | Loss: 0.69
Epoch:5 | Optim:0.0800 | Loss: 0.69
Epoch:5 | Optim:0.0800 | Loss: 0.68
Epoch:5 | Optim:0.0800 | Loss: 0.71
Epoch:5 | Optim:0.0800 | Loss: 0.68
Epoch:5 | Optim:0.0800 | Loss: 0.71
Epoch:5 | Optim:0.0800 | Loss: 0.69
Epoch:5 | Optim:0.0800 | Loss: 0.71
Epoch:5 | Optim:0.0800 | Loss: 0.70

简而言之，我想问我如何找到正确的学习速率?如果有人能告诉我如何用loss绘制learning rate图，我将不胜感激。

Answer 1

From https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CyclicLR.html#torch.optim.lr_scheduler.CyclicLR

您必须使用get_last_lr()来获得该调度程序的最后学习率。