如何在 Pytorch 中冻结模型的选定层?

Question

我正在使用mobileNetV2，我只想冻结模型的一部分。我知道我可以使用以下代码来冻结整个模型

MobileNet = models.mobilenet_v2(pretrained = True)
for param in MobileNet.parameters():
param.requires_grad = False

但我希望从 (15( 开始的所有内容都保持不冻结。如何在冻结所需层之前有选择地冻结所有内容？

(15): InvertedResidual(
(conv): Sequential(
(0): ConvBNReLU(
(0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU6(inplace=True)
)
(1): ConvBNReLU(
(0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)
(1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU6(inplace=True)
)
(2): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)
(3): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(16): InvertedResidual(
(conv): Sequential(
(0): ConvBNReLU(
(0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU6(inplace=True)
)
(1): ConvBNReLU(
(0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)
(1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU6(inplace=True)
)
(2): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)
(3): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(17): InvertedResidual(
(conv): Sequential(
(0): ConvBNReLU(
(0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU6(inplace=True)
)
(1): ConvBNReLU(
(0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)
(1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU6(inplace=True)
)
(2): Conv2d(960, 320, kernel_size=(1, 1), stride=(1, 1), bias=False)
(3): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(18): ConvBNReLU(
(0): Conv2d(320, 1280, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(1280, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU6(inplace=True)
)   )   (classifier): Sequential(
(0): Dropout(p=0.2, inplace=False)
(1): Linear(in_features=1280, out_features=1000, bias=True)   ) )

Answer 1

Pytorch 的模型实现具有良好的模块化，所以像你一样

for param in MobileNet.parameters():
param.requires_grad = False

，你也可以做

for param in MobileNet.features[15].parameters():
param.requires_grad = True

之后解冻 (15( 中的参数。

从 15 循环到 18 以解冻最后几层。

Answer 2

为了完整起见，只是在这里添加它。您还可以就地冻结参数，而无需使用requires_grad_(API( 迭代它们。

例如，假设您有一个RetinaNet，并且只想对头部进行微调

class RetinaNet(torch.nn.Module):
def __init__(self, ...):
self.backbone = ResNet(...)
self.fpn = FPN(...)
self.box_head = torch.nn.Sequential(...)
self.cls_head = torch.nn.Sequential(...)

然后你可以像这样冻结主干和 FPN：

# Getting the model
retinanet = RetinaNet(...)
# Freezing backbone and FPN
retinanet.backbone.requires_grad_(False)
retinanet.fpn.requires_grad_(False)

Answer 3

如果你想按名称定义一些层，然后解冻它们，我提出了@JVGD答案的变体：

class RetinaNet(torch.nn.Module):
def __init__(self, ...):
self.backbone = ResNet(...)
self.fpn = FPN(...)
self.box_head = torch.nn.Sequential(...)
self.cls_head = torch.nn.Sequential(...)
# Getting the model
retinanet = RetinaNet(...)
# The param name is f'{module_name}.weight' or f'{module_name}.bias'.
# Some layers, e.g., batch norm, have additional params.
# In some circumstances, e.g., when using DataParallel(), 
# the param name is prefixed by 'module.'.
params_to_train = ['cls_head.weight', 'cls_head.bias']
for name, param in retinanet.named_parameters():
# Set True only for params in the list 'params_to_train'
param.requires_grad = True if name in params_to_train else False
...

优点是您可以在一个Iterable中定义要解冻的所有图层。

Answer 4

由于有不同类型的模型，有时单独在块上设置required_grad=True是行不通的*。

选项 1

# freeze everything
for param in model.parameters():
param.requires_grad = False
# and Un-Freeze lower 4 layers of encoder 
for i in range(0,num_encoder_layers-8,1):
for param in model.encoder.block[i].parameters():
param.requires_grad = True
#verify
for name, param in model.named_parameters():
print(name,param.requires_grad)

选项 2

# Freeze upper 3 layers of encoder (lower is unfreezed)
for i in range(num_encoder_layers-1,num_encoder_layers-4,-1):
for param in model.encoder.block[i].parameters():
param.requires_grad = False
# Freeze all layers of decoder
for i in range(num_decoder_layers):
for param in model.decoder.block[i].parameters():
param.requires_grad = False
for name, param in model.named_parameters():
print(name,param.requires_grad)

根据您冻结的内容，您会得到这样的东西

输出

shared.weight False
encoder.block.0.layer.0.SelfAttention.q.weight True
encoder.block.0.layer.0.SelfAttention.k.weight True
encoder.block.0.layer.0.SelfAttention.v.weight True
encoder.block.0.layer.0.SelfAttention.o.weight True
encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight True
encoder.block.0.layer.0.layer_norm.weight True
encoder.block.0.layer.1.DenseReluDense.wi_0.weight True
encoder.block.0.layer.1.DenseReluDense.wi_1.weight True
encoder.block.0.layer.1.DenseReluDense.wo.weight True
encoder.block.0.layer.1.layer_norm.weight True
encoder.block.1.layer.0.SelfAttention.q.weight True
encoder.block.1.layer.0.SelfAttention.k.weight True
encoder.block.1.layer.0.SelfAttention.v.weight True
encoder.block.1.layer.0.SelfAttention.o.weight True
encoder.block.1.layer.0.layer_norm.weight True
encoder.block.1.layer.1.DenseReluDense.wi_0.weight True
encoder.block.1.layer.1.DenseReluDense.wi_1.weight True
encoder.block.1.layer.1.DenseReluDense.wo.weight True
encoder.block.1.layer.1.layer_norm.weight True
encoder.block.2.layer.0.SelfAttention.q.weight True
encoder.block.2.layer.0.SelfAttention.k.weight True
encoder.block.2.layer.0.SelfAttention.v.weight True
encoder.block.2.layer.0.SelfAttention.o.weight True
encoder.block.2.layer.0.layer_norm.weight True
encoder.block.2.layer.1.DenseReluDense.wi_0.weight True
encoder.block.2.layer.1.DenseReluDense.wi_1.weight True
encoder.block.2.layer.1.DenseReluDense.wo.weight True
encoder.block.2.layer.1.layer_norm.weight True
encoder.block.3.layer.0.SelfAttention.q.weight True
encoder.block.3.layer.0.SelfAttention.k.weight True
encoder.block.3.layer.0.SelfAttention.v.weight True
encoder.block.3.layer.0.SelfAttention.o.weight True
encoder.block.3.layer.0.layer_norm.weight True
encoder.block.3.layer.1.DenseReluDense.wi_0.weight True
encoder.block.3.layer.1.DenseReluDense.wi_1.weight True
encoder.block.3.layer.1.DenseReluDense.wo.weight True
encoder.block.3.layer.1.layer_norm.weight True
encoder.block.4.layer.0.SelfAttention.q.weight False
encoder.block.4.layer.0.SelfAttention.k.weight False
encoder.block.4.layer.0.SelfAttention.v.weight False
encoder.block.4.layer.0.SelfAttention.o.weight False
encoder.block.4.layer.0.layer_norm.weight False
encoder.block.4.layer.1.DenseReluDense.wi_0.weight False
encoder.block.4.layer.1.DenseReluDense.wi_1.weight False
encoder.block.4.layer.1.DenseReluDense.wo.weight False
encoder.block.4.layer.1.layer_norm.weight False
encoder.block.5.layer.0.SelfAttention.q.weight False
encoder.block.5.layer.0.SelfAttention.k.weight False
encoder.block.5.layer.0.SelfAttention.v.weight False
encoder.block.5.layer.0.SelfAttention.o.weight False
encoder.block.5.layer.0.layer_norm.weight False
encoder.block.5.layer.1.DenseReluDense.wi_0.weight False
encoder.block.5.layer.1.DenseReluDense.wi_1.weight False
encoder.block.5.layer.1.DenseReluDense.wo.weight False
encoder.block.5.layer.1.layer_norm.weight False
encoder.block.6.layer.0.SelfAttention.q.weight False
encoder.block.6.layer.0.SelfAttention.k.weight False
encoder.block.6.layer.0.SelfAttention.v.weight False
encoder.block.6.layer.0.SelfAttention.o.weight False
encoder.block.6.layer.0.layer_norm.weight False
encoder.block.6.layer.1.DenseReluDense.wi_0.weight False
encoder.block.6.layer.1.DenseReluDense.wi_1.weight False
encoder.block.6.layer.1.DenseReluDense.wo.weight False
encoder.block.6.layer.1.layer_norm.weight False
encoder.block.7.layer.0.SelfAttention.q.weight False
encoder.block.7.layer.0.SelfAttention.k.weight False
encoder.block.7.layer.0.SelfAttention.v.weight False
encoder.block.7.layer.0.SelfAttention.o.weight False
encoder.block.7.layer.0.layer_norm.weight False
encoder.block.7.layer.1.DenseReluDense.wi_0.weight False
encoder.block.7.layer.1.DenseReluDense.wi_1.weight False
encoder.block.7.layer.1.DenseReluDense.wo.weight False
encoder.block.7.layer.1.layer_norm.weight False
encoder.block.8.layer.0.SelfAttention.q.weight False
encoder.block.8.layer.0.SelfAttention.k.weight False
encoder.block.8.layer.0.SelfAttention.v.weight False
encoder.block.8.layer.0.SelfAttention.o.weight False
encoder.block.8.layer.0.layer_norm.weight False
encoder.block.8.layer.1.DenseReluDense.wi_0.weight False
encoder.block.8.layer.1.DenseReluDense.wi_1.weight False
encoder.block.8.layer.1.DenseReluDense.wo.weight False
encoder.block.8.layer.1.layer_norm.weight False
encoder.block.9.layer.0.SelfAttention.q.weight False
encoder.block.9.layer.0.SelfAttention.k.weight False
encoder.block.9.layer.0.SelfAttention.v.weight False
encoder.block.9.layer.0.SelfAttention.o.weight False
encoder.block.9.layer.0.layer_norm.weight False
encoder.block.9.layer.1.DenseReluDense.wi_0.weight False
encoder.block.9.layer.1.DenseReluDense.wi_1.weight False
encoder.block.9.layer.1.DenseReluDense.wo.weight False
encoder.block.9.layer.1.layer_norm.weight False
encoder.block.10.layer.0.SelfAttention.q.weight False
encoder.block.10.layer.0.SelfAttention.k.weight False
encoder.block.10.layer.0.SelfAttention.v.weight False
encoder.block.10.layer.0.SelfAttention.o.weight False
encoder.block.10.layer.0.layer_norm.weight False
encoder.block.10.layer.1.DenseReluDense.wi_0.weight False
encoder.block.10.layer.1.DenseReluDense.wi_1.weight False
encoder.block.10.layer.1.DenseReluDense.wo.weight False
encoder.block.10.layer.1.layer_norm.weight False
encoder.block.11.layer.0.SelfAttention.q.weight False
encoder.block.11.layer.0.SelfAttention.k.weight False
encoder.block.11.layer.0.SelfAttention.v.weight False
encoder.block.11.layer.0.SelfAttention.o.weight False
encoder.block.11.layer.0.layer_norm.weight False
encoder.block.11.layer.1.DenseReluDense.wi_0.weight False
encoder.block.11.layer.1.DenseReluDense.wi_1.weight False
encoder.block.11.layer.1.DenseReluDense.wo.weight False
encoder.block.11.layer.1.layer_norm.weight False
encoder.final_layer_norm.weight False
decoder.block.0.layer.0.SelfAttention.q.weight False
decoder.block.0.layer.0.SelfAttention.k.weight False
decoder.block.0.layer.0.SelfAttention.v.weight False
decoder.block.0.layer.0.SelfAttention.o.weight False
decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight False
decoder.block.0.layer.0.layer_norm.weight False
decoder.block.0.layer.1.EncDecAttention.q.weight False
decoder.block.0.layer.1.EncDecAttention.k.weight False
decoder.block.0.layer.1.EncDecAttention.v.weight False
decoder.block.0.layer.1.EncDecAttention.o.weight False
decoder.block.0.layer.1.layer_norm.weight False
decoder.block.0.layer.2.DenseReluDense.wi_0.weight False
decoder.block.0.layer.2.DenseReluDense.wi_1.weight False
decoder.block.0.layer.2.DenseReluDense.wo.weight False
decoder.block.0.layer.2.layer_norm.weight False
decoder.block.1.layer.0.SelfAttention.q.weight False
decoder.block.1.layer.0.SelfAttention.k.weight False
decoder.block.1.layer.0.SelfAttention.v.weight False
decoder.block.1.layer.0.SelfAttention.o.weight False
decoder.block.1.layer.0.layer_norm.weight False
decoder.block.1.layer.1.EncDecAttention.q.weight False
decoder.block.1.layer.1.EncDecAttention.k.weight False
decoder.block.1.layer.1.EncDecAttention.v.weight False
decoder.block.1.layer.1.EncDecAttention.o.weight False
decoder.block.1.layer.1.layer_norm.weight False
decoder.block.1.layer.2.DenseReluDense.wi_0.weight False
decoder.block.1.layer.2.DenseReluDense.wi_1.weight False
decoder.block.1.layer.2.DenseReluDense.wo.weight False
decoder.block.1.layer.2.layer_norm.weight False
decoder.block.2.layer.0.SelfAttention.q.weight False
decoder.block.2.layer.0.SelfAttention.k.weight False
decoder.block.2.layer.0.SelfAttention.v.weight False
decoder.block.2.layer.0.SelfAttention.o.weight False
decoder.block.2.layer.0.layer_norm.weight False
decoder.block.2.layer.1.EncDecAttention.q.weight False
decoder.block.2.layer.1.EncDecAttention.k.weight False
decoder.block.2.layer.1.EncDecAttention.v.weight False
decoder.block.2.layer.1.EncDecAttention.o.weight False
decoder.block.2.layer.1.layer_norm.weight False
decoder.block.2.layer.2.DenseReluDense.wi_0.weight False
decoder.block.2.layer.2.DenseReluDense.wi_1.weight False
decoder.block.2.layer.2.DenseReluDense.wo.weight False
decoder.block.2.layer.2.layer_norm.weight False
decoder.block.3.layer.0.SelfAttention.q.weight False
decoder.block.3.layer.0.SelfAttention.k.weight False
decoder.block.3.layer.0.SelfAttention.v.weight False
decoder.block.3.layer.0.SelfAttention.o.weight False
decoder.block.3.layer.0.layer_norm.weight False
decoder.block.3.layer.1.EncDecAttention.q.weight False
decoder.block.3.layer.1.EncDecAttention.k.weight False
decoder.block.3.layer.1.EncDecAttention.v.weight False
decoder.block.3.layer.1.EncDecAttention.o.weight False
decoder.block.3.layer.1.layer_norm.weight False
decoder.block.3.layer.2.DenseReluDense.wi_0.weight False
decoder.block.3.layer.2.DenseReluDense.wi_1.weight False
decoder.block.3.layer.2.DenseReluDense.wo.weight False
decoder.block.3.layer.2.layer_norm.weight False
decoder.block.4.layer.0.SelfAttention.q.weight False
decoder.block.4.layer.0.SelfAttention.k.weight False
decoder.block.4.layer.0.SelfAttention.v.weight False
decoder.block.4.layer.0.SelfAttention.o.weight False
decoder.block.4.layer.0.layer_norm.weight False
decoder.block.4.layer.1.EncDecAttention.q.weight False
decoder.block.4.layer.1.EncDecAttention.k.weight False
decoder.block.4.layer.1.EncDecAttention.v.weight False
decoder.block.4.layer.1.EncDecAttention.o.weight False
decoder.block.4.layer.1.layer_norm.weight False
decoder.block.4.layer.2.DenseReluDense.wi_0.weight False
decoder.block.4.layer.2.DenseReluDense.wi_1.weight False
decoder.block.4.layer.2.DenseReluDense.wo.weight False
decoder.block.4.layer.2.layer_norm.weight False
decoder.block.5.layer.0.SelfAttention.q.weight False
decoder.block.5.layer.0.SelfAttention.k.weight False
decoder.block.5.layer.0.SelfAttention.v.weight False
decoder.block.5.layer.0.SelfAttention.o.weight False
decoder.block.5.layer.0.layer_norm.weight False
decoder.block.5.layer.1.EncDecAttention.q.weight False
decoder.block.5.layer.1.EncDecAttention.k.weight False
decoder.block.5.layer.1.EncDecAttention.v.weight False
decoder.block.5.layer.1.EncDecAttention.o.weight False
decoder.block.5.layer.1.layer_norm.weight False
decoder.block.5.layer.2.DenseReluDense.wi_0.weight False
decoder.block.5.layer.2.DenseReluDense.wi_1.weight False
decoder.block.5.layer.2.DenseReluDense.wo.weight False
decoder.block.5.layer.2.layer_norm.weight False
decoder.block.6.layer.0.SelfAttention.q.weight False
decoder.block.6.layer.0.SelfAttention.k.weight False
decoder.block.6.layer.0.SelfAttention.v.weight False
decoder.block.6.layer.0.SelfAttention.o.weight False
decoder.block.6.layer.0.layer_norm.weight False
decoder.block.6.layer.1.EncDecAttention.q.weight False
decoder.block.6.layer.1.EncDecAttention.k.weight False
decoder.block.6.layer.1.EncDecAttention.v.weight False
decoder.block.6.layer.1.EncDecAttention.o.weight False
decoder.block.6.layer.1.layer_norm.weight False
decoder.block.6.layer.2.DenseReluDense.wi_0.weight False
decoder.block.6.layer.2.DenseReluDense.wi_1.weight False
decoder.block.6.layer.2.DenseReluDense.wo.weight False
decoder.block.6.layer.2.layer_norm.weight False
decoder.block.7.layer.0.SelfAttention.q.weight False
decoder.block.7.layer.0.SelfAttention.k.weight False
decoder.block.7.layer.0.SelfAttention.v.weight False
decoder.block.7.layer.0.SelfAttention.o.weight False
decoder.block.7.layer.0.layer_norm.weight False
decoder.block.7.layer.1.EncDecAttention.q.weight False
decoder.block.7.layer.1.EncDecAttention.k.weight False
decoder.block.7.layer.1.EncDecAttention.v.weight False
decoder.block.7.layer.1.EncDecAttention.o.weight False
decoder.block.7.layer.1.layer_norm.weight False
decoder.block.7.layer.2.DenseReluDense.wi_0.weight False
decoder.block.7.layer.2.DenseReluDense.wi_1.weight False
decoder.block.7.layer.2.DenseReluDense.wo.weight False
decoder.block.7.layer.2.layer_norm.weight False
decoder.block.8.layer.0.SelfAttention.q.weight False
decoder.block.8.layer.0.SelfAttention.k.weight False
decoder.block.8.layer.0.SelfAttention.v.weight False
decoder.block.8.layer.0.SelfAttention.o.weight False
decoder.block.8.layer.0.layer_norm.weight False
decoder.block.8.layer.1.EncDecAttention.q.weight False
decoder.block.8.layer.1.EncDecAttention.k.weight False
decoder.block.8.layer.1.EncDecAttention.v.weight False
decoder.block.8.layer.1.EncDecAttention.o.weight False
decoder.block.8.layer.1.layer_norm.weight False
decoder.block.8.layer.2.DenseReluDense.wi_0.weight False
decoder.block.8.layer.2.DenseReluDense.wi_1.weight False
decoder.block.8.layer.2.DenseReluDense.wo.weight False
decoder.block.8.layer.2.layer_norm.weight False
decoder.block.9.layer.0.SelfAttention.q.weight False
decoder.block.9.layer.0.SelfAttention.k.weight False
decoder.block.9.layer.0.SelfAttention.v.weight False
decoder.block.9.layer.0.SelfAttention.o.weight False
decoder.block.9.layer.0.layer_norm.weight False
decoder.block.9.layer.1.EncDecAttention.q.weight False
decoder.block.9.layer.1.EncDecAttention.k.weight False
decoder.block.9.layer.1.EncDecAttention.v.weight False
decoder.block.9.layer.1.EncDecAttention.o.weight False
decoder.block.9.layer.1.layer_norm.weight False
decoder.block.9.layer.2.DenseReluDense.wi_0.weight False
decoder.block.9.layer.2.DenseReluDense.wi_1.weight False
decoder.block.9.layer.2.DenseReluDense.wo.weight False
decoder.block.9.layer.2.layer_norm.weight False
decoder.block.10.layer.0.SelfAttention.q.weight False
decoder.block.10.layer.0.SelfAttention.k.weight False
decoder.block.10.layer.0.SelfAttention.v.weight False
decoder.block.10.layer.0.SelfAttention.o.weight False
decoder.block.10.layer.0.layer_norm.weight False
decoder.block.10.layer.1.EncDecAttention.q.weight False
decoder.block.10.layer.1.EncDecAttention.k.weight False
decoder.block.10.layer.1.EncDecAttention.v.weight False
decoder.block.10.layer.1.EncDecAttention.o.weight False
decoder.block.10.layer.1.layer_norm.weight False
decoder.block.10.layer.2.DenseReluDense.wi_0.weight False
decoder.block.10.layer.2.DenseReluDense.wi_1.weight False
decoder.block.10.layer.2.DenseReluDense.wo.weight False
decoder.block.10.layer.2.layer_norm.weight False
decoder.block.11.layer.0.SelfAttention.q.weight False
decoder.block.11.layer.0.SelfAttention.k.weight False
decoder.block.11.layer.0.SelfAttention.v.weight False
decoder.block.11.layer.0.SelfAttention.o.weight False
decoder.block.11.layer.0.layer_norm.weight False
decoder.block.11.layer.1.EncDecAttention.q.weight False
decoder.block.11.layer.1.EncDecAttention.k.weight False
decoder.block.11.layer.1.EncDecAttention.v.weight False
decoder.block.11.layer.1.EncDecAttention.o.weight False
decoder.block.11.layer.1.layer_norm.weight False
decoder.block.11.layer.2.DenseReluDense.wi_0.weight False
decoder.block.11.layer.2.DenseReluDense.wi_1.weight False
decoder.block.11.layer.2.DenseReluDense.wo.weight False
decoder.block.11.layer.2.layer_norm.weight False
decoder.final_layer_norm.weight False
lm_head.weight False

• 示例此不起作用

for i in range(-num_encoder_layers//2, 0):  
model.encoder.block[i].requires_grad = True

Answer 5

上述第一个答案的优化答案是仅冻结前 15 层 [0-14]，因为最后一层 [15-18] 默认未冻结 (param.requires_grad = True(。

因此，我们只需要这样编码：

MobileNet = torchvision.models.mobilenet_v2(pretrained = True)
for param in MobileNet.features[0:14].parameters():
param.requires_grad = False

Answer 6

您可以使用以下代码片段迭代所需的层：

for name, child in MobileNet.named_children:
if name < 15 :
for param in child.parameters():
param.requires_grad = False