保存微调后的Tensorflow模型时，列表索引超出范围

Question

我正在尝试使用Tensorflow对Huggingface的预训练BERT模型进行微调。一切运行顺利，模型构建和训练没有错误。但是当我试图保存模型时，它会停止错误"IndexError:列表索引超出范围"。我使用谷歌Colab与TPU。

任何帮助将非常感激!

代码:

import tensorflow as tf
from tensorflow.keras import activations, optimizers, losses
from transformers import TFBertModel
def create_model(max_sequence, model_name, num_labels):
bert_model = TFBertModel.from_pretrained(model_name)
input_ids = tf.keras.layers.Input(shape=(max_sequence,), dtype=tf.int32, name='input_ids')
attention_mask = tf.keras.layers.Input((max_sequence,), dtype=tf.int32, name='attention_mask')
output = bert_model([input_ids, attention_mask])[0]
output = output[:, 0, :]
output = tf.keras.layers.Dense(num_labels, activation='sigmoid')(output)
model = tf.keras.models.Model(inputs=[input_ids, attention_mask], outputs=output)
return model
with strategy.scope():
model = create_model(20, 'bert-base-uncased', 1)
opt = optimizers.Adam(learning_rate=3e-5)
loss = 'binary_crossentropy'
model.compile(optimizer=opt, loss=loss, metrics=['accuracy'])
model.fit(tfdataset_train, batch_size=32, epochs=2)
SAVE_PATH = 'path/to/save/location'
model.save(SAVE_PATH)

错误:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-22-255116b49022> in <module>()
1 SAVE_PATH = 'path/to/save/location'
----> 2 model.save(SAVE_PATH,save_format='tf')
50 frames
/usr/local/lib/python3.7/dist-packages/transformers/modeling_tf_utils.py in input_processing(func, config, input_ids, **kwargs)
372                     output[tensor_name] = input
373                 else:
--> 374                     output[parameter_names[i]] = input
375             elif isinstance(input, allowed_types) or input is None:
376                 output[parameter_names[i]] = input
IndexError: list index out of range

用图形绘制的模型:Tensorflow模型

Answer 1

解决方案是:

output = bert_model([input_ids, attention_mask])[0]

output = bert_model.bert([input_ids, attention_mask])[0]

参考:https://github.com/huggingface/transformers/issues/3627

我支持你发布的解决方案，但后来我在训练时发现了模型的问题。它不收敛。

Answer 2

在尝试保存一个微调的BERT模型(使用huggingface和tensorflow)时，我得到了相同的错误，该模型使用所有三个输入张量- input_ids, token_type_ids和attention_mask。

适合我的解决方案是对user_007答案进行稍微修改，如下所示:

output = bert_model.bert(
input_ids=input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask
)[0]

Answer 3

问题解决了!

删除两个输入层中的一个(即attention_mask)解决了这个问题。

工作代码->

import tensorflow as tf
from tensorflow.keras import activations, optimizers, losses
from transformers import TFBertModel
def create_model(max_sequence, model_name, num_labels):
bert_model = TFBertModel.from_pretrained(model_name)
input_ids = tf.keras.layers.Input(shape=(max_sequence,), dtype=tf.int32, name='input_ids')
#attention_mask = tf.keras.layers.Input((max_sequence,), dtype=tf.int32, name='attention_mask')
#output = bert_model([input_ids, attention_mask])[0]
output = bert_model([input_ids])[0]
output = output[:, 0, :]
output = tf.keras.layers.Dense(num_labels, activation='sigmoid')(output)
#model = tf.keras.models.Model(inputs=[input_ids, attention_mask], outputs=output)
model = tf.keras.models.Model(inputs=[input_ids], outputs=output)
return model
with strategy.scope():
model = create_model(20, 'bert-base-uncased', 1)
opt = optimizers.Adam(learning_rate=3e-5)
loss = 'binary_crossentropy'
model.compile(optimizer=opt, loss=loss, metrics=['accuracy'])
model.fit(tfdataset_train, batch_size=32, epochs=2)
SAVE_PATH = 'path/to/save/location'
model.save(SAVE_PATH)