Keras:如何保存模型并继续训练?

Question

我有一个模型，我已经训练了40个epoch。我保留了每个纪元的检查点，并且我还用model.save()保存了模型。训练代码为：

n_units = 1000
model = Sequential()
model.add(LSTM(n_units, input_shape=(None, vec_size), return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(n_units, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(n_units))
model.add(Dropout(0.2))
model.add(Dense(vec_size, activation='linear'))
model.compile(loss='mean_squared_error', optimizer='adam')
# define the checkpoint
filepath="word2vec-{epoch:02d}-{loss:.4f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
callbacks_list = [checkpoint]
# fit the model
model.fit(x, y, epochs=40, batch_size=50, callbacks=callbacks_list)

但是，当我加载模型并尝试再次训练它时，它会重新开始，就好像以前没有训练过一样。损失不是从上次训练开始的。

让我感到困惑的是，当我加载模型并重新定义模型结构并使用load_weight时，model.predict()运行良好。因此，我相信模型权重已加载：

model = Sequential()
model.add(LSTM(n_units, input_shape=(None, vec_size), return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(n_units, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(n_units))
model.add(Dropout(0.2))
model.add(Dense(vec_size, activation='linear'))
filename = "word2vec-39-0.0027.hdf5"
model.load_weights(filename)
model.compile(loss='mean_squared_error', optimizer='adam')

但是，当我继续用这个训练时，损失与初始阶段一样高：

filepath="word2vec-{epoch:02d}-{loss:.4f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
callbacks_list = [checkpoint]
# fit the model
model.fit(x, y, epochs=40, batch_size=50, callbacks=callbacks_list)

我搜索并在这里和这里找到了一些保存和加载模型的示例。但是，它们都不起作用。

---

更新 1

我看了这个问题，尝试了一下，它有效：

model.save('partly_trained.h5')
del model
load_model('partly_trained.h5')

但是当我关闭 Python 并重新打开它，然后再次运行load_model时，它失败了。损失与初始状态一样高。

---

更新 2

我尝试了Yu-Yang的示例代码，它可以工作。但是，当我再次使用我的代码时，它仍然失败了。

这是原始训练的结果。第二个纪元应以损失 = 3.1*** 开头：

13700/13846 [============================>.] - ETA: 0s - loss: 3.0519
13750/13846 [============================>.] - ETA: 0s - loss: 3.0511
13800/13846 [============================>.] - ETA: 0s - loss: 3.0512Epoch 00000: loss improved from inf to 3.05101, saving model to LPT-00-3.0510.h5
13846/13846 [==============================] - 81s - loss: 3.0510    
Epoch 2/60
50/13846 [..............................] - ETA: 80s - loss: 3.1754
100/13846 [..............................] - ETA: 78s - loss: 3.1174
150/13846 [..............................] - ETA: 78s - loss: 3.0745

我关闭了 Python，重新打开它，用model = load_model("LPT-00-3.0510.h5")加载模型，然后用：

filepath="LPT-{epoch:02d}-{loss:.4f}.h5"
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
callbacks_list = [checkpoint]
# fit the model
model.fit(x, y, epochs=60, batch_size=50, callbacks=callbacks_list)

损失从 4.54 开始：

Epoch 1/60
50/13846 [..............................] - ETA: 162s - loss: 4.5451
100/13846 [..............................] - ETA: 113s - loss: 4.3835

Answer 1

由于很难澄清问题出在哪里，我从您的代码创建了一个玩具示例，它似乎工作正常。

import numpy as np
from numpy.testing import assert_allclose
from keras.models import Sequential, load_model
from keras.layers import LSTM, Dropout, Dense
from keras.callbacks import ModelCheckpoint
vec_size = 100
n_units = 10
x_train = np.random.rand(500, 10, vec_size)
y_train = np.random.rand(500, vec_size)
model = Sequential()
model.add(LSTM(n_units, input_shape=(None, vec_size), return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(n_units, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(n_units))
model.add(Dropout(0.2))
model.add(Dense(vec_size, activation='linear'))
model.compile(loss='mean_squared_error', optimizer='adam')
# define the checkpoint
filepath = "model.h5"
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
callbacks_list = [checkpoint]
# fit the model
model.fit(x_train, y_train, epochs=5, batch_size=50, callbacks=callbacks_list)
# load the model
new_model = load_model(filepath)
assert_allclose(model.predict(x_train),
new_model.predict(x_train),
1e-5)
# fit the model
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
callbacks_list = [checkpoint]
new_model.fit(x_train, y_train, epochs=5, batch_size=50, callbacks=callbacks_list)

模型加载后，损失继续减少。(重新启动 Python 也没有问题(

Using TensorFlow backend.
Epoch 1/5
500/500 [==============================] - 2s - loss: 0.3216     Epoch 00000: loss improved from inf to 0.32163, saving model to model.h5
Epoch 2/5
500/500 [==============================] - 0s - loss: 0.2923     Epoch 00001: loss improved from 0.32163 to 0.29234, saving model to model.h5
Epoch 3/5
500/500 [==============================] - 0s - loss: 0.2542     Epoch 00002: loss improved from 0.29234 to 0.25415, saving model to model.h5
Epoch 4/5
500/500 [==============================] - 0s - loss: 0.2086     Epoch 00003: loss improved from 0.25415 to 0.20860, saving model to model.h5
Epoch 5/5
500/500 [==============================] - 0s - loss: 0.1725     Epoch 00004: loss improved from 0.20860 to 0.17249, saving model to model.h5
Epoch 1/5
500/500 [==============================] - 0s - loss: 0.1454     Epoch 00000: loss improved from inf to 0.14543, saving model to model.h5
Epoch 2/5
500/500 [==============================] - 0s - loss: 0.1289     Epoch 00001: loss improved from 0.14543 to 0.12892, saving model to model.h5
Epoch 3/5
500/500 [==============================] - 0s - loss: 0.1169     Epoch 00002: loss improved from 0.12892 to 0.11694, saving model to model.h5
Epoch 4/5
500/500 [==============================] - 0s - loss: 0.1097     Epoch 00003: loss improved from 0.11694 to 0.10971, saving model to model.h5
Epoch 5/5
500/500 [==============================] - 0s - loss: 0.1057     Epoch 00004: loss improved from 0.10971 to 0.10570, saving model to model.h5

顺便说一句，重新定义模型后跟load_weight()肯定行不通，因为save_weight()和load_weight()不会保存/加载优化器。

Answer 2

我将我的代码与此示例进行了比较 http://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/通过小心地逐行屏蔽并再次运行。过了一整天，终于，我发现了问题所在。

在制作 char-int 映射时，我使用了

# title_str_reduced is a string
chars = list(set(title_str_reduced))
# make char to int index mapping
char2int = {}
for i in range(len(chars)):
char2int[chars[i]] = i    

集合是一种无序数据结构。在python中，当一个集合被转换为一个有序的列表时，顺序是被随意给出的。因此，每次我重新打开 python 时，我的 char2int 字典都是随机的。 我通过添加排序((修复了我的代码

chars = sorted(list(set(title_str_reduced)))

这会强制转换为固定顺序。

Answer 3

上面的答案使用tensorflow 1.x。这是使用Tensorflow 2.x的更新版本。

import numpy as np
from numpy.testing import assert_allclose
from tensorflow.keras.models import Sequential, load_model
from tensorflow.keras.layers import LSTM, Dropout, Dense
from tensorflow.keras.callbacks import ModelCheckpoint
vec_size = 100
n_units = 10
x_train = np.random.rand(500, 10, vec_size)
y_train = np.random.rand(500, vec_size)
model = Sequential()
model.add(LSTM(n_units, input_shape=(None, vec_size), return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(n_units, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(n_units))
model.add(Dropout(0.2))
model.add(Dense(vec_size, activation='linear'))
model.compile(loss='mean_squared_error', optimizer='adam')
# define the checkpoint
filepath = "model.h5"
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
callbacks_list = [checkpoint]
# fit the model
model.fit(x_train, y_train, epochs=5, batch_size=50, callbacks=callbacks_list)
# load the model
new_model = load_model("model.h5")
assert_allclose(model.predict(x_train),
new_model.predict(x_train),
1e-5)
# fit the model
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
callbacks_list = [checkpoint]
new_model.fit(x_train, y_train, epochs=5, batch_size=50, callbacks=callbacks_list)

Answer 4

打勾的答案不正确;真正的问题更微妙。

创建ModelCheckpoint()时，请检查最佳：

cp1 = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
print(cp1.best)

您将看到这设置为np.inf，不幸的是，这不是您停止训练时的最后最佳成绩。因此，当您重新训练并重新创建ModelCheckpoint()时，如果您调用fit并且损失小于先前已知的值，那么它似乎有效，但在更复杂的问题中，您最终会保存一个糟糕的模型并失去最好的。

您可以通过覆盖cp.best参数来解决此问题，如下所示：

import numpy as np
from numpy.testing import assert_allclose
from keras.models import Sequential, load_model
from keras.layers import LSTM, Dropout, Dense
from keras.callbacks import ModelCheckpoint
vec_size = 100
n_units = 10
x_train = np.random.rand(500, 10, vec_size)
y_train = np.random.rand(500, vec_size)
model = Sequential()
model.add(LSTM(n_units, input_shape=(None, vec_size), return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(n_units, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(n_units))
model.add(Dropout(0.2))
model.add(Dense(vec_size, activation='linear'))
model.compile(loss='mean_squared_error', optimizer='adam')
# define the checkpoint
filepath = "model.h5"
cp1= ModelCheckpoint(filepath=filepath, monitor='loss',     save_best_only=True, verbose=1, mode='min')
callbacks_list = [cp1]
# fit the model
model.fit(x_train, y_train, epochs=5, batch_size=50, shuffle=True, validation_split=0.1, callbacks=callbacks_list)
# load the model
new_model = load_model(filepath)
#assert_allclose(model.predict(x_train),new_model.predict(x_train), 1e-5)
score = model.evaluate(x_train, y_train, batch_size=50)
cp1 = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
cp1.best = score # <== ****THIS IS THE KEY **** See source for  ModelCheckpoint
# fit the model
callbacks_list = [cp1]
new_model.fit(x_train, y_train, epochs=5, batch_size=50, callbacks=callbacks_list)

Answer 5

我想你可以写

model.save('partly_trained.h5' )

和

model = load_model('partly_trained.h5')

而不是

model = Sequential()
model.add(LSTM(n_units, input_shape=(None, vec_size), return_sequences=True))    
model.add(Dropout(0.2)) 
model.add(LSTM(n_units, return_sequences=True))  
model.add(Dropout(0.2)) 
model.add(LSTM(n_units))
model.add(Dropout(0.2))
model.add(Dense(vec_size, activation='linear')) 
model.compile(loss='mean_squared_error', optimizer='adam')

然后继续训练。 因为model.save存储架构和权重，你可以在文档中读到。

Answer 6

以下是官方 kera 保存模型的文档：

https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model

在这篇文章中，作者提供了两个将模型保存和加载到文件的示例：

• JSON 格式。
• 亚姆尔有孔虫。

Answer 7

假设你有这样的代码：

model = some_model_you_made(input_img) # you compiled your model in this 
model.summary()
model_checkpoint = ModelCheckpoint('yours.h5', monitor='val_loss', verbose=1, save_best_only=True)
model_json = model.to_json()
with open("yours.json", "w") as json_file:
json_file.write(model_json)
model.fit_generator(#stuff...) # or model.fit(#stuff...)

现在将你的代码变成这样：

model = some_model_you_made(input_img) #same model here
model.summary()
model_checkpoint = ModelCheckpoint('yours.h5', monitor='val_loss', verbose=1, save_best_only=True) #same ckeckpoint
model_json = model.to_json()
with open("yours.json", "w") as json_file:
json_file.write(model_json)
with open('yours.json', 'r') as f:
old_model = model_from_json(f.read()) # open the model you just saved (same as your last train) with a different name
old_model.load_weights('yours.h5') # the model checkpoint you trained before
old_model.compile(#stuff...) # need to compile again (exactly like the last compile)
# now start training with the checkpoint...
old_model.fit_generator(#same stuff like the last train) # or model.fit(#stuff...)

Answer 8

由于 Keras 和 Tensorflow 现在是捆绑在一起的，您可以使用更新的 Tensorflow 格式，该格式将保存所有模型信息，包括优化器及其状态(来自文档，强调我的(：

您可以将整个模型保存到单个工件中。它将包括：

• 模型的体系结构/配置
• 模型的权重值(在训练期间学习(
• 模型的编译信息(如果调用了 compile(( (
• 优化器及其状态(如果有((这使您能够从离开的地方重新开始训练(

蜜蜂属

• model.save()或tf.keras.models.save_model()
• tf.keras.models.load_model((

因此，以这种方式保存模型后，您可以加载它并继续训练：它将从中断的地方继续。