我有什么：
- 一个神经网络模型
- 10个结构相同的数据集
我想要什么：
- 分别在所有数据集上训练模型
- 单独保存模型
我可以单独训练数据集，并一次保存一个模型。但我想加载我的10个数据集，并在一次运行中用它们创建10个模型。解决方案可能很明显，但我对此还相当陌生。我该如何做到这一点？
提前谢谢。
您可以使用concurrency and parallelism的一个概念，即Multi-Threading，或者在某些情况下使用Multi-Processing来实现这一点
最简单的编码方法是使用python的concurrent-futures模块
您可以为要使用的每个数据集调用模型上的训练函数，所有这些都在ThreadPoolExecutor下，以便激发并行线程来执行单独的训练。
代码可能有点像这样：
步骤1：必要的导入
from concurrent.futures import ThreadPoolExecutor, as_completed
import tensorflow as tf
from tensorflow.keras.models import load_model, Sequential
from tensorflow.keras.layers import Dense, Activation, Flatten
步骤2：创建和构建模型
def create_model(): # responsible for creating model
model = Sequential()
model.add(Flatten()) # adding NN layers
model.add(Dense(64))
model.add(Activation('relu'))
# ........ so on
model.compile(optimizer='..', loss='..', metrics=[...]) # compiling the model
return model # finally returning the model
步骤3：定义拟合函数(执行模型训练(
def fit(model, XY_train): # performs model.fit(...parameters...)
model.fit(XY_train[0], XY_train[1], epochs=5, validation_split=0.3) # use your already defined x_train, y_train
return model # finally returns trained model
步骤4：并行训练器方法，使用TPE上下文管理器启动同步训练
# trains provided model on each dataset parallelly by using multi-threading
def parallel_trainer(model, XY_train_datasets : list[tuple]):
with ThreadPoolExecutor(max_workers = len(XY_train_datasets)) as executor:
futureObjs = [
executor.submit(
lambda ds: fit(model, ds), XY_train_datasets) # Call Fit for each dataset iterate through the datasets
]
for i, obj in enumerate(as_completed(futureObjs)): # iterate through trained models
(obj.result()).save(f"{i}.model") # save models
步骤5：创建模型，加载数据集，调用并行训练器
model = create_model() # create the model
mnist = tf.keras.datasets.mnist # get dataset - for example :- mnist dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data() # get (x_train, y_train), (x_test, y_test)
datasets = [(x_train, y_train)]*10 # list of dataset paths (in your case, same dataset used 10 times)
parallel_trainer(model, datasets) # call parallel trainer
整个程序
from concurrent.futures import ThreadPoolExecutor, as_completed
import tensorflow as tf
from tensorflow.keras.models import load_model, Sequential
from tensorflow.keras.layers import Dense, Activation, Flatten
def create_model(): # responsible for creating model
model = Sequential()
model.add(Flatten()) # adding NN layers
model.add(Dense(64))
model.add(Activation('relu'))
# ........ so on
model.compile(optimizer='..', loss='..', metrics=[...]) # compiling the model
return model # finally returning the model
def fit(model, XY_train): # performs model.fit(...parameters...)
model.fit(XY_train[0], XY_train[1], epochs=5, validation_split=0.3) # use your already defined x_train, y_train
return model # finally returns trained model
# trains provided model on each dataset parallelly by using multi-threading
def parallel_trainer(model, XY_train_datasets : list[tuple]):
with ThreadPoolExecutor(max_workers = len(XY_train_datasets)) as executor:
futureObjs = [
executor.submit(
lambda ds: fit(model, ds), XY_train_datasets) # Call Fit for each dataset iterate through the datasets
]
for i, obj in enumerate(as_completed(futureObjs)): # iterate through trained models
(obj.result()).save(f"{i}.model") # save models
model = create_model() # create the model
mnist = tf.keras.datasets.mnist # get dataset - for example :- mnist dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data() # get (x_train, y_train), (x_test, y_test)
datasets = [(x_train, y_train)]*10 # list of dataset paths (in your case, same dataset used 10 times)
parallel_trainer(model, datasets) # call parallel trainer