使用keras上的多_gpu_model-引起资源耗尽

Question

我以以下方式构建了我的网络：

# Build U-Net model
inputs = Input((IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS))
s = Lambda(lambda x: x / 255) (inputs)
width = 64
c1 = Conv2D(width, (3, 3), activation='relu', padding='same') (s)
c1 = Conv2D(width, (3, 3), activation='relu', padding='same') (c1)
p1 = MaxPooling2D((2, 2)) (c1)
c2 = Conv2D(width*2, (3, 3), activation='relu', padding='same') (p1)
c2 = Conv2D(width*2, (3, 3), activation='relu', padding='same') (c2)
p2 = MaxPooling2D((2, 2)) (c2)
c3 = Conv2D(width*4, (3, 3), activation='relu', padding='same') (p2)
c3 = Conv2D(width*4, (3, 3), activation='relu', padding='same') (c3)
p3 = MaxPooling2D((2, 2)) (c3)
c4 = Conv2D(width*8, (3, 3), activation='relu', padding='same') (p3)
c4 = Conv2D(width*8, (3, 3), activation='relu', padding='same') (c4)
p4 = MaxPooling2D(pool_size=(2, 2)) (c4)
c5 = Conv2D(width*16, (3, 3), activation='relu', padding='same') (p4)
c5 = Conv2D(width*16, (3, 3), activation='relu', padding='same') (c5)
u6 = Conv2DTranspose(width*8, (2, 2), strides=(2, 2), padding='same') (c5)
u6 = concatenate([u6, c4])
c6 = Conv2D(width*8, (3, 3), activation='relu', padding='same') (u6)
c6 = Conv2D(width*8, (3, 3), activation='relu', padding='same') (c6)
u7 = Conv2DTranspose(width*4, (2, 2), strides=(2, 2), padding='same') (c6)
u7 = concatenate([u7, c3])
c7 = Conv2D(width*4, (3, 3), activation='relu', padding='same') (u7)
c7 = Conv2D(width*4, (3, 3), activation='relu', padding='same') (c7)
u8 = Conv2DTranspose(width*2, (2, 2), strides=(2, 2), padding='same') (c7)
u8 = concatenate([u8, c2])
c8 = Conv2D(width*2, (3, 3), activation='relu', padding='same') (u8)
c8 = Conv2D(width*2, (3, 3), activation='relu', padding='same') (c8)
u9 = Conv2DTranspose(width, (2, 2), strides=(2, 2), padding='same') (c8)
u9 = concatenate([u9, c1], axis=3)
c9 = Conv2D(width, (3, 3), activation='relu', padding='same') (u9)
c9 = Conv2D(width, (3, 3), activation='relu', padding='same') (c9)
outputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)
with tf.device('/cpu:0'):
    model = Model(inputs=[inputs], outputs=[outputs])
sgd = optimizers.SGD(lr=0.03, decay=1e-6, momentum=0.9, nesterov=True)
parallel_model = multi_gpu_model(model, gpus=8)
parallel_model.compile(optimizer=sgd, loss='binary_crossentropy', metrics=[mean_iou])
model.summary()

请注意，我正在实例化CPU上的基本模型，如Keras Documentation所建议。然后，我使用以下行运行网络：

# Fit model
earlystopper = EarlyStopping(patience=20, verbose=1)
checkpointer = ModelCheckpoint('test.h5', verbose=1, save_best_only=True)
results = parallel_model.fit(X_train, Y_train, validation_split=0.05, batch_size = 256, verbose=1, epochs=100, 
                    callbacks=[earlystopper, checkpointer])

但是，即使我使用的是multiple_gpu_model，我的代码仍然会导致以下错误：

oom用形状分配张量[32,128,256,256]并输入float on/job：local -host/epplica：0/任务：0/device：0/device：gpu：0/gpu：0，分配器GPU_0_BFC

表明网络正在尝试仅在单个GPU上运行256的批处理大小而不是8。我需要在示例中使用Xception吗？

Answer 1

张量的第一个凹点是batch_size，因此在您的情况下一切都很好。您已将批次_size指定为256，并且使用8 GPU。因此，如错误所述，您所产生的batch_size为32。此外，错误表明您的模型仍然太大了，batch_size的gpus batch_size无法处理。