如何在Keras制作端到端3D-2D CNN

Question

我有两个CAE模型，一个是3D模型，另一个是2D模型。该2D CAE将第一个生成的新表示作为输入。我的目标是找出如何组合它们，这样我就可以拥有一个端到端的完整3D-2D CAE模型，我如何训练它？

以下是每个型号的代码：

#3D CAE (I have just implemented the first encoding part since my aim is to generate the new representation z)
in_3D = Input((100,100, 288, 1))
model_3D = Conv3D(8, (5, 5, 5), activation='relu', padding='same')(in_3D)
model_3D = MaxPooling3D((2, 2, 2), strides=(1, 1, 4), padding='same')(model_3D)
model_3D = Reshape((10000,72*8))(model_3D)
model_3D = Dense(350, activation="relu")(model_3D)
model_3D = Dense(250, activation="relu")(model_3D)
model_3D = Dense(198, activation="relu")(model_3D)
model_3D = Reshape((100,100, 198))(model_3D)
z = Permute((3,2, 1))(model_3D)
_________________________________________________________________
Layer (type)                Output Shape              Param #   
=================================================================
input_1 (InputLayer)        [(None, 100, 100, 288, 1  0         
)]                                  

conv3d_1 (Conv3D)           (None, 100, 100, 288, 8)  1008      

max_pooling3d_1 (MaxPooling  (None, 100, 100, 72, 8)  0         
3D)                                                             

reshape (Reshape)         (None, 10000, 576)        0         

dense (Dense)            (None, 10000, 350)        201950    

dense_1 (Dense)            (None, 10000, 250)        87750     

dense_2 (Dense)            (None, 10000, 198)        49698     

reshape_1 (Reshape)         (None, 100, 100, 198)     0         

permute (Permute)           (None, 198, 100, 100)     0         

以及接收由第一模型生成的新z(198100100(作为输入的第二2D CAE模型。此处198作为无通过

#2D CAE  
in_2D = Input((100,100, 1))
model_2D= Conv2D(16, (3, 3), activation='relu', padding='same', name='Conv1')(in_2D)
model_2D = MaxPooling2D((2, 2), padding='same')(model_2D)
model_2D = Flatten()(model_2D)
model_2D = Dense(48, activation='relu')(model_2D)
model_2D = Dense(36, activation='relu')(model_2D)
model_2D = Dense(12)(model_2D)
model_2D= Dense(100*100, activation='linear')(model_2D)
_________________________________________________________________
Layer (type)                Output Shape              Param #   
=================================================================
input_1 (InputLayer)        [(None, 100, 100, 1)]     0         

Conv1 (Conv2D)              (None, 100, 100, 16)      160       

max_pooling2d_1 (MaxPooling  (None, 50, 50, 16)       0         
2D)                                                             

flatten (Flatten)          (None, 40000)             0         

dense (Dense)              (None, 48)                1920048   

dense_1 (Dense)             (None, 36)                1764      

dense_2 (Dense)             (None, 12)                444       

dense_3 (Dense)            (None, 10000)             130000    

任何帮助都将不胜感激。

Answer 1

要组合这两个模型，您可以从单独声明它们开始，如下所示：

import keras
import tensorflow as tf
from tensorflow.keras.layers import *
in_3D = Input((100,100, 288, 1))
model_3D = Conv3D(8, (5, 5, 5), activation='relu', padding='same')(in_3D)
model_3D = MaxPooling3D((2, 2, 2), strides=(1, 1, 4), padding='same')(model_3D)
model_3D = Reshape((10000,72*8))(model_3D)
model_3D = Dense(350, activation="relu")(model_3D)
model_3D = Dense(250, activation="relu")(model_3D)
model_3D = Dense(198, activation="relu")(model_3D)
model_3D = Reshape((100,100, 198))(model_3D)
z = Permute((3,2, 1))(model_3D)
cae_model_3D = keras.Model(in_3D, z)
in_2D = Input((100,100, 1))
model_2D= Conv2D(16, (3, 3), activation='relu', padding='same', name='Conv1')(in_2D)
model_2D = MaxPooling2D((2, 2), padding='same')(model_2D)
model_2D = Flatten()(model_2D)
model_2D = Dense(48, activation='relu')(model_2D)
model_2D = Dense(36, activation='relu')(model_2D)
model_2D = Dense(12)(model_2D)
model_2D= Dense(100*100, activation='linear')(model_2D)
cae_model_2D = keras.Model(in_2D, model_2D)

然后声明一个组合模型，将第一个模型的输出作为输入传递给第二个：

combined_model_input = Input((100, 100, 288, 1))
cae_model_3D_output = cae_model_3D(combined_model_input)
cae_model_3D_output_reshaped = tf.reshape(cae_model_3D_output, (-1, 100, 100, 1))
combined_model_output = cae_model_2D(cae_model_3D_output_reshaped)
combined_model = keras.Model(combined_model_input, combined_model_output)

请注意，我们必须重塑第一个模型的输出，使其与您希望198作为批处理维度(None(传递的想法一致。

训练模型的最简单方法是调用它的compile和fit方法。传递给这些方法的确切参数将取决于您试图解决的问题和您自己的偏好。这里有一个官方文档的链接以获得帮助。