如何用tf.data在tensorflow管道中加载.npy文件

Question

我正试图在tf.data管道中使用np.load()从.npy文件中读取我的X和y数据。但得到以下错误，如果我调用model.fit()。有人能解决这个问题吗?我认为我必须给tf.py_funciton X_data和y_data的形状。我正在使用Tensorflow 2.4

错误:

Input 0 of layer sequential_13 is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: (None, None)

描述:

train_filenames,train_label_filenames是包含每个.npy文件路径的列表。

使print(train_filenames[0])显示'E:UserDataMustermann2_DataX_DataSample0.npy' np.load(train_filenames[0]).shape为(12, 8002)
np.load(label_filenames[0]).reshape(-1,1).shape为(1, 1)

所以一个样本有12个时间步长和8002个特征。

代码:

def load_files_py(train_filenames, train_label_filenames):
   
   X_data = np.load(train_filenames)
   label = np.load(train_label_filenames).reshape(-1,1)
   
   return X_data, label, X_data.shape, label.shape

def parse_function(train_filenames, train_label_filenames):
    
    temp = tf.py_function(load_files_py, inp=[train_filenames, train_label_filenames], Tout=[tf.float32, tf.float32, tf.int32, tf.int32])
    X_data = tf.reshape(temp[0], [temp[2]])
    label = tf.reshape(temp[1], [temp[3]])
    return X_data, label

batch_size = 64
train_dataset = tf.data.Dataset.from_tensor_slices((train_filenames,train_label_filenames))
train_dataset = train_dataset.shuffle(len(train_filenames))
train_dataset = train_dataset.map(parse_function, num_parallel_calls=2)
train_dataset = train_dataset.batch(batch_size)
train_dataset = train_dataset.prefetch(1)
test_dataset = tf.data.Dataset.from_tensor_slices((test_filenames,test_label_filenames))
test_dataset = test_dataset.shuffle(len(test_filenames))
test_dataset = test_dataset.map(parse_function, num_parallel_calls=2)
test_dataset = test_dataset.batch(batch_size)
test_dataset = test_dataset.prefetch(1)

model = tf.keras.models.Sequential([
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(16, input_shape = (12, 8002), return_sequences=True)),
    tf.keras.layers.LSTM(16),
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(8, activation='relu'),
    tf.keras.layers.Dense(4, activation='relu'),
    tf.keras.layers.Dense(1, activation = 'linear')   
]) 
model.compile(optimizer='adam', loss='mse')
EPOCHS =300
early = tf.keras.callbacks.EarlyStopping('val_loss', patience=20)
history = model.fit(train_dataset,
                    epochs=EPOCHS,
                    validation_data = test_dataset)

Answer 1

从我所看到的，加载.npy文件没有问题，因为错误表明模型有问题。

在你的第一层，你使用的参数return_sequences=True，这实际上是问题。这个参数返回一个3-D Array，它与下一层不兼容。我将通过删除该参数来尝试一下。