在 Tensorflow 中实现自定义损失函数，导致"ValueError: 'outputs' must be defined before the loop."

Question

我正在使用Python进行我的第一个机器学习项目-使用TensorFlow尝试使用Moby Hyphenator II数据集对单词进行音节化。

我将其视为一个多标签分类问题，其中单词及其音节以以下格式编码:

T e n - s o r - f l o w
0 0 1   0 0 1   0 0 0 0

当我开始阅读这个指南时，我看到作者使用了一个自定义函数——他们在PyTorch中平均加权二元交叉熵和均方根误差:

def bce_rmse(pred, target, pos_weight = 1.3, epsilon = 1e-12):
# Weighted binary cross entropy
loss_pos = target * torch.log(pred + epsilon)
loss_neg = (1 - target) * torch.log(1 - pred + epsilon)
bce = torch.mean(torch.neg(pos_weight * loss_pos + loss_neg))
# Root mean squared error
mse = (torch.sum(pred, dim = 0) - torch.sum(target, dim = 0)) ** 2
rmse = torch.mean(torch.sqrt(mse + epsilon))
return (bce + rmse) / 2

我试图在TensorFlow中以以下方式实现这一点:

def weighted_bce_mse(y_true, y_prediction):
# Binary crossentropy with weighting
epsilon = 1e-12
positive_weight = 4.108897148948174
loss_positive = y_true * tf.math.log(y_prediction + epsilon)
loss_negative = (1 - y_true) * tf.math.log(1 - y_prediction + epsilon)
bce_loss = np.mean(tf.math.negative(positive_weight * loss_positive + loss_negative))

# Mean squared error
mse = tf.keras.losses.MeanSquaredError()
mse_loss = mse(y_true, y_prediction)
averaged_bce_mse = (bce_loss + mse_loss) / 2
return averaged_bce_mse

在这样做时，我收到错误ValueError: 'outputs' must be defined before the loop.，我不知道为什么我在构建和编译模型之前定义了这个函数。

我正在使用Keras Functional API，我的编译和适配阶段是:

model.compile(optimizer="adam", loss=weighted_bce_mse, metrics=["accuracy"], steps_per_execution=64)
history = model.fit(padded_inputs, padded_outputs, validation_data=(validation_inputs, validation_outputs), epochs=10, verbose=2)

Answer 1

如前所述，显示的错误与自定义损失函数无关。您所展示的代码还有许多其他错误，比如没有正确导入tf.keras.layers。修复这些错误后，请参阅下面的代码并在下面的版本上进行测试(工作正常):

tensorflow 2.4.1
numpy 1.19.5
python 3.9.6

import tensorflow as tf
# Custom loss function - mean of binary crossentropy and mean squared error
def mean_weighted_bce_mse(y_true, y_prediction):
# Binary crossentropy with weighting
epsilon = 1e-12
positive_weight = 4.108897148948174
loss_positive = y_true * tf.math.log(y_prediction + epsilon)
loss_negative = (1 - y_true) * tf.math.log(1 - y_prediction + epsilon)
bce_loss = np.mean(tf.math.negative(positive_weight * loss_positive + loss_negative))

# Mean squared error
mse = tf.keras.losses.MeanSquaredError()
mse_loss = mse(y_true, y_prediction)

averaged_bce_mse = (bce_loss + mse_loss) / 2
return tf.math.reduce_mean(averaged_bce_mse, axis=-1)


inputs = tf.keras.Input(shape=(15,))
x = tf.keras.layers.Embedding(64, 64, mask_zero=True)(inputs)

x = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(128, return_sequences=True))(x)
x = tf.keras.layers.Dropout(0.2)(x)
x = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(128, return_sequences=True))(x)
x = tf.keras.layers.Dropout(0.2)(x)
x = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(128, return_sequences=True))(x)
x = tf.keras.layers.Dropout(0.3)(x)


x = tf.keras.layers.Conv1D(64, kernel_size=1)(x)
x = tf.keras.layers.Dropout(0.2)(x)
x = tf.keras.layers.Conv1D(64, kernel_size=1)(x)
x = tf.keras.layers.Dropout(0.2)(x)
x = tf.keras.layers.GlobalMaxPool1D()(x)
x = tf.keras.layers.Dropout(0.5)(x)

x = tf.keras.layers.Dense(32, activation="relu")(x)
x = tf.keras.layers.Dense(15, activation="sigmoid")(x)


model = tf.keras.models.Model(inputs=inputs, outputs=x)
model.compile(optimizer="adam", loss=mean_weighted_bce_mse, metrics=["accuracy"], steps_per_execution=64)

# history = model.fit(padded_inputs,
#                     padded_outputs,
#                     validation_data=(validation_inputs, validation_outputs),
#                     epochs=20,
#                     batch_size=8)

Answer 2

在下一行代码中:

model.compile(optimizer="adam", loss=mean_weighted_bce_mse, metrics=["accuracy"], steps_per_execution=64)
history = model.fit(padded_inputs,
padded_outputs,
validation_data=(validation_inputs, validation_outputs),
epochs=20,
batch_size=8)

输入数据的长度是多少?steps_per_execution应该是len(input_data)/Batch_size。拆下steps_per_execution，重新检查。

Answer 3

我发现错误源于我在自定义损失函数中使用的操作:

bce_loss = np.mean(tf.math.negative(positive_weight * loss_positive + loss_negative))

这行使用np.mean导致错误-将其替换为tf.math.reduce_mean以及通过tf.cast将y_true和y_prediction转换为tf.float32解决了问题:

# Custom loss function - mean of binary crossentropy and mean squared error
def mean_weighted_bce_mse(y_true, y_prediction):
y_true = tf.cast(y_true, tf.float32)
y_prediction = tf.cast(y_prediction, tf.float32)
# Binary crossentropy with weighting
epsilon = 1e-12
positive_weight = 4.108897148948174
loss_positive = y_true * tf.math.log(y_prediction + epsilon)
loss_negative = (1 - y_true) * tf.math.log(1 - y_prediction + epsilon)
bce_loss = tf.math.reduce_mean(tf.math.negative(positive_weight * loss_positive + loss_negative))

# Mean squared error
mse = tf.keras.losses.MeanSquaredError()
mse_loss = mse(y_true, y_prediction)
averaged_bce_mse = (bce_loss + mse_loss) / 2
return averaged_bce_mse