如何在 Python 中将 scikit-learn 评估指标函数与 Keras 结合使用

Question

Keras

提供了定义自定义评估指标的可能性--我对scikit learn提供的F指标的变体感兴趣，例如F1，F2等-但指示我们通过调用在这方面受到限制的Keras后端函数来做到这一点。

我的目标是将这些指标与 Keras 的提前停止方法结合使用。 所以我应该找到一种方法来将指标与 Keras 模型的学习过程集成在一起。 （当然，在学习/拟合过程之外，我可以简单地调用Scikit-Learn的结果）。

我在这里有什么选择？

更新

使用 Kaggle 的数据集实现 Aaron 的解决方案后titanic_all_numeric我得到了以下内容：

# Compile the model
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy', f1])
# Fit the model
hist = model.fit(predictors, target, validation_split = 0.3)
Train on 623 samples, validate on 268 samples
Epoch 1/1
623/623 [==============================] - 0s 642us/step - loss: 0.8037 - acc: 0.6132 - f1: 0.6132 - val_loss: 0.5815 - val_acc: 0.7537 - val_f1: 0.7537
# Compile the model
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fit the model
hist = model.fit(predictors, target, validation_split = 0.3)
Train on 623 samples, validate on 268 samples
Epoch 1/1
623/623 [==============================] - 0s 658us/step - loss: 0.8148 - acc: 0.6404 - val_loss: 0.7056 - val_acc: 0.7313
# Compile the model
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = [f1])
# Fit the model
hist = model.fit(predictors, target, validation_split = 0.3)
Train on 623 samples, validate on 268 samples
Epoch 1/1
623/623 [==============================] - 0s 690us/step - loss: 0.6554 - f1: 0.6709 - val_loss: 0.5107 - val_f1: 0.7612

我想知道这些结果是否正常。 这一次，准确性和 f1 分数是相同的。

Answer 1

您可以将预测和标签从 keras 模型传递到任何 scikit-learn 函数以进行评估。例如，如果您正在解决分类问题，则可以使用scikit-learn的classification_report，该提供了诸如精度，召回率，f1分数之类的指标，例如（示例代码直接取自他们的文档）：

from sklearn.metrics import classification_report
y_true = [0, 1, 2, 2, 2]
y_pred = [0, 0, 2, 2, 1]
target_names = ['class 0', 'class 1', 'class 2']
print(classification_report(y_true, y_pred, target_names=target_names))
          precision    recall  f1-score   support
 class 0       0.50      1.00      0.67         1
 class 1       0.00      0.00      0.00         1
 class 2       1.00      0.67      0.80         3
 micro avg     0.60      0.60      0.60         5
 macro avg     0.50      0.56      0.49         5
 weighted avg  0.70      0.60      0.61         5

更新：如果您想将指标合并到 keras 训练中，请使用：

from keras import backend as K
def f1(y_true, y_pred):
    def recall(y_true, y_pred):
        """Recall metric.
        Only computes a batch-wise average of recall.
        Computes the recall, a metric for multi-label classification of
        how many relevant items are selected.
        """
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
        recall = true_positives / (possible_positives + K.epsilon())
        return recall
    def precision(y_true, y_pred):
        """Precision metric.
        Only computes a batch-wise average of precision.
        Computes the precision, a metric for multi-label classification of
        how many selected items are relevant.
        """
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
        precision = true_positives / (predicted_positives + K.epsilon())
        return precision
    precision = precision(y_true, y_pred)
    recall = recall(y_true, y_pred)
    return 2*((precision*recall)/(precision+recall+K.epsilon()))

model.compile(loss='binary_crossentropy',
          optimizer= "adam",
          metrics=[f1])