Apache Spark 中的多类分类错误指标

Question

是否可以在Apache Spark中的多类分类问题中找到错误指标(精度和召回率(。我正在使用Spark的MlLib的Logistic Regression来构建我的模型，并希望使用误差指标评估我的模型。

Answer 1

来自 MLlib 文档

假设您的测试数据处于test

import org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS
import org.apache.spark.mllib.evaluation.MulticlassMetrics
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.util.MLUtils
val predictionAndLabels = test.map { case LabeledPoint(label,
features) =>
val prediction = model.predict(features)
(prediction, label)
}
val metrics = new MulticlassMetrics(predictionAndLabels)

混淆矩阵

println("Confusion matrix:")
println(metrics.confusionMatrix)

总体统计

val accuracy = metrics.accuracy
println("Summary Statistics")
println(s"Accuracy = $accuracy")

按标签计算的精度

val labels = metrics.labels
labels.foreach { l =>
println(s"Precision($l) = " + metrics.precision(l))
}

按标签召回

labels.foreach { l =>
println(s"Recall($l) = " + metrics.recall(l))
}

按标签划分的误报率

labels.foreach { l =>
println(s"FPR($l) = " + metrics.falsePositiveRate(l))
}

按标签测量

F

labels.foreach { l =>
println(s"F1-Score($l) = " + metrics.fMeasure(l))
}

加权统计数据

println(s"Weighted precision: ${metrics.weightedPrecision}")
println(s"Weighted recall: ${metrics.weightedRecall}")
println(s"Weighted F1 score: ${metrics.weightedFMeasure}")
println(s"Weighted false positive rate: ${metrics.weightedFalsePositiveRate}")