分类指标无法处理二进制目标和连续目标的混合

Question

我尝试训练和测试几个scikit-learn模型，并尝试打印出准确性。这些模型中只有一些有效，其他模型则因

ValueError: Classification metrics can't handle a mix of binary and continuous targets. 

此错误是什么意思？如何修改下面的代码以成功评估失败的模型？

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn import model_selection
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn import linear_model
from sklearn import svm
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn import preprocessing
from sklearn import utils
# Shuffle pandas rows randomly
from sklearn.utils import shuffle
# Disable annoying warnings
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=UserWarning)
# Performance
import time
# Import the data and assign the column names
colNames = []
for colName in range(0,3780):
    colNames.append("%s"%(colName))
colNames.append('class')
df = pd.read_csv("HoGTestData.csv", names=colNames)
# Randomly shuffle rows
df = shuffle(df)
df = df.head(20)
# Print some info on the dataset
print("Head of Data:")
print(df.head())
print("Shape of Data:")
print(df.shape)
# descriptions
print("Describe Data:")
#print(df.describe())
# class distribution
print(df.groupby('class').size())
# Split-out validation dataset
datasetData = df.values
# Determine shape and portion of data that is real data as opposed to labels
shape = datasetData.shape
thresh = int(shape[1]) - 1
# Extract labels and feature vectors
featureVectors = datasetData[:,0:thresh]
labels = datasetData[:,thresh:]
# Perform a standard scaler on the data
scaler = preprocessing.StandardScaler()
featureVectors = scaler.fit_transform(featureVectors)
# Encode labels to be acceptable
labelEncoder = preprocessing.LabelEncoder()
labels = labelEncoder.fit_transform(labels)
# Split data into training and testing data
test_size = 0.20
seed = 7
featureVectorTrain, featureVectorTest, labelsTrain, labelsTest = model_selection.train_test_split(featureVectors, labels, test_size=test_size, random_state=seed)
# Spot Check Algorithms
models = []
models.append(('SVM', svm.SVC()))
models.append(('LR', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
#models.append(('SGDRegressor', linear_model.SGDRegressor())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('BayesianRidge', linear_model.BayesianRidge())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('LassoLars', linear_model.LassoLars())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('ARDRegression', linear_model.ARDRegression())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('PassiveAggressiveRegressor', linear_model.PassiveAggressiveRegressor())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('TheilSenRegressor', linear_model.TheilSenRegressor())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('LinearRegression', linear_model.LinearRegression())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
# Test options and evaluation metric
seed = 42
scoring = 'accuracy'
# evaluate each model in turn
results = []
names = []
print("---------------------------------------")
for name, model in models:
    start_time = time.time()
    kfold = model_selection.KFold(n_splits=10, random_state=seed)
    cv_results = model_selection.cross_val_score(model, featureVectorTrain, labelsTrain, cv=kfold, scoring=scoring)
    elapsed_time = time.time() - start_time
    results.append(cv_results)
    names.append(name)
    msg = "{:3.2f} ({:3.2f})  Time elapsed: {:6.2f}".format(cv_results.mean(), cv_results.std(), elapsed_time)
    msg = "%s "%(name) + msg
    print(msg)
print("---------------------------------------")
print("Done")

下面是脚本输出：

Head of Data:
              0         1         2  ...        3778      3779  class
20573  0.124282  0.090376  0.088723  ...    0.148411  0.120542     -1
20461  0.154031  0.110177  0.087799  ...    0.100416  0.119484     -1
10416  0.340767  0.150863  0.025489  ...    0.047592  0.036171      1
52404  0.000000  0.000000  0.000000  ...    0.000000  0.000000     -1
42785  0.159105  0.118963  0.090405  ...    0.009996  0.027460     -1
[5 rows x 3781 columns]
Shape of Data:
(1024, 3781)
Describe Data:
class
-1    794
 1    230
dtype: int64
---------------------------------------
SVM 0.9878 (0.0123)  Time elapsed:  10.20
LR 0.9414 (0.0187)  Time elapsed:   7.09
LDA 0.9768 (0.0128)  Time elapsed:   6.60
KNN 0.8511 (0.0384)  Time elapsed:   3.06
CART 0.9047 (0.0358)  Time elapsed:   8.84
NB 0.9292 (0.0209)  Time elapsed:   0.36
---------------------------------------
Done

下面是标签训练变量：

print(labelsTrain)
[0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 1 1 1 0 1 1 0 0
 1 1 0 1 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0
 0 0 1 1 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 1 0 0 1 0 1 0 0 1 0 0 0 1 1 0 0 0
 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 1 1 1 0 1 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1
 1 0 1 1 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 1 1 0 0 0 0 1 0 0 1 0 0 0 1
 0 1 0 1 0 0 0 0 1 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 1 0 0 0 0 0 0 1 0 0 1 0 0
 0 0 0 0 1 1 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0
 1 0 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 1 1 0 1 0 1 1 0 1 0 0 0 0 0 0 1 0 1
 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0
 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0
 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0
 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0
 0 0 0 1 1 1 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 0 0
 1 0 0 0 0 0 1 1 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0
 1 0 0 0 0 0 0 1 1 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 0 1
 0 0 0 0 1 0 1 0 0 1 1 1 1 0 1 0 0 0 0 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0
 0 0 1 1 0 0 1 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0
 0 0 1 0 0]

错误发生在cross_val_score函数期间：

# evaluate each model in turn
results = []
names = []
print("---------------------------------------")
for name, model in models:
    start_time = time.time()
    kfold = model_selection.KFold(n_splits=10, random_state=seed)
    print("start cross_val_score")
    cv_results = model_selection.cross_val_score(model, featureVectorTrain, labelsTrain, cv=kfold, scoring=scoring)
    print("done cross_val_score")
    elapsed_time = time.time() - start_time
    results.append(cv_results)
    #print(results)
    names.append(name)
    msg = "{:3.4f} ({:3.4f})  Time elapsed: {:6.2f}".format(cv_results.mean(), cv_results.std(), elapsed_time)
    msg = "%s "%(name) + msg
    print(msg)
print("---------------------------------------")
... 
---------------------------------------
start cross_val_score
done cross_val_score
SVM 0.9744 (0.0127)  Time elapsed:  10.46
start cross_val_score
done cross_val_score
LR 0.9194 (0.0390)  Time elapsed:   9.56
start cross_val_score
done cross_val_score
LDA 0.9780 (0.0106)  Time elapsed:   8.04
start cross_val_score
done cross_val_score
KNN 0.8657 (0.0319)  Time elapsed:   3.20
start cross_val_score
done cross_val_score
CART 0.9072 (0.0326)  Time elapsed:  10.20
start cross_val_score
done cross_val_score
NB 0.9182 (0.0327)  Time elapsed:   0.38
start cross_val_score
Traceback (most recent call last):
  File "/Users/me/Desktop/MachineLearning/Initial.py", line 112, in <module>
    cv_results = model_selection.cross_val_score(model, featureVectorTrain, labelsTrain, cv=kfold, scoring=scoring)
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 402, in cross_val_score
    error_score=error_score)
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 240, in cross_validate
    for train, test in cv.split(X, y, groups))
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 917, in __call__
    if self.dispatch_one_batch(iterator):
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 759, in dispatch_one_batch
    self._dispatch(tasks)
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 716, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 182, in apply_async
    result = ImmediateResult(func)
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 549, in __init__
    self.results = batch()
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 225, in __call__
    for func, args, kwargs in self.items]
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 225, in <listcomp>
    for func, args, kwargs in self.items]
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 568, in _fit_and_score
    test_scores = _score(estimator, X_test, y_test, scorer, is_multimetric)
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 605, in _score
    return _multimetric_score(estimator, X_test, y_test, scorer)
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 635, in _multimetric_score
    score = scorer(estimator, X_test, y_test)
  File "/usr/local/lib/python3.7/site-packages/sklearn/metrics/scorer.py", line 98, in __call__
    **self._kwargs)
  File "/usr/local/lib/python3.7/site-packages/sklearn/metrics/classification.py", line 176, in accuracy_score
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)
  File "/usr/local/lib/python3.7/site-packages/sklearn/metrics/classification.py", line 81, in _check_targets
    "and {1} targets".format(type_true, type_pred))
ValueError: Classification metrics can't handle a mix of binary and continuous targets

Answer 1

所有注释掉的模型都不是分类器，而是回归模型，准确性毫无意义。

你得到错误是因为这些回归模型不产生二进制结果，而是连续（浮点）数（就像所有回归模型一样）;因此，当scikit-learn尝试通过将二进制数（真标签）与浮点数（预测值）进行比较来计算准确性时，它不会意外地给出错误。而这个原因在错误消息本身中清楚地暗示了：

Classification metrics can't handle a mix of binary and continuous target

Answer 2

我使用了一些模型来使用vecstack进行堆叠并设置needs_proba=True然后得到此错误。我通过更改堆栈内的指标解决了它。因为堆叠默认使用类预测，所以如果你想有概率，你也应该改变指标。我定义了一个新函数作为度量：

def get_classification_metric(testy, probs):
    from sklearn.metrics import precision_recall_curve
    precision, recall, thresholds = precision_recall_curve(testy, probs[:,1])
    # convert to f score
    fscore = (2 * precision * recall) / (precision + recall)
    # locate the index of the largest f score
    ix = np.argmax(fscore)
    return fscore[ix]

此函数在最佳阈值处查找最高的 F1 分数。所以只需要设置metric=get_classification_metric内部的堆叠功能。