Scikit Learn OneHotEncoder 拟合和变换 错误: 值错误: X 的形状与拟合期间的形状不同

Question

下面是我的代码。

我知道为什么在转换过程中会发生错误。这是因为拟合和转换期间的功能列表不匹配。 我该如何解决这个问题？如何获得所有其余功能的 0？

在此之后，我想将其用于 SGD 分类器的部分拟合。

Jupyter QtConsole 4.3.1
Python 3.6.2 |Anaconda custom (64-bit)| (default, Sep 21 2017, 18:29:43) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.
import pandas as pd
from sklearn.preprocessing import OneHotEncoder
input_df = pd.DataFrame(dict(fruit=['Apple', 'Orange', 'Pine'], 
color=['Red', 'Orange','Green'],
is_sweet = [0,0,1],
country=['USA','India','Asia']))
input_df
Out[1]: 
color country   fruit  is_sweet
0     Red     USA   Apple         0
1  Orange   India  Orange         0
2   Green    Asia    Pine         1
filtered_df = input_df.apply(pd.to_numeric, errors='ignore')
filtered_df.info()
# apply one hot encode
refreshed_df = pd.get_dummies(filtered_df)
refreshed_df
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 4 columns):
color       3 non-null object
country     3 non-null object
fruit       3 non-null object
is_sweet    3 non-null int64
dtypes: int64(1), object(3)
memory usage: 176.0+ bytes
Out[2]: 
is_sweet  color_Green  color_Orange  color_Red  country_Asia  
0         0            0             0          1             0   
1         0            0             1          0             0   
2         1            1             0          0             1   
country_India  country_USA  fruit_Apple  fruit_Orange  fruit_Pine  
0              0            1            1             0           0  
1              1            0            0             1           0  
2              0            0            0             0           1  
enc = OneHotEncoder()
enc.fit(refreshed_df)
Out[3]: 
OneHotEncoder(categorical_features='all', dtype=<class 'numpy.float64'>,
handle_unknown='error', n_values='auto', sparse=True)
new_df = pd.DataFrame(dict(fruit=['Apple'], 
color=['Red'],
is_sweet = [0],
country=['USA']))
new_df
Out[4]: 
color country  fruit  is_sweet
0   Red     USA  Apple         0
filtered_df1 = new_df.apply(pd.to_numeric, errors='ignore')
filtered_df1.info()
# apply one hot encode
refreshed_df1 = pd.get_dummies(filtered_df1)
refreshed_df1
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1 entries, 0 to 0
Data columns (total 4 columns):
color       1 non-null object
country     1 non-null object
fruit       1 non-null object
is_sweet    1 non-null int64
dtypes: int64(1), object(3)
memory usage: 112.0+ bytes
Out[5]: 
is_sweet  color_Red  country_USA  fruit_Apple
0         0          1            1            1
enc.transform(refreshed_df1)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-33a6a884ba3f> in <module>()
----> 1 enc.transform(refreshed_df1)
~/anaconda3/lib/python3.6/site-packages/sklearn/preprocessing/data.py in transform(self, X)
2073         """
2074         return _transform_selected(X, self._transform,
-> 2075                                    self.categorical_features, copy=True)
2076 
2077 
~/anaconda3/lib/python3.6/site-packages/sklearn/preprocessing/data.py in _transform_selected(X, transform, selected, copy)
1810 
1811     if isinstance(selected, six.string_types) and selected == "all":
-> 1812         return transform(X)
1813 
1814     if len(selected) == 0:
~/anaconda3/lib/python3.6/site-packages/sklearn/preprocessing/data.py in _transform(self, X)
2030             raise ValueError("X has different shape than during fitting."
2031                              " Expected %d, got %d."
-> 2032                              % (indices.shape[0] - 1, n_features))
2033 
2034         # We use only those categorical features of X that are known using fit.
ValueError: X has different shape than during fitting. Expected 10, got 4.

Answer 1

而不是使用pd.get_dummies()而是需要LabelEncoder + OneHotEncoder，它可以存储原始值，然后将它们用于新数据。

像下面这样更改代码将为您提供所需的结果。

import pandas as pd
from sklearn.preprocessing import OneHotEncoder, LabelEncoder
input_df = pd.DataFrame(dict(fruit=['Apple', 'Orange', 'Pine'], 
color=['Red', 'Orange','Green'],
is_sweet = [0,0,1],
country=['USA','India','Asia']))
filtered_df = input_df.apply(pd.to_numeric, errors='ignore')
# This is what you need
le_dict = {}
for col in filtered_df.columns:
le_dict[col] = LabelEncoder().fit(filtered_df[col])
filtered_df[col] = le_dict[col].transform(filtered_df[col])
enc = OneHotEncoder()
enc.fit(filtered_df)
refreshed_df = enc.transform(filtered_df).toarray()
new_df = pd.DataFrame(dict(fruit=['Apple'], 
color=['Red'],
is_sweet = [0],
country=['USA']))
for col in new_df.columns:
new_df[col] = le_dict[col].transform(new_df[col])
new_refreshed_df = enc.transform(new_df).toarray()
print(filtered_df)
color  country  fruit  is_sweet
0      2        2      0         0
1      1        1      1         0
2      0        0      2         1
print(refreshed_df)
[[ 0.  0.  1.  0.  0.  1.  1.  0.  0.  1.  0.]
[ 0.  1.  0.  0.  1.  0.  0.  1.  0.  1.  0.]
[ 1.  0.  0.  1.  0.  0.  0.  0.  1.  0.  1.]]
print(new_df)
color  country  fruit  is_sweet
0      2        2      0         0
print(new_refreshed_df)
[[ 0.  0.  1.  0.  0.  1.  1.  0.  0.  1.  0.]]

Answer 2

您的编码器安装在包含 10 列的refreshed_df上，而您的refreshed_df1只包含 4 列，字面意思是错误中报告的内容。您必须删除未出现在refreshed_df1上的列，或者只是将编码器调整到仅包含refreshed_df1中显示的 4 列的新版本refreshed_df。