我正在尝试使用不平衡数据集对数据进行过度采样。按照 sklearn 约定,我将 X,y 作为特征矩阵和目标向量。它们是 pandas.core.frame.DataFrame 类型,形状分别为 (200000, 17) 和 (200000,)。
我首先使用 sklean 的train_test_split拆分数据。然后应用 SMOTE 方法对训练数据集进行过采样,导致以下错误:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
C:Users...Anaconda3libsite-packagespandasindexesbase.py in get_loc(self, key, method, tolerance)
1944 try:
-> 1945 return self._engine.get_loc(key)
1946 except KeyError:
pandasindex.pyx in pandas.index.IndexEngine.get_loc (pandasindex.c:4154)()
pandasindex.pyx in pandas.index.IndexEngine.get_loc (pandasindex.c:4018)()
pandashashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandashashtable.c:12368)()
pandashashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandashashtable.c:12322)()
KeyError: 1143
During handling of the above exception, another exception occurred:
KeyError Traceback (most recent call last)
<ipython-input-99-1c5830417b3f> in <module>()
6 # 'SMOTE'
7 SM = SMOTE(ratio=ratio, verbose=verbose, kind='regular')
----> 8 smx, smy = SM.fit_transform(Xtrain, ytrain)
C:Users...Anaconda3libsite-packagesunbalanceddataset-0.1-py3.5.eggunbalanced_datasetunbalanced_dataset.py in fit_transform(self, x, y)
274 return self.out_x, self.out_y, self.out_idx
275 else:
--> 276 self.out_x, self.out_y = self.resample()
277
278 return self.out_x, self.out_y
C:Users...Anaconda3libsite-packagesunbalanceddataset-0.1-py3.5.eggunbalanced_datasetover_sampling.py in resample(self)
358 step_size=1.0,
359 random_state=self.rs,
--> 360 verbose=self.verbose)
361
362 if self.verbose:
C:Users...Anaconda3libsite-packagesunbalanceddataset-0.1-py3.5.eggunbalanced_datasetunbalanced_dataset.py in make_samples(x, nn_data, y_type, nn_num, n_samples, step_size, random_state, verbose)
388
389 # Construct synthetic sample
--> 390 new[i] = x[row] - step * (x[row] - nn_data[nn_num[row, col]])
391
392 # The returned target vector is simply a repetition of the
C:Users...Anaconda3libsite-packagespandascoreframe.py in __getitem__(self, key)
1995 return self._getitem_multilevel(key)
1996 else:
-> 1997 return self._getitem_column(key)
1998
1999 def _getitem_column(self, key):
C:Users...Anaconda3libsite-packagespandascoreframe.py in _getitem_column(self, key)
2002 # get column
2003 if self.columns.is_unique:
-> 2004 return self._get_item_cache(key)
2005
2006 # duplicate columns & possible reduce dimensionality
C:Users...Anaconda3libsite-packagespandascoregeneric.py in _get_item_cache(self, item)
1348 res = cache.get(item)
1349 if res is None:
-> 1350 values = self._data.get(item)
1351 res = self._box_item_values(item, values)
1352 cache[item] = res
C:Users...Anaconda3libsite-packagespandascoreinternals.py in get(self, item, fastpath)
3288
3289 if not isnull(item):
-> 3290 loc = self.items.get_loc(item)
3291 else:
3292 indexer = np.arange(len(self.items))[isnull(self.items)]
C:Users...Anaconda3libsite-packagespandasindexesbase.py in get_loc(self, key, method, tolerance)
1945 return self._engine.get_loc(key)
1946 except KeyError:
-> 1947 return self._engine.get_loc(self._maybe_cast_indexer(key))
1948
1949 indexer = self.get_indexer([key], method=method, tolerance=tolerance)
pandasindex.pyx in pandas.index.IndexEngine.get_loc (pandasindex.c:4154)()
pandasindex.pyx in pandas.index.IndexEngine.get_loc (pandasindex.c:4018)()
pandashashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandashashtable.c:12368)()
pandashashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandashashtable.c:12322)()
KeyError: 1143
我得到了这个错误,而不平衡数据集对相同数据的所有欠采样方法都工作正常。有什么建议可以解决过采样问题吗?
更新:
正如glemaitre所提到的,为了解决这个问题,需要将Pandas DataFrame转换为Numpy数组。因此,以下转换将解决问题:
Xc = Xtrain.as_matrix()
不平衡数据集需要 numpy 数组。尝试将其插入函数,看看是否有效。
干杯