用于混合数据类型的转换器

Question

我遇到了问题将不同的转换器同时应用于不同类型的列(文本与数字(，并将这些转换器连接到一个单独的转换器中以供以后使用。

我试着按照混合类型的Column Transformer文档中的步骤进行操作，该文档解释了如何对分类数据和数字数据的混合进行操作，但它似乎不适用于文本数据。

TL；DR

如何创建一个可存储的转换器，它遵循文本和数字数据的不同管道？

数据下载和准备

# imports
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.datasets import fetch_openml
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import FeatureUnion, Pipeline
from sklearn.preprocessing import StandardScaler
np.random.seed(0)
# download Titanic data
X, y = fetch_openml("titanic", version=1, as_frame=True, return_X_y=True)
# data preparation
numeric_features = ['age', 'fare']
text_features = ['name', 'cabin', 'home.dest']
X.fillna({text_col: '' for text_col in text_features}, inplace=True)
# train test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

转换数字特征：好

按照上面链接中的步骤，可以为以下数字特征创建一个转换器：

# handling missing data and normalization
numeric_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='median')),
('scaler', StandardScaler())])
num_preprocessor = ColumnTransformer(transformers=[('num', numeric_transformer, numeric_features)])
# this works
num_preprocessor.fit(X_train)
train_feature_set = num_preprocessor.transform(X_train)
test_feature_set = num_preprocessor.transform(X_test)
# verify shape = (number of data points, number of numerical features (2) )
train_feature_set.shape  # (1047, 2)
test_feature_set.shape  # (262, 2)

转换文本功能：确定

为了处理文本特性，我使用TF-IDF对每个文本列进行矢量化(而不是连接所有文本列，只应用TF-IDF一次(：

# Tfidf of max 30 features
text_transformer = TfidfVectorizer(use_idf=True,
max_features=30)
# apply separately to each column
text_transformer_list = [(x + '_vectorizer', text_transformer, x) for x in text_features]
text_preprocessor = ColumnTransformer(transformers=text_transformer_list)
# this works
text_preprocessor.fit(X_train)
train_feature_set = text_preprocessor.transform(X_train)
test_feature_set = text_preprocessor.transform(X_test)
# verify shape = (number of data points, number of text features (3) times max_features(30) )
train_feature_set.shape  # (1047, 90)
test_feature_set.shape  # (262, 90)

你是如何同时做到这两件事的

我尝试了各种策略来将上述两个过程保存在一个转换器中，但由于不同的错误，它们都失败了。

尝试1：遵循记录在案的策略

一旦文本数据取代了分类数据，以下文档(具有混合类型的列转换器(就不起作用：

# documented strategy
sum_preprocessor = ColumnTransformer(transformers=[('num', numeric_transformer, numeric_features),
('text', text_transformer, text_features)])
# fails
sum_preprocessor.fit(X_train)

返回以下错误消息：

ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 1047 and the array at index 1 has size 3

尝试2:FeatureUnion在变压器列表上

# create a list of numerical transformer, like those for text
numerical_transformer_list = [(x + '_scaler', numeric_transformer, x) for x in numeric_features]
# fails
column_trans = FeatureUnion([text_transformer_list, numerical_transformer_list])

返回以下错误消息：

TypeError: All estimators should implement fit and transform. '('cabin_vectorizer', TfidfVectorizer(max_features=30), 'cabin')' (type <class 'tuple'>) doesn't

尝试3：变压器列表上的ColumnTransformer

# create a list of all transformers, text and numerical
sum_transformer_list = text_transformer_list + numerical_transformer_list
# works
sum_preprocessor = ColumnTransformer(transformers=sum_transformer_list)
# fails
sum_preprocessor.fit(X_train)

返回以下错误消息：

ValueError: Expected 2D array, got 1D array instead:
array=[54. nan nan ... 20. nan nan].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

我的问题

如何创建可以混合文本和数字类型的fit和transform数据的单个对象

Answer 1

简短回答：

all_transformers = text_transformer_list + [('num', numeric_transformer, numeric_features)]
all_preprocessor = ColumnTransformer(transformers=all_transformers)
all_preprocessor.fit(X_train)
train_all = all_preprocessor.transform(X_train)
test_all = all_preprocessor.transform(X_test)
print(train_all.shape, test_all.shape)
# prints (1047, 92) (262, 92)

这里的困难在于(大多数(文本转换器期望一维输入，但(大多数(数字转换器期望二维输入。ColumnTransformer通过允许您指定单列或列列表来处理此问题：在第一种情况下，将1d数组传递给transformer，在第二种情况下将2d数组传递。

因此，为了解释三次尝试中的错误：

尝试1：TF-IDF正在接收一个2d数组，并将列视为文档而非单个条目，因此仅产生三个输出。当它试图将其连接到1047行的数字输出时，它失败了。

尝试2:FeatureUnion的输入格式与ColumnTransformer不同：在这种情况下，不应该有三元组(name, transformer, columns)。不管怎样，FeatureUnion并不适合你在这里做什么。

尝试3：这一次你试图将1d数据发送到数字转换器，但这些都是2d数据。