快速文本：无法cross_validation

Question

我正在努力将FastText（FTTransformer）实现到一个迭代不同矢量器的管道中。更具体地说，我无法获得交叉验证分数。使用以下代码：

%%time
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from gensim.utils import simple_preprocess
from gensim.sklearn_api.ftmodel import FTTransformer
np.random.seed(0)
data = pd.read_csv('https://pastebin.com/raw/dqKFZ12m')
X_train, X_test, y_train, y_test = train_test_split(data.text, data.label, random_state=0)
w2v_texts = [simple_preprocess(doc) for doc in X_train]
models = [FTTransformer(size=10, min_count=0, seed=42)]
classifiers = [LogisticRegression(random_state=0)]
for model in models:
    for classifier in classifiers:
        model.fit(w2v_texts)
        classifier.fit(model.transform(X_train), y_train)
        pipeline = Pipeline([
                ('vec', model),
                ('clf', classifier)
            ])
        print(pipeline.score(X_train, y_train))
        #print(model.gensim_model.wv.most_similar('kirk'))
        cross_val_score(pipeline, X_train, y_train, scoring='accuracy', cv=5)

KeyError： 'all ngram for word "Machine Learning can be useful 品牌有时"在模型中缺席"

如何解决问题？

旁注：我的其他管道D2VTransformer或TfIdfVectorizer工作正常。在这里，我可以在定义管道后简单地应用 pipeline.fit(X_train, y_train)，而不是如上所示的两个拟合。FTTransformer 似乎没有与其他给定的矢量化器很好地集成？

Answer 1

是的

，要在管道中使用，需要修改FTTransformer以在其fit方法中将文档拆分为单词。可以按如下方式进行操作：

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from gensim.utils import simple_preprocess
from gensim.sklearn_api.ftmodel import FTTransformer
np.random.seed(0)

class FTTransformer2(FTTransformer):
    def fit(self, x, y):
        super().fit([simple_preprocess(doc) for doc in x])
        return self

data = pd.read_csv('https://pastebin.com/raw/dqKFZ12m')
X_train, X_test, y_train, y_test = train_test_split(data.text, data.label, random_state=0)
classifiers = [LogisticRegression(random_state=0)]
for classifier in classifiers:
    pipeline = Pipeline([
            ('ftt', FTTransformer2(size=10, min_count=0, seed=0)),
            ('clf', classifier)
        ])
    score = cross_val_score(pipeline, X_train, y_train, scoring='accuracy', cv=5)
    print(score)