如何存储TfidfVectorizer以供将来在scikit-learn中使用

Question

我有一个TfidfVectorizer，它对文章集合进行矢量化，然后进行特征选择。

vectroizer = TfidfVectorizer()
X_train = vectroizer.fit_transform(corpus)
selector = SelectKBest(chi2, k = 5000 )
X_train_sel = selector.fit_transform(X_train, y_train)

现在，我想存储它并在其他程序中使用它。我不想在训练数据集上重新运行TfidfVectorizer()和特征选择器。我怎么做呢?我知道如何使用joblib使模型持久，但我想知道这是否与使模型持久相同。

Answer 1

您可以简单地使用内置的pickle库:

import pickle
pickle.dump(vectorizer, open("vectorizer.pickle", "wb"))
pickle.dump(selector, open("selector.pickle", "wb"))

并加载:

vectorizer = pickle.load(open("vectorizer.pickle", "rb"))
selector = pickle.load(open("selector.pickle", "rb"))

Pickle将对象序列化到磁盘，并在需要时再次加载到内存中

pickle lib docs

Answer 2

下面是我使用joblib的答案:

import joblib
joblib.dump(vectorizer, 'vectorizer.pkl')
joblib.dump(selector, 'selector.pkl')

稍后，我可以加载它并准备好:

vectorizer = joblib.load('vectorizer.pkl')
selector = joblib.load('selector.pkl')
test = selector.trasnform(vectorizer.transform(['this is test']))

Answer 3

"使对象持久"基本上意味着你要将存储在内存中的二进制代码转储到硬盘驱动器上的一个文件中，以便稍后在你的程序或任何其他程序中，对象可以从硬盘驱动器中的文件中重新加载到内存中。

scikit-learn包含joblib或标准库pickle和cPickle都可以完成这项工作。我倾向于cPickle，因为它明显更快。使用ippython的%timeit命令:

>>> from sklearn.feature_extraction.text import TfidfVectorizer as TFIDF
>>> t = TFIDF()
>>> t.fit_transform(['hello world'], ['this is a test'])
# generic serializer - deserializer test
>>> def dump_load_test(tfidf, serializer):
...:    with open('vectorizer.bin', 'w') as f:
...:        serializer.dump(tfidf, f)
...:    with open('vectorizer.bin', 'r') as f:
...:        return serializer.load(f)
# joblib has a slightly different interface
>>> def joblib_test(tfidf):
...:    joblib.dump(tfidf, 'tfidf.bin')
...:    return joblib.load('tfidf.bin')
# Now, time it!
>>> %timeit joblib_test(t)
100 loops, best of 3: 3.09 ms per loop
>>> %timeit dump_load_test(t, pickle)
100 loops, best of 3: 2.16 ms per loop
>>> %timeit dump_load_test(t, cPickle)
1000 loops, best of 3: 879 µs per loop

现在，如果您想在单个文件中存储多个对象，您可以轻松地创建一个数据结构来存储它们，然后转储数据结构本身。这将适用于tuple, list或dict。从你的问题的例子:

# train
vectorizer = TfidfVectorizer()
X_train = vectorizer.fit_transform(corpus)
selector = SelectKBest(chi2, k = 5000 )
X_train_sel = selector.fit_transform(X_train, y_train)
# dump as a dict
data_struct = {'vectorizer': vectorizer, 'selector': selector}
# use the 'with' keyword to automatically close the file after the dump
with open('storage.bin', 'wb') as f: 
    cPickle.dump(data_struct, f)

稍后或在另一个程序中，下列语句将在程序内存中恢复该数据结构:

# reload
with open('storage.bin', 'rb') as f:
    data_struct = cPickle.load(f)
    vectorizer, selector = data_struct['vectorizer'], data_struct['selector']
# do stuff...
vectors = vectorizer.transform(...)
vec_sel = selector.transform(vectors)