构建Sklearn文本分类器并使用Coremltools转换

Question

我想用Sklearn构建一个文本分类器，然后使用CoremlTools软件包将其转换为IOS11机器学习文件。我构建了三个不同的分类器，具有逻辑回归，随机森林和线性SVC，并且它们在Python中都很好。问题是Coremltools软件包及其将Sklearn模型转换为iOS文件的方式。正如其文档所说，它仅支持这些模型：

• 线性和逻辑回归
• Linearsvc和Linearsvr
• SVC和SVR
• NUSVC和NUSVR
• 梯度提升分类器和回归器
• 决策树分类器和回归量
• 随机森林分类器和回归
• 归一化器
• 螺母
• 标准秤
• dictvectorizer
• 一个热编码器

因此，它不允许我对文本数据集进行矢量化(我在分类器中使用了tfidfvectorizer软件包(：

import coremltools
coreml_model = coremltools.converters.sklearn.convert(model, input_features='text', output_feature_names='category')

---

Traceback (most recent call last):
File "<ipython-input-3-97beddbdad10>", line 1, in <module>
    coreml_model = coremltools.converters.sklearn.convert(pipeline, input_features='Message', output_feature_names='Label')
  File "/usr/local/lib/python2.7/dist-packages/coremltools/converters/sklearn/_converter.py", line 146, in convert
    sk_obj, input_features, output_feature_names, class_labels = None)
  File "/usr/local/lib/python2.7/dist-packages/coremltools/converters/sklearn/_converter_internal.py", line 147, in _convert_sklearn_model
    for sk_obj_name, sk_obj in sk_obj_list]
  File "/usr/local/lib/python2.7/dist-packages/coremltools/converters/sklearn/_converter_internal.py", line 97, in _get_converter_module
    ",".join(k.__name__ for k in _converter_module_list)))
ValueError: Transformer 'TfidfVectorizer(analyzer='word', binary=False, decode_error=u'strict',
        dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content',
        lowercase=True, max_df=1.0, max_features=None, min_df=3,
        ngram_range=(1, 2), norm=u'l2', preprocessor=None, smooth_idf=1,
        stop_words='english', strip_accents='unicode', sublinear_tf=1,
        token_pattern='\w+', tokenizer=None, use_idf=1, vocabulary=None)' not supported; 
supported transformers are coremltools.converters.sklearn._dict_vectorizer,coremltools.converters.sklearn._one_hot_encoder,coremltools.converters.sklearn._normalizer,coremltools.converters.sklearn._standard_scaler,coremltools.converters.sklearn._imputer,coremltools.converters.sklearn._NuSVC,coremltools.converters.sklearn._NuSVR,coremltools.converters.sklearn._SVC,coremltools.converters.sklearn._SVR,coremltools.converters.sklearn._linear_regression,coremltools.converters.sklearn._LinearSVC,coremltools.converters.sklearn._LinearSVR,coremltools.converters.sklearn._logistic_regression,coremltools.converters.sklearn._random_forest_classifier,coremltools.converters.sklearn._random_forest_regressor,coremltools.converters.sklearn._decision_tree_classifier,coremltools.converters.sklearn._decision_tree_regressor,coremltools.converters.sklearn._gradient_boosting_classifier,coremltools.converters.sklearn._gradient_boosting_regressor.

有什么方法可以构建Sklearn文本分类器，而不使用TFIDFECTORIZER或CountVectorizer模型？

Answer 1

现在，如果要将其转换为.mlModel格式，则无法在管道中包含tf-idf vectorizer。解决此问题的方法是分别矢量化数据，然后使用矢量化数据训练模型(线性SVC，随机森林，...(。然后，您需要在设备上计算TF-IDF表示形式，然后可以将其插入模型。这是我写的TF-IDF函数的副本。

func tfidf(document: String) -> MLMultiArray{
    let wordsFile = Bundle.main.path(forResource: "words_ordered", ofType: "txt")
    let dataFile = Bundle.main.path(forResource: "data", ofType: "txt")
    do {
        let wordsFileText = try String(contentsOfFile: wordsFile!, encoding: String.Encoding.utf8)
        var wordsData = wordsFileText.components(separatedBy: .newlines)
        let dataFileText = try String(contentsOfFile: dataFile!, encoding: String.Encoding.utf8)
        var data = dataFileText.components(separatedBy: .newlines)
        let wordsInMessage = document.split(separator: " ")
        var vectorized = try MLMultiArray(shape: [NSNumber(integerLiteral: wordsData.count)], dataType: MLMultiArrayDataType.double)
        for i in 0..<wordsData.count{
            let word = wordsData[i]
            if document.contains(word){
                var wordCount = 0
                for substr in wordsInMessage{
                    if substr.elementsEqual(word){
                        wordCount += 1
                    }
                }
                let tf = Double(wordCount) / Double(wordsInMessage.count)
                var docCount = 0
                for line in data{
                    if line.contains(word) {
                        docCount += 1
                    }
                }
                let idf = log(Double(data.count) / Double(docCount))
                vectorized[i] = NSNumber(value: tf * idf)
            } else {
                vectorized[i] = 0.0
            }
        }
        return vectorized
    } catch {
        return MLMultiArray()
    }
}

编辑：在http://gokulswamy.me/imessage-spam-detection/。