如何使用Sklearn Library使用幼稚的贝叶斯进行文本分类

Question

我正在尝试使用幼稚的贝叶斯文本分类器尝试文本分类。我的数据是以下格式，基于问题和摘录，我必须决定问题的主题。培训数据的记录超过20K。我知道SVM在这里是一个更好的选择，但我想使用Sklearn Library和Naive Bayes一起去。

{[{"topic":"electronics","question":"What is the effective differencial effective of this circuit","excerpt":"I'm trying to work out, in general terms, the effective capacitance of this circuit (see diagram: https://i.stack.imgur.com/BS85b.png).  nnWhat is the effective capacitance of this circuit and will the ...rn        "},
{"topic":"electronics","question":"Outlet Installation--more wires than my new outlet can use [on hold]","excerpt":"I am replacing a wall outlet with a Cooper Wiring USB outlet (TR7745).  The new outlet has 3 wires coming out of it--a black, a white, and a green.  Each one needs to be attached with a wire nut to ...rn        "}]}

这是我到目前为止尝试过的，

import numpy as np
import json
from sklearn.naive_bayes import *
topic = []
question = []
excerpt = []
with open('training.json') as f:
    for line in f:
        data = json.loads(line)
        topic.append(data["topic"])
        question.append(data["question"])
        excerpt.append(data["excerpt"])
unique_topics = list(set(topic))
new_topic = [x.encode('UTF8') for x in topic]
numeric_topics = [name.replace('gis', '1').replace('security', '2').replace('photo', '3').replace('mathematica', '4').replace('unix', '5').replace('wordpress', '6').replace('scifi', '7').replace('electronics', '8').replace('android', '9').replace('apple', '10') for name in new_topic]
numeric_topics = [float(i) for i in numeric_topics]
x1 = np.array(question)
x2 = np.array(excerpt)
X = zip(*[x1,x2])
Y = np.array(numeric_topics)
print X[0]
clf = BernoulliNB()
clf.fit(X, Y)
print "Prediction:", clf.predict( ['hello'] )

，但是正如预期的那样，我将获得ValueError：无法将字符串转换为float。我的问题是如何创建一个简单的分类器来对问题进行分类并摘录到相关主题中？

Answer 1

Sklearn中的所有分类器都要求输入表示为某些固定维度的向量。对于文本，有CountVectorizer，HashingVectorizer和TfidfVectorizer，可以将您的字符串转换为浮动数的向量。

vect = TfidfVectorizer()
X = vect.fit_transform(X)

显然，您需要以相同的方式对测试集进行矢量化

clf.predict( vect.transform(['hello']) )

请参阅有关将Sklearn与文本数据一起使用的教程。