如何编写一个返回两个文档之间余弦相似性的方法

Question

我正在编写一个返回两个文档之间余弦相似性的方法。使用sklearn CountVector((我试过

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity
def doc_cos_similar(doc1:str, doc2:str) -> float:
vectorizer= CountVectorizer()
doc1="Good morning"
doc2="Good evening"
documents = [doc1, doc2]
count_vectorizer = CountVectorizer()
sparse_matrix = count_vectorizer.fit_transform(documents)
doc_term_matrix = sparse_matrix.todense()
return doc_term_matrix

#输入

doc1="Good morning"
doc2="Good afternoon"

输出应该是0.60(差不多(

但是输出是

矩阵([0,1,1]，[1,1,0]](

Answer 1

您就快到了。

cosine_similarity(doc_term_matrix)返回

array([[1. , 0.5],
[0.5, 1. ]])

所以你可以使用cosine_similarity(doc_term_matrix)[0][1](或者[1][0]，这并不重要，因为余弦是对称的(。

请注意，您应该将doc1和doc2作为参数传递，而不是对它们进行硬编码。

Answer 2

你可以试试这个：

from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
# X = input("Enter first string: ").lower()
# Y = input("Enter second string: ").lower()
X ="Good morning! Welcome"
Y ="Good evening! Welcome"
# tokenization
X_list = word_tokenize(X)
Y_list = word_tokenize(Y)
# sw contains the list of stopwords
sw = stopwords.words('english')
l1 =[];l2 =[]
# remove stop words from the string
X_set = {w for w in X_list if not w in sw}
Y_set = {w for w in Y_list if not w in sw}
# form a set containing keywords of both strings
rvector = X_set.union(Y_set)
for w in rvector:
if w in X_set: l1.append(1) # create a vector
else: l1.append(0)
if w in Y_set: l2.append(1)
else: l2.append(0)
c = 0
# cosine formula
for i in range(len(rvector)):
c+= l1[i]*l2[i]
cosine = c / float((sum(l1)*sum(l2))**0.5)
print("similarity: ", cosine)