如何在Scikit-learn聚类中使用Pearson相关作为距离度量



我有以下数据:

State   Murder  Assault UrbanPop    Rape
Alabama 13.200  236 58  21.200
Alaska  10.000  263 48  44.500
Arizona 8.100   294 80  31.000
Arkansas    8.800   190 50  19.500
California  9.000   276 91  40.600
Colorado    7.900   204 78  38.700
Connecticut 3.300   110 77  11.100
Delaware    5.900   238 72  15.800
Florida 15.400  335 80  31.900
Georgia 17.400  211 60  25.800
Hawaii  5.300   46  83  20.200
Idaho   2.600   120 54  14.200
Illinois    10.400  249 83  24.000
Indiana 7.200   113 65  21.000
Iowa    2.200   56  57  11.300
Kansas  6.000   115 66  18.000
Kentucky    9.700   109 52  16.300
Louisiana   15.400  249 66  22.200
Maine   2.100   83  51  7.800
Maryland    11.300  300 67  27.800
Massachusetts   4.400   149 85  16.300
Michigan    12.100  255 74  35.100
Minnesota   2.700   72  66  14.900
Mississippi 16.100  259 44  17.100
Missouri    9.000   178 70  28.200
Montana 6.000   109 53  16.400
Nebraska    4.300   102 62  16.500
Nevada  12.200  252 81  46.000
New Hampshire   2.100   57  56  9.500
New Jersey  7.400   159 89  18.800
New Mexico  11.400  285 70  32.100
New York    11.100  254 86  26.100
North Carolina  13.000  337 45  16.100
North Dakota    0.800   45  44  7.300
Ohio    7.300   120 75  21.400
Oklahoma    6.600   151 68  20.000
Oregon  4.900   159 67  29.300
Pennsylvania    6.300   106 72  14.900
Rhode Island    3.400   174 87  8.300
South Carolina  14.400  279 48  22.500
South Dakota    3.800   86  45  12.800
Tennessee   13.200  188 59  26.900
Texas   12.700  201 80  25.500
Utah    3.200   120 80  22.900
Vermont 2.200   48  32  11.200
Virginia    8.500   156 63  20.700
Washington  4.000   145 73  26.200
West Virginia   5.700   81  39  9.300
Wisconsin   2.600   53  66  10.800
Wyoming 6.800   161 60  15.600

我用它来执行基于状态的分层聚类。这是完整的工作代码:

import pandas as pd 
from sklearn.cluster import AgglomerativeClustering
df = pd.io.parsers.read_table("http://dpaste.com/031VZPM.txt")
samples = df["State"].tolist()
ndf = df[["Murder", "Assault", "UrbanPop","Rape"]]
X = ndf.as_matrix()
cluster = AgglomerativeClustering(n_clusters=3, 
                               linkage='complete',affinity='euclidean').fit(X)
label = cluster.labels_
outclust = list(zip(label, samples))  
outclust_df = pd.DataFrame(outclust,columns=["Clusters","Samples"])  
for clust in outclust_df.groupby("Clusters"):
    print (clust)

注意,在这个方法中,我使用了euclidean距离。我要做的是用1-Pearson correlation distance。在R中是这样的:

dat <- read.table("http://dpaste.com/031VZPM.txt",sep="t",header=TRUE)
dist2 = function(x) as.dist(1-cor(t(x), method="pearson"))
dat = dat[c("Murder","Assault","UrbanPop","Rape")]
hclust(dist2(dat), method="ward.D")

我如何使用Scikit-learn agglomerativeclu群集实现这一点?我理解有"预先计算"的亲和力的论据。但不知道如何使用它来解决我的问题

你可以定义一个自定义的关联矩阵作为一个函数,它接受你的数据并返回关联矩阵:

from scipy.stats import pearsonr
import numpy as np
def pearson_affinity(M):
   return 1 - np.array([[pearsonr(a,b)[0] for a in M] for b in M])

那么你就可以把这个称为亲和函数的凝聚聚类(你必须改变链接,因为'ward'只适用于欧几里得距离。

cluster = AgglomerativeClustering(n_clusters=3, linkage='average',
                           affinity=pearson_affinity)
cluster.fit(X)

请注意,由于某些原因,它似乎不能很好地处理您的数据:

cluster.labels_
Out[107]: 
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1,
       0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 1, 0])

最新更新