将函数应用于每个子集组合并返回方阵

Question

如果没有

四个嵌套的for循环，我不知道该怎么做。

我想将自定义函数应用于 hour 和 day 的每个可能的子集组合，返回该值，然后将数据框透视为方阵。但是，这些for循环似乎过多，因此我正在寻找一种更有效的方法来执行此操作。我拥有的数据相当大，因此任何速度的提高都是有益的。

编辑：我更新了问题以包含自定义函数。

这里有一个例子，

示例数据

import pandas as pd
import numpy as np
dat = pd.DataFrame({'day': {0: 1, 1: 1, 2: 1, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1, 9: 1, 10: 2, 11: 2, 12: 2, 13: 2, 14: 2, 15: 2, 16: 2, 17: 2, 18: 2, 19: 2}, 'hour': {0: 1, 1: 1, 2: 1, 3: 1, 4: 1, 5: 2, 6: 2, 7: 2, 8: 2, 9: 2, 10: 1, 11: 1, 12: 1, 13: 1, 14: 1, 15: 2, 16: 2, 17: 2, 18: 2, 19: 2}, 'distance': {0: 1.2898851269657656, 1: 0.0, 2: 0.8371526423804061, 3: 0.8703856587273138, 4: 0.6257425922449789, 5: 0.0, 6: 0.0, 7: 0.0, 8: 1.2895328696587023, 9: 0.0, 10: 0.6875527848294374, 11: 0.0, 12: 0.0, 13: 0.9009031833559706, 14: 0.0, 15: 1.1040652963428623, 16: 0.0, 17: 0.0, 18: 0.0, 19: 0.0}})

法典

def custom_fn(x, y):
    x = pd.Series(x)
    y = pd.Series(y)
    x = x**2
    y = np.sqrt(y)
    return x.sum() - y.sum()
# Empty data.frame to append to
dmat = pd.DataFrame()
# For i, j = hour; k, l = day
for i in range(1, 3):
    for j in range(1, 3):
        for k in range(1, 3):
            for l in range(1, 3):
                x = dat[(dat['hour'] == i) & (dat['day'] == k)].distance
                y = dat[(dat['hour'] == j) & (dat['day'] == l)].distance
                # Calculate difference
                jds = custom_fn(x, y)
                # Build data frame and append
                outdat = pd.DataFrame({'day_hour_a': f"{k}_{i}", 'day_hour_b': f"{l}_{j}", 'jds': [round(jds, 4)]})
                dmat = dmat.append(outdat, ignore_index=True)
# Pivot data to get matrix
distMatrix = dmat.pivot(index='day_hour_a', columns='day_hour_b', values='jds')

输出

> print(distMatrix)
day_hour_b     1_1     1_2     2_1     2_2
day_hour_a                                
1_1        -0.2609  2.3782  1.7354  2.4630
1_2        -2.1118  0.5273 -0.1155  0.6121
2_1        -2.4903  0.1488 -0.4940  0.2336
2_2        -2.5557  0.0834 -0.5594  0.1682

Answer 1

如果我

理解正确，您正在做的事情与以下内容相同：

def f(x):
    return x.mean()
x = df.groupby(['day', 'hour'])['distance'].apply(f)
x = x.values[:,None] - x.values
print(x)

输出：

[[ 0.          0.46672663  0.40694201  0.50382014]
 [-0.46672663  0.         -0.05978462  0.03709351]
 [-0.40694201  0.05978462  0.          0.09687813]
 [-0.50382014 -0.03709351 -0.09687813  0.        ]]

更新

：对于更新的自定义函数，您仍然可以将其分解为单独的groupby：

g = df.groupby(['day', 'hour'])['distance']
x = g.apply(lambda z: (z**2).sum())
y = g.apply(lambda z: np.sqrt(z).sum())
x.values[:,None] - y.values

输出：

array([[-0.26092193,  2.37817717,  1.73540595,  2.46300806],
       [-2.11178008,  0.52731901, -0.1154522 ,  0.61214991],
       [-2.49031973,  0.14877937, -0.49399185,  0.23361026],
       [-2.55571493,  0.08338417, -0.55938705,  0.16821506]])

更新 2：如果无法分离计算，另一种选择是：

def f(x, y):
    return distance.jensenshannon(x, y)
x = []
g = df.groupby(['day', 'hour'])['distance']
for k1, g1 in g:
    for k2, g2 in g:
        x += [(k1, k2, f(g1, g2))]
x = pd.DataFrame(x).pivot(index=0, columns=1, values=2)
print(x)

输出：

1         (1, 1)    (1, 2)    (2, 1)    (2, 2)
0                                             
(1, 1)  0.000000  0.623167  0.419371  0.550291
(1, 2)  0.623167  0.000000  0.424608  0.832555
(2, 1)  0.419371  0.424608  0.000000  0.504233
(2, 2)  0.550291  0.832555  0.504233  0.000000