我正在寻找一种方法来标记pd.dataframe(例如df(中连续常量值(例如n(,这些值是常量。
我编写了一些代码,如果值与下一个 n/2 和前一个数据点的 n/2 之差为零,则通过这些代码将标记该值。
n = 5 # the minimum number of sequential constant values
#to create a adatframe example
df=pd.DataFrame(np.random.randn(25), index=pd.date_range(start='2010-1-1',end='2010-1-2',freq='H'), columns=['value'])
#to modify the dataframe to have several sets of constant values
df[1:10]=23
df[20:26]=10
df[15:17]=15
for i in np.arange(1, int(n/2)):
# to calcualte the difference between value and ith previous values
df_diff['delta_' + str(i)] = (df['value'].diff(periods=i)).abs()
# to calcualte the difference between value and ith next values
df_diff['delta_' + str(-i)] = (df['value'].diff(periods=-i)).abs()
# to filter the results (e.g. as a boolean)
result_1 = (df_diff[:] <= 0).all(axis=1)
result_2 = (df_diff[:] <= 0).any(axis=1)
此示例中的result_1和results_2未提供正确答案。
我的期望是:
2010-01-01 00:00:00 False
2010-01-01 01:00:00 True
2010-01-01 02:00:00 True
2010-01-01 03:00:00 True
2010-01-01 04:00:00 True
2010-01-01 05:00:00 True
2010-01-01 06:00:00 True
2010-01-01 07:00:00 True
2010-01-01 08:00:00 True
2010-01-01 09:00:00 True
2010-01-01 10:00:00 False
2010-01-01 11:00:00 False
2010-01-01 12:00:00 False
2010-01-01 13:00:00 False
2010-01-01 14:00:00 False
2010-01-01 15:00:00 False
2010-01-01 16:00:00 False
2010-01-01 17:00:00 False
2010-01-01 18:00:00 False
2010-01-01 19:00:00 False
2010-01-01 20:00:00 True
2010-01-01 21:00:00 True
2010-01-01 22:00:00 True
2010-01-01 23:00:00 True
2010-01-02 00:00:00 True
IIUC,使用DataFrame.groupby
石斑鱼Series.diff
,.ne(0)
然后.cumsum
:
df.groupby(df.value.diff().ne(0).cumsum())['value'].transform('size').ge(n)
[出]
2010-01-01 00:00:00 False
2010-01-01 01:00:00 True
2010-01-01 02:00:00 True
2010-01-01 03:00:00 True
2010-01-01 04:00:00 True
2010-01-01 05:00:00 True
2010-01-01 06:00:00 True
2010-01-01 07:00:00 True
2010-01-01 08:00:00 True
2010-01-01 09:00:00 True
2010-01-01 10:00:00 False
2010-01-01 11:00:00 False
2010-01-01 12:00:00 False
2010-01-01 13:00:00 False
2010-01-01 14:00:00 False
2010-01-01 15:00:00 False
2010-01-01 16:00:00 False
2010-01-01 17:00:00 False
2010-01-01 18:00:00 False
2010-01-01 19:00:00 False
2010-01-01 20:00:00 True
2010-01-01 21:00:00 True
2010-01-01 22:00:00 True
2010-01-01 23:00:00 True
2010-01-02 00:00:00 True
Freq: H, Name: value, dtype: bool
解释
我们分组的序列将是具有相等值的连续组:
s = df.value.diff().ne(0).cumsum()
2010-01-01 00:00:00 1
2010-01-01 01:00:00 2
2010-01-01 02:00:00 2
2010-01-01 03:00:00 2
2010-01-01 04:00:00 2
2010-01-01 05:00:00 2
2010-01-01 06:00:00 2
2010-01-01 07:00:00 2
2010-01-01 08:00:00 2
2010-01-01 09:00:00 2
2010-01-01 10:00:00 3
2010-01-01 11:00:00 4
2010-01-01 12:00:00 5
2010-01-01 13:00:00 6
2010-01-01 14:00:00 7
2010-01-01 15:00:00 8
2010-01-01 16:00:00 8
2010-01-01 17:00:00 9
2010-01-01 18:00:00 10
2010-01-01 19:00:00 11
2010-01-01 20:00:00 12
2010-01-01 21:00:00 12
2010-01-01 22:00:00 12
2010-01-01 23:00:00 12
2010-01-02 00:00:00 12
Freq: H, Name: value, dtype: int32
当您按这些"组 id"分组时,使用 transform
返回与原始DataFrame
相同的形状的对象,聚合到"大小",您将获得:
s.groupby(s).transform('size')
2010-01-01 00:00:00 1
2010-01-01 01:00:00 9
2010-01-01 02:00:00 9
2010-01-01 03:00:00 9
2010-01-01 04:00:00 9
2010-01-01 05:00:00 9
2010-01-01 06:00:00 9
2010-01-01 07:00:00 9
2010-01-01 08:00:00 9
2010-01-01 09:00:00 9
2010-01-01 10:00:00 1
2010-01-01 11:00:00 1
2010-01-01 12:00:00 1
2010-01-01 13:00:00 1
2010-01-01 14:00:00 1
2010-01-01 15:00:00 2
2010-01-01 16:00:00 2
2010-01-01 17:00:00 1
2010-01-01 18:00:00 1
2010-01-01 19:00:00 1
2010-01-01 20:00:00 5
2010-01-01 21:00:00 5
2010-01-01 22:00:00 5
2010-01-01 23:00:00 5
2010-01-02 00:00:00 5
Freq: H, Name: value, dtype: int64
从这里开始,这是一个简单的Series.ge
(>=
(与您的价值n
比较
请参阅 https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.duplicated.html
import pandas as pd
df = pd.DataFrame({
'brand': ['Yum Yum', 'Yum Yum', 'Indomie', 'Indomie', 'Indomie'],
'style': ['cup', 'cup', 'cup', 'pack', 'pack'],
'rating': [4, 4, 3.5, 15, 5]
})
df
# brand style rating
# 0 Yum Yum cup 4.0
# 1 Yum Yum cup 4.0
# 2 Indomie cup 3.5
# 3 Indomie pack 15.0
# 4 Indomie pack 5.0
df.duplicated(keep=False)
# 0 True
# 1 True
# 2 False
# 3 False
# 4 False
# dtype: bool