如何选择最大(日期)与spark数据框架API

Question

给定以下数据集

id v  date
1  a1 1
1  a2 2
2  b1 3
2  b2 4

我想只选择每个id的最后一个值(关于日期)。

我想出了这个代码:

scala> val df = sc.parallelize(List((41,"a1",1), (1, "a2", 2), (2, "b1", 3), (2, "b2", 4))).toDF("id", "v", "date")
df: org.apache.spark.sql.DataFrame = [id: int, v: string, date: int]
scala> val agg = df.groupBy("id").max("date")
agg: org.apache.spark.sql.DataFrame = [id: int, max(date): int]
scala> val res = df.join(agg, df("id") === agg("id") && df("date") === agg("max(date)"))
16/11/14 22:25:01 WARN sql.Column: Constructing trivially true equals predicate, 'id#3 = id#3'. Perhaps you need to use aliases.
res: org.apache.spark.sql.DataFrame = [id: int, v: string, date: int, id: int, max(date): int]

有没有更好的方法(更习惯的，…)?

红利:如何在日期列上执行max并避免此错误Aggregation function can only be applied on a numeric column. ?

Answer 1

您可以尝试agg()与max函数:

import static org.apache.spark.sql.functions.* df.groupBy("id").agg(max("date"))

Answer 2

对我来说，它只是这样工作的:

df = df.groupBy('CPF').agg({'DATA': 'max'})