使用滞后函数后，将 null 替换为数据帧中的另一个值

Question

df1.show(10):
+--------+---------+-------------+-------------------+
|issue_id|letter_id|read_duration|read_dttm          |
+--------+---------+-------------+-------------------+
|300     |186      |null         |2017-02-03 14:34:19|
|300     |186      |null         |2017-02-03 14:34:18|
|336     |2643     |null         |2017-04-14 15:29:36|
|300     |1860971  |null         |2017-02-03 14:34:17|
|336     |2647574  |null         |2017-04-14 15:29:36|
|276     |12421353 |null         |2017-01-17 10:31:43|
|276     |12421354 |null         |2016-12-29 22:15:14|
|276     |12421355 |null         |2016-12-28 14:37:00|
|276     |12421355 |null         |2017-03-03 11:31:38|
|276     |12421355 |null         |2017-01-18 18:01:07|
+--------+---------+-------------+-------------------+

接下来，我正在使用lag函数：

windowSpec = W.partitionBy(df1.issue_id, df1.letter_id).orderBy(df1.issue_id, df1.letter_id, df1.read_dttm)
df1_lag = df1.where((df1.issue_id == '276') & (df1.letter_id == '12421355'))
.select(df1.issue_id, df1.letter_id, df1.read_duration, df1.read_dttm
        , lag(df1.read_dttm, 1).over(windowSpec).alias('previous_read_dttm')).show()

现在，我得到了这个：

+--------+---------+-------------+-------------------+-------------------+
|issue_id|letter_id|read_duration|          read_dttm| previous_read_dttm|
+--------+---------+-------------+-------------------+-------------------+
|     276| 12421355|         null|2016-12-28 12:31:06|               null|
|     276| 12421355|         null|2016-12-28 13:11:30|2016-12-28 12:31:06|
|     276| 12421355|         null|2016-12-28 14:37:00|2016-12-28 13:11:30|
|     276| 12421355|         null|2017-01-18 18:01:07|2016-12-28 14:37:00|
|     276| 12421355|         null|2017-01-24 12:56:35|2017-01-18 18:01:07|
|     276| 12421355|         null|2017-03-03 11:31:38|2017-01-24 12:56:35|
+--------+---------+-------------+-------------------+-------------------+

如何将列previous_read_dttm中的空值替换为"1900-01-01 00：00：00"？

Answer 1

抱歉，我找到了答案，很简单：

df1_lag = df1.where((df1.issue_id == '276') & (df1.letter_id == '12421355'))
.select(df1.issue_id, df1.letter_id, df1.read_duration, df1.read_dttm
        , lag(df1.read_dttm, 1, '1900-01-01 00:00:00').over(windowSpec).alias('previous_read_dttm')).show()

结果是：

+--------+---------+-------------+-------------------+-------------------+
|issue_id|letter_id|read_duration|          read_dttm| previous_read_dttm|
+--------+---------+-------------+-------------------+-------------------+
|     276| 12421355|         null|2016-12-28 12:31:06|1900-01-01 00:00:00|
|     276| 12421355|         null|2016-12-28 13:11:30|2016-12-28 12:31:06|
|     276| 12421355|         null|2016-12-28 14:37:00|2016-12-28 13:11:30|
|     276| 12421355|         null|2017-01-18 18:01:07|2016-12-28 14:37:00|
|     276| 12421355|         null|2017-01-24 12:56:35|2017-01-18 18:01:07|
|     276| 12421355|         null|2017-03-03 11:31:38|2017-01-24 12:56:35|
+--------+---------+-------------+-------------------+-------------------+