如何将字符串转换为数据块中各种长度的日期?

Question

我有一个不同长度的字符串，例如11/8/16和1/27/16。格式是Month/Day/Year，我如何将这些转换为日期?我试过各种组合的MM, MM, DD等，但不能让它工作。

Answer 1

我检查了Spark 3，必须将timeParserPolicy更改为LEGACY

import pyspark.sql.functions as F
spark.conf.set("spark.sql.legacy.timeParserPolicy","LEGACY")
data1 = [
["11/8/16"],
["1/27/16"]
]
df = spark.createDataFrame(data1).toDF("source")
tmp = df.withColumn("parsed_to_date", F.to_date(F.col("source"), "MM/dd/yy"))
tmp.show(truncate = False)

输出

+-------+--------------+
|source |parsed_to_date|
+-------+--------------+
|11/8/16|2016-11-08    |
|1/27/16|2016-01-27    |
+-------+--------------+