从sparkapi中的字符串列(日期时间戳)中提取星期天编号

Question

我是Spark API的新手。我正试图从字符串列col_date（具有日期时间戳，例如"13AUG15:09:40:15"）中提取工作日编号，并添加另一列作为工作日（整数）。我做不成功。

Answer 1

下面的方法对我有效，使用了一个"单行"udf-与上面类似但不同：

from  pyspark.sql  import SparkSession, functions
spark = SparkSession.builder.appName('dayofweek').getOrCreate()

设置数据帧：

df = spark.createDataFrame(
    [(1, "2018-05-12")
     ,(2, "2018-05-13")
     ,(3, "2018-05-14")
      ,(4, "2018-05-15")
      ,(5, "2018-05-16")
      ,(6, "2018-05-17")
      ,(7, "2018-05-18")
      ,(8, "2018-05-19")
      ,(9, "2018-05-20")
    ], ("id", "date")) 

设置udf:

from pyspark.sql.functions import udf,desc
from datetime import datetime
weekDay =  udf(lambda x: datetime.strptime(x, '%Y-%m-%d').strftime('%w'))
df = df.withColumn('weekDay', weekDay(df['date'])).sort(desc("date"))

结果：

    df.show()
+---+----------+-------+
| id|      date|weekDay|
+---+----------+-------+
|  9|2018-05-20|      0|
|  8|2018-05-19|      6|
|  7|2018-05-18|      5|
|  6|2018-05-17|      4|
|  5|2018-05-16|      3|
|  4|2018-05-15|      2|
|  3|2018-05-14|      1|
|  2|2018-05-13|      0|
|  1|2018-05-12|      6|
+---+----------+-------+

Answer 2

这很简单。

这个简单的函数生成所有作业，并将工作日返回为数字（星期一=1）：

from time import time
from datetime import datetime
# get weekdays and daily hours from timestamp
def toWeekDay(x):
#     v = datetime.strptime(datetime.fromtimestamp(int(x)).strftime("%Y %m %d %H"), "%Y %m %d %H").strftime('%w') - from unix timestamp
    v = datetime.strptime(x, '%d%b%y:%H:%M:%S').strftime('%w')
    return v
days = ['13AUG15:09:40:15','27APR16:20:04:35'] # create example dates
days = sc.parallelize(days) # for example purposes - transform python list to RDD so we can do it in a 'Spark [parallel] way'
days.take(2) # to see whats in RDD
> ['13AUG15:09:40:15', '27APR16:20:04:35']
result = v.map(lambda x: (toWeekDay(x))) # apply functon toWeekDay on each element of RDD
result.take(2) # lets see results
> ['4', '3']

有关日期时间处理的更多详细信息，请参阅Python文档。