如何在 Spark-CSV 中编写"DATE"数据类型

Question

我有类似的数据。

no,full_name,Ocupation,DOB
1,Eugene Perry,Physical Therapy Assistant,11/22/1972
2,Scott Evans,Account Executive,8/25/1961
3,Harold Porter,Community Outreach Specialist,4/5/1986
4,Roger Turner,Account Coordinator,12/24/1989
5,Patrick Freeman,Geological Engineer,2/11/1990
6,Jonathan Hart,Safety Technician I,7/24/1963
7,Susan Jenkins,Nurse,4/17/1967
8,Jason Graham,Internal Auditor,10/18/1966
9,Mark Wells,VP Marketing,2/20/1990

(Java)

 SparkSession spark = SparkSession
                .builder()
                .appName("Java Spark SQL Example")
                .config("spark.some.config.option", "some-value")
                .config("spark.sql.warehouse.dir", "file:///C:/tmp/") 
                .getOrCreate();
       StructType Const = new StructType(new StructField[] {
            new StructField("no", DataTypes.IntegerType, true, Metadata.empty()),
            new StructField("full_name", DataTypes.StringType, true, Metadata.empty()),
            new StructField("Ocupation", DataTypes.StringType, true, Metadata.empty()),
            new StructField("DOB", DataTypes.DateType, true, Metadata.empty())
        });
                Dataset<Row> Tem  =  spark.read().format("com.databricks.spark.csv")
                        .schema(Const).option("header", "true").option("delimiter", ",").option("dateFormat", "MM/DD/YYYY")
                        .load("E:\Datasets\people.csv");
                Tem.show();
                Tem.printSchema();
                Tem.coalesce(1).write()
                    .format("com.databricks.spark.csv")
                    .option("header", "true").option("dateFormat", "MM/DD/YYYY")
                    .save("E:\Datasets\output");

上面的代码工作正常，没有错误，但是DATE数据类型("DOB")列在平面文件中没有正确打印。

my实际结果是:

no,full_name,Ocupation,DOB
1,Eugene Perry,Physical Therapy Assistant,724
2,Scott Evans,Account Executive,-3287
3,Harold Porter,Community Outreach Specialist,5841
4,Roger Turner,Account Coordinator,6940
5,Patrick Freeman,Geological Engineer,7304
6,Jonathan Hart,Safety Technician I,-2559

，即使我使用.option("dateFormat", "MM/DD/YYYY")。

见上一列，"DATE"格式在.csv文件中不工作。我不知道我错过了什么。

Answer 1

此问题与spark 2.0.0版本有关。

升级到spark 2.0.1并使用内置的spark CSV格式可以修复此问题。

参考链接