我想将RDD
保存为parquet文件。为此,我将RDD传递给DataFrame
,然后使用结构将DataFrame
保存为parquet文件:
val aStruct = new StructType(Array(StructField("id",StringType,nullable = true),
StructField("role",StringType,nullable = true)))
val newDF = sqlContext.createDataFrame(filtered, aStruct)
问题是如何为所有列自动创建aStruct
假设它们都是StringType
?另外,nullable = true
的含义是什么?这是否意味着所有的空值都将被Null
代替?
为什么不使用内置的toDF
呢?
scala> val myRDD = sc.parallelize(Seq(("1", "roleA"), ("2", "roleB"), ("3", "roleC")))
myRDD: org.apache.spark.rdd.RDD[(String, String)] = ParallelCollectionRDD[60] at parallelize at <console>:27
scala> val colNames = List("id", "role")
colNames: List[String] = List(id, role)
scala> val myDF = myRDD.toDF(colNames: _*)
myDF: org.apache.spark.sql.DataFrame = [id: string, role: string]
scala> myDF.show
+---+-----+
| id| role|
+---+-----+
| 1|roleA|
| 2|roleB|
| 3|roleC|
+---+-----+
scala> myDF.printSchema
root
|-- id: string (nullable = true)
|-- role: string (nullable = true)
scala> myDF.write.save("myDF.parquet")
nullable=true
仅仅意味着指定的列可以包含null
值(这对于通常没有null
值的int
列尤其有用——Int
没有NA
或null
)。