如何在Spark2中将结构类型的映射转换为Json

Question

我在具有以下模式的数据集中有一个映射字段

|-- party: map (nullable = true)
|    |-- key: string
|    |-- value: struct (valueContainsNull = true)
|    |    |-- partyName: string (nullable = true)
|    |    |-- cdrId: string (nullable = true)
|    |    |-- legalEntityId: string (nullable = true)
|    |    |-- customPartyId: string (nullable = true)
|    |    |-- partyIdScheme: string (nullable = true)
|    |    |-- customPartyIdScheme: string (nullable = true)
|    |    |-- bdrId: string (nullable = true)

需要将其转换为JSON类型。请建议如何做。提前感谢

Answer 1

Spark为DataFrame操作提供了to_json功能：

import org.apache.spark.sql.functions._
import spark.implicits._
val df =
List(
("key1", "party01", "cdrId01"),
("key2", "party02", "cdrId02"),
)
.toDF("key", "partyName", "cdrId")
.select(struct($"key", struct($"partyName", $"cdrId")).as("col1"))
.agg(map_from_entries(collect_set($"col1")).as("map_col"))
.select($"map_col", to_json($"map_col").as("json_col"))