Spark无法处理递归avro数据

Question

我有如下avsc模式：

{
"name": "address",
"type": [
"null", 
{
"type":"record",
"name":"Address",
"namespace":"com.data",
"fields":[
{
"name":"address",
"type":[ "null","com.data.Address"],
"default":null
}
]
}
],
"default": null
}

在pyspark:中加载此数据时

jsonFormatSchema = open("Address.avsc", "r").read()
spark = SparkSession.builder.appName('abc').getOrCreate()
df = spark.read.format("avro")
.option("avroSchema", jsonFormatSchema)
.load("xxx.avro")

我得到了这样的例外：

"Found recursive reference in Avro schema, which can not be processed by Spark"

我尝试了许多其他配置，但都没有成功。执行我使用的火花提交：

--packages org.apache.spark:spark-avro_2.12:3.0.1

Answer 1

这是一个预期的功能，您可以查看；问题"：

https://issues.apache.org/jira/browse/SPARK-25718