Spark使用继承scala函数(Java/SparkSQL)

Question

我试图使用继承的Scala函数(StuctType.diff())，我得到一个NoSuchMethodError。

---

线程"main"中的异常java.lang.NoSuchMethodError:org.apache.spark.sql.types.StructType.diff (Lscala/收集/GenSeq;) Lscala/收集/Seq;TableNode.neighborNode (SparkSQLTest.java: 112)SparkSQLTest.main (SparkSQLTest.java: 58)在sun.reflect.NativeMethodAccessorImpl。invoke0(本地方法)sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java: 62)sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java: 43)java.lang.reflect.Method.invoke (Method.java: 497)在.org apache引发美元美元org.apache.spark.deploy.SparkSubmit部署SparkSubmit美元$ $ runMain (SparkSubmit.scala: 731)在org.apache.spark.deploy.SparkSubmit .doRunMain 1美元(SparkSubmit.scala: 181)在org.apache.spark.deploy.SparkSubmit美元。submit (SparkSubmit.scala: 206)org.apache.spark.deploy.SparkSubmit .main美元(SparkSubmit.scala: 121)org.apache.spark.deploy.SparkSubmit.main (SparkSubmit.scala) *

---

代码如下所示:

public StructType foo(DataFrame df){
    StructField sf = this.schema.diff(df.schema()).last();
    StructType schema_tmp = new StructType().add(sf);
    return schema_tmp;
}

有人有什么想法吗?我使用Spark 1.6.2和Scala 2.10

Answer 1

我使用Spark 2.0和StructType方法是支持的。请看看下面的代码片段，希望对您有所帮助:

val schema = new StructType(数组
)StructField("可以",StringType,真的),
StructField("SALE_DATE"， DateType,true))