Databricks Connect:无法从IntelliJ在Databricks集群中运行scala程序



遵循本文档中提到的步骤。databricks-connect test命令工作正常。然而,当我从Intellij启动测试scala程序时,我看到以下错误:

Exception in thread "main" java.lang.NoSuchMethodError: com.databricks.spark.util.MetricDefinitions$.EVENT_INITIAL_CONFIG_LOG()Lcom/databricks/spark/util/MetricDefinition;
at com.databricks.spark.util.InitialConfigLogging$.recordInitialConfigs(InitialConfigLogging.scala:47)
at org.apache.spark.sql.internal.SQLConf.recordSessionInitialConfs(SQLConf.scala:4166)
at org.apache.spark.sql.SparkSession.<init>(SparkSession.scala:150)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1033)
at com.srikanth.Demo$.main(Demo.scala:13)
at com.srikanth.Demo.main(Demo.scala)

环境详细信息:

python - 3.8
java - 1.8
databricks-connect - 9.1

我收到了同样的错误消息,结果发现我没有正确添加databricks连接库。您应该像这里描述的那样添加库:https://docs.databricks.com/dev-tools/databricks-connect.html#intellij-scala或java

最新更新