我的代码在emr 中不工作
finalDF.createOrReplaceGlobalTempView("temp_visits")
spark.sql(s"insert overwrite table test PARTITION (date) SELECT * from temp_visits")
我得到以下异常
Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: temp_visits; line 1 pos 100
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:663)
我需要添加什么设置才能识别这个临时表吗?
需要在引用的全局视图中添加数据库名称
默认火花设置具有数据库名称-global_temp
finalDF.createOrReplaceGlobalTempView("temp_visits")
spark.sql(s"insert overwrite table test PARTITION (date) SELECT * from global_temp.temp_visits")
您可以通过在sparkSession初始化-期间提供配置来更改此名称
spark.sql.globalTempDatabase