SPARK SQL在联合上选择别名列

Question

我正在与自己进行联合，然后尝试选择我创建的别名列。

这是我拥有的代码：

val mergedDF = sparkSession.sqlContext.sql(" SELECT COLUMN1 as COLUMN3 FROM MY_VIEW Union SELECT COLUMN2 as COLUMN3 from MY_VIEW")
val mergedView = mergedDF.createOrReplaceTempView("MERGED_VIEW")
val distinctColumnDF =sparkSession.sqlContext.sql(" SELECT distinct COLUMN3 from MERGED_VIEW WHERE node like '%city%')
logger.debug("No.of Distinct City Rows="+distinctColumnDF.count());

我收到以下错误：

org.apache.spark.sql.catalyst.parser.ParseException: mismatched input 'from' expecting { <EOF>, 'WHERE' , 'GROUP', 'ORDER', 'HAVING', 'LIMIT', 'LATERAL', 'WINDOW', 'UNION', 'EXCEPT', 'MINUS', 'INTERSECT','SORT','CLUSTER','DISTRIBUTE' }(line 1 , pos 21)
== SQL ==
SELECT distinct COLUMN3 from MERGED_VIEW where node like '%city%'
---------------------^^^^

感谢任何帮助。

谢谢

Answer 1

以下似乎正在工作。

val dfA = sparkSession.sqlContext.sql("SELECT COLUMN1 FROM MY_VIEW")
val dfB = sparkSession.sqlContext.sql("SELECT COLUMN2 FROM MY_VIEW")     
val mergedDF = dfA.select(dfA.col("COLUMN1").alias("COLUMN3")).union(dfB.select(dfB.col("COLUMN2").alias("COLUMN3")))
val str = mergedDF.select(mergedDF.col("COLUMN3")).filter(mergedDF.col("COLUMN3").contains("test")).distinct().count()
logger.debug("No.of Distinct Rows="+str);

不确定为什么纯SQL语法不起作用。

感谢FIAGB的建议。

Answer 2

您可以在dataFrame上使用别名方法，例如：

val dfA = sparkSession.sqlContext.sql("SELECT COLUMN1 FROM MY_VIEW")
val dfB = sparkSession.sqlContext.sql("SELECT COLUMN2 FROM MY_VIEW")     
val mergedDF = dfA.select(dfA.col("COLUMN1").alias("COLUMN3")).union(dfB.select(dfB.col("COLUMN2").alias("COLUMN3")))
val distinctColumnDF = mergedDF.filter(mergedDF.col("COLUMN3").contains("city")).distinct().collect()
logger.debug("No.of Distinct Rows="+distinctColumnDF.count());