Spark SQL:催化剂是扫描不希望的列

Question

我有两个方案：

scala> val dfA = sqlContext.read.parquet("/home/mohit/ruleA")
dfA: org.apache.spark.sql.DataFrame = [aid: int, aVal: string]
scala> val dfB = sqlContext.read.parquet("/home/mohit/ruleB")
dfB: org.apache.spark.sql.DataFrame = [bid: int, bVal: string]
scala> dfA.registerTempTable("A")
scala> dfB.registerTempTable("B")

1。左连接到

的过滤器

sqlContext.sql("select A.aid, B.bid from A left join B on A.aid=B.bid where B.bid<2").explain
== Physical Plan ==
Project [aid#15,bid#17]
+- Filter (bid#17 < 2)
   +- BroadcastHashOuterJoin [aid#15], [bid#17], LeftOuter, None
      :- Scan ParquetRelation[aid#15,aVal#16] InputPaths: file:/home/mohit/ruleA
      +- Scan ParquetRelation[bid#17,bVal#18] InputPaths: file:/home/mohit/ruleB

2。左将与

的过滤器连接

sqlContext.sql("select A.aid, B.bid from A left join B on A.aid=B.bid and B.bid<2").explain
== Physical Plan ==
Project [aid#15,bid#17]
+- BroadcastHashOuterJoin [aid#15], [bid#17], LeftOuter, None
   :- Scan ParquetRelation[aid#15] InputPaths: file:/home/mohit/ruleA
   +- Filter (bid#17 < 2)
      +- Scan ParquetRelation[bid#17] InputPaths: file:/home/mohit/ruleB, PushedFilters: [LessThan(bid,2)]

问题

在任何一种情况下，Catalyst都有信息，从表B中，仅需要B.bid（BID＃17）。为什么在WHERE情况下需要进行整个表扫描。表B的projection列是隐式和确定性的。

注意：这是生产问题中浇水的例子。火花版-1.6.2。

Answer 1

我在jira-18642的Spark上提出了这一点。这是Spark 1.6的真正错误。