避免在 PySpark 中连接两个数据帧时出现列重复的列名

Question

我有以下代码：

from pyspark.sql import SQLContext
ctx = SQLContext(sc)
a = ctx.createDataFrame([("1","a",1),("2","a",1),("3","a",0),("4","a",0),("5","b",1),("6","b",0),("7","b",1)],["id","group","value1"])
b = ctx.createDataFrame([("1","a",8),("2","a",1),("3","a",1),("4","a",2),("5","b",1),("6","b",3),("7","b",4)],["id","group","value2"])
c = a.join(b,"id")
c.select("group")

它返回一个错误：

pyspark.sql.utils.AnalysisException: Reference 'group' is ambiguous, could be: group#1406, group#1409.;

问题是c有两倍相同的列"group"：

>>> c.columns
['id', 'group', 'value1', 'group', 'value2']

例如，我希望能够做c.select("a.group")但我不知道如何在进行连接时自动调整列名。

Answer 1

只需删除引号：c.select(a.group)，它将从数据框中选择group列a。