GraphFrames api是否支持创建二分图

Question

GraphFrames api是否支持在当前版本中创建二分图？

当前版本：0.1.0

Spark版本：1.6.1

Answer 1

正如这个问题的评论中所指出的，GraphFrames和GraphX都没有内置的对二分图的支持。然而，它们都有足够的灵活性来创建二分图。有关GraphX解决方案，请参见上一个答案。该解决方案使用不同顶点/对象类型之间的共享特性。虽然这对RDDs有效，但对DataFrames无效。DataFrame中的一行有一个固定的模式——它有时不能包含price列，有时也不能。它可以有一个price列，有时是null，但该列必须存在于每一行中。

相反，GraphFrames的解决方案似乎是，您需要定义一个DataFrame，它本质上是二分图中两种类型对象的线性子类型——它必须包含这两种类型的对象的所有字段。这其实很简单——一个join和full_outer会给你这个。类似这样的东西：

val players = Seq(
  (1,"dave", 34),
  (2,"griffin", 44)
).toDF("id", "name", "age")
val teams = Seq(
  (101,"lions","7-1"),
  (102,"tigers","5-3"),
  (103,"bears","0-9")
).toDF("id","team","record")

然后，您可以创建一个超级集合DataFrame，如下所示：

val teamPlayer = players.withColumnRenamed("id", "l_id").join(
  teams.withColumnRenamed("id", "r_id"),
  $"r_id" === $"l_id", "full_outer"
).withColumn("l_id", coalesce($"l_id", $"r_id"))
 .drop($"r_id")
 .withColumnRenamed("l_id", "id")
teamPlayer.show
+---+-------+----+------+------+
| id|   name| age|  team|record|
+---+-------+----+------+------+
|101|   null|null| lions|   7-1|
|102|   null|null|tigers|   5-3|
|103|   null|null| bears|   0-9|
|  1|   dave|  34|  null|  null|
|  2|griffin|  44|  null|  null|
+---+-------+----+------+------+

你可以用structs:做得更干净一点

val tpStructs = players.select($"id" as "l_id", struct($"name", $"age") as "player").join(
  teams.select($"id" as "r_id", struct($"team",$"record") as "team"),
  $"l_id" === $"r_id",
  "full_outer"
).withColumn("l_id", coalesce($"l_id", $"r_id"))
 .drop($"r_id")
 .withColumnRenamed("l_id", "id")
tpStructs.show
+---+------------+------------+
| id|      player|        team|
+---+------------+------------+
|101|        null| [lions,7-1]|
|102|        null|[tigers,5-3]|
|103|        null| [bears,0-9]|
|  1|   [dave,34]|        null|
|  2|[griffin,44]|        null|
+---+------------+------------+

我还将指出，或多或少相同的解决方案将在GraphX和RDDs中工作。您总是可以通过连接两个不共享任何traits:的case classes来创建顶点

case class Player(name: String, age: Int)
val playerRdd = sc.parallelize(Seq(
  (1L, Player("date", 34)),
  (2L, Player("griffin", 44))
))
case class Team(team: String, record: String)
val teamRdd = sc.parallelize(Seq(
  (101L, Team("lions", "7-1")),
  (102L, Team("tigers", "5-3")),
  (103L, Team("bears", "0-9"))
))
playerRdd.fullOuterJoin(teamRdd).collect foreach println
(101,(None,Some(Team(lions,7-1))))
(1,(Some(Player(date,34)),None))
(102,(None,Some(Team(tigers,5-3))))
(2,(Some(Player(griffin,44)),None))
(103,(None,Some(Team(bears,0-9))))

考虑到前面的答案，这似乎是一种更灵活的处理方式——不必在组合的对象之间共享trait。