使用映射创建一个新列

Question

是否有一种方法(不使用UDF)通过获取现有列并从映射中提取其等效值来获取现有数据框架并创建新列?

df.withColumn("newCol", transform(col("existing").using(map)))

其中map的键类型与existing相同，其值为我想要的输出

Answer 1

可以将Map转换为DataFrame和join:

val df = sc.parallelize(Seq(
    (1, "foo"), (2, "bar"), (3, "foobar")
)).toDF("id", "existing")
val map = Map("foo" -> 1, "bar" -> 2)
val lookup = sc.parallelize(map.toSeq).toDF("key", "value")
df
 .join(lookup, $"existing" <=> $"key", "left")
 .drop("key")
 .withColumnRenamed("value", "newCol")

Answer 2

import sqtx.implicits._
val x = Map("foo" -> 1,"bar"-> 2, "baz"->3)
val df = sc.parallelize(Seq(
  (1, "foo"), (2, "bar"), (3, "foobar")
)).toDF("id", "existing")
df.map(r => (r.getInt(0),x.getOrElse(r.getString(1),0))).toDF("id","new")