将 Spark 数据帧列值替换为随机值(例如 UUID)

Question

>我有一个带有列user_tag的数据帧，我想在其中获得新的随机UUID值，我该怎么做？

--------------------------------------
| user_tag  |  pref_code  |  name    |
--------------------------------------
| abc123    |  Reg        |  Richard |
| abc123    |  Reg        |  Mort    |
| abc123    |  Disc       |  Jack    |

我想为火花中的user_tag生成随机UUID。要有

-------------------------------------------------------------------
| user_tag                                |  pref_code  |  name    |
-------------------------------------------------------------------
| af3fb8b8-7ceb-4cec-ac27-2a034bb44bb9    |  Reg        |  Richard |
| snc22fls-2cgb-sas2-hc26-43d35ggg4522    |  Reg        |  Mort    |
| afgdw8b8-4fss-ycec-ycd7-haj3jbbj4bj9    |  Disc       |  Jack    |

我试过这个：但它导致每一行都有相同的 UUID

val withUUID = dataFrame.withColumn("user_tag", 
  when(col("user_tag") === "abc123", randomUUID.toString).otherwise(col("user_tag")))

Answer 1

你可以

尝试通过创建udf然后在case when-then statement中调用udf。

例：

val rand_UUID = udf(() => java.util.UUID.randomUUID().toString) //udf to generate randomUUID
val df=Seq(("abc123","Reg","Richard"),("abc123","Reg","Mort"))
       .toDF("user_tag","pref_code","name")
df.withColumn("user_tag",when('user_tag === "abc123",rand_UUID())
  .otherwise('user_tag))
  .show(false)

结果：

+------------------------------------+---------+-------+
|user_tag                            |pref_code|name   |
+------------------------------------+---------+-------+
|e0b3c917-dcc5-4c42-bfe3-32af18b1cfec|Reg      |Richard|
|90098d7d-8dc7-42df-a89b-5bd7f2c5cd99|Reg      |Mort   |
+------------------------------------+---------+-------+

基本上每个匹配都会调用udf然后生成randomUUID。