将pyspark中特定列的null值替换为mean

Question

我想将年龄和身高列的空值替换为平均值。我知道有个帖子用同一列的平均值填充Pyspark数据帧列null值但在这篇文章中，给定的函数抛出了一个错误。

df = spark.createDataFrame([(1, 'John', 1.79, 28,'M', 'Doctor'),
(2, 'Steve', 1.78, 45,'M', None),
(3, 'Emma', 1.75, None, None, None),
(4, 'Ashley',1.6, 33,'F', 'Analyst'),
(5, 'Olivia', 1.8, 54,'F', 'Teacher'),
(6, 'Hannah', 1.82, None, 'F', None),
(7, 'William', 1.7, 42,'M', 'Engineer'),
(None,None,None,None,None,None),
(8,'Ethan',1.55,38,'M','Doctor'),
(9,'Hannah',1.65,None,'F','Doctor')]
, ['Id', 'Name', 'Height', 'Age', 'Gender', 'Profession'])

后给定中的函数

def fill_with_mean(df, exclude=set()): 
stats = df.agg(*(
avg(c).alias(c) for c in df.columns if c not in exclude
))
return df.na.fill(stats.first().asDict())
fill_with_mean(df, ["Age", "Height"])

当我运行这个函数时，它显示

NameError：未定义名称"avg">

有人能解决这个问题吗？非常感谢。

Answer 1

固定示例。它对我的作用正如你所期望的那样！

from pyspark.sql.functions import avg
df = spark.createDataFrame(
[
(1, 'John', 1.79, 28, 'M', 'Doctor'),
(2, 'Steve', 1.78, 45, 'M', None),
(3, 'Emma', 1.75, None, None, None),
(4, 'Ashley', 1.6, 33, 'F', 'Analyst'),
(5, 'Olivia', 1.8, 54, 'F', 'Teacher'),
(6, 'Hannah', 1.82, None, 'F', None),
(7, 'William', 1.7, 42, 'M', 'Engineer'),
(None, None, None, None, None, None),
(8, 'Ethan', 1.55, 38, 'M', 'Doctor'),
(9, 'Hannah', 1.65, None, 'F', 'Doctor')
],
['Id', 'Name', 'Height', 'Age', 'Gender', 'Profession']
)

def fill_with_mean(this_df, exclude=set()):
stats = this_df.agg(*(avg(c).alias(c) for c in this_df.columns if c not in exclude))
return this_df.na.fill(stats.first().asDict())

res = fill_with_mean(df, ["Gender", "Profession", "Id", "Name"])
res.show()