我想通过使用pyspark
中的列表来生成查询list = ["hi@gmail.com", "goodbye@gmail.com"]
query = "SELECT * FROM table WHERE email IN (" + list + ")"
这是我所需的输出：
query
SELECT * FROM table WHERE email IN ("hi@gmail.com", "goodbye@gmail.com")
我得到了：TypeError: cannot concatenate 'str' and 'list' objects
谁能帮助我实现这一目标？谢谢
如果有人遇到相同的问题，我发现您可以使用以下代码：
"'"+"','".join(map(str, emails))+"'"
您将具有以下输出：
SELECT * FROM table WHERE email IN ('hi@gmail.com', 'goodbye@gmail.com')
尝试以下：
基于数据框的方法 -
df = spark.createDataFrame([(1,"hi@gmail.com") ,(2,"goodbye@gmail.com",),(3,"abc@gmail.com",),(4,"xyz@gmail.com")], ['id','email_id'])
email_filter_list = ["hi@gmail.com", "goodbye@gmail.com"]
df.where(col('email_id').isin(email_filter_list)).show()
SPARK SQL基于SQL的方法 -
df = spark.createDataFrame([(1,"hi@gmail.com") ,(2,"goodbye@gmail.com",),(3,"abc@gmail.com",),(4,"xyz@gmail.com")], ['id','email_id'])
df.createOrReplaceTempView('t1')
sql_filter = ','.join(["'" +i + "'" for i in email_filter_list])
spark.sql("SELECT * FROM t1 WHERE email_id IN ({})".format(sql_filter)).show()