从Spotify API调用创建PySpark Dataframe

Question

我目前正在学习更多关于pySpark的知识，并想把一个项目放在一起，将spotify api数据放入spark数据框架中进行分析。

我熟悉如何在Pandas中做到这一点，并认为方法应该是相似的，但我很快意识到这里的情况并非如此。我已经检查了其他SO问题，但仍然没有找到适合我的方法。

我现在的代码是这样的:

from spotipy.oauth2 import SpotifyClientCredentials, SpotifyOAuth
import datetime
import os
from pyspark.sql import SparkSession, types
from pyspark import SparkContext,SparkConf

# Extract Data
def get_data():
CLIENT_ID = os.environ.get('SP_CLIENT_ID')
CLIENT_SEC = os.environ.get('SP_CLIENT_SECRET')
scope = "user-library-read user-read-recently-played"
today = datetime.datetime.now()
yesterday = today - datetime.timedelta(days=1)
yesterday_unix_timestamp = int(yesterday.timestamp()) * 1000
client_creds_manager = SpotifyClientCredentials(client_id=CLIENT_ID,client_secret=CLIENT_SEC)
sp = spotipy.Spotify(auth_manager=SpotifyOAuth(client_id=CLIENT_ID,client_secret=CLIENT_SEC,scope=scope,redirect_uri='http://localhost:3000/callback'))

return sp.current_user_recently_played(after=yesterday_unix_timestamp, limit=30)
# Transform Data in pyspark
def transform():
data = get_data()
song_names = []
artist_names = []
played_at = []
timestamps = []
for song in data['items']:
song_names.append(song['track']['name'])
artist_names.append(song['track']['album']['artists'][0]['name'])
played_at.append(song['played_at'])
timestamps.append(song['played_at'][0:10])
song_dict = {
'song_names': song_names,
'artist_names': artist_names,
'played_at': played_at,
'timestamps': timestamps
}
spark = SparkSession.builder.appName('Spotify').getOrCreate()
sc = spark.sparkContext
json_rdd = sc.parallelize(song_dict)
df = spark.read.json(json_rdd, multiLine=True)
df.show(truncate=False)
transform()

任何帮助将非常感激，spark是一个有趣的技术来学习和加载api数据是我想真正理解的东西。

谢谢!

Answer 1

所以你有多个Python的列表，你想创建一个Spark的DataFrame。您应该使用SparkSession的createDataFrame方法。

songs = "a b c x y z".split()
names = "i h k l m n".split()
df = spark.createDataFrame(zip(songs,names),['songs','names'])
df.show()

你会得到一个Spark的DataFramedf，像这样:

+-----+-----+
|songs|names|
+-----+-----+
|    a|    i|
|    b|    h|
|    c|    k|
|    x|    l|
|    y|    m|
|    z|    n|
+-----+-----+