>我需要读取文件行并将每行拆分为单词并对单词进行操作。
我该怎么做?
我写了下面的代码:
logFile = "/home/hadoop/spark-2.3.1-bin-hadoop2.7/README.md" # Should be
some file on your system
spark = SparkSession.builder.appName("SimpleApp1").getOrCreate()
logData = spark.read.text(logFile).cache()
logData.printSchema()
logDataLines = logData.collect()
#The line variable below seems to be of type row. How I perform similar operations
on row or how do I convert row to a string.
for line in logDataLines:
words = line.select(explode(split(line,"s+")))
for word in words:
print(word)
print("----------------------------------")
我认为您应该对行应用map
函数。 您可以在自创建函数中应用任何内容:
data = spark.read.text("/home/spark/test_it.txt").cache()
def someFunction(row):
wordlist = row[0].split(" ")
result = list()
for word in wordlist:
result.append(word.upper())
return result
data.rdd.map(someFunction).collect()
输出:
[[u'THIS', u'IS', u'JUST', u'A', u'TEST'], [u'TO', u'UNDERSTAND'], [u'THE', u'PROCESSING']]