Spark结构化流媒体可视化

Question

我正在尝试在结构化流中可视化流查询。我怎么能那样做？我应该使用仪表板还是有其他工具？

我在网上找不到类似的东西。

DF = spark 
.readStream 
.format("kafka") 
.option("kafka.bootstrap.servers", bootstrapServers)
.option("subscribe", topics)
.load()
.selectExpr("CAST(value AS STRING)")
...
query1 = prediction.writeStream.outputMode("update").format('console').start()
query1.awaitTermination()

Answer 1

试一下这样的方法-queryName the coil：

Scala

// Have all the aggregates in an in-memory table
val aggDF
.writeStream
.queryName("aggregates")    // this query name will be the table name
.outputMode("complete")
.format("memory")
.start()
spark.sql("select * from aggregates").show() 

pyspark

# Have all the aggregates in an in-memory table. The query name will be the table name
aggDF 
.writeStream 
.queryName("aggregates") 
.outputMode("complete") 
.format("memory") 
.start()
spark.sql("select * from aggregates").show()   # interactively query in-memory table

DataBricks中的笔记本具有display功能。