我正在流处理框架和之间进行一些基准比较
我选择了WordCount；你好世界"这方面的任务(有些曲折(，到目前为止测试了Flink和Hazelcast Jet，结果是Flink需要80+秒才能完成，而Jet只需要30+秒的
我知道Flink很受欢迎，我在这里做错了什么？真的很好奇这个
我的样本代码在这里
https://github.com/ChinW/stream-processing-compare
以下是详细信息(规范、管道、日志(
测试的WordCount管道
Source (read from file, 5MB)
-> Process: Split line into words (Here here is a bomb, every word emit 1000 times)
-> Group/Count
-> Sink (do nothing)
- MacBook Pro(13英寸，2020，四个Thunderbolt 3端口(
- 2 GHz四核Intel Core i5(8个逻辑处理器(
- 16 GB 3733 MHz LPDDR4X
- JDK 11
喷气式飞机4.4管道：
digraph DAG {
"items" [localParallelism=1];
"fused(flat-map, filter)" [localParallelism=8];
"group-and-aggregate-prepare" [localParallelism=8];
"group-and-aggregate" [localParallelism=8];
"do-nothing-sink" [localParallelism=1];
"items" -> "fused(flat-map, filter)" [queueSize=1024];
"fused(flat-map, filter)" -> "group-and-aggregate-prepare" [label="partitioned", queueSize=1024];
subgraph cluster_0 {
"group-and-aggregate-prepare" -> "group-and-aggregate" [label="distributed-partitioned", queueSize=1024];
}
"group-and-aggregate" -> "do-nothing-sink" [queueSize=1024];
}
日志：
Start time: 2021-04-18T13:52:52.106
Duration: 00:00:36.459
Jet: finish in 36.45935081 seconds.
Start time: 2021-04-19T16:51:53.806
Duration: 00:00:30.143
Jet: finish in 30.625740453 seconds.
Start time: 2021-04-19T16:52:48.906
Duration: 00:00:37.207
Jet: finish in 37.862554137 seconds.
Flink 1.12.2 for Scala 2.11flink-config.yaml配置：
jobmanager.rpc.address: localhost
jobmanager.rpc.port: 6123
jobmanager.memory.process.size: 2096m
taskmanager.memory.process.size: 12288m
taskmanager.numberOfTaskSlots: 8
parallelism.default: 8
管道：
{
"nodes" : [ {
"id" : 1,
"type" : "Source: Custom Source",
"pact" : "Data Source",
"contents" : "Source: Custom Source",
"parallelism" : 1
}, {
"id" : 2,
"type" : "Flat Map",
"pact" : "Operator",
"contents" : "Flat Map",
"parallelism" : 8,
"predecessors" : [ {
"id" : 1,
"ship_strategy" : "REBALANCE",
"side" : "second"
} ]
}, {
"id" : 4,
"type" : "Keyed Aggregation",
"pact" : "Operator",
"contents" : "Keyed Aggregation",
"parallelism" : 8,
"predecessors" : [ {
"id" : 2,
"ship_strategy" : "HASH",
"side" : "second"
} ]
}, {
"id" : 5,
"type" : "Sink: Unnamed",
"pact" : "Data Sink",
"contents" : "Sink: Unnamed",
"parallelism" : 8,
"predecessors" : [ {
"id" : 4,
"ship_strategy" : "FORWARD",
"side" : "second"
} ]
} ]
}
日志：
❯ flink run -c chiw.spc.flink.FlinkWordCountKt stream-processing-compare-1.0-SNAPSHOT.jar
Job has been submitted with JobID 163ce849a663e45f3c3028a98f260e7c
Program execution finished
Job with JobID 163ce849a663e45f3c3028a98f260e7c has finished.
Job Runtime: 88614 ms
❯ flink run -c chiw.spc.flink.FlinkWordCountKt stream-processing-compare-1.0-SNAPSHOT.jar
Job has been submitted with JobID fcf12488204969299e4e5d7f23f4ea6e
Program execution finished
Job with JobID fcf12488204969299e4e5d7f23f4ea6e has finished.
Job Runtime: 90165 ms
❯ flink run -c chiw.spc.flink.FlinkWordCountKt stream-processing-compare-1.0-SNAPSHOT.jar
Job has been submitted with JobID 37e349e4fad90cd7405546d30239afa4
Program execution finished
Job with JobID 37e349e4fad90cd7405546d30239afa4 has finished.
Job Runtime: 78908 ms
非常感谢您的帮助！
我不认为你做错了什么，我们的测试表明Jet比Spark和Flink快得多，单词计数是我们用来衡量这一点的例子之一。
考虑到炸弹会创建大量的小项目(而不是少量的大项目(，我对Jet在这里可能具有优势的最佳猜测是它的单生产者-单消费者(SPSC(队列与类似协程的并发性。
您有8个平面映射阶段和8个聚合阶段。Jet将在总共8个线程上执行此操作(假设您有8个availableProcessors(，因此在操作系统级别上几乎不会执行线程调度。数据将在线程之间以大块的形式移动：flatMap将一次排队1024个，然后每个聚合器将提取所有指定给它的项目。SPSC队列上的通信不会受到其他线程的任何干扰：每个聚合处理器有8个输入队列，每个平面映射器专用一个。
在Flink中，每个阶段都将启动另外8个线程，我还注意到接收器的并行度为8，所以这是24个线程，而另一个线程用于源。操作系统必须将它们安排在8个物理核心上。通信将发生在多生产者-单消费者(MPSC(队列上，这意味着所有平面映射器线程必须进行协调，以便每次只有单个线程将项目排入任何给定的聚合器，并且争用会导致所有线程中的热CAS循环。
为了证实这种怀疑，请尝试收集一些分析数据。如果上面的故事是正确的，那么Flink将花费大量的CPU时间将数据排入队列。