我应该在使用 Apach Flink 的节点上的防火墙中打开哪些端口



当我尝试在Apache Flink独立集群上运行我的流程时,我看到以下异常:

java.lang.IllegalStateException: Update task on instance aaa0859f6af25decf1f5fc1821ffa55d @ app-2 - 4 slots - URL: akka.tcp://flink@192.168.38.98:46369/user/taskmanager failed due to:
    at org.apache.flink.runtime.executiongraph.Execution$6.onFailure(Execution.java:954)
    at akka.dispatch.OnFailure.internal(Future.scala:228)
    at akka.dispatch.OnFailure.internal(Future.scala:227)
    at akka.dispatch.japi$CallbackBridge.apply(Future.scala:174)
    at akka.dispatch.japi$CallbackBridge.apply(Future.scala:171)
    at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
    at scala.runtime.AbstractPartialFunction.applyOrElse(AbstractPartialFunction.scala:28)
    at scala.concurrent.Future$$anonfun$onFailure$1.apply(Future.scala:136)
    at scala.concurrent.Future$$anonfun$onFailure$1.apply(Future.scala:134)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
    at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://flink@192.168.38.98:46369/user/taskmanager#1804590378]] after [10000 ms]
    at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333)
    at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
    at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599)
    at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
    at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597)
    at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)
    at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)
    at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)
    at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
    at java.lang.Thread.run(Thread.java:745)

似乎端口 46369 被防火墙阻止。这是真的,因为我阅读了配置部分并仅打开这些端口:

6121:
  comment: Apache Flink TaskManager (Data Exchange)
6122:
  comment: Apache Flink TaskManager (IPC)
6123:
  comment: Apache Flink JobManager
6130:
  comment: Apache Flink JobManager (BLOB Server)
8081:
  comment: Apache Flink JobManager (Web UI)

flink-conf.yaml中所述的相同端口:

jobmanager.rpc.address: app-1.stag.local
jobmanager.rpc.port: 6123
jobmanager.heap.mb: 1024
taskmanager.heap.mb: 2048
taskmanager.numberOfTaskSlots: 4
taskmanager.memory.preallocate: false
blob.server.port: 6130
parallelism.default: 4
jobmanager.web.port: 8081
state.backend: jobmanager
restart-strategy: none
restart-strategy.fixed-delay.attempts: 2
restart-strategy.fixed-delay.delay: 60s

所以,我有两个问题:

  1. 此异常与阻塞的端口有关。右?
  2. 我应该在防火墙上为独立的 Apache Flink 集群打开哪些端口?

更新 1

我在mastersslaves文件中发现了配置问题(我跳过了这些文件中所述的主机之间的新行分隔符)。我修复了它,现在我看到其他异常:

  • flink--taskmanager-0-app-1.stag.local.log
  • flink--taskmanager-0-app-2.stag.local.log

我有 2 个节点:

  • app-1.stag.local(使用正在运行的作业和任务管理器)
  • app-2.stag.local(运行任务管理器)

从这些日志中可以看出,app-1.stag.local 任务管理器无法连接到其他任务管理器:

java.io.IOException:连接通道失败:连接到远程任务管理器 + 'app-2.stag.local/192.168.38.98:35806' 失败。这可能表示远程任务管理器已丢失。

但是app-2.stag.local有开放端口:

2016-03-18 16:24:14,347 INFO org.apache.flink.runtime.io.network.netty.NettyServer - 初始化成功(耗时 39 毫秒)。侦听套接字地址/192.168.38.98:35806

所以,我认为问题与防火墙有关,但我不明白我可以在 Apache Flink 中在哪里配置此端口(或端口范围)。

我发现了一个问题:默认情况下taskmanager.data.port参数设置为 0(但文档说明它应该设置为 6121 )。

因此,我将此端口设置为flink-conf.yaml,现在一切正常。

相关内容

  • 没有找到相关文章

最新更新