Apache ActiveMQ Artemis HA集群部署在Kubernetes中，Istio代理注入到Artemis

Question

Apache ActiveMQ Artemis集群在Kubernetes环境中，Istio代理注入到Artemis pod中，记录了大量显示主(活动)和从(备份)实例之间连接丢失的错误。我们还经历了从JBoss EAP到Artemis的间歇性连接丢失，在随后的调用中恢复(例如，在发送JMS消息时)。

活动(主)实例的tcpdump显示了备份和活动artemis实例之间连接的大量RST(与没有注入Istio代理的环境的tcpdump相比)。

Artemis acceptor允许/使用的协议是CORE, AMQP。为Artemis集群保留默认的通信端口。

Artemis集群用于JMS消息传递。JMS通信是从Jboss EAP执行的，该EAP"位于";在同一个kubernetes命名空间的另一个pod中。

静态连接器用于形成Artemis集群。复制机制用于数据交换。静态连接器也在Jboss配置中使用。

Istio代理中TCP和HTTP连接的idleTimeout设置为无限大。

当Istio代理没有被注入时，Artemis日志不会显示任何错误，在这种情况下也没有观察到JMS消息传递的问题。

注意:在安装了artemis/jboss的容器上没有ping命令可用(如果需要执行"活检查")

来自活动(主)Artemis pod(多个片段)的堆栈跟踪:

2023-04-10 20:56:19,436 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to <***artemis backup instance's DNS***>/<***artemis backup instance's IP***>:61616 has been detected: AMQ219011: Did not receive data from server for org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection@6243c958[ID=fef52fc9, local= /<***artemis active instance's IP***>:55842, remote=<***artemis backup instance's DNS***>/<***artemis backup instance's IP***>:61616] [code=CONNECTION_TIMEDOUT]
2023-04-10 20:56:19,562 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection
java.lang.IllegalStateException: No ActiveMQChannelHandler has been found while connecting to <***artemis backup instance's DNS***>/<***artemis backup instance's IP***>:61616 from Channel with id = cf68f05a
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:954) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:840) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:822) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(ClientSessionFactoryImpl.java:1105) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1212) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1146) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(ClientSessionFactoryImpl.java:1375) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:967) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:858) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:799) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:656) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:534) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:527) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:390) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58) ~[artemis-commons-2.27.0.jar:?]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33) ~[artemis-commons-2.27.0.jar:?]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69) ~[artemis-commons-2.27.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.27.0.jar:?]

2023-04-11 01:22:33,180 WARN  [org.apache.activemq.artemis.core.server] AMQ222092: Connection to the backup node failed, removing replication now
org.apache.activemq.artemis.api.core.ActiveMQConnectionTimedOutException: AMQ229014: Did not receive data from /127.0.0.6:55653 within the 60000ms connection TTL. The connection will now be closed.
at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread$2.run(RemotingServiceImpl.java:781) ~[artemis-server-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58) ~[artemis-commons-2.27.0.jar:?]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33) ~[artemis-commons-2.27.0.jar:?]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69) ~[artemis-commons-2.27.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.27.0.jar:?]
2023-04-11 01:22:33,405 ERROR [org.apache.activemq.artemis.core.client] AMQ214016: Failed to create netty connection
java.lang.IllegalStateException: No ActiveMQChannelHandler has been found while connecting to <***artemis backup instance's DNS***>/<***artemis backup instance's IP***>:61616 from Channel with id = f979331a
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:954) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:840) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:822) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(ClientSessionFactoryImpl.java:1105) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1212) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1146) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(ClientSessionFactoryImpl.java:1375) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:967) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:858) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:799) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:656) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:534) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:527) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:390) ~[artemis-core-client-2.27.0.jar:2.27.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:58) ~[artemis-commons-2.27.0.jar:?]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:33) ~[artemis-commons-2.27.0.jar:?]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:69) ~[artemis-commons-2.27.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.27.0.jar:?]

来自Jboss pod的堆栈跟踪:

2023-04-10 20:56:10,649 AMQ214016: Failed to create netty connection: java.lang.IllegalStateException: No ActiveMQChannelHandler has been found while connecting to <***artemis backup instance's DNS***>/<***artemis backup instance's IP***>:61616 from Channel with id = 98ac7598
at [e-mail]-00045//org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:970)
at [e-mail]-00045//org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:856)
at [e-mail]-00045//org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:838)
at [e-mail]-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.openTransportConnection(ClientSessionFactoryImpl.java:1097)
at [e-mail]-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createTransportConnection(ClientSessionFactoryImpl.java:1146)
at [e-mail]-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.establishNewConnection(ClientSessionFactoryImpl.java:1378)
at [e-mail]-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:952)
at [e-mail]-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:841)
at [e-mail]-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:779)
at [e-mail]-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:638)
at [e-mail]-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:525)
at [e-mail]-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:518)
at [e-mail]-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$100(ClientSessionFactoryImpl.java:74)
at [e-mail]-00045//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:381)
at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

执行的操作:我们尝试使用EnvoyFilter配置将artemis和jboss Istio代理的入站和出站连接的idleTimeout设置为infinite (EnvoyFilter是Istio代理的配置资源——帮助配置入站/出站流量)。我们还将reconnect-attempts设置为"-1";连接ttl到"86400000";对于jboss (standallone-full.xml文件)中的pooled-connection-factory .

我们期望与Artemis的连接至少保持24小时，但它没有发生。

我想知道可能的根本原因是什么，以及需要对Artemis和Jboss安装应用哪些额外配置以保持连接正常。这个问题与"保持活力"有关吗?支票?

Answer 1

这个问题似乎与Artemis无关，而是与kubernetes中用于Artemis集群的配置有关。

更新5/2:执行了额外的测试，似乎allow-direct-connections-only="true">如下:

<static-connectors allow-direct-connections-only="true">

解决了这个问题，直到Artemis 2.24，但从2.26开始，它仍然不能与Istio一起工作(上面共享的日志仍然相关)。