使用kube_ping缩放后,在野生蝇中重复警告消息



我试图使用kube_ping jgroups协议在kubernetes上以HA全模式运行野生蝇。一切都很好,我可以扩大群集,节点互相识别而没有任何问题。

当我尝试缩放群集时,就会发生问题。ActiveMQ Artemis不断抱怨它无法连接到断开的节点,即使JGroups承认旧节点已离开群集。

我想知道我在JGroups配置中可能做错了什么。我已经附加了一些日志消息,以及我的jgroups配置KUBE_PING

只是为了确保我提供了尽可能多的信息,我正在使用最新的Wildfly官方Docker映像, 15.0.1.final ,它在JDK 11上运行。/p>

事先感谢您的任何帮助!

编辑:固定错字

jGroups node disconnect

wildfly-kube 12:48:36,514 INFO  [org.apache.activemq.artemis.core.server] (Thread-22 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@10f88645)) AMQ221027: Bridge ClusterConnectionBridge@379d51e3 [name=$.artemis.internal.sf.my-cluster.7ee91868-337b-11e9-9849-ce422226aad5, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.7ee91868-337b-11e9-9849-ce422226aad5, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=314721ae-337b-11e9-9cfa-0e8a9828b1cb], temp=false]@195607a8 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@379d51e3 [name=$.artemis.internal.sf.my-cluster.7ee91868-337b-11e9-9849-ce422226aad5, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.7ee91868-337b-11e9-9849-ce422226aad5, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=314721ae-337b-11e9-9cfa-0e8a9828b1cb], temp=false]@195607a8 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=100-116-0-4], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1699294977[nodeUUID=314721ae-337b-11e9-9cfa-0e8a9828b1cb, connector=TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=100-122-0-6, address=jms, server=ActiveMQServerImpl::serverUUID=314721ae-337b-11e9-9cfa-0e8a9828b1cb])) [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=100-116-0-4], discoveryGroupConfiguration=null]] is connected
wildfly-kube-b6f69fb9-b2hd5 wildfly-kube 12:48:38,905 WARN  [org.apache.activemq.artemis.core.server] (Thread-5 (ActiveMQ-client-global-threads)) AMQ222095: Connection failed with failedOver=false
wildfly-kube-b6f69fb9-b2hd5 wildfly-kube 12:48:43,758 ERROR [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,wildfly-kube-b6f69fb9-b2hd5) JGRP000034: wildfly-kube-b6f69fb9-b2hd5: failure sending message to wildfly-kube-b6f69fb9-nshvn: java.net.SocketTimeoutException: connect timed out
wildfly-kube-b6f69fb9-b2hd5 wildfly-kube 12:48:44,759 INFO  [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-13,ejb,wildfly-kube-b6f69fb9-b2hd5) ISPN000094: Received new cluster view for channel ejb: [wildfly-kube-b6f69fb9-b2hd5|2] (1) [wildfly-kube-b6f69fb9-b2hd5]
wildfly-kube-b6f69fb9-b2hd5 wildfly-kube 12:48:44,772 INFO  [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-13,ejb,wildfly-kube-b6f69fb9-b2hd5) ISPN100001: Node wildfly-kube-b6f69fb9-nshvn left the cluster
wildfly-kube-b6f69fb9-b2hd5 wildfly-kube 12:48:44,777 INFO  [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-13,ejb,wildfly-kube-b6f69fb9-b2hd5) ISPN000094: Received new cluster view for channel ejb: [wildfly-kube-b6f69fb9-b2hd5|2] (1) [wildfly-kube-b6f69fb9-b2hd5]
wildfly-kube-b6f69fb9-b2hd5 wildfly-kube 12:48:44,779 INFO  [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-13,ejb,wildfly-kube-b6f69fb9-b2hd5) ISPN100001: Node wildfly-kube-b6f69fb9-nshvn left the cluster
wildfly-kube-b6f69fb9-b2hd5 wildfly-kube 12:48:44,787 INFO  [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-13,ejb,wildfly-kube-b6f69fb9-b2hd5) ISPN000094: Received new cluster view for channel ejb: [wildfly-kube-b6f69fb9-b2hd5|2] (1) [wildfly-kube-b6f69fb9-b2hd5]
wildfly-kube-b6f69fb9-b2hd5 wildfly-kube 12:48:44,788 INFO  [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-13,ejb,wildfly-kube-b6f69fb9-b2hd5) ISPN100001: Node wildfly-kube-b6f69fb9-nshvn left the cluster
wildfly-kube-b6f69fb9-b2hd5 wildfly-kube 12:48:44,791 INFO  [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-13,ejb,wildfly-kube-b6f69fb9-b2hd5) ISPN000094: Received new cluster view for channel ejb: [wildfly-kube-b6f69fb9-b2hd5|2] (1) [wildfly-kube-b6f69fb9-b2hd5]
wildfly-kube-b6f69fb9-b2hd5 wildfly-kube 12:48:44,792 INFO  [org.infinispan.CLUSTER] (VERIFY_SUSPECT.TimerThread-13,ejb,wildfly-kube-b6f69fb9-b2hd5) ISPN100001: Node wildfly-kube-b6f69fb9-nshvn left the cluster

重复ActiveMQ Artemis警告(每3秒一次)

wildfly-kube-b6f69fb9-b2hd5 wildfly-kube 13:02:11,825 WARN  [org.apache.activemq.artemis.core.server] (Thread-55 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@866e807)) AMQ224091: Bridge ClusterConnectionBridge@39836857 [name=$.artemis.internal.sf.my-cluster.314721ae-337b-11e9-9cfa-0e8a9828b1cb, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.314721ae-337b-11e9-9cfa-0e8a9828b1cb, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=7ee91868-337b-11e9-9849-ce422226aad5], temp=false]@39425add targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@39836857 [name=$.artemis.internal.sf.my-cluster.314721ae-337b-11e9-9cfa-0e8a9828b1cb, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.314721ae-337b-11e9-9cfa-0e8a9828b1cb, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=7ee91868-337b-11e9-9849-ce422226aad5], temp=false]@39425add targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=100-122-0-6], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1432944139[nodeUUID=7ee91868-337b-11e9-9849-ce422226aad5, connector=TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=100-116-0-4, address=jms, server=ActiveMQServerImpl::serverUUID=7ee91868-337b-11e9-9849-ce422226aad5])) [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=100-122-0-6], discoveryGroupConfiguration=null]] is unable to connect to destination. Retrying
wildfly-kube-b6f69fb9-b2hd5 wildfly-kube 13:02:14,897 WARN  [org.apache.activemq.artemis.core.server] (Thread-68 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@866e807)) AMQ224091: Bridge ClusterConnectionBridge@39836857 [name=$.artemis.internal.sf.my-cluster.314721ae-337b-11e9-9cfa-0e8a9828b1cb, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.314721ae-337b-11e9-9cfa-0e8a9828b1cb, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=7ee91868-337b-11e9-9849-ce422226aad5], temp=false]@39425add targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@39836857 [name=$.artemis.internal.sf.my-cluster.314721ae-337b-11e9-9cfa-0e8a9828b1cb, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.314721ae-337b-11e9-9cfa-0e8a9828b1cb, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=7ee91868-337b-11e9-9849-ce422226aad5], temp=false]@39425add targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=100-122-0-6], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1432944139[nodeUUID=7ee91868-337b-11e9-9849-ce422226aad5, connector=TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=100-116-0-4, address=jms, server=ActiveMQServerImpl::serverUUID=7ee91868-337b-11e9-9849-ce422226aad5])) [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=100-122-0-6], discoveryGroupConfiguration=null]] is unable to connect to destination. Retrying

JGroups配置

<subsystem xmlns="urn:jboss:domain:jgroups:6.0">
<channels default="ee">
    <channel name="ee" stack="tcp" cluster="ejb"/>
</channels>
<stacks>
    <stack name="tcp">
        <transport type="TCP" socket-binding="jgroups-tcp">
            <property name="logical_addr_cache_expiration">360000</property>
        </transport>
        <protocol type="kubernetes.KUBE_PING">
            <property name="namespace">${KUBERNETES_CLUSTER_NAMESPACE:default}</property>
            <property name="labels">${KUBERNETES_CLUSTER_LABEL:cluster=nyc}</property>
            <property name="port_range">0</property>
        </protocol>
        <protocol type="MERGE3"/>
        <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
        <protocol type="FD_ALL"/>
        <protocol type="VERIFY_SUSPECT"/>
        <protocol type="pbcast.NAKACK2">
            <property name="use_mcast_xmit">false</property>
        </protocol>
        <protocol type="UNICAST3"/>
        <protocol type="pbcast.STABLE"/>
        <protocol type="pbcast.GMS">
            <property name="join_timeout">30000</property>
            <property name="print_local_addr">true</property>
            <property name="print_physical_addrs">true</property>
        </protocol>
        <protocol type="MFC"/>
        <protocol type="FRAG3"/>
    </stack>
</stacks>

ActiveMQ Artemis配置

<subsystem xmlns="urn:jboss:domain:messaging-activemq:5.0">
<server name="default">
    <cluster user="my_admin" password="my_password"/>
    <security-setting name="#">
        <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
    </security-setting>
    <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>
    <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
    <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
        <param name="batch-delay" value="50"/>
    </http-connector>
    <in-vm-connector name="in-vm" server-id="0">
        <param name="buffer-pooling" value="false"/>
    </in-vm-connector>
    <http-acceptor name="http-acceptor" http-listener="default"/>
    <http-acceptor name="http-acceptor-throughput" http-listener="default">
        <param name="batch-delay" value="50"/>
        <param name="direct-deliver" value="false"/>
    </http-acceptor>
    <in-vm-acceptor name="in-vm" server-id="0">
        <param name="buffer-pooling" value="false"/>
    </in-vm-acceptor>
    <broadcast-group name="bg-group1" jgroups-cluster="activemq-cluster" connectors="http-connector"/>
    <discovery-group name="dg-group1" jgroups-cluster="activemq-cluster"/>
    <cluster-connection name="my-cluster" address="jms" connector-name="http-connector" discovery-group="dg-group1"/>
    <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
    <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
    <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
    <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" ha="true" block-on-acknowledge="true" reconnect-attempts="-1"/>
    <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" transaction="xa"/>
</server>

更新:我要添加的一件事,如果容器优雅地关闭,Artemis似乎可以正确处理断开连接。在我的kubernetes部署中的容器定义中添加 prestop 命令在终止容器之前将Wildfly关闭,以便将容器优雅地从集群中取出。

ActiveMQ Artemis仅使用JGroup(或任何其他发现机制)进行 Discover 其他经纪人,以将它们聚集在一起。一旦发现了另一个经纪人,他们在自己之间建立了TCP连接,此后,JGroups不发挥任何作用,这意味着看到经纪人离开集群的JGroup是无关紧要的。

集群桥失败的事实足以告诉ActiveMQ Artemis经纪人离开了集群。那时的问题是经纪人应该做什么以响应死节点。默认情况下,它将尝试无限期地重新连接,因为它希望该节点在某个时候返回。在传统用例中,这是一个合理的期望,但在云中没有那么多。此行为由cluster-connection上的reconnect-attempts属性控制。将reconnect-attempts设置为您认为合理的东西(例如10),您会看到桥梁重新连接放弃并停止记录。

我只是遇到了这个,贾斯汀的答案似乎正确。

这是您可以在容器init

中使用的CLI命令

/subsystem = messaging-activemq/server =默认/cluster-connection = my-cluster:write-attribute(name = reconnect-atempts,value = 3)

这是针对Bitnami/Wildfly容器图像中运行的Wildfly 28。

相关内容

  • 没有找到相关文章

最新更新