No members active in cluster group

以你之姓@ 2022-05-16 12:58 121阅读 0赞

使用Tomcat8.5.32和nginx14做负载均衡，登录系统时输入用户名和密码、验证码后很大概率会重新进入登录页面而不是直接进入系统，一开始怀疑是不是nginx重定向的问题，nginx.conf配置了半天仍然无效。。。后来就在登录代码里都加上打印输出语句，在测试环境观察终于发现是验证码在其中一台Tomcat服务器上生成，然后登录请求到另一台Tomcat服务器，导致校验验证码时不成功所以重新进入了登录页面。因为之前Tomcat进行session共享是成功的，所以也没往这方面想，现在问题找到了，tail -f logs/catalina.out查看启动日志发现报：

org.apache.catalina.ha.session.DeltaManager.getAllClusterSessions Manager [localhost#/chat]: skipping state transfer. No members active in cluster group.

这说明确实是没有检测到另一台机器集群的机器。session共享的配置也是按照官网推荐的来的：

[https://tomcat.apache.org/tomcat-8.5-doc/cluster-howto.html][https_tomcat.apache.org_tomcat-8.5-doc_cluster-howto.html]

server.xml配置如下：为了防止4000端口被防火墙限制特意改成了14000

<?xml version="1.0" encoding="UTF-8"?>
    
    
    <Server port="8005" shutdown="SHUTDOWN">
      <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
      
      
      <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
      
      <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
      <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
      <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
    
      
      <GlobalNamingResources>
        
        <Resource name="UserDatabase" auth="Container"
                  type="org.apache.catalina.UserDatabase"
                  description="User database that can be updated and saved"
                  factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
                  pathname="conf/tomcat-users.xml" />
      </GlobalNamingResources>
    
      
      <Service name="Catalina">
    
        
        
    
    
        
        <Connector port="18080" protocol="HTTP/1.1"
                   connectionTimeout="20000"
                   redirectPort="8443" />
        
        
        
        
        
        
    
        
        <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
    
    
        
    
        
        <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
          <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
            <Manager className="org.apache.catalina.ha.session.DeltaManager" expireSessionsOnShutdown="false" notifyListenersOnReplication="true"/>
              <Channel className="org.apache.catalina.tribes.group.GroupChannel">
                  <Membership className="org.apache.catalina.tribes.membership.McastService" address="228.0.0.4" port="45564" frequency="500" dropTime="3000"/>
                      <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="192.168.60.139" port="14000" autoBind="100" selectorTimeout="5000" maxThreads="6"/>
                          <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
                                <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
                                    </Sender>
                                        <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
                                            <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor"/>
                                              </Channel>
                                                <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>
                                                  <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
                                                  <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer" tempDir="/tmp/war-temp/" deployDir="/tmp/war-deploy/" watchDir="/tmp/war-listen/" watchEnabled="false"/>
      <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
                                                      </Cluster>
          
          
    
          
          <Realm className="org.apache.catalina.realm.LockOutRealm">
            
            <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
                   resourceName="UserDatabase"/>
          </Realm>
    
          <Host name="localhost"  appBase="webapps"
                unpackWARs="true" autoDeploy="true">
    
            
            
    
            
            <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
                   prefix="localhost_access_log" suffix=".txt"
                   pattern="%h %l %u %t "%r" %s %b" />
    
          </Host>
        </Engine>
      </Service>
    </Server>

context.xml也加上了distributable=true，项目的web.xml也加入了<distributable/>元素，然而还是，不行，后来把防火墙关了，竟然成功了。。。考虑是不是linux没有开通组播的原因使用下面的命令开通组播

route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

然后netstat -gn和route -e查看组播列表和路由表也没问题，但是还是不行。然而即使把组播的地址和端口都加白名单也不行，找了半天原因实在想不到防火墙还有什么限制，后来突然想到组播使用的45564端口是使用的UDP协议，而我在防火墙加的白名单是TCP协议。。。改成UDP的白名单：

\-A INPUT -p udp -m udp --dport 45564 -j ACCEPT

![70][]

再测试终于可以了。。排查问题还是不能心急，需要逐条过滤。

[https_tomcat.apache.org_tomcat-8.5-doc_cluster-howto.html]: https://tomcat.apache.org/tomcat-8.5-doc/cluster-howto.html
[70]: /images/20220516/d60fa68c5b29407d97d60fc496e88a88.png