【Zookeeper】zookeeper not connected

迈不过友情╰ 2023-02-21 09:18 9阅读 0赞

# 一、问题分析和代码跟踪 #

原来跑的好好的代码，晚上运行的时候提示连接不上 zookeeper。提示信息如下：

java.lang.IllegalStateException: zookeeper not connected
    	at org.apache.dubbo.remoting.zookeeper.curator.CuratorZookeeperClient.<init>(CuratorZookeeperClient.java:83) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.remoting.zookeeper.curator.CuratorZookeeperTransporter.createZookeeperClient(CuratorZookeeperTransporter.java:26) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.remoting.zookeeper.support.AbstractZookeeperTransporter.connect(AbstractZookeeperTransporter.java:70) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.remoting.zookeeper.ZookeeperTransporter$Adaptive.connect(ZookeeperTransporter$Adaptive.java) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.metadata.store.zookeeper.ZookeeperMetadataReport.<init>(ZookeeperMetadataReport.java:61) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.metadata.store.zookeeper.ZookeeperMetadataReportFactory.createMetadataReport(ZookeeperMetadataReportFactory.java:37) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.metadata.report.support.AbstractMetadataReportFactory.getMetadataReport(AbstractMetadataReportFactory.java:49) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.metadata.report.MetadataReportFactory$Adaptive.getMetadataReport(MetadataReportFactory$Adaptive.java) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.metadata.report.MetadataReportInstance.init(MetadataReportInstance.java:49) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.config.bootstrap.DubboBootstrap.startMetadataReport(DubboBootstrap.java:630) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.config.bootstrap.DubboBootstrap.initMetadataService(DubboBootstrap.java:716) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.config.bootstrap.DubboBootstrap.initialize(DubboBootstrap.java:517) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.config.bootstrap.DubboBootstrap.start(DubboBootstrap.java:744) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.config.spring.context.DubboBootstrapApplicationListener.onContextRefreshedEvent(DubboBootstrapApplicationListener.java:59) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.config.spring.context.DubboBootstrapApplicationListener.onApplicationContextEvent(DubboBootstrapApplicationListener.java:52) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.apache.dubbo.config.spring.context.OneTimeExecutionApplicationContextEventListener.onApplicationEvent(OneTimeExecutionApplicationContextEventListener.java:40) ~[dubbo-2.7.7.jar:2.7.7]
    	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
    	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
    	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
    	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:404) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
    	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:361) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
    	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:898) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
    	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:554) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
    	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
    	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) [spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
    	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) [spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
    	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
    	at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
    	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) [spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
    	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
    	at com.tom.bootdubboorderprovider.BootDubboOrderProviderApplication.main(BootDubboOrderProviderApplication.java:12) [classes/:na]
    Caused by: java.lang.IllegalStateException: zookeeper not connected
    	at org.apache.dubbo.remoting.zookeeper.curator.CuratorZookeeperClient.<init>(CuratorZookeeperClient.java:80) ~[dubbo-2.7.7.jar:2.7.7]
    	... 30 common frames omitted
    
    2020-06-29 23:20:58.928  INFO 5200 --- [           main] .b.c.e.AwaitingNonWebApplicationListener :  [Dubbo] Current Spring Boot Application is about to shutdown...
    2020-06-29 23:20:58.935  INFO 5200 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'applicationTaskExecutor'
    2020-06-29 23:20:58.935  INFO 5200 --- [           main] f.a.ReferenceAnnotationBeanPostProcessor : class org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor was destroying!
    
    Process finished with exit code 1

点开报错信息，跟踪进源码，发现异常的抛出是由于在等待连接返回时的返回值false

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70][]

跟进方法，可以看到他在不断循环等待连接成功，若在最大等待时间内连接上，则返回true，否则false

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 1][]

综合所述，猜测是由于网络原因，超时所致（毕竟穷逼买的是100多块钱一年的服务器~~~）

# 二、继续深入 #

猜出了大概原因，如果是超时所致，且使用 dubbo-admin 发现 Zookeeper 运行状态正常，那么只要提高最大等待时间就可以了

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 2][]

根据关键字发现，可能跟 metadata-report 的设置有关，且 timeout 有个默认值 5 秒

![20200629233226407.png][]

继续跟踪，查找 CuratorZookeeperClient 方法的调用

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 3][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 4][]

结果很明显，这个 timeout 是从配置文件中读取出来的，若没有，则是 5000 ms

# 三、修改配置文件 #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 5][]

# 设置最大等待时间为10秒
    dubbo.metadata-report.timeout=10000

# 四、重试 => 翻车 #

重试依旧报错，查看

![20200629234155755.png][]

再去找配置文件

![202006292342455.png][]

# 配置中心连接时间改为10秒
    dubbo.config-center.timeout=10000

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 6][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 7][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200629232518324.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200629232635593.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200629233111276.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx,size_16,color_FFFFFF,t_70
[20200629233226407.png]: https://img-blog.csdnimg.cn/20200629233226407.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200629233356521.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200629233601685.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200629233758271.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx,size_16,color_FFFFFF,t_70
[20200629234155755.png]: https://img-blog.csdnimg.cn/20200629234155755.png
[202006292342455.png]: https://img-blog.csdnimg.cn/202006292342455.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200629234354879.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200629234427763.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM0NDE2MzMx,size_16,color_FFFFFF,t_70