NGINX + TOMCAT出现大量的TIME-WAIT状态的TCP连接解决

缺乏、安全感 2023-10-06 08:50 7阅读 0赞

昨天把两个tomcat服务停止后，更新项目，更新完启动，两个tomcat都报了异常，其中一个tomcat报的异常如下：

[com.alibaba.druid.proxy.DruidDriver]，但在Web应用程序停止时无法注销它。 为防止内存泄漏，JDBC驱动程序已被强制取消注册。
    14-May-2020 14:38:24.188 警告 [Thread-6] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc Web应用程序 [ROOT] 注册了JDBC驱动程序 [com.mysql.jdbc.Driver]，但在Web应用程序停止时无法注销它。 为防止内存泄漏，JDBC驱动程序已被强制取消注册。
    14-May-2020 14:38:24.189 警告 [Thread-6] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
     java.lang.Object.wait(Native Method)
     java.lang.ref.ReferenceQueue.remove(Unknown Source)
     com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
    14-May-2020 14:38:24.231 警告 [Thread-6] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc Web应用程序 [WebLogin] 注册了JDBC驱动程序 [com.alibaba.druid.proxy.DruidDriver]，但在Web应用程序停止时无法注销它。 为防止内存泄漏，JDBC驱动程序已被强制取消注册。
    14-May-2020 14:38:24.231 警告 [Thread-6] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc Web应用程序 [WebLogin] 注册了JDBC驱动程序 [com.mysql.jdbc.Driver]，但在Web应用程序停止时无法注销它。 为防止内存泄漏，JDBC驱动程序已被强制取消注册。
    14-May-2020 14:38:24.232 警告 [Thread-6] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [WebLogin] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
     java.lang.Object.wait(Native Method)
     java.lang.ref.ReferenceQueue.remove(Unknown Source)
     com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
    14-May-2020 14:38:24.245 信息 [Thread-6] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-9026"]
    14-May-2020 14:38:24.247 信息 [Thread-6] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-9029"]
    14-May-2020 14:38:24.250 信息 [Thread-6] org.apache.coyote.AbstractProtocol.destroy 正在摧毁协议处理器 ["http-nio-9026"]
    14-May-2020 14:38:24.251 信息 [Thread-6] org.apache.coyote.AbstractProtocol.destroy 正在摧毁协议处理器 ["ajp-nio-9029"]

另一个tomcat报的异常如下：

14-May-2020 19:50:17.160 警告 [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [myProject] registered the JDBC driver [com.alibaba.druid.proxy.DruidDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
    14-May-2020 19:50:17.162 警告 [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [myProject] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
    14-May-2020 19:50:17.230 警告 [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [myProject] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
     java.lang.Object.wait(Native Method)
     java.lang.ref.ReferenceQueue.remove(Unknown Source)
     com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
    14-May-2020 19:50:17.310 信息 [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [D:\tomcatServer\tomcat-9.0.10\webapps\myProject.war] has finished in [7,240] ms
    14-May-2020 19:50:17.314 信息 [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [D:\tomcatServer\tomcat-9.0.10\webapps\docs]
    14-May-2020 19:50:17.361 信息 [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [D:\tomcatServer\tomcat-9.0.10\webapps\docs] has finished in [47] ms
    14-May-2020 19:50:17.362 信息 [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [D:\tomcatServer\tomcat-9.0.10\webapps\manager]
    14-May-2020 19:50:17.391 信息 [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [D:\tomcatServer\tomcat-9.0.10\webapps\manager] has finished in [28] ms
    14-May-2020 19:50:17.395 信息 [main] org.apache.catalina.startup.Catalina.start Server startup in 18478 ms
    14-May-2020 19:50:17.397 严重 [main] org.apache.catalina.core.StandardServer.await StandardServer.await: create[localhost:9015]: 
     java.net.BindException: Address already in use: JVM_Bind
    	at java.net.DualStackPlainSocketImpl.bind0(Native Method)
    	at java.net.DualStackPlainSocketImpl.socketBind(Unknown Source)
    	at java.net.AbstractPlainSocketImpl.bind(Unknown Source)
    	at java.net.PlainSocketImpl.bind(Unknown Source)
    	at java.net.ServerSocket.bind(Unknown Source)
    	at java.net.ServerSocket.<init>(Unknown Source)
    	at org.apache.catalina.core.StandardServer.await(StandardServer.java:422)
    	at org.apache.catalina.startup.Catalina.await(Catalina.java:770)
    	at org.apache.catalina.startup.Catalina.start(Catalina.java:716)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    	at java.lang.reflect.Method.invoke(Unknown Source)
    	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:350)
    	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)

当时是百思不得其解，我是正常关闭tomcat服务以后才更新的文件，怎么会报内存溢出呢？当时也是一时手足无措，把想到的关联因素redis和nginx关闭后重新启动还是老样子。今天有点头绪，会不会是关闭tomcat服务时还有残余线程存在呢？于是今天再现昨天的情景，先把两个tomcat关闭，然后通过cmd命令提示符的**netstat -aon|findstr "9010"**命令查看指定端口号的占用情况，

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoZW54aWFvbW8xNjg4_size_16_color_FFFFFF_t_70][]

可以看到，尽管两个tomcat服务已关闭，但依然有TIME\_WAIT进程在占用。所以重新启动服务时报了内存溢出的异常。由于我通过nginx反向代理了tomcat，受到TCP协议的重传机制的限制。那么问题就出在tomcat和nginx的配置上了，查了下网上资料，主要原因是：

1.nginx开启了keepalive ，而且默认用的http1.1

2.tomcat开启了keepalive，默认也使用http1.1

3.但是nginx代理却使用的http1.0，默认不开启keepalive，所以就造成了大量的TIME-WAID状态的TCP连接

同时也给出两种解决方法，第二种我没试，感兴趣可以前往一试[https://my.oschina.net/haitaohu/blog/3043113][https_my.oschina.net_haitaohu_blog_3043113]，不知效果如何，我是参照第一种方法即nginx官网的方法，修改我的nginx配置，

nginx官网提供

For HTTP, the [proxy\_http\_version][proxy_http_version] directive should be set to “`1.1`” and the “Connection” header field should be cleared:  
参考地址:[http://nginx.org/en/docs/http/ngx\_http\_upstream\_module.html][http_nginx.org_en_docs_http_ngx_http_upstream_module.html]

upstream http_backend {
        server 127.0.0.1:8080;
    
        keepalive 16;
    }
    
    server {
        ...
    
        location /http/ {
            proxy_pass http://http_backend;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            ...
        }
    }

修改好后，重新加载nginx,再次使用cmd命令查看端口号占用情况，结果如下：

![20200515180700400.png][]

可以看到，原先TIME\_WAIT那些进程消失了，然后去启动tomcat,，去看日志，发现异常已经消失，也可以访问项目了。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoZW54aWFvbW8xNjg4_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200515172752786.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3NoZW54aWFvbW8xNjg4,size_16,color_FFFFFF,t_70
[https_my.oschina.net_haitaohu_blog_3043113]: https://my.oschina.net/haitaohu/blog/3043113
[proxy_http_version]: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_http_version
[http_nginx.org_en_docs_http_ngx_http_upstream_module.html]: http://nginx.org/en/docs/http/ngx_http_upstream_module.html
[20200515180700400.png]: https://img-blog.csdnimg.cn/20200515180700400.png