诊断weblogic告警必须要有jstack

Dear 丶 2022-06-02 10:05 382阅读 0赞

上午10点收到GX现场weblogic告警，由于server日志是每天生成一份，所以看到的是今天的堵塞，连接池不够用了。

\#\#\#\#<2018-1-10 上午07时47分03秒 CST> <Error> <WebLogicServer> <zcyy71> <mm\_app01\_8001> <\[ACTIVE\] ExecuteThread: '110' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1515541623723> <BEA-000337> <\[STUCK\] ExecuteThread: '100' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "651" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@56e1f809\[

POST /web/gg/controlquality/common/getReadyData.sc HTTP/1.1  
X-Requested-With: XMLHttpRequest  
Accept: application/json, text/javascript, \*/\*; q=0.01  
Referer: http://10.100.81.120/web/gg/purchase/contracteventtrack/ContractEventTrackToSignNew.jsp?tSession=1515540971309  
Accept-Language: zh-cn  
Accept-Encoding: gzip, deflate  
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; BOIE9;ZHCN)  
Content-Length: 0  
Connection: Keep-Alive  
Cache-Control: no-cache  
Cookie: DWRSESSIONID=nxVxEsqGULlin2EyUpDZ2NBht4m; JSESSIONIDMINI=vv70hVRMJyQJmMh29fFJBQGWSPC2GF5LhZCkDnvDGZXJ49mSBVj1!-1775855159; ComtopSessionSID=<SNAID>mJ1xhVRfFMGbGkwHqfvrLwwM7125Q4mGTFDt0VV81hJyF50Ll2JJ!-2095660264!1515540863114</SNAID>; cs\_cStorage\_=1; JSESSIONID=HJTdRl9KgW1m34fvJJ-FvpiQUAs02L2XUTFqCkTMxmTcinSf-KzC!-189191805  
\]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:  
java.lang.Object.wait(Native Method)  
weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:643)  
weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1230)  
weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1164)  
weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:285)  
weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:522)  
weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:449)  
weblogic.jdbc.jts.Driver.createLocalConnection(Driver.java:349)  
weblogic.jdbc.jts.Driver.connect(Driver.java:173)  
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:364)  
org.apache.ibatis.transaction.managed.ManagedTransaction.openConnection(ManagedTransaction.java:84)  
org.apache.ibatis.transaction.managed.ManagedTransaction.getConnection(ManagedTransaction.java:58)  
org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:271)  
org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:69)  
org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:56)  
org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:259)  
org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:132)  
org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:105)

org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:81)

随手做了一份jstack，发现了有一个stuck的进程,持有了锁，堵塞了几个进程：

"\[STUCK\] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f54f8008800 nid=0x628b runnable \[0x00007f53cdf89000\]  
   java.lang.Thread.State: RUNNABLE  
at java.net.SocketInputStream.socketRead0(Native Method)  
at java.net.SocketInputStream.read(SocketInputStream.java:129)  
at oracle.net.ns.Packet.receive(Packet.java:293)  
at oracle.net.ns.DataPacket.receive(DataPacket.java:104)  
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)  
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)  
at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)  
at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)  
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)  
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)  
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1136)  
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1113)  
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)  
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)  
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)  
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:204)  
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1006)  
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1315)  
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576)  
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3677)  
\- locked <0x000000074602ea10> (a oracle.jdbc.driver.T4CConnection)  
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4714)  
\- locked <0x000000074602ea10> (a oracle.jdbc.driver.T4CConnection)  
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1374)  
at weblogic.jdbc.wrapper.PreparedStatement.execute(PreparedStatement.java:99)  
at org.apache.ibatis.executor.statement.CallableStatementHandler.update(CallableStatementHandler.java:45)  
at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:66)  
at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:45)  
at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:100)  
at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:75)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)  
at java.lang.reflect.Method.invoke(Method.java:597)  
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:59)  
at sun.proxy.$Proxy1263.update(Unknown Source)  
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:148)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)  
at java.lang.reflect.Method.invoke(Method.java:597)  
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:358)  
at sun.proxy.$Proxy1203.update(Unknown Source)  
at org.mybatis.spring.SqlSessionTemplate.update(SqlSessionTemplate.java:254)  
at com.gg.common.dao.CommonDAO.execUpdate(CommonDAO.java:220)  
at com.gg.material.controlquality.workflow.arrivalsampling.appservice.impl.CqWorkflowASAppService.postASDataToCSG(CqWorkflowASAppService.java:406)

根据jstack的结果分析功能，到今天的server日志中找不到对于的功能，我推测是昨天产生的，于是翻看了昨天的server日志，果然找到了，分析功能，是一个跨区域的dblink hang住了。

这个给我反思：以前分析告警比较少看jstack，就看server日志，往往发现很多stuck，不管三七二十一，都发给开发组，这种无疑增加了开发的负担。现在结合jstack的结果，可以非常方便的看出谁是始作俑者。