weblogic大量功能慢导致的问题

矫情吗；* 2022-06-02 03:48 839阅读 0赞

现场报节点不能使用。

\#\#\#\#<2018-1-2 上午09时41分49秒 CST> <Error> <WebLogicServer> <AQSCGLXT15> <FWMS\_23\_8004> <\[ACTIVE\] ExecuteThread: '155' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1514857309373> <BEA-000337> <\[STUCK\] ExecuteThread: '59' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "600" seconds working on the request "Http Request Information: weblogic.servlet.internal.ServletRequestImpl@1b831c3d\[POST /web/forinstance/instrument/add\]

", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:  
java.lang.Object.wait(Native Method)  
weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:685)  
weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:650)  
weblogic.transaction.internal.XAServerResourceInfo.end(XAServerResourceInfo.java:1305)  
weblogic.transaction.internal.XAServerResourceInfo.internalDelist(XAServerResourceInfo.java:424)  
weblogic.transaction.internal.XAServerResourceInfo.delist(XAServerResourceInfo.java:344)  
weblogic.transaction.internal.ServerTransactionImpl.delistAll(ServerTransactionImpl.java:1689)  
weblogic.transaction.internal.ServerTransactionImpl.delistAll(ServerTransactionImpl.java:1677)  
weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTransactionImpl.java:2243)  
weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:290)  
weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:250)  
org.glassfish.transaction.TransactionManagerImplCommon.commit(TransactionManagerImplCommon.java:535)  
org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1011)  
org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:755)  
org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724)  
org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:475)  
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:270)  
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)  
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)  
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)  
  
从jstack可以看到由于13号线程堵塞了接近400个线程，如下所示：  
"\[ACTIVE\] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007fe3cc008800 nid=0x15b03 runnable \[0x00007fe3e8b7f000\]  
   java.lang.Thread.State: RUNNABLE  
at java.net.SocketInputStream.socketRead0(Native Method)  
at java.net.SocketInputStream.read(SocketInputStream.java:152)  
at java.net.SocketInputStream.read(SocketInputStream.java:122)  
at oracle.net.ns.Packet.receive(Packet.java:300)  
at oracle.net.ns.DataPacket.receive(DataPacket.java:106)  
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)  
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)  
at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)  
at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)  
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)  
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)  
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)  
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)  
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)  
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)  
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)  
at oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1119)  
at oracle.jdbc.driver.OracleResultSetImpl.close\_or\_fetch\_from\_next(OracleResultSetImpl.java:375)  
\- locked <0x000000070113ad78> (a oracle.jdbc.driver.T4CConnection)  
at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:279)  
\- locked <0x000000070113ad78> (a oracle.jdbc.driver.T4CConnection)  
at weblogic.jdbc.wrapper.ResultSet\_oracle\_jdbc\_driver\_OracleResultSetImpl.next(Unknown Source)  
at gg.bpms.db.EnginePostJdbcAdapterImpl.queryUsersByPostAndOrgCode(EnginePostJdbcAdapterImpl.java:321)  
at gg.bpms.db.EnginePostJdbcAdapterImpl.recycleGetUsersByPostIdAndOrgId(EnginePostJdbcAdapterImpl.java:285)  
  
  
"\[ACTIVE\] ExecuteThread: '157' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007fe3cc0e1800 nid=0x3878 waiting for monitor entry \[0x00007fe053e6d000\]  
   java.lang.Thread.State: BLOCKED (on object monitor)  
at oracle.jdbc.xa.client.OracleXAResource.end(OracleXAResource.java:342)  
\- waiting to lock <0x000000070113ad78> (a oracle.jdbc.driver.T4CConnection)  
at weblogic.jdbc.jta.DataSource.end(DataSource.java:953)  
at weblogic.transaction.internal.XAServerResourceInfo.end(XAServerResourceInfo.java:1306)  
at weblogic.transaction.internal.XAServerResourceInfo.internalDelist(XAServerResourceInfo.java:424)  
at weblogic.transaction.internal.XAServerResourceInfo.delist(XAServerResourceInfo.java:344)  
at weblogic.transaction.internal.ServerTransactionImpl.delistAll(ServerTransactionImpl.java:1689)  
at weblogic.transaction.internal.ServerTransactionImpl.localRollback(ServerTransactionImpl.java:2082)  
at weblogic.transaction.internal.ServerTransactionImpl.globalRetryRollback(ServerTransactionImpl.java:3238)  
at weblogic.transaction.internal.ServerTransactionImpl.access$100(ServerTransactionImpl.java:59)  
at weblogic.transaction.internal.ServerTransactionImpl$1.run(ServerTransactionImpl.java:3480)  
at weblogic.transaction.internal.PlatformHelperImpl$3.run(PlatformHelperImpl.java:1052)  
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:312)  
at weblogic.work.ExecuteThread.run(ExecuteThread.java:264)  
  
总结：这个功能查询比较慢，执行一次大概在60s，当出现大量这样的请求时，weblogic就出现过载了。打一个基线：正常的weblogic是1一百多个，出问题的时候，线程多大405个（不算GC的线程）。