记一次由于Druid连接池版本问题导致的系统崩溃

报错信息如下：

ERROR [com.alibaba.druid.pool.DruidDataSource] - recyle error
java.sql.SQLRecoverableException: 关闭的连接
    at oracle.jdbc.driver.PhysicalConnection.setAutoCommit(PhysicalConnection.java:2232)
    at com.alibaba.druid.filter.FilterChainImpl.connection_setAutoCommit(FilterChainImpl.java:553)
    at com.alibaba.druid.filter.FilterAdapter.connection_setAutoCommit(FilterAdapter.java:984)
    at com.alibaba.druid.filter.FilterChainImpl.connection_setAutoCommit(FilterChainImpl.java:549)
    at com.alibaba.druid.proxy.jdbc.ConnectionProxyImpl.setAutoCommit(ConnectionProxyImpl.java:430)
    at com.alibaba.druid.pool.DruidConnectionHolder.reset(DruidConnectionHolder.java:246)
    at com.alibaba.druid.pool.DruidDataSource.recycle(DruidDataSource.java:1301)
    at com.alibaba.druid.pool.DruidPooledConnection.recycle(DruidPooledConnection.java:305)
    at com.alibaba.druid.filter.FilterChainImpl.dataSource_recycle(FilterChainImpl.java:4534)
    at com.alibaba.druid.filter.stat.StatFilter.dataSource_releaseConnection(StatFilter.java:647)
    at com.alibaba.druid.filter.FilterChainImpl.dataSource_recycle(FilterChainImpl.java:4530)
    at com.alibaba.druid.pool.DruidPooledConnection.close(DruidPooledConnection.java:253)
    at org.springframework.jdbc.datasource.DataSourceUtils.doCloseConnection(DataSourceUtils.java:341)
    at org.springframework.jdbc.datasource.DataSourceUtils.doReleaseConnection(DataSourceUtils.java:328)
    at org.springframework.jdbc.datasource.DataSourceUtils.releaseConnection(DataSourceUtils.java:294)
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doCleanupAfterCompletion(DataSourceTransactionManager.java:329)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:1012)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:879)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:826)
    at org.springframework.transaction.support.TransactionTemplate.rollbackOnException(TransactionTemplate.java:164)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137)
    at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)
    at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
    at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
    at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
    at org.activiti.engine.impl.jobexecutor.AcquireJobsRunnableImpl.run(AcquireJobsRunnableImpl.java:54)
    at java.lang.Thread.run(Unknown Source)

Caused by: oracle.net.ns.NetException: Socket read timed out
    at oracle.net.ns.Packet.receive(Packet.java:350)
    at oracle.net.ns.DataPacket.receive(DataPacket.java:105)
    at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:249)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:171)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:89)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
    at oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429)
    at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)
    at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
    at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
    at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225)
    at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:53)
    at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:943)
    ... 36 more
2020-11-04 06:36:21,853 [Thread-7] ERROR [org.activiti.engine.impl.jobexecutor.AcquireJobsRunnableImpl] - exception during job acquisition: Could not roll back JDBC transaction; nested exception is java.sql.SQLRecoverableException: 关闭的连接
org.springframework.transaction.TransactionSystemException: Could not roll back JDBC transaction; nested exception is java.sql.SQLRecoverableException: 关闭的连接
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:290)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:849)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:826)
    at org.springframework.transaction.support.TransactionTemplate.rollbackOnException(TransactionTemplate.java:164)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137)
    at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)
    at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
    at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
    at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
    at org.activiti.engine.impl.jobexecutor.AcquireJobsRunnableImpl.run(AcquireJobsRunnableImpl.java:54)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.sql.SQLRecoverableException: 关闭的连接
    at oracle.jdbc.driver.PhysicalConnection.getAutoCommit(PhysicalConnection.java:2254)
    at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:2424)
    at com.alibaba.druid.filter.FilterChainImpl.connection_rollback(FilterChainImpl.java:533)
    at com.alibaba.druid.filter.stat.StatFilter.connection_rollback(StatFilter.java:268)
    at com.alibaba.druid.filter.FilterChainImpl.connection_rollback(FilterChainImpl.java:529)
    at com.alibaba.druid.proxy.jdbc.ConnectionProxyImpl.rollback(ConnectionProxyImpl.java:400)
    at com.alibaba.druid.pool.DruidPooledConnection.rollback(DruidPooledConnection.java:757)
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:287)
    ... 10 more
2020-11-04 06:36:40,697 [Druid-ConnectionPool-Create-2106295893] ERROR [com.alibaba.druid.pool.DruidDataSource] - create connection error, 
java.sql.SQLRecoverableException: IO 错误: Connection reset
    at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:752)
    at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666)
    at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
    at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566)
    at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:148)
    at com.alibaba.druid.filter.stat.StatFilter.connection_connect(StatFilter.java:211)
    at com.alibaba.druid.filter.FilterChainImpl.connection_connect(FilterChainImpl.java:142)
    at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1410)
    at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1464)
    at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:1969)
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(Unknown Source)
    at java.net.SocketInputStream.read(Unknown Source)
    at oracle.net.ns.Packet.receive(Packet.java:311)
    at oracle.net.ns.NSProtocolStream.negotiateConnection(NSProtocolStream.java:159)
    at oracle.net.ns.NSProtocol.connect(NSProtocol.java:264)
    at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1452)
    at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:496)
    ... 9 more

我是从第一个报错定位感觉是druid连接池，然后就照着百度去搜了一下，发现：

由于项目中使用的数据库连接池druid版本过低，源码中使用的是可中断式加锁方式，即如果在此之前线程被中断
那么druid连接池持有的数据库连接是不会被正常回收的，会直接捕获中断异常。
看了一下也没有发现所以然，然后去GitHub上的Druid官方开源处，看了一下历史问题修复，
发现这个是旧版本已知的一个Bug。链接：https://github.com/alibaba/druid/issues/785

将druid更新到相对新版本后翻到出错那里的源码，发现原来抛异常的地方已经改为了lock加锁方式。
 更新到最新版本后，上述问题得到解决。

数据库连接被中断的原因有很多，根据报错信息猜测本次项目中出现的问题有可能是上述druid连接池中无效的数据库连接未被正常回收，
有客户端访问时（即用户操作APP时）使用的是无效的数据库连接，日志一直打印输出连接重置，超时等与druid相关的错误。

借鉴的博客如下：druid问题原因链接

druid官方github描述