在上一章节 jedis源码分析(六)-扩展哨兵监控多主节点连接池的中写了一个测试多个主节点的分片储存连接池的测试类ShardedJedisSentinelPoolUtil,之后经过反复测试发现在多线程并发情况下,运行一段时候后发现会出现获取不到连接资源的问题,
异常信息如下:
redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:51)
at com.redis.demo.MultiShardedJedisSentinelPool.getResource(MultiShardedJedisSentinelPool.java:202)
at com.redis.util.ShardedJedisSentinelPoolUtil.set(ShardedJedisSentinelPoolUtil.java:75)
at com.redis.util.Thread1.run(ShardedJedisSentinelPoolUtil.java:206)
Caused by: java.util.NoSuchElementException: Unable to validate object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:495)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:362)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 3 more
六月 21, 2018 6:29:02 下午 com.redis.util.ShardedJedisSentinelPoolUtil set
信息: null
经过测试发现由于多线程操作连接池导致连接资源不够导致,所以在操作redis的方法上加上synchronized锁来解决资源竞争问题,代码如下:
public static synchronized String set(String key, String value) {
ShardedJedis jedis = null;
String result = null;
try {
if (jedis == null) {
jedis = pool.getResource();
}
}catch(Exception e) {
e.printStackTrace();
}
try {
result = jedis.set(key, value);
} catch (Exception e) {
if(jedis !=null) {
jedis.close();
}
logger.info(e.getMessage());
} finally {
if(jedis !=null) {
jedis.close();
}
}
return result;
}
修改之后测试正常,没有发现获取不到资源的情况,我这里的测试使用两台主节点,各自有两个从节点,有两台哨兵节点监控两台主节点,结构图如下:
但是如果有一台主节点挂掉之后,在redis哨兵sentinel没有选举出新的主节点重新创建连接池的过程中,依然会报错,也会出现获取不到连接资源的问题,具体错误信息如下:
线程id::37: 06:43:14: hello=OK
线程id::45: 06:43:14: hello=OK
线程id::161: 06:43:14: hello=OK
线程id::49: 06:43:14: hello=OK
线程id::53: 06:43:14: hello=null
redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:51)
at com.redis.demo.MultiShardedJedisSentinelPool.getResource(MultiShardedJedisSentinelPool.java:202)
at com.redis.util.ShardedJedisSentinelPoolUtil.set(ShardedJedisSentinelPoolUtil.java:75)
at com.redis.util.Thread1.run(ShardedJedisSentinelPoolUtil.java:206)
Caused by: java.util.NoSuchElementException: Unable to validate object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:495)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:362)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 3 more
六月 21, 2018 6:43:20 下午 com.redis.util.ShardedJedisSentinelPoolUtil set
信息: null
redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:51)
at com.redis.demo.MultiShardedJedisSentinelPool.getResource(MultiShardedJedisSentinelPool.java:202)
at com.redis.util.ShardedJedisSentinelPoolUtil.set(ShardedJedisSentinelPoolUtil.java:75)
at com.redis.util.Thread1.run(ShardedJedisSentinelPoolUtil.java:206)
Caused by: java.util.NoSuchElementException: Unable to validate object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:495)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:362)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 3 more
六月 21, 2018 6:43:22 下午 com.redis.util.ShardedJedisSentinelPoolUtil set
信息: null
线程id::57: 06:43:14: hello=null
redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:51)
at com.redis.demo.MultiShardedJedisSentinelPool.getResource(MultiShardedJedisSentinelPool.java:202)
at com.redis.util.ShardedJedisSentinelPoolUtil.set(ShardedJedisSentinelPoolUtil.java:75)
at com.redis.util.Thread1.run(ShardedJedisSentinelPoolUtil.java:206)
Caused by: java.util.NoSuchElementException: Unable to validate object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:495)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:362)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 3 more
六月 21, 2018 6:43:24 下午 com.redis.util.ShardedJedisSentinelPoolUtil set
信息: null
线程id::61: 06:43:14: hello=null
redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:51)
at com.redis.demo.MultiShardedJedisSentinelPool.getResource(MultiShardedJedisSentinelPool.java:202)
at com.redis.util.ShardedJedisSentinelPoolUtil.set(ShardedJedisSentinelPoolUtil.java:75)
at com.redis.util.Thread1.run(ShardedJedisSentinelPoolUtil.java:206)
Caused by: java.util.NoSuchElementException: Unable to validate object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:495)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:362)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 3 more
六月 21, 2018 6:43:26 下午 com.redis.util.ShardedJedisSentinelPoolUtil set
信息: null
线程id::65: 06:43:14: hello=null
六月 21, 2018 6:43:26 下午 com.redis.demo.MultiShardedJedisSentinelPool$MasterListener$1 onMessage
信息: Sentinel 127.0.0.1:4200 published: master1 127.0.0.1 4102 127.0.0.1 4103.
六月 21, 2018 6:43:26 下午 com.redis.demo.MultiShardedJedisSentinelPool$MasterListener$1 onMessage
信息: Sentinel masters before change map: {master1=127.0.0.1:4102, master2=127.0.0.1:4101}.
六月 21, 2018 6:43:26 下午 com.redis.demo.MultiShardedJedisSentinelPool$MasterListener$1 onMessage
信息: Sentinel masters before change: currentHostMasters[127.0.0.1:4102, 127.0.0.1:4101].
六月 21, 2018 6:43:26 下午 com.redis.demo.MultiShardedJedisSentinelPool$MasterListener$1 onMessage
信息: Sentinel masters after change: [127.0.0.1:4103, 127.0.0.1:4101].
六月 21, 2018 6:43:26 下午 com.redis.demo.MultiShardedJedisSentinelPool initPool
信息: 赋值前currentHostMasters的值是:[127.0.0.1:4102, 127.0.0.1:4101]
六月 21, 2018 6:43:26 下午 com.redis.demo.MultiShardedJedisSentinelPool initPool
信息: 赋值后currentHostMasters的值是:[127.0.0.1:4103, 127.0.0.1:4101]
六月 21, 2018 6:43:27 下午 com.redis.demo.MultiShardedJedisSentinelPool initPool
信息: Created JedisPool to master at [127.0.0.1:4103, 127.0.0.1:4101]
redis.clients.jedis.exceptions.JedisException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:51)
at com.redis.demo.MultiShardedJedisSentinelPool.getResource(MultiShardedJedisSentinelPool.java:202)
at com.redis.util.ShardedJedisSentinelPoolUtil.set(ShardedJedisSentinelPoolUtil.java:75)
at com.redis.util.Thread1.run(ShardedJedisSentinelPoolUtil.java:206)
Caused by: java.util.NoSuchElementException: Unable to validate object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:495)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:362)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 3 more
六月 21, 2018 6:43:28 下午 com.redis.util.ShardedJedisSentinelPoolUtil set
信息: null
线程id::180: 06:43:14: hello=null
线程id::179: 06:43:14: hello=OK
线程id::157: 06:43:14: hello=OK
线程id::178: 06:43:14: hello=OK
线程id::174: 06:43:14: hello=OK
线程id::175: 06:43:14: hello=OK
线程id::176: 06:43:14: hello=OK
,我的思路是想在redis重新选举出新的主节点的过程中让代码在获取连接资源等待一段时间,等sentinel选举出新的主节点重新创建连接池后,在重新操作redis。所以有一下进一步修改代码
public
static
synchronized
String set(String
key
, String
value
) {
ShardedJedis
jedis
=
null
;
String
result
=
null
;
try
{
if
(
jedis
==
null
) {
jedis
=
pool
.getResource();
}
}
catch
(JedisException | NoSuchElementException
e
) {
logger
.info(
"获取连接池中jedis对象异常,等待连接池恢复,线程等待10000毫秒"
);
try
{
Thread.
sleep
(10000L);
}
catch
(InterruptedException
e1
) {
e1
.printStackTrace();
}
if
(
jedis
==
null
) {
jedis
=
pool
.getResource();
}
}
try
{
result
=
jedis
.set(
key
,
value
);
}
catch
(Exception
e
) {
if
(
jedis
!=
null
) {
jedis
.close();
}
logger
.info(
e
.getMessage());
}
finally
{
if
(
jedis
!=
null
) {
jedis
.close();
}
}
return
result
;
}
运行代码测试,关闭一个主节点结果如下:
六月 21, 2018 6:52:22 下午 com.redis.demo.MultiShardedJedisSentinelPool initSentinels
信息: Trying to find master from available Sentinels...[127.0.0.1:4200, 127.0.0.1:4201]
六月 21, 2018 6:52:23 下午 com.redis.demo.MultiShardedJedisSentinelPool initSentinels
信息: Connecting to Sentinel 127.0.0.1:4200
六月 21, 2018 6:52:23 下午 com.redis.demo.MultiShardedJedisSentinelPool initSentinels
信息: Found Redis master at [127.0.0.1:4103]
六月 21, 2018 6:52:23 下午 com.redis.demo.MultiShardedJedisSentinelPool initSentinels
信息: Connecting to Sentinel 127.0.0.1:4201
六月 21, 2018 6:52:23 下午 com.redis.demo.MultiShardedJedisSentinelPool initSentinels
信息: Found Redis master at [127.0.0.1:4103, 127.0.0.1:4101]
六月 21, 2018 6:52:23 下午 com.redis.demo.MultiShardedJedisSentinelPool initSentinels
信息: Redis master running at [127.0.0.1:4103, 127.0.0.1:4101], starting Sentinel listeners...
六月 21, 2018 6:52:23 下午 com.redis.demo.MultiShardedJedisSentinelPool initPool
信息: 赋值前currentHostMasters的值是:null
六月 21, 2018 6:52:23 下午 com.redis.demo.MultiShardedJedisSentinelPool initPool
信息: 赋值后currentHostMasters的值是:[127.0.0.1:4103, 127.0.0.1:4101]
六月 21, 2018 6:52:23 下午 com.redis.demo.MultiShardedJedisSentinelPool initPool
信息: Created JedisPool to master at [127.0.0.1:4103, 127.0.0.1:4101]
六月 21, 2018 6:52:23 下午 com.redis.util.ShardedJedisSentinelPoolUtil <clinit>
信息: 多节点连接池创建成功
线程id::29: 06:52:23: hello=OK
线程id::302: 06:52:23: hello=OK
线程id::179: 06:52:23: hello=OK
................
线程id::4842: 06:52:24: hello=OK
线程id::4832: 06:52:24: hello=OK
六月 21, 2018 6:52:29 下午 com.redis.util.ShardedJedisSentinelPoolUtil set
信息: 获取连接池中jedis对象异常,等待连接池恢复,线程等待10000毫秒
六月 21, 2018 6:52:35 下午 com.redis.demo.MultiShardedJedisSentinelPool$MasterListener$1 onMessage
信息: Sentinel 127.0.0.1:4200 published: master1 127.0.0.1 4103 127.0.0.1 4102.
六月 21, 2018 6:52:35 下午 com.redis.demo.MultiShardedJedisSentinelPool$MasterListener$1 onMessage
信息: Sentinel masters before change map: {master1=127.0.0.1:4103, master2=127.0.0.1:4101}.
六月 21, 2018 6:52:35 下午 com.redis.demo.MultiShardedJedisSentinelPool$MasterListener$1 onMessage
信息: Sentinel masters before change: currentHostMasters[127.0.0.1:4103, 127.0.0.1:4101].
六月 21, 2018 6:52:35 下午 com.redis.demo.MultiShardedJedisSentinelPool$MasterListener$1 onMessage
信息: Sentinel masters after change: [127.0.0.1:4102, 127.0.0.1:4101].
六月 21, 2018 6:52:35 下午 com.redis.demo.MultiShardedJedisSentinelPool initPool
信息: 赋值前currentHostMasters的值是:[127.0.0.1:4103, 127.0.0.1:4101]
六月 21, 2018 6:52:35 下午 com.redis.demo.MultiShardedJedisSentinelPool initPool
信息: 赋值后currentHostMasters的值是:[127.0.0.1:4102, 127.0.0.1:4101]
六月 21, 2018 6:52:35 下午 com.redis.demo.MultiShardedJedisSentinelPool initPool
信息: Created JedisPool to master at [127.0.0.1:4102, 127.0.0.1:4101]
线程id::4864: 06:52:24: hello=OK
线程id::4872: 06:52:24: hello=OK
线程id::4862: 06:52:24: hello=OK
线程id::4836: 06:52:24: hello=OK
线程id::4852: 06:52:24: hello=OK
线程id::4856: 06:52:24: hello=OK
....................
结果没有异常发生,数据正常存入,可以解决在sentinel重新选举过程中获取不到连接资源导致异常发生,但是等待的具体时间不好界定,如果超过10000毫秒之后sentinel仍然没有选举出新的主节点重新初始化连接池,这里获取不到连接资源的问题仍然存在。
不知道有没碰到类似问题的同学是怎么解决这个问题的。
本文探讨了在多主节点的Redis环境中使用哨兵监控时遇到的连接池资源不足问题。通过添加同步锁解决了多线程并发导致的资源争用,并提出了一种等待策略以应对主节点故障期间的资源获取问题。

2万+

被折叠的 条评论
为什么被折叠?



