容器化redis-cluster使用（二）java客戶端刷新cluster topology問題

          前一篇提到了容器ip變化後集羣自發現的問題，現在接着講這個問題引申出的另兩個問題：

1.雖然集羣ip問題解決了，但java client還是連接報錯，似乎連的是老的ip地址。

2.當key所在的master掛了，slave切換到master後，java客戶端卻一直嘗試重連，直到超時，並沒有隨着master的切換去連主拿key。以下代碼和例子爲環境模擬。

集羣信息：

192.168.27.128:7000> cluster nodes
ffd1619ad6b8f8db6d5764ab6739f4b03661cbe3 192.168.27.128:7001@17001 master - 0 1568640842000 70 connected 11404 11589-16383
8ca3773af76228ac6a864ef660a7e701d207aa3c 192.168.27.129:7002@17002 master - 0 1568640844271 35 connected 666-5460
a92e5ab3de251dd49a25c313380fb2ce8a81513c 192.168.27.128:7003@17003 slave a2d327c3be89c685524fe12e26b8e7227b500e20 0 1568640842000 37 connected
8c81e214847a2e0ed609a432f15f8ef048cff047 192.168.27.129:7001@17001 slave ffd1619ad6b8f8db6d5764ab6739f4b03661cbe3 0 1568640842355 70 connected
7e0686f3bec7c103a794aa6402c7b9ad0f6c632e 192.168.27.128:7000@17000 myself,slave 8ca3773af76228ac6a864ef660a7e701d207aa3c 0 1568640842000 1 connected
88ecdc0b150b3cbfdc4d791204bfd0d58cabd2ff 192.168.27.129:7003@17003 master - 0 1568640842264 8 connected 0-665 5461-6127 10923-11403 11405-11588
a2d327c3be89c685524fe12e26b8e7227b500e20 192.168.27.129:7000@17000 master - 0 1568640843260 37 connected 6128-10922

模擬客戶端程序，循環插入key和獲取key的值：

@Test
public void redisClusterTest() throws InterruptedException {
        ValueOperations<String,String> operations=redisTemplate.opsForValue();
        int i =0;
        while (true){         
                operations.set("mchf","mchf");        
                System.out.println(operations.get("mchf"));
                i++;
                System.out.println(i);
            }catch (Exception e){
                e.printStackTrace();
            }

            Thread.sleep(2000);
        }

獲取key所在的slot和節點：

192.168.27.128:7000> get mchf
(error) MOVED 11404 192.168.27.128:7001

我們kill掉192.168.27.128:7001這個redis實例，看看客戶端會發生什麼：

3
mchf
4
2019-09-16 21:37:42.393  INFO 3612 --- [xecutorLoop-1-3] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was /192.168.27.128:7001
2019-09-16 21:37:44.400  WARN 3612 --- [ioEventLoop-6-4] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /192.168.27.128:7001
2019-09-16 21:37:48.693  INFO 3612 --- [xecutorLoop-1-5] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was 192.168.27.128:7001
2019-09-16 21:37:50.698  WARN 3612 --- [ioEventLoop-6-1] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /192.168.27.128:7001
2019-09-16 21:37:54.992  INFO 3612 --- [xecutorLoop-1-2] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was 192.168.27.128:7001
2019-09-16 21:37:57.000  WARN 3612 --- [ioEventLoop-6-4] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /192.168.27.128:7001
2019-09-16 21:38:02.091  INFO 3612 --- [xecutorLoop-1-5] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was 192.168.27.128:7001
2019-09-16 21:38:04.096  WARN 3612 --- [ioEventLoop-6-1] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /192.168.27.128:7001
2019-09-16 21:38:09.292  INFO 3612 --- [xecutorLoop-1-3] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was 192.168.27.128:7001
2019-09-16 21:38:11.298  WARN 3612 --- [ioEventLoop-6-3] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /192.168.27.128:7001
org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: Command timed out after 30 second(s)

這邊一直嘗試重連，超時拋出異常。這樣當slave提升到master後，客戶端還在嘗試連接原來的master，而原來的master已經被kill掉了，客戶端並沒有去連接新的master。

我們是基於springboot2.0做的開發，而springboot2.0中redis的連接池用的是lettuce，查文檔後發現，需要開啓自適應或定期刷新ClusterTopology，以下爲修改後的代碼：

  @Autowired
    private RedisProperties redisProperties;


    @Bean
    public LettuceConnectionFactory redisConnectionFactory() {

        RedisClusterConfiguration redisClusterConfiguration = new RedisClusterConfiguration(redisProperties.getCluster().getNodes());


        GenericObjectPoolConfig genericObjectPoolConfig = new GenericObjectPoolConfig();
        //支持自適應集羣拓撲刷新和靜態刷新源
        ClusterTopologyRefreshOptions clusterTopologyRefreshOptions =  ClusterTopologyRefreshOptions.builder()
               //.enablePeriodicRefresh(Duration.ofSeconds(5))
                .enableAllAdaptiveRefreshTriggers()
                .adaptiveRefreshTriggersTimeout(Duration.ofSeconds(10))
                //.enablePeriodicRefresh(Duration.ofSeconds(10))
                .build();

        ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder().timeoutOptions(TimeoutOptions.enabled(Duration.ofSeconds(30)))  超時修改爲30秒
                //.autoReconnect(false)  是否自動重連
                //.pingBeforeActivateConnection(Boolean.TRUE) 
                //.cancelCommandsOnReconnectFailure(Boolean.TRUE)
                //.disconnectedBehavior(ClientOptions.DisconnectedBehavior.REJECT_COMMANDS)
                .topologyRefreshOptions(clusterTopologyRefreshOptions).build();

        LettuceClientConfiguration lettuceClientConfiguration = LettucePoolingClientConfiguration.builder()
                .poolConfig(genericObjectPoolConfig)
                //.readFrom(ReadFrom.NEAREST)
                .clientOptions(clusterClientOptions).build();

        return new LettuceConnectionFactory(redisClusterConfiguration, lettuceClientConfiguration);
    }


    @Bean
    public RedisTemplate<String,String> redisTemplate(RedisConnectionFactory redisConnectionfactory){
        RedisTemplate<String,String> redisTemplate=new RedisTemplate<>();
        redisTemplate.setConnectionFactory(redisConnectionfactory);
        redisTemplate.setKeySerializer(new StringRedisSerializer());
        redisTemplate.setValueSerializer(new StringRedisSerializer());
        redisTemplate.afterPropertiesSet();
        return redisTemplate;
    }

這邊需要注意兩點：

1.默認客戶端重連超時時間爲60秒，這邊修改到了30秒，redis集羣認爲master宕機slave切換到master的默認時間爲15秒，可以在redis.conf中修改

2.啓用adaptive cluster topology view當某個節點無法連接上之後，會生效，並且會和集羣中所有的節點每隔一段時間嘗試建立ESTABLISH連接，時間間隔取決於代碼中的刷新頻率，而啓用periodic cluster topology不管集羣有沒有發生改變，都會每隔一段時間和集羣中所有節點嘗試建立ESTABLISH連接。集羣使用客戶端比較多的時候需要注意這點。

代碼修改後可以從變化後的master拿到key：

前面發生了主從切換所以現在192.168.27.129:7001是主，我們kill掉，查看客戶端變化

192.168.27.128:7000> get mchf
(error) MOVED 11404 192.168.27.129:7001

Caused by: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 30 second(s)
	at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
	at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
	at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
	at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
	at com.sun.proxy.$Proxy85.set(Unknown Source)
	at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.set(LettuceStringCommands.java:146)
	... 39 more
mchf
5
2019-09-16 22:07:05.372  INFO 13520 --- [xecutorLoop-1-1] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was 192.168.27.129:7001
mchf
6

當配置自適應或定期刷新cluster topology後，上面兩個問題得以解決。

參考：https://github.com/lettuce-io/lettuce-core/wiki/Redis-Cluster