rocketmq API 生产者与消费者，springboot集成RocketMQ

前言

       rocketmq在Linux上搭建好了，现在说说rocketmq的默认producer与consumer方式。

1. rocketmq设计

我画了一张架构图

rocketmq的每条队列是顺序的，跟kafka的partition很相似；rocketmq默认通过随机正整数+1取模方式来选取队列的。rocketmq通过MessageQueueSelector保证消息的顺序发收，默认通过hash对队列数取模来实现消息发送到某条队列上，实现消息的顺序发送。

但如果任意一台broker异常，如连接中断，宕机，当Broker恢复前（主备切换或者broker重启等），因为队列总数改变，哈希取模路由的队列就会不同，就会造成短暂顺序不一致。

除了严格的顺序要求，一般使用默认方式就可以保证顺序，严格顺序会造成broker异常的短暂不可用。

rocketmq的每个组仅有一个节点消费消息。

下面说说默认实现方式

2. demo & pom

依赖client

<dependencies>
        <dependency>
            <groupId>org.apache.rocketmq</groupId>
            <artifactId>rocketmq-client</artifactId>
            <version>4.5.2</version>
        </dependency>

    </dependencies>

3. Producer

package com.feng.rocketmq.base;

import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.remoting.common.RemotingHelper;
import org.apache.rocketmq.remoting.exception.RemotingException;

import java.io.UnsupportedEncodingException;

public class Producer {

    public static void main(String[] args) throws MQClientException, RemotingException, InterruptedException, MQBrokerException, UnsupportedEncodingException {
        //instance
        DefaultMQProducer producer = new DefaultMQProducer();
        //group, for producer load balance
        producer.setProducerGroup("demo-producer-group");
        //namesrvAddr,cluster nameserver with ; spit
        producer.setNamesrvAddr("127.0.0.1:9876");
        //start
        producer.start();
        // send msg
        int num = 20;
        for (int i = 0; i < num; i++) {
            //message，topic,tags is a mark in consumer,keys is a mark for message query
            Message message = new Message("demoTopic","tags-1", "instanceKeys", ("I`m a " + i + " rocket mq msg!").getBytes(RemotingHelper.DEFAULT_CHARSET));
            SendResult result = producer.send(message, 1000);
            System.out.println("send result is\t" + result);
        }
        //close, can use for rocket mq switch
        producer.shutdown();
    }
}

访问console

查看status

每个topic默认创建4个队列，即上面的queueId字段

4. consumer

consumer分为推和拉模式。推就是回调通知，监听器，来了消息就通知；拉是自己写代码定时拉取一定数量的消息。消费者需要自己写消费策略，需要考虑重复消费，消费业务考虑幂等性；需要处理不能消费的消息，防止消息堆积。

推：

缺点：消费过慢，不能处理的消息会造成消息堆积；无效的消息可能会反复推送

优点：消费及时，消息来了，监听器触发

拉：

缺点：需要大量的创建长连接，拉取的频率，拉取的数目需要严格考虑，是否使用长轮询

比如：消费者拉取失败,并不return，把连接挂起wait, 等待一段时间继续拉取，直到成功。

优点：不用考虑消息堆积；无效消息offset指向后即可处理

推的代码：

package com.feng.rocketmq.base;

import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.remoting.common.RemotingHelper;

import java.util.List;

public class PushConsumer {
    public static void main(String[] args) throws MQClientException {
        //instance
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer();
        //group
        consumer.setConsumerGroup("demo-consumer-group");
        //setNamesrvAddr，cluster with ; spit
        consumer.setNamesrvAddr("127.0.0.1:9876");

        //consumer offset
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);

        //subscribe, the subExpression is tags in message send
        //subscribe topic store in map
        consumer.subscribe("demoTopic", "tags-1");
        //can subscribe more
        //consumer.subscribe("demoTopic2", "*");
        //or use setSubscription, method is deprecated
        //consumer.setSubscription();

        //batch consumer max message limit
        consumer.setConsumeMessageBatchMaxSize(1000);
        //min thread
        consumer.setConsumeThreadMin(10);

        //listener
        consumer.registerMessageListener((MessageListenerConcurrently) (list, consumeConcurrentlyContext) -> {
            try {
                for (MessageExt messageExt : list) {
                    if (messageExt.getReconsumeTimes() > 1){
                        continue;
                    }
                    String topic = messageExt.getTopic();
                    int queueId = messageExt.getQueueId();
                    String message = new String(messageExt.getBody(), RemotingHelper.DEFAULT_CHARSET);
                    System.out.println("the topic: " + topic + "\tqueueId:" + queueId + "\t body：" + message);
                }
            } catch (Exception e) {
                //retry
                return ConsumeConcurrentlyStatus.RECONSUME_LATER;
            }
            return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
        });
        consumer.start();

        //consumer.shutdown();
    }
}

订阅topic存入map

消费成功

拉的代码：

package com.feng.rocketmq.base;

import org.apache.rocketmq.client.consumer.DefaultMQPullConsumer;
import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.PullResult;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
import org.apache.rocketmq.remoting.common.RemotingHelper;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

public class PullConsumer {
    //record offset use key topic_queueId
    private static final Map<String, Long> offsetMap = new ConcurrentHashMap<>();

    public static void main(String[] args) throws MQClientException, InterruptedException {

        //instance
        DefaultMQPullConsumer consumer = new DefaultMQPullConsumer();
        //group
        consumer.setConsumerGroup("demo-consumer-group");
        //namesrv
        consumer.setNamesrvAddr("127.0.0.1:9876");
        //cluster model
        consumer.setMessageModel(MessageModel.CLUSTERING);

        consumer.start();
        //pull all mq with cluster or broadcast
        Set<MessageQueue> mqs = null;

        switch (consumer.getMessageModel()) {
            case BROADCASTING:
                mqs = consumer.fetchSubscribeMessageQueues("demoTopic");
                break;
            case CLUSTERING:
                //set topic load balance
                consumer.registerMessageQueueListener("demoTopic", null);
                //cluster, each node get some mq
                mqs = consumer.fetchMessageQueuesInBalance("demoTopic");
                if (mqs == null || mqs.isEmpty()) {
                    while (mqs == null || mqs.isEmpty()){
                        Thread.sleep(1000l);
                        mqs = consumer.fetchMessageQueuesInBalance("demoTopic");
                    }
                }
                break;
        }

        ScheduledThreadPoolExecutor scheduledThreadPoolExecutor = new ScheduledThreadPoolExecutor(4);
        for (MessageQueue mq : mqs) {
            scheduledThreadPoolExecutor.scheduleAtFixedRate(() -> {
                try {
                    PullResult pullResult = consumer.pullBlockIfNotFound(mq, "tags-1", offerMQOffset(mq, consumer.fetchConsumeOffset(mq, true)), 10);
                    System.out.println(pullResult);
                    switch (pullResult.getPullStatus()) {
                        case FOUND:
                            List<MessageExt> messageExtList = pullResult.getMsgFoundList();
                            //can async deal
                            for (MessageExt m : messageExtList) {
                                System.out.println(new String(m.getBody()));
                            }
                            saveMQOffset(mq, pullResult.getNextBeginOffset());
                            break;
                        case NO_MATCHED_MSG:
                        case NO_NEW_MSG:
                        case OFFSET_ILLEGAL:
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }, 5l, 5l, TimeUnit.SECONDS);
            System.out.println("topic & queueId : \t"+mq.getTopic()+"_"+mq.getQueueId());
        }
        //consumer.shutdown();
    }

    private static void saveMQOffset(MessageQueue mq, long offset) {
        offsetMap.put(mq.getTopic() + "_" + mq.getQueueId(), offset);
    }

    private static long offerMQOffset(MessageQueue mq, long defaultOffset) {
        Long offset = offsetMap.get(mq.getTopic() + "_" + mq.getQueueId());
        if (defaultOffset < 0) {
            defaultOffset = 0;
        }
        return offset == null ? defaultOffset : offset;
    }
}

这里非常关键 

consumer.registerMessageQueueListener("demoTopic", null);

源码可以看到将topic注册进loadbalance列表中了，笔者最开始各种load balance拉取都拉取不到message queue；原因是没有负责均衡。

另外在拉取的messageExtList，一般是要异步线程池处理的，提高并发量。

rocketmq官方提供

MQPullConsumerScheduleService 

用于支持定时pull 消息，相当于封装好了拉取逻辑，推荐使用，不用自己写一堆代码，下一章写一个demo，分析原理。

5. spring-boot集成rocketmq

在spring-boot框架中集成rocketmq非常简单，将上面的代码转变成springboot方式即可

5.1 配置属性

package com.feng.springboot.rocketmq.config;

import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties(prefix = "org.rocketmq")
@Data
public class RocketMQProperties {

    private String nameSrvs;
    private String producerGroup;
}

application.properties文件，yml同理

org.rocketmq.nameSrvs=127.0.0.1:9876
org.rocketmq.producerGroup=producer_demo_group

5.2 配置bean

package com.feng.springboot.rocketmq.config;

import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.client.producer.MQProducer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@EnableConfigurationProperties(RocketMQProperties.class)
@Configuration
public class RocketMQInitConfig {

    @Autowired
    private RocketMQProperties rocketMQProperties;

    @Bean
    public MQProducer getMQProducer() throws MQClientException {
        DefaultMQProducer mqProducer = new DefaultMQProducer();
        mqProducer.setNamesrvAddr(rocketMQProperties.getNameSrvs());
        mqProducer.setProducerGroup(rocketMQProperties.getProducerGroup());
        mqProducer.setSendMsgTimeout(1000);
        mqProducer.setVipChannelEnabled(false);
        //set other properties
        //............

        mqProducer.start();

        return mqProducer;
    }
}

好了，集成进来了，可以使用了

@SpringBootApplication
@RestController
public class RocketMQMain {

    public static void main(String[] args) {
        SpringApplication.run(RocketMQMain.class, args);
    }

    @Autowired
    private MQProducer producer;

    @RequestMapping(value = "messages", method = RequestMethod.GET)
    public Map<String, String> sendMsg() throws InterruptedException, RemotingException, MQClientException, MQBrokerException, UnsupportedEncodingException {
        Map<String, String> map = new HashMap<>(2);

        // send msg
        int num = 20;
        for (int i = 0; i < num; i++) {
            //message，topic,tags is a mark in consumer,keys is a mark for message query
            Message message = new Message("demoTopic","tags-1", "instanceKeys", ("I`m a " + i + " rocket mq msg!").getBytes(RemotingHelper.DEFAULT_CHARSET));
            SendResult result = producer.send(message, 1000);
            System.out.println("send result is\t" + result);
        }

        map.put("isSend", "OK");
        return map;
    }
}

运行后访问http://localhost:8080/messages

控制台

说明集成成功，同理集成consumer

    @Autowired
    private RocketMQDemoListener rocketMQDemoListener;

    @Bean
    public MQConsumer getMQConsumer() throws MQClientException {
        //instance
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer();
        //group
        consumer.setConsumerGroup(rocketMQProperties.getConsumerGroup());
        //setNamesrvAddr，cluster with ; spit
        consumer.setNamesrvAddr(rocketMQProperties.getNameSrvs());

        //consumer offset
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);

        //subscribe, the subExpression is tags in message send
        //subscribe topic store in map
        consumer.subscribe("demoTopic", "tags-1");
        //can subscribe more
        //consumer.subscribe("demoTopic2", "*");
        //or use setSubscription, method is deprecated
        //consumer.setSubscription();

        //batch consumer max message limit
        consumer.setConsumeMessageBatchMaxSize(1000);
        //min thread
        consumer.setConsumeThreadMin(10);
        consumer.registerMessageListener(rocketMQDemoListener);


        consumer.start();

        return consumer;
    }

listener逻辑，在自定义的listener里面可以做一个代理层，埋下一个钩子，方便以后业务处理。

@Component
public class RocketMQDemoListener implements MessageListenerConcurrently {

    @Override
    public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
        try {
            for (MessageExt messageExt : msgs) {
                if (messageExt.getReconsumeTimes() > 1){
                    continue;
                }
                String topic = messageExt.getTopic();
                int queueId = messageExt.getQueueId();
                String message = new String(messageExt.getBody(), RemotingHelper.DEFAULT_CHARSET);
                System.out.println("the topic: " + topic + "\tqueueId:" + queueId + "\t body：" + message);
            }
        } catch (Exception e) {
            //retry
            return ConsumeConcurrentlyStatus.RECONSUME_LATER;
        }
        return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
    }
}

将RocketMQInitConfig加入springboot的EnableAutoConfiguration即可做成rocketmq插件starter

 在src/main/resource目录下创建META-INF目录，然后创建文件spring.factories：

org.springframework.boot.autoconfigure.EnableAutoConfiguration=自己写的config类（全路径）

总结

       rocketmq 其实生产与消费很简单，复杂的地方在group的loadbalance，消息在broker上的分配。rocketmq有严格执行顺序的生产消费方式，并支持分布式事务，由消息传递分布式各节点的事务情况，生产者根据结果中心化处理事务情况。