rocketmq API 生產者與消費者，springboot集成RocketMQ

前言

rocketmq在Linux上搭建好了，現在說說rocketmq的默認producer與consumer方式。

1. rocketmq設計

我畫了一張架構圖

rocketmq的每條隊列是順序的，跟kafka的partition很相似；rocketmq默認通過隨機正整數+1取模方式來選取隊列的。rocketmq通過MessageQueueSelector保證消息的順序發收，默認通過hash對隊列數取模來實現消息發送到某條隊列上，實現消息的順序發送。

但如果任意一臺broker異常，如連接中斷，宕機，當Broker恢復前（主備切換或者broker重啓等），因爲隊列總數改變，哈希取模路由的隊列就會不同，就會造成短暫順序不一致。

除了嚴格的順序要求，一般使用默認方式就可以保證順序，嚴格順序會造成broker異常的短暫不可用。

rocketmq的每個組僅有一個節點消費消息。

下面說說默認實現方式

2. demo & pom

依賴client

<dependencies>
        <dependency>
            <groupId>org.apache.rocketmq</groupId>
            <artifactId>rocketmq-client</artifactId>
            <version>4.5.2</version>
        </dependency>

    </dependencies>

3. Producer

package com.feng.rocketmq.base;

import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.remoting.common.RemotingHelper;
import org.apache.rocketmq.remoting.exception.RemotingException;

import java.io.UnsupportedEncodingException;

public class Producer {

    public static void main(String[] args) throws MQClientException, RemotingException, InterruptedException, MQBrokerException, UnsupportedEncodingException {
        //instance
        DefaultMQProducer producer = new DefaultMQProducer();
        //group, for producer load balance
        producer.setProducerGroup("demo-producer-group");
        //namesrvAddr,cluster nameserver with ; spit
        producer.setNamesrvAddr("127.0.0.1:9876");
        //start
        producer.start();
        // send msg
        int num = 20;
        for (int i = 0; i < num; i++) {
            //message，topic,tags is a mark in consumer,keys is a mark for message query
            Message message = new Message("demoTopic","tags-1", "instanceKeys", ("I`m a " + i + " rocket mq msg!").getBytes(RemotingHelper.DEFAULT_CHARSET));
            SendResult result = producer.send(message, 1000);
            System.out.println("send result is\t" + result);
        }
        //close, can use for rocket mq switch
        producer.shutdown();
    }
}

訪問console

查看status

每個topic默認創建4個隊列，即上面的queueId字段

4. consumer

consumer分爲推和拉模式。推就是回調通知，監聽器，來了消息就通知；拉是自己寫代碼定時拉取一定數量的消息。消費者需要自己寫消費策略，需要考慮重複消費，消費業務考慮冪等性；需要處理不能消費的消息，防止消息堆積。

推：

缺點：消費過慢，不能處理的消息會造成消息堆積；無效的消息可能會反覆推送

優點：消費及時，消息來了，監聽器觸發

拉：

缺點：需要大量的創建長連接，拉取的頻率，拉取的數目需要嚴格考慮，是否使用長輪詢

比如：消費者拉取失敗,並不return，把連接掛起wait, 等待一段時間繼續拉取，直到成功。

優點：不用考慮消息堆積；無效消息offset指向後即可處理

推的代碼：

package com.feng.rocketmq.base;

import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.remoting.common.RemotingHelper;

import java.util.List;

public class PushConsumer {
    public static void main(String[] args) throws MQClientException {
        //instance
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer();
        //group
        consumer.setConsumerGroup("demo-consumer-group");
        //setNamesrvAddr，cluster with ; spit
        consumer.setNamesrvAddr("127.0.0.1:9876");

        //consumer offset
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);

        //subscribe, the subExpression is tags in message send
        //subscribe topic store in map
        consumer.subscribe("demoTopic", "tags-1");
        //can subscribe more
        //consumer.subscribe("demoTopic2", "*");
        //or use setSubscription, method is deprecated
        //consumer.setSubscription();

        //batch consumer max message limit
        consumer.setConsumeMessageBatchMaxSize(1000);
        //min thread
        consumer.setConsumeThreadMin(10);

        //listener
        consumer.registerMessageListener((MessageListenerConcurrently) (list, consumeConcurrentlyContext) -> {
            try {
                for (MessageExt messageExt : list) {
                    if (messageExt.getReconsumeTimes() > 1){
                        continue;
                    }
                    String topic = messageExt.getTopic();
                    int queueId = messageExt.getQueueId();
                    String message = new String(messageExt.getBody(), RemotingHelper.DEFAULT_CHARSET);
                    System.out.println("the topic: " + topic + "\tqueueId:" + queueId + "\t body：" + message);
                }
            } catch (Exception e) {
                //retry
                return ConsumeConcurrentlyStatus.RECONSUME_LATER;
            }
            return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
        });
        consumer.start();

        //consumer.shutdown();
    }
}

訂閱topic存入map

消費成功

拉的代碼：

package com.feng.rocketmq.base;

import org.apache.rocketmq.client.consumer.DefaultMQPullConsumer;
import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.PullResult;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
import org.apache.rocketmq.remoting.common.RemotingHelper;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

public class PullConsumer {
    //record offset use key topic_queueId
    private static final Map<String, Long> offsetMap = new ConcurrentHashMap<>();

    public static void main(String[] args) throws MQClientException, InterruptedException {

        //instance
        DefaultMQPullConsumer consumer = new DefaultMQPullConsumer();
        //group
        consumer.setConsumerGroup("demo-consumer-group");
        //namesrv
        consumer.setNamesrvAddr("127.0.0.1:9876");
        //cluster model
        consumer.setMessageModel(MessageModel.CLUSTERING);

        consumer.start();
        //pull all mq with cluster or broadcast
        Set<MessageQueue> mqs = null;

        switch (consumer.getMessageModel()) {
            case BROADCASTING:
                mqs = consumer.fetchSubscribeMessageQueues("demoTopic");
                break;
            case CLUSTERING:
                //set topic load balance
                consumer.registerMessageQueueListener("demoTopic", null);
                //cluster, each node get some mq
                mqs = consumer.fetchMessageQueuesInBalance("demoTopic");
                if (mqs == null || mqs.isEmpty()) {
                    while (mqs == null || mqs.isEmpty()){
                        Thread.sleep(1000l);
                        mqs = consumer.fetchMessageQueuesInBalance("demoTopic");
                    }
                }
                break;
        }

        ScheduledThreadPoolExecutor scheduledThreadPoolExecutor = new ScheduledThreadPoolExecutor(4);
        for (MessageQueue mq : mqs) {
            scheduledThreadPoolExecutor.scheduleAtFixedRate(() -> {
                try {
                    PullResult pullResult = consumer.pullBlockIfNotFound(mq, "tags-1", offerMQOffset(mq, consumer.fetchConsumeOffset(mq, true)), 10);
                    System.out.println(pullResult);
                    switch (pullResult.getPullStatus()) {
                        case FOUND:
                            List<MessageExt> messageExtList = pullResult.getMsgFoundList();
                            //can async deal
                            for (MessageExt m : messageExtList) {
                                System.out.println(new String(m.getBody()));
                            }
                            saveMQOffset(mq, pullResult.getNextBeginOffset());
                            break;
                        case NO_MATCHED_MSG:
                        case NO_NEW_MSG:
                        case OFFSET_ILLEGAL:
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }, 5l, 5l, TimeUnit.SECONDS);
            System.out.println("topic & queueId : \t"+mq.getTopic()+"_"+mq.getQueueId());
        }
        //consumer.shutdown();
    }

    private static void saveMQOffset(MessageQueue mq, long offset) {
        offsetMap.put(mq.getTopic() + "_" + mq.getQueueId(), offset);
    }

    private static long offerMQOffset(MessageQueue mq, long defaultOffset) {
        Long offset = offsetMap.get(mq.getTopic() + "_" + mq.getQueueId());
        if (defaultOffset < 0) {
            defaultOffset = 0;
        }
        return offset == null ? defaultOffset : offset;
    }
}

這裏非常關鍵

consumer.registerMessageQueueListener("demoTopic", null);

源碼可以看到將topic註冊進loadbalance列表中了，筆者最開始各種load balance拉取都拉取不到message queue；原因是沒有負責均衡。

另外在拉取的messageExtList，一般是要異步線程池處理的，提高併發量。

rocketmq官方提供

MQPullConsumerScheduleService

用於支持定時pull 消息，相當於封裝好了拉取邏輯，推薦使用，不用自己寫一堆代碼，下一章寫一個demo，分析原理。

5. spring-boot集成rocketmq

在spring-boot框架中集成rocketmq非常簡單，將上面的代碼轉變成springboot方式即可

5.1 配置屬性

package com.feng.springboot.rocketmq.config;

import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties(prefix = "org.rocketmq")
@Data
public class RocketMQProperties {

    private String nameSrvs;
    private String producerGroup;
}

application.properties文件，yml同理

org.rocketmq.nameSrvs=127.0.0.1:9876
org.rocketmq.producerGroup=producer_demo_group

5.2 配置bean

package com.feng.springboot.rocketmq.config;

import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.client.producer.MQProducer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@EnableConfigurationProperties(RocketMQProperties.class)
@Configuration
public class RocketMQInitConfig {

    @Autowired
    private RocketMQProperties rocketMQProperties;

    @Bean
    public MQProducer getMQProducer() throws MQClientException {
        DefaultMQProducer mqProducer = new DefaultMQProducer();
        mqProducer.setNamesrvAddr(rocketMQProperties.getNameSrvs());
        mqProducer.setProducerGroup(rocketMQProperties.getProducerGroup());
        mqProducer.setSendMsgTimeout(1000);
        mqProducer.setVipChannelEnabled(false);
        //set other properties
        //............

        mqProducer.start();

        return mqProducer;
    }
}

好了，集成進來了，可以使用了

@SpringBootApplication
@RestController
public class RocketMQMain {

    public static void main(String[] args) {
        SpringApplication.run(RocketMQMain.class, args);
    }

    @Autowired
    private MQProducer producer;

    @RequestMapping(value = "messages", method = RequestMethod.GET)
    public Map<String, String> sendMsg() throws InterruptedException, RemotingException, MQClientException, MQBrokerException, UnsupportedEncodingException {
        Map<String, String> map = new HashMap<>(2);

        // send msg
        int num = 20;
        for (int i = 0; i < num; i++) {
            //message，topic,tags is a mark in consumer,keys is a mark for message query
            Message message = new Message("demoTopic","tags-1", "instanceKeys", ("I`m a " + i + " rocket mq msg!").getBytes(RemotingHelper.DEFAULT_CHARSET));
            SendResult result = producer.send(message, 1000);
            System.out.println("send result is\t" + result);
        }

        map.put("isSend", "OK");
        return map;
    }
}

運行後訪問http://localhost:8080/messages

控制檯

說明集成成功，同理集成consumer

    @Autowired
    private RocketMQDemoListener rocketMQDemoListener;

    @Bean
    public MQConsumer getMQConsumer() throws MQClientException {
        //instance
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer();
        //group
        consumer.setConsumerGroup(rocketMQProperties.getConsumerGroup());
        //setNamesrvAddr，cluster with ; spit
        consumer.setNamesrvAddr(rocketMQProperties.getNameSrvs());

        //consumer offset
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);

        //subscribe, the subExpression is tags in message send
        //subscribe topic store in map
        consumer.subscribe("demoTopic", "tags-1");
        //can subscribe more
        //consumer.subscribe("demoTopic2", "*");
        //or use setSubscription, method is deprecated
        //consumer.setSubscription();

        //batch consumer max message limit
        consumer.setConsumeMessageBatchMaxSize(1000);
        //min thread
        consumer.setConsumeThreadMin(10);
        consumer.registerMessageListener(rocketMQDemoListener);


        consumer.start();

        return consumer;
    }

listener邏輯，在自定義的listener裏面可以做一個代理層，埋下一個鉤子，方便以後業務處理。

@Component
public class RocketMQDemoListener implements MessageListenerConcurrently {

    @Override
    public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
        try {
            for (MessageExt messageExt : msgs) {
                if (messageExt.getReconsumeTimes() > 1){
                    continue;
                }
                String topic = messageExt.getTopic();
                int queueId = messageExt.getQueueId();
                String message = new String(messageExt.getBody(), RemotingHelper.DEFAULT_CHARSET);
                System.out.println("the topic: " + topic + "\tqueueId:" + queueId + "\t body：" + message);
            }
        } catch (Exception e) {
            //retry
            return ConsumeConcurrentlyStatus.RECONSUME_LATER;
        }
        return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
    }
}

將RocketMQInitConfig加入springboot的EnableAutoConfiguration即可做成rocketmq插件starter

在src/main/resource目錄下創建META-INF目錄，然後創建文件spring.factories：

org.springframework.boot.autoconfigure.EnableAutoConfiguration=自己寫的config類（全路徑）

總結

rocketmq 其實生產與消費很簡單，複雜的地方在group的loadbalance，消息在broker上的分配。rocketmq有嚴格執行順序的生產消費方式，並支持分佈式事務，由消息傳遞分佈式各節點的事務情況，生產者根據結果中心化處理事務情況。

rocketmq API 生產者與消費者，springboot集成RocketMQ

comparator和comparable的使用

Java結束線程

rocketmq API 生產者與消費者，springboot集成RocketMQ

ArrayList與HashMap遍歷刪除元素，HashMap與ArrayList的clone體修改之間影響

InheritableThreadLocal父線程傳遞子線程線程安全

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結