可靠mq（三）——mq順序消費

問題

假設order.bp.fdd服務發出訂單創建mq，adc.nh.fdd接收到mq消息，就會插入一個訂單記錄；

假設order.bp.fdd服務發出訂單變更mq，adc.nh.fdd接收到mq消息，就會根據訂單id修改訂單信息；

如果order.bp.fdd服務在極短的時間內，發出創建和變更mq，那麼adc的兩臺服務，可能就會是一臺處理創建，一臺處理更新。

這種情況下，處理更新的那個adc服務就會因爲查詢不到訂單，導致更新操作執行不了。

方案選擇

順序消息，是指同一個消息組內的消息，消費順序跟發送的順序一致（重點：同一個分組，不是所有消息都是順序的）

要實現順序消息，需要依賴消費記錄，必須知道自己消費到了哪裏，才能判斷接收到的mq能不能處理！！！這個可以跟《mq只被消費一次》使用同一套消費記錄的查詢和保存代碼

我們使用rabbitMq發送mq，都是將消息發送到一個類型爲fanout的exchange，那些要監聽這個消息的服務，各自建一個隊列綁定到該exchange。即同一個消息，同時會被幾個服務消費，每個服務的消費速度和進度不一樣。所以消費記錄適合放在各個服務自己的業務庫中。

目前採用了兩種方案，不同點在於定時拉取消費失敗的順序消息時的邏輯有所不同。

方案一：

生產者

在生產者服務新建一張表sequence_message,用來保存發送的順序消息，生產者服務調用mq-platform-server提供的順序消息發送方法，在發送消息之前首先將消息保存到sequence_message 中，然後

獲取該消息的自增主鍵id和message_id,通過自增主鍵id,消息所在分組group_name ,生產者服務名稱application獲取到前置消息id ：previous_message_id，然後將消息的自增id，所在分組，服務名稱，

前置消息id 設置到消息頭中。

消費者

增加一個RemoteSequenceOperationsInterceptor類，利用RabbitListenerContainerBeanPostProcessor和SimpleRabbitListenerContainerFactoryBeanPostProcessor兩個後置處理類，將SequenceOperationsInterceptor設置到所有SimpleRabbitListenerContainer對象的adviceChain屬性，並且RemoteSequenceOperationsInterceptor的優先級最高。如果消息是順序消息並

且前置消息未消費，則將消息保存到mq-platform服務的rabbit_sequence_record表中，同時消費方需要配置一個強制消費時間（默認30分鐘）。

mq-platform-server

在mq-platform-server新增一張數據表rabbit_sequence_record來保存那些前置消息還未消費的消息，服務在啓動時會啓動一個工作線程實時的去這個表中拉取到達強制時間的記錄，將消息記錄的消息類型

由順序消息改爲普通消息進行強制消費。如果消費再次失敗，則保存到rabbit_consume_fail_record表中。

流程設計：

代碼實現

mq-platform-producer：

@Slf4j
public class ClientRabbitServiceImpl implements RabbitService {
    private SequenceManager sequenceManager;
    private RabbitTemplate rabbitTemplate;
    private Alarm alarm;
 
    public ClientRabbitServiceImpl(SequenceManager sequenceManager, RabbitTemplate rabbitTemplate,
        ObjectProvider<Alarm> alarmProvider) {
        this.sequenceManager = sequenceManager;
        this.rabbitTemplate = rabbitTemplate;
        if (Objects.nonNull(alarmProvider)) {
            this.alarm = alarmProvider.getIfAvailable();
        }
    }
 
    public ClientRabbitServiceImpl(SequenceManager sequenceManager, RabbitTemplate rabbitTemplate, Alarm alarm) {
        this.sequenceManager = sequenceManager;
        this.rabbitTemplate = rabbitTemplate;
        this.alarm = alarm;
    }
 
 
    @Async
    @Override
    public void send(List<RabbitProducer> producerList) {
        producerList.forEach(producer -> {
            try {
                send(producer);
            } catch (Throwable t) {
                log.error("fail to invoke rabbitTemplate to send message", t);
            }
        });
    }
 
    @Async
    @Override
    public void send(RabbitProducer producer) {
        try {
            Message message;
            if (Objects.nonNull(sequenceManager)
                && Objects.equals(producer.getType(), MessageTypeEnum.SEQUENCE.getValue())) {
                String previousMessageId = sequenceManager.getPreviousMessageId(producer.getMessageId(),
                    producer.getApplication());
                message = RabbitUtils.generateMessage(producer, previousMessageId);
            } else {
                message = RabbitUtils.generateMessage(producer);
            }
            CorrelationData correlationData = RabbitUtils.generateCorrelationData(producer);
            rabbitTemplate.send(producer.getExchange(), producer.getRoutingKey(), message, correlationData);
        } catch (Throwable throwable) {
            log.error("send message failed producer={}", producer.toString(), throwable);
            if (Objects.nonNull(alarm)) {
                alarm.failWhenProduce(JSON.toJSONString(producer), throwable);
            }
        }
    }
 
    @Override
    public String getVirtualHost() {
        return rabbitTemplate.getConnectionFactory().getVirtualHost();
    }
}

mq-platform-consumer:

/**
 * 順序消費攔截器基類
 *
 * @author chenxudong
 * @date 2020/02/21
 */
@Slf4j
public abstract class AbstractSequenceOperationsInterceptor extends AbstractConsumerOperationsInterceptor {
    protected ConsumeRecordManager consumeRecordManager;
    protected int consumeFailDelay;
    protected int faultTolerantTime;
 
    public AbstractSequenceOperationsInterceptor(
            ConsumeRecordManager consumeRecordManager,
            int consumeFailDelay,
            int faultTolerantTime
    ) {
        this.consumeRecordManager = consumeRecordManager;
        this.consumeFailDelay = consumeFailDelay;
        if (faultTolerantTime < 0) {
            this.faultTolerantTime = Integer.MAX_VALUE;
        } else {
            this.faultTolerantTime = faultTolerantTime;
        }
    }
 
    @Override
    public Object invoke(MethodInvocation methodInvocation) throws Throwable {
        Message message = getMessage(methodInvocation);
        if (Objects.isNull(message)
                || Objects.isNull(message.getMessageProperties())
                || CollectionUtils.isEmpty(message.getMessageProperties().getHeaders())) {
            return methodInvocation.proceed();
        }
        String application = RabbitUtils.getProduceApplication(message);
        String messageId = RabbitUtils.getMessageId(message);
        if (StringUtils.isEmpty(application) || StringUtils.isEmpty(messageId)) {
            return methodInvocation.proceed();
        }
        // 不是順序消息，直接執行
        if (!Objects.equals(RabbitUtils.getMessageType(message), MessageTypeEnum.SEQUENCE.getValue())) {
            return methodInvocation.proceed();
        }
        // 沒有分組內的上一個消息id，直接執行
        Object previousMessageIdObject =
                message.getMessageProperties().getHeaders().get(RabbitHeaderConstants.MESSAGE_PREVIOUS_ID_HEADER);
        if (Objects.isNull(previousMessageIdObject)
                || !(previousMessageIdObject instanceof String)
                || StringUtils.isEmpty(previousMessageIdObject)) {
            return methodInvocation.proceed();
        }
        String previousMessageId = (String) previousMessageIdObject;
        Date msgTimestamp = RabbitUtils.getTimestamp(message);
        // 判斷是否已經可以消費
        if (isCanConsume(application, messageId, previousMessageId, msgTimestamp)) {
            return consumeMessage(methodInvocation, message);
        }
        if (consumeFailDelay < 1) {
            stashMessage(message);
            return null;
        }
        try {
            Thread.sleep(consumeFailDelay);
        } catch (InterruptedException ignore) {}
        // 延遲後還不能消費，就要丟回隊列了
        if (!isCanConsume(application, messageId, previousMessageId, msgTimestamp)) {
            stashMessage(message);
            return null;
        }
        return consumeMessage(methodInvocation, message);
    }
 
    private boolean isCanConsume(String application, String messageId, String previousMessageId, Date msgTimestamp) {
        boolean canConsume = false;
        try {
            canConsume = consumeRecordManager.isConsumed(application, previousMessageId);
        } catch (Throwable e) {
            log.error("fail to check sequence is can consume, message id: {}", messageId, e);
        }
        if (!canConsume
                && Objects.nonNull(msgTimestamp)
                && Math.abs(msgTimestamp.getTime() - System.currentTimeMillis()) >= faultTolerantTime) {
            canConsume = true;
        }
        return canConsume;
    }
 
    private Object consumeMessage(MethodInvocation methodInvocation, Message message) throws Throwable {
        String application = RabbitUtils.getProduceApplication(message);
        String messageId = RabbitUtils.getMessageId(message);
        Object result = methodInvocation.proceed();
        // 成功執行，嘗試添加消費記錄，catch所有異常，不影響mq的消費
        // 如果開啓了冪等消費，則由冪等攔截器添加消費記錄，下面語句不會添加成功
        try {
            consumeRecordManager.insertConsumeRecord(application, messageId);
            afterSuccessConsume(message);
        } catch (Throwable e) {
            log.error("fail to save consume record", e);
        }
        return result;
    }
 
    abstract protected void stashMessage(Message message);
 
    protected void afterSuccessConsume(Message message) {};
}

/**
 * 順序消費攔截器——遠程模式
 *
 * @author chenxudong
 * @date 2019/09/18
 */
@Slf4j
public class RemoteSequenceOperationsInterceptor extends AbstractSequenceOperationsInterceptor {
    private static final String CAN_NOT_CONSUME_YET = "消息還不能被消費，前置消息爲：";
    private ConsumeService consumeService;
 
    public RemoteSequenceOperationsInterceptor(ObjectProvider<ConsumeService> consumeService,
                                               ConsumeRecordManager consumeRecordManager, int consumeFailDelay, int faultTolerantTime) {
        super(consumeRecordManager, consumeFailDelay, faultTolerantTime);
        this.consumeService = consumeService.getIfUnique();
    }
 
    @Override
    protected void stashMessage(Message message) {
        String application = RabbitUtils.getProduceApplication(message);
        String previousMessageId =
                message.getMessageProperties().getHeaders().get(RabbitHeaderConstants.MESSAGE_PREVIOUS_ID_HEADER).toString();
        if (Objects.isNull(consumeService)) {
            throw new ImmediateRequeueAmqpException(CAN_NOT_CONSUME_YET + previousMessageId);
        } else {
            boolean success = false;
            // 構造保存請求
            Date forceConsumeTime = DateUtils.addMilliseconds(new Date(), faultTolerantTime);
            SaveRabbitSequenceRecordReq request = new SaveRabbitSequenceRecordReq();
            request.setBody(RabbitUtils.getBodyAsString(message));
            request.setConsumeApplication(application);
            request.setProduceApplication(RabbitUtils.getProduceApplication(message));
            request.setVirtualHost(RabbitUtils.getVirtualHost(message));
            request.setQueue(RabbitUtils.getConsumerQueue(message));
            request.setPreviousMessageId(previousMessageId);
            request.setMessageId(RabbitUtils.getMessageId(message));
            request.setProperties(RabbitUtils.getPropertiesAsJsonString(message));
            request.setGroupName(RabbitUtils.getMessageGroup(message));
            request.setProduceMessageId(RabbitUtils.getProduceMessageId(message));
            request.setStatus((byte) 0);
            request.setForceConsumeTime(forceConsumeTime);
            try {
                success = consumeService.saveRabbitSequenceRecord(request);
            } catch (Throwable e) {
                log.error("fail to save sequence record=" + JSON.toJSONString(request), e);
            }
            if (!success) {
                log.warn("fail to save sequence record:{}", JSON.toJSONString(request));
                throw new ImmediateRequeueAmqpException("can not consume this message yet, previous message id: " + previousMessageId);
            }
        }
    }
 
    @Override
    protected void afterSuccessConsume(Message message) {
 
    }
 
}

mq-platform-server：

/**
 * @author wanglongzhao
 * @2020-02-24 15:21
 * description
 */
@Slf4j
@Component
public class ConsumeSequenceRunner extends AbstractConsumeRunner implements ApplicationRunner {
    private static final String CONSUME_SEQUENCE_DISTRIBUTION_LOCK_KEY = "sequenceLockKey";
 
    @Autowired
    private RabbitSequenceRecordManager rabbitSequenceRecordManager;
    @Autowired
    private RabbitSequenceRecordService rabbitSequenceRecordService;
    @Autowired
    private ConsumeSequenceConfirmCallback callback;
 
    @Override
    public void run(ApplicationArguments args) throws Exception {
        AsyncHandlerUtil.excute(() -> {
            executeConsumeSequence();
            return Boolean.TRUE;
        });
    }
 
    public void executeConsumeSequence() {
 
        DistributionLock lock = createLock();
        while (!getDistributionLock(lock)) {
        }
        // 獲取鎖成功，開始執行任務
        try {
            while (working) {
                try {
                    // 檢索“未順序消費”的任務，發送消息
                    fetchSequenceRecordAndSendMessage();
                } catch (Exception e) {
                    try {
                        //延時3s
                        Thread.sleep(3000);
                    } catch (InterruptedException ignore) {
                    }
                }
            }
        } finally {
            this.releaseDistributionLock(lock);
        }
    }
 
    private void fetchSequenceRecordAndSendMessage() {
        //查詢未順序消費的任務
        GetRabbitConsumeSequenceRecordFilter sequenceRecordFilter = new GetRabbitConsumeSequenceRecordFilter();
        sequenceRecordFilter.setForceConsumeTime(new Date());
        sequenceRecordFilter.setStatus(0);
        RabbitSequenceRecord record = rabbitSequenceRecordManager.getRecord(sequenceRecordFilter);
        if (Objects.isNull(record)) {
            //未查詢到記錄，休眠500ms
            try {
                Thread.sleep(500);
            } catch (InterruptedException ignore) {
            }
            return;
        }
        //更新數據庫記錄的狀態，待發送更新爲發送中
        Boolean result = rabbitSequenceRecordService.disableStatus(record);
        if (result) {
            //更新成功，發送mq
            this.sendMessage2Queue(record);
        }
    }
 
    private void sendMessage2Queue(RabbitSequenceRecord record) {
        try {
            Message message = ServerRabbitUtils.sequence2Message(record);
            CorrelationData correlationData = new CorrelationData();
            correlationData.setId(record.getRecordId().toString());
            sendRabbitMqService.send(record.getVirtualHost(), record.getQueue(), message, correlationData, callback);
        } catch (Throwable e) {
            log.error("fail to requeue message, record=" + JSON.toJSONString(record), e);
        }
    }
 
    private DistributionLock createLock() {
        LockInfo lockInfo = new LockInfo();
        lockInfo.setId(CONSUME_SEQUENCE_DISTRIBUTION_LOCK_KEY);
        lockInfo.setModule(moduleName);
        //單位：ms 超時獲取時間
        lockInfo.setWaitTime(5000);
        lockInfo.setLockProvider(LockProviderTypeEnum.ZOOKEEPER);
        return lockManager.createLock(lockInfo);
    }
 
}

方案二

生產者

生產者同方案一邏輯一致

消費者

基本同方案一一致，區別在於方案四中會將前置消息未消費的消息都保存在消費者服務的數據庫中的consume_sequence_record表中，消費重試的時候沒有mq-platform的參與。

流程設計：

代碼實現：

/**
 * 順序消費攔截器——本地模式
 *
 * @author chenxudong
 * @date 2020/02/21
 */
@Slf4j
public class LocalSequenceOperationsInterceptor extends AbstractSequenceOperationsInterceptor {
    private RabbitConsumeSequenceRecordManager consumeSequenceRecordManager;
 
    public LocalSequenceOperationsInterceptor(ObjectProvider<RabbitConsumeSequenceRecordManager> consumeSequenceRecordManagerObjectProvider,
                                       ConsumeRecordManager consumeRecordManager, int consumeFailDelay, int faultTolerantTime) {
        super(consumeRecordManager, consumeFailDelay, faultTolerantTime);
        try {
            consumeSequenceRecordManager = consumeSequenceRecordManagerObjectProvider.getIfUnique();
        } catch (Throwable ignore) {}
    }
 
    @Override
    protected void stashMessage(Message message) {
        try {
            consumeSequenceRecordManager.saveRecord(message, faultTolerantTime);
        } catch (Throwable e) {
            log.error("fail to save consume sequence record", e);
            String messageId = RabbitUtils.getMessageId(message);
            throw new ImmediateRequeueAmqpException("fail to save consume sequence record, messageId=" + messageId);
        }
    }
 
    @Override
    protected void afterSuccessConsume(Message message) {
        Object clearSequenceRecord = RabbitUtils.getMessageHeader(message,
                RabbitHeaderConstants.MESSAGE_SEQUENCE_RETRY_MARK);
        if (Objects.isNull(clearSequenceRecord)) {
            return;
        }
        try {
            String produceApplication = RabbitUtils.getProduceApplication(message);
            String messageId = RabbitUtils.getMessageId(message);
            consumeSequenceRecordManager.successConsumeMessage(produceApplication, messageId);
        } catch (Throwable e) {
            log.error("fail to save consume sequence record", e);
            // TODO 告警
        }
    }
}

 

/**
 * 工作線程管理類
 *
 * @author chenxudong
 * @date 2020/02/21
 */
@Slf4j
public class WorkThreadManager {
    private static final String WORK_THREAD_LOCK_KEY = "/work_thread_lock_key";
    private static final ExecutorService WORK_THREAD_POOL = new ThreadPoolExecutor(1, 1, 10, TimeUnit.SECONDS,
            new SynchronousQueue<>(), new CustomizedThreadFactory());
 
    private volatile boolean working = true;
    private CuratorFramework client;
    private String applicationName;
    private RabbitConsumeSequenceRecordManager recordManager;
    private RabbitTemplateManager rabbitTemplateManager;
    private long defaultRetryPeriod = 2000;
 
    public WorkThreadManager(
            String applicationName,
            String zookeeperAddress,
            RabbitConsumeSequenceRecordManager recordManager,
            RabbitTemplateManager rabbitTemplateManager
    ) {
        this.applicationName = applicationName;
        this.client = ZookeeperSupport.createCuratorFramework(zookeeperAddress, 20, 20);
        this.recordManager = recordManager;
        this.rabbitTemplateManager = rabbitTemplateManager;
    }
 
    @PostConstruct
    public void postConstruct() {
        WORK_THREAD_POOL.submit(() -> {
            InterProcessMutex lock = getLock();
            if (Objects.isNull(lock)) {
                throw new RuntimeException("fail to get distribution lock");
            }
            while (!tryLock(lock)) {}
            try {
                while (working) {
                    // TODO 發送順序消息
                    List<RabbitConsumeSequenceRecord> records = recordManager.getCanBeRetryRecords();
                    if (CollectionUtils.isEmpty(records)) {
                        try {
                            Thread.sleep(500L);
                        } catch (InterruptedException ignore) {}
                        continue;
                    }
                    for (RabbitConsumeSequenceRecord record : records) {
                        try {
                            Optional<RabbitTemplate> rabbitTemplateOptional =
                                    rabbitTemplateManager.getRabbitTemplate(record.getVirtualHost());
                            if (rabbitTemplateOptional.isPresent()) {
                                Date nextRetryTime = new Date(System.currentTimeMillis() + defaultRetryPeriod);
                                recordManager.changeRetryInfo(record.getId(), record.getRetryTimes() + 1, nextRetryTime);
                                rabbitTemplateOptional.get().send(RabbitUtils.generateMessage(record));
                            } else {
                                // TODO 告警，修改任務狀態
                            }
                        } catch (Throwable e) {
                            log.error("fail to resend sequence message", e);
                        }
                    }
                }
            } finally {
                unLock(lock);
                ZookeeperSupport.closeCuratorFramework(client);
            }
        });
    }
 
    @PreDestroy
    public void preDestroy() {
        working = false;
        WORK_THREAD_POOL.shutdown();
        int retryTimes = 0;
        if (retryTimes < 3 && !WORK_THREAD_POOL.isTerminated()) {
            retryTimes++;
            try {
                WORK_THREAD_POOL.awaitTermination(3, TimeUnit.SECONDS);
            } catch (InterruptedException ignore) {}
        }
    }
 
    private InterProcessMutex getLock() {
        if (Objects.nonNull(client)) {
            return ZookeeperSupport.getLock(client,
                    applicationName + WORK_THREAD_LOCK_KEY);
        }
        return null;
    }
 
    private Boolean tryLock(InterProcessMutex lock) {
        try {
            while (!lock.acquire(60, TimeUnit.SECONDS)) {
            }
        } catch (Exception e) {
            log.error("fail to get distribution lock", e);
            return Boolean.FALSE;
        }
        return Boolean.TRUE;
    }
 
    private void unLock(InterProcessMutex lock) {
        try {
            lock.release();
 
        } catch (Throwable e){
            log.error("fail to release distribution lock", e);
        }
    }
 
    /**
     * 自定義線程工廠,新的線程利用該類進行創建
     */
    static class CustomizedThreadFactory implements ThreadFactory {
        private AtomicInteger sequence = new AtomicInteger(1);
        @Override
        public Thread newThread(Runnable runnable) {
            Thread thread = new Thread(runnable);
            thread.setName("work-thread-" + sequence.getAndIncrement());
            return thread;
        }
    }
}