RocketMQ消息消費六：消息過濾

概述

RocketMQ的消息過濾分爲兩種：表達式過濾和類過濾。表達式過濾針對消息的屬性過濾，適合於簡單的場景，類過濾可以實現複雜的邏輯。

表達式過濾

表達式過濾分爲tag過濾和SQL92過濾。SQL92在這裏不展開描述，只介紹下tag過濾。
在客戶端發送消息的時候可以指定消息的tag，並根據消息的tag生成哈希值，爲tagcode，存儲在CommitLog中。所以tag過濾分爲兩部分：拉取過濾和消費過濾，分別發生在broker端和消費端。

拉取過濾

消費者過濾規則存儲

消費者訂閱消費組時，將消息的過濾保存，以便消息拉取時過濾使用。詳細代碼見：org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#subscribe(java.lang.String, java.lang.String)

    public void subscribe(String topic, String subExpression) throws MQClientException {
        try {
            SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(this.defaultMQPushConsumer.getConsumerGroup(),
                topic, subExpression);
            this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
            if (this.mQClientFactory != null) {
                this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
            }
        } catch (Exception e) {
            throw new MQClientException("subscription exception", e);
        }
    }

來看下SubscriptionData的核心屬性：

public class SubscriptionData implements Comparable<SubscriptionData> {
    public final static String SUB_ALL = "*"; //訂閱所有消息
    private boolean classFilterMode = false; // 類過濾模式標記
    private String topic;
    private String subString; //表達式，可用豎線隔開
    private Set<String> tagsSet = new HashSet<String>();// tag集合
    private Set<Integer> codeSet = new HashSet<Integer>();// tagcode集合
    private long subVersion = System.currentTimeMillis();
    private String expressionType = ExpressionType.TAG;// 過濾類型
}

消費者拉取消息時生成過濾標記

根據上面的存儲，生成拉取請求中的過濾標記，見org.apache.rocketmq.client.impl.consumer.PullAPIWrapper#pullKernelImpl
中構造requestHeader部分，這裏不再貼源碼。

broker過濾tagcode

根據請求requestHeader中的過濾信息，構造MessageFilter，見 org.apache.rocketmq.broker.processor.PullMessageProcessor#processRequest(io.netty.channel.Channel, org.apache.rocketmq.remoting.protocol.RemotingCommand, boolean)

        MessageFilter messageFilter;
        if (this.brokerController.getBrokerConfig().isFilterSupportRetry()) {
            messageFilter = new ExpressionForRetryMessageFilter(subscriptionData, consumerFilterData,
                this.brokerController.getConsumerFilterManager());
        } else {
            messageFilter = new ExpressionMessageFilter(subscriptionData, consumerFilterData,
                this.brokerController.getConsumerFilterManager());
        }

直接看最終的過濾邏輯，這裏以ExpressionMessageFilter爲例：

public boolean isMatchedByConsumeQueue(Long tagsCode, ConsumeQueueExt.CqExtUnit cqExtUnit) {
        if (null == subscriptionData) {
            return true;
        }

        if (subscriptionData.isClassFilterMode()) { //類過濾這裏不處理
            return true;
        }

        // tag模式
        if (ExpressionType.isTagType(subscriptionData.getExpressionType())) {

            if (tagsCode == null) {// tag爲空
                return true;
            }

            if (subscriptionData.getSubString().equals(SubscriptionData.SUB_ALL)) {// 全訂閱
                return true;
            }

            return subscriptionData.getCodeSet().contains(tagsCode.intValue()); // 包含即命中
        } else {
            // no expression or no bloom
            if (consumerFilterData == null || consumerFilterData.getExpression() == null
                || consumerFilterData.getCompiledExpression() == null || consumerFilterData.getBloomFilterData() == null) {
                return true;
            }

            // message is before consumer
            if (cqExtUnit == null || !consumerFilterData.isMsgInLive(cqExtUnit.getMsgStoreTime())) {
                log.debug("Pull matched because not in live: {}, {}", consumerFilterData, cqExtUnit);
                return true;
            }

            byte[] filterBitMap = cqExtUnit.getFilterBitMap();
            BloomFilter bloomFilter = this.consumerFilterManager.getBloomFilter();
            if (filterBitMap == null || !this.bloomDataValid
                || filterBitMap.length * Byte.SIZE != consumerFilterData.getBloomFilterData().getBitNum()) {
                return true;
            }

            BitsArray bitsArray = null;
            try {
                bitsArray = BitsArray.create(filterBitMap);
                boolean ret = bloomFilter.isHit(consumerFilterData.getBloomFilterData(), bitsArray);
                log.debug("Pull {} by bit map:{}, {}, {}", ret, consumerFilterData, bitsArray, cqExtUnit);
                return ret;
            } catch (Throwable e) {
                log.error("bloom filter error, sub=" + subscriptionData
                    + ", filter=" + consumerFilterData + ", bitMap=" + bitsArray, e);
            }
        }

        return true;
    }

消費過濾

因爲在broker端過濾的時候只比較了tagcode，並沒有比較tag的值，在消息消費的時候還需要對tag值做校驗，具體邏輯詳見：org.apache.rocketmq.client.impl.consumer.PullAPIWrapper#processPullResult

    public PullResult processPullResult(final MessageQueue mq, final PullResult pullResult,
        final SubscriptionData subscriptionData) {
        PullResultExt pullResultExt = (PullResultExt) pullResult;

        this.updatePullFromWhichNode(mq, pullResultExt.getSuggestWhichBrokerId());
        if (PullStatus.FOUND == pullResult.getPullStatus()) {
            ByteBuffer byteBuffer = ByteBuffer.wrap(pullResultExt.getMessageBinary());
            List<MessageExt> msgList = MessageDecoder.decodes(byteBuffer);

            List<MessageExt> msgListFilterAgain = msgList;
            if (!subscriptionData.getTagsSet().isEmpty() && !subscriptionData.isClassFilterMode()) {
                msgListFilterAgain = new ArrayList<MessageExt>(msgList.size());
                for (MessageExt msg : msgList) {
                    if (msg.getTags() != null) {
                    // 對比tag值
                        if (subscriptionData.getTagsSet().contains(msg.getTags())) {
                            msgListFilterAgain.add(msg);
                        }
                    }
                }
            }

			...

        return pullResult;
    }

小結

在broker端的CommitLog過濾時，使用的是tagcode過濾，原因是CommitLog中只保存了tagcode。爲什麼只保留tagcode呢？是爲了將CommitLog設計爲定長的，提高存儲效率。所以需要在消息真正消費的時候，再用消息的真實tag過濾一遍

類過濾模式

待補充