RocketMQ學習-Broker-1

領域模型

MQ領域語言描述RocketMQ做的事情，producer構建Message，發送給broker的指定topic，broker負責將消息投遞到指定topic下的隊列，並記錄消息隊列的offset，consumer利用拉模式拉取消息進行消費。

ddd-for-rmq.png

啓動過程

BrokerController

BrokerController是broker模塊的核心控制類，負責broker的初始化、啓動、停止、資源管理，以及接受外部的請求並作出相應的動作。看下BrokerController中主要的屬性，藉此可以看下broker的基本功能

主要屬性

BrokerConfig，用於維護broker的配置信息
NettyServerConfig，對於producer和consumer來說，broker是服務端
NettyClientConfig，對於name server來說，broker是客戶端
MessageStoreConfig，消息存儲的配置，RocketMQ一個非常厲害的特性就是上億消息的堆積能力，堆積的消息是存儲在broker的磁盤上的，那麼這個類就是維護broker的消息存儲的配置信息
ConsumerManager，消費者管理
ConsumerFilterManager，消費者消息過濾管理
ProducerManager，生產者管理
MessageArrivingListener，消息到達監聽器
BrokerOuterAPI，broker和外部系統溝通的適配層，有幾個功能：（1）和name server交互，進行broker節點的註冊和取消；（2）和其他broker節點交互；

上面這些不是全部，除此之外，還有幾個線程池和線程池對應的隊列，以及用於做HA的管理模塊。顯然，broker功能非常多，我們在接下來的幾篇中慢慢梳理其中的代碼。

public class BrokerController {
    private static final Logger log = LoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
    private static final Logger LOG_PROTECTION = LoggerFactory.getLogger(LoggerName.PROTECTION_LOGGER_NAME);
    private static final Logger LOG_WATER_MARK = LoggerFactory.getLogger(LoggerName.WATER_MARK_LOGGER_NAME);
    /**
     * Broker的配置
     */
    private final BrokerConfig brokerConfig;
    /**
     * netty服務端配置，對於生產者和消費者來說，broker是服務端
     */
    private final NettyServerConfig nettyServerConfig;
    /**
     * netty客戶端，對於name server來說，broker是客戶端
     */
    private final NettyClientConfig nettyClientConfig;
    /**
     * 消息存儲配置
     */
    private final MessageStoreConfig messageStoreConfig;
    /**
     * 消費者的offset管理
     */
    private final ConsumerOffsetManager consumerOffsetManager;
    /**
     * 消費者管理
     */
    private final ConsumerManager consumerManager;
    /**
     * 消費過濾管理
     */
    private final ConsumerFilterManager consumerFilterManager;
    /**
     * 生產者管理
     */
    private final ProducerManager producerManager;
    /**
     * 監聽客戶端和broker建立的通信通道，當通道關閉時候清理信息
     */
    private final ClientHousekeepingService clientHousekeepingService;
    /**
     * 拉取消息處理器
     */
    private final PullMessageProcessor pullMessageProcessor;
    /**
     * ???暫時不理解
     */
    private final PullRequestHoldService pullRequestHoldService;
    /**
     * 消息到達監聽器
     */
    private final MessageArrivingListener messageArrivingListener;
    /**
     * 用於broker對client發起指令
     */
    private final Broker2Client broker2Client;
    private final SubscriptionGroupManager subscriptionGroupManager;
    private final ConsumerIdsChangeListener consumerIdsChangeListener;
    /**
     * 負載均衡管理器
     */
    private final RebalanceLockManager rebalanceLockManager = new RebalanceLockManager();
    /**
     * broker對外暴露的API
     */
    private final BrokerOuterAPI brokerOuterAPI;

    /**
     * 調度線程池
     */
    private final ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(new ThreadFactoryImpl(
        "BrokerControllerScheduledThread"));
    /**
     * 子節點同步器
     */
    private final SlaveSynchronize slaveSynchronize;
    /**
     * 發送消息的線程池任務隊列
     */
    private final BlockingQueue<Runnable> sendThreadPoolQueue;
    /**
     * 拉取消息的線程池任務隊列
     */
    private final BlockingQueue<Runnable> pullThreadPoolQueue;
    private final BlockingQueue<Runnable> queryThreadPoolQueue;
    private final BlockingQueue<Runnable> clientManagerThreadPoolQueue;
    private final BlockingQueue<Runnable> consumerManagerThreadPoolQueue;
    private final FilterServerManager filterServerManager;
    private final BrokerStatsManager brokerStatsManager;
    private final List<SendMessageHook> sendMessageHookList = new ArrayList<SendMessageHook>();
    private final List<ConsumeMessageHook> consumeMessageHookList = new ArrayList<ConsumeMessageHook>();
    private MessageStore messageStore;
    private RemotingServer remotingServer;
    private RemotingServer fastRemotingServer;
    private TopicConfigManager topicConfigManager;
    private ExecutorService sendMessageExecutor;
    private ExecutorService pullMessageExecutor;
    private ExecutorService queryMessageExecutor;
    private ExecutorService adminBrokerExecutor;
    private ExecutorService clientManageExecutor;
    private ExecutorService consumerManageExecutor;
    private boolean updateMasterHAServerAddrPeriodically = false;
    private BrokerStats brokerStats;
    private InetSocketAddress storeHost;
    private BrokerFastFailure brokerFastFailure;
    private Configuration configuration;
    
    //省略其他代碼
}

initialize

這個方法用於初始化broker節點，主要的工作可以列舉如下：

加載主要模塊的配置信息

這個部分，會從外存總加載各個模塊的配置信息，包括：topicConfigManager、consumerOffsetManager、subscriptionGroupManager、consumerFilterManager、messageStore。這裏代碼寫得非常漂亮，使用了配置外化的思路和實現、應用了模板設計模式、插件設計模式和工廠設計模式。
- 模板設計模式
RocketMQ中的模板方法設計模式

插件設計模式

插件設計模式和工廠設計模式一起使用，需要包含一個插件上下文、一個抽象插件類（AbstractPluginMessageStore），主要模塊入下圖所示。

插件設計模式

插件上下文（MessageStorePluginContext）用於保存跟插件相關的信息，看下插件上下文的代碼：

public class MessageStorePluginContext {
    private MessageStoreConfig messageStoreConfig;
    private BrokerStatsManager brokerStatsManager;
    private MessageArrivingListener messageArrivingListener;
    private BrokerConfig brokerConfig;

    public MessageStorePluginContext(MessageStoreConfig messageStoreConfig,
        BrokerStatsManager brokerStatsManager, MessageArrivingListener messageArrivingListener,
        BrokerConfig brokerConfig) {
        super();
        this.messageStoreConfig = messageStoreConfig;
        this.brokerStatsManager = brokerStatsManager;
        this.messageArrivingListener = messageArrivingListener;
        this.brokerConfig = brokerConfig;
    }
    //省略了getter和setter方法
}

工廠設計模式

public final class MessageStoreFactory {
    public final static MessageStore build(MessageStorePluginContext context, MessageStore messageStore)
        throws IOException {
        //從配置文件中取出配置好的插件
        String plugin = context.getBrokerConfig().getMessageStorePlugIn();
        if (plugin != null && plugin.trim().length() != 0) {
            String[] pluginClasses = plugin.split(",");
            //依次加載插件類對象，並生成對應的MessageStore對象
            for (int i = pluginClasses.length - 1; i >= 0; --i) {
                String pluginClass = pluginClasses[i];
                try {
                    @SuppressWarnings("unchecked")
                    Class<AbstractPluginMessageStore> clazz = (Class<AbstractPluginMessageStore>) Class.forName(pluginClass);
                    Constructor<AbstractPluginMessageStore> construct = clazz.getConstructor(MessageStorePluginContext.class, MessageStore.class);
                    messageStore = construct.newInstance(context, messageStore);
                } catch (Throwable e) {
                    throw new RuntimeException(String.format(
                        "Initialize plugin's class %s not found!", pluginClass), e);
                }
            }
        }
        return messageStore;
    }
}

啓動服務器

            this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.clientHousekeepingService);
            NettyServerConfig fastConfig = (NettyServerConfig) this.nettyServerConfig.clone();
            fastConfig.setListenPort(nettyServerConfig.getListenPort() - 2);
            this.fastRemotingServer = new NettyRemotingServer(fastConfig, this.clientHousekeepingService);

初始化各種線程池

            this.sendMessageExecutor = new BrokerFixedThreadPoolExecutor(
                this.brokerConfig.getSendMessageThreadPoolNums(),
                this.brokerConfig.getSendMessageThreadPoolNums(),
                1000 * 60,
                TimeUnit.MILLISECONDS,
                this.sendThreadPoolQueue,
                new ThreadFactoryImpl("SendMessageThread_"));

            this.pullMessageExecutor = new BrokerFixedThreadPoolExecutor(
                this.brokerConfig.getPullMessageThreadPoolNums(),
                this.brokerConfig.getPullMessageThreadPoolNums(),
                1000 * 60,
                TimeUnit.MILLISECONDS,
                this.pullThreadPoolQueue,
                new ThreadFactoryImpl("PullMessageThread_"));

            this.queryMessageExecutor = new BrokerFixedThreadPoolExecutor(
                this.brokerConfig.getQueryMessageThreadPoolNums(),
                this.brokerConfig.getQueryMessageThreadPoolNums(),
                1000 * 60,
                TimeUnit.MILLISECONDS,
                this.queryThreadPoolQueue,
                new ThreadFactoryImpl("QueryMessageThread_"));

            this.adminBrokerExecutor =
                Executors.newFixedThreadPool(this.brokerConfig.getAdminBrokerThreadPoolNums(), new ThreadFactoryImpl(
                    "AdminBrokerThread_"));

            this.clientManageExecutor = new ThreadPoolExecutor(
                this.brokerConfig.getClientManageThreadPoolNums(),
                this.brokerConfig.getClientManageThreadPoolNums(),
                1000 * 60,
                TimeUnit.MILLISECONDS,
                this.clientManagerThreadPoolQueue,
                new ThreadFactoryImpl("ClientManageThread_"));

            this.consumerManageExecutor =
                Executors.newFixedThreadPool(this.brokerConfig.getConsumerManageThreadPoolNums(), new ThreadFactoryImpl(
                    "ConsumerManageThread_"));

註冊請求處理器

這個方法類似於name server裏的用法，這裏不仔細展開講
設置各種定時任務，包括：獲取broker狀態的、週期性將消費者的offset刷到硬盤、週期性檢查消費者的消費能力以保護broker、定期打印消息消費標記等等。

獲取name server的地址

            if (this.brokerConfig.getNamesrvAddr() != null) {
                this.brokerOuterAPI.updateNameServerAddressList(this.brokerConfig.getNamesrvAddr());
                log.info("Set user specified name server address: {}", this.brokerConfig.getNamesrvAddr());
            } else if (this.brokerConfig.isFetchNamesrvAddrByAddressServer()) {
                this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

                    @Override
                    public void run() {
                        try {
                            BrokerController.this.brokerOuterAPI.fetchNameServerAddr();
                        } catch (Throwable e) {
                            log.error("ScheduledTask fetchNameServerAddr exception", e);
                        }
                    }
                }, 1000 * 10, 1000 * 60 * 2, TimeUnit.MILLISECONDS);
            }

slave節點和master節點的不同處理

如果當前節點是slave節點，則設置一個定時任務：每隔一段時間，就將配置信息從master節點同步到當前節點；如果當前節點是master節點，則設置一個定時任務：每隔一段時間，就對比master節點和slave節點的配置信息，並打印出不相同的配置。

其他

其他還有start、shutdown和registerBrokerAll等方法，其中reigsterBrokerAll方法的作用是將broker節點註冊到name server，這樣producer和consumer就可以拿到broker節點的地址信息。

RocketMQ學習-Broker-1

領域模型

啓動過程

BrokerController

主要屬性

initialize

其他

druid數據源 xml配置

RSA 加密，解密，加簽，驗籤

Linux -crontab 定時任務時間配置

深入理解Java中this關鍵字的使用

項目啓動中報錯Address already in use: bind

java的設計模式建造者模式

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結