消息隊列之推還是拉，RocketMQ 和 Kafka 是如何做的？

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每個時代，都不會虧待會學習的人"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家好，我是 yes。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天我們就來談一談消息隊列的推拉模式，這也是一個面試熱點，例如你在簡歷裏面寫了 RocketMQ ，基本上會問你 RocketMQ 採用的是推模式還是拉模式啊？是拉模式？不是有 PushConsumer 嗎？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天我們就來談談推拉模式，並且再來看看 RocketMQ 和 Kafka 是如何做的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"推拉模式"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先明確一下推拉模式到底是在討論消息隊列的哪一個步驟，一般而言我們在談論"},{"type":"text","marks":[{"type":"strong"}],"text":"推拉模式的時候指的是 Comsumer 和 Broker 之間的交互"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"默認的認爲 Producer 與 Broker 之間就是推的方式，即 Producer 將消息推送給 Broker，而不是 Broker 主動去拉取消息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"想象一下，如果需要 Broker 去拉取消息，那麼 Producer 就必須在本地通過日誌的形式保存消息來等待 Broker 的拉取，如果有很多生產者的話，那麼消息的可靠性不僅僅靠 Broker 自身，還需要靠成百上千的 Producer。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Broker 還能靠多副本等機制來保證消息的存儲可靠，而成百上千的 Producer 可靠性就有點難辦了，所以默認的 Producer 都是推消息給 Broker。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以說有些情況分佈式好，而有些時候還是集中管理好。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"推模式"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"推模式指的是消息從 Broker 推向 Consumer，即 Consumer 被動的接收消息，由 Broker 來主導消息的發送。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"我們來想一下推模式有什麼好處？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"消息實時性高"},{"type":"text","text":"， Broker 接受完消息之後可以立馬推送給 Consumer。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"對於消費者使用來說更簡單"},{"type":"text","text":"，簡單啊就等着，反正有消息來了就會推過來。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"推模式有什麼缺點？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"推送速率難以適應消費速率"},{"type":"text","text":"，推模式的目標就是以最快的速度推送消息，當生產者往 Broker 發送消息的速率大於消費者消費消息的速率時，隨着時間的增長消費者那邊可能就“爆倉”了，因爲根本消費不過來啊。當推送速率過快就像 DDos 攻擊一樣消費者就傻了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"並且不同的消費者的消費速率還不一樣，身爲 Broker 很難平衡每個消費者的推送速率，如果要實現自適應的推送速率那就需要在推送的時候消費者告訴 Broker ，我不行了你推慢點吧，然後 Broker 需要維護每個消費者的狀態進行推送速率的變更。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這其實就增加了 Broker 自身的複雜度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以說推模式難以根據消費者的狀態控制推送速率，適用於消息量不大、消費能力強要求實時性高的情況下。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"拉模式"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"拉模式指的是 Consumer 主動向 Broker 請求拉取消息，即 Broker 被動的發送消息給 Consumer。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"我們來想一下拉模式有什麼好處？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"拉模式主動權就在消費者身上了，"},{"type":"text","marks":[{"type":"strong"}],"text":"消費者可以根據自身的情況來發起拉取消息的請求"},{"type":"text","text":"。假設當前消費者覺得自己消費不過來了，它可以根據一定的策略停止拉取，或者間隔拉取都行。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"拉模式下 Broker 就相對輕鬆了"},{"type":"text","text":"，它只管存生產者發來的消息，至於消費的時候自然由消費者主動發起，來一個請求就給它消息唄，從哪開始拿消息，拿多少消費者都告訴它，它就是一個沒有感情的工具人，消費者要是沒來取也不關它的事。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"拉模式可以更合適的進行消息的批量發送"},{"type":"text","text":"，基於推模式可以來一個消息就推送，也可以緩存一些消息之後再推送，但是推送的時候其實不知道消費者到底能不能一次性處理這麼多消息。而拉模式就更加合理，它可以參考消費者請求的信息來決定緩存多少消息之後批量發送。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"拉模式有什麼缺點？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"消息延遲"},{"type":"text","text":"，畢竟是消費者去拉取消息，但是消費者怎麼知道消息到了呢？所以它只能不斷地拉取，但是又不能很頻繁地請求，太頻繁了就變成消費者在攻擊 Broker 了。因此需要降低請求的頻率，比如隔個 2 秒請求一次，你看着消息就很有可能延遲 2 秒了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"消息忙請求"},{"type":"text","text":"，忙請求就是比如消息隔了幾個小時纔有，那麼在幾個小時之內消費者的請求都是無效的，在做無用功。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ac/ac63076945c784d644cfb864340f12e2.png","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"那到底是推還是拉"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到推模式和拉模式各有優缺點，到底該如何選擇呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" RocketMQ 和 Kafka 都選擇了拉模式，當然業界也有基於推模式的消息隊列如 ActiveMQ。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我個人覺得拉模式更加的合適，因爲現在的消息隊列都有持久化消息的需求，也就是說本身它就有個存儲功能，它的使命就是接受消息，保存好消息使得消費者可以消費消息即可。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而消費者各種各樣，身爲 Broker 不應該有依賴於消費者的傾向，我已經爲你保存好消息了，你要就來拿好了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雖說一般而言 Broker 不會成爲瓶頸，因爲消費端有業務消耗比較慢，但是 Broker 畢竟是一箇中心點，能輕量就儘量輕量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼竟然 RocketMQ 和 Kafka 都選擇了拉模式，它們就不怕拉模式的缺點麼？ 怕，所以它們操作了一波，減輕了拉模式的缺點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"長輪詢"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ 和 Kafka 都是利用“長輪詢”來實現拉模式，我們就來看看它們是如何操作的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了簡單化，下面我把消息不滿足本次拉取的條數啊、總大小啊等等都統一描述成還沒有消息，反正都是不滿足條件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"RocketMQ 中的長輪詢"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ 中的 PushConsumer 其實是披着"},{"type":"text","marks":[{"type":"strong"}],"text":"推"},{"type":"text","text":"模式實際上是"},{"type":"text","marks":[{"type":"strong"}],"text":"拉"},{"type":"text","text":"模式的方法，"},{"type":"text","marks":[{"type":"strong"}],"text":"只是看起來像推模式而已"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲 RocketMQ 在被背後偷偷的幫我們去 Broker 請求數據了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"後臺會有個 RebalanceService 線程，這個線程會根據 topic 的隊列數量和當前消費組的消費者個數做負載均衡，每個隊列產生的 pullRequest 放入阻塞隊列 pullRequestQueue 中。然後又有個 PullMessageService 線程不斷的從阻塞隊列 pullRequestQueue 中獲取 pullRequest，然後通過網絡請求 broker，這樣實現的準實時拉取消息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這一部分代碼我不截了，就是這麼個事兒，稍後會用圖來展示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然後 Broker 的 PullMessageProcessor 裏面的 processRequest 方法是用來處理拉消息請求的，有消息就直接返回，如果沒有消息怎麼辦呢？我們來看一下代碼。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d7/d798be2d0cd99e726d1705921f2a2365.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們再來看下 suspendPullRequest 方法做了什麼。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b3/b385722d848b552ddde8b14c956bd744.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而 PullRequestHoldService 這個線程會每 5 秒從 pullRequestTable 取PullRequest請求，然後看看待拉取消息請求的偏移量是否小於當前消費隊列最大偏移量，如果條件成立則說明有新消息了，則會調用 notifyMessageArriving ，最終調用 PullMessageProcessor 的 executeRequestWhenWakeup() 方法重新嘗試處理這個消息的請求，也就是再來一次，整個長輪詢的時間默認 30 秒。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/94/94c31eec6915f3a108472b8179142a0a.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"簡單的說就是 5 秒會檢查一次消息時候到了，如果到了則調用 processRequest 再處理一次。這好像不太實時啊？ 5秒？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"別急，還有個 ReputMessageService 線程，這個線程用來不斷地從 commitLog 中解析數據並分發請求，構建出 ConsumeQueue 和 IndexFile 兩種類型的數據，"},{"type":"text","marks":[{"type":"strong"}],"text":"並且也會有喚醒請求的操作，來彌補每 5s 一次這麼慢的延遲"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"代碼我就不截了，就是消息寫入並且會調用 pullRequestHoldService#notifyMessageArriving。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後我再來畫個圖，描述一下整個流程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4f/4fcfec456923d467f226dd3904559e72.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Kafka 中的長輪詢"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"像 Kafka 在拉請求中有參數，可以使得消費者請求在 “長輪詢” 中阻塞等待。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"簡單的說就是消費者去 Broker 拉消息，定義了一個超時時間，也就是說消費者去請求消息，如果有的話馬上返回消息，如果沒有的話消費者等着直到超時，然後再次發起拉消息請求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"並且 Broker 也得配合，如果消費者請求過來，有消息肯定馬上返回，沒有消息那就建立一個延遲操作，等條件滿足了再返回。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們來簡單的看一下源碼，爲了突出重點，我會刪減一些代碼。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先來看消費者端的代碼。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5a/5addf1afb88134bad4714677881e4185.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面那個 poll 接口想必大家都很熟悉，其實從註解直接就知道了確實是等待數據的到來或者超時，我們再簡單的往下看。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/98/9806d9c520c8fa63faa607a1bf7a6d4a.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們再來看下最終 client.poll 調用的是什麼。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/10/104530a90bf9753e929148315c2a3619.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後"},{"type":"text","marks":[{"type":"strong"}],"text":"調用的就是 Kafka 包裝過的 selector，而最終會調用 Java nio 的 select(timeout)"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在消費者端的代碼已經清晰了，"},{"type":"text","marks":[{"type":"strong"}],"text":"我們再來看看 Broker 如何做的"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Broker 處理所有請求的入口其實我在之前的文章介紹過，就在 KafkaApis.scala 文件的 handle 方法下，這次的主角就是 handleFetchRequest 。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4b/4bcd7974bb248e69ee8b79ba65420d5c.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個方法進來，我截取最重要的部分。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/77/77993b7061eb535d0350abaff144c297.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面的圖片就是 fetchMessages 方法內部實現，源碼給的註釋已經很清晰了，大家放大圖片看下即可。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/6d/6d35420320124867727c71267239197b.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個煉獄名字取得很有趣，簡單的說就是利用我之前文章提到的時間輪，來執行定時任務，例如這裏是"},{"type":"codeinline","content":[{"type":"text","text":"delayedFetchPurgatory"}]},{"type":"text","text":"，專門用來處理延遲拉取操作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們先簡單想一下，這個延遲操作都需要實現哪些方法，首先構建的延遲操作需要有檢查機制，來查看消息是否已經到了，然後呢還得有個消息到了之後該執行的方法，還需要有執行完畢之後該幹啥的方法，當然還得有個超時之後得幹啥的方法。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這幾個方法其實對應的就是代碼裏的 DelayedFetch ，這個類繼承了 DelayedOperation 內部有："}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"isCompleted 檢查條件是否滿足的方法"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"tryComplete 條件滿足之後執行的方法"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"onComplete 執行完畢之後調用的方法"}]}]},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"onExpiration 過期之後需要執行的方法"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"判斷是否過期就是由時間輪來推動判斷的，但是總不能等過期的時候再去看消息到了沒吧？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏 Kafka 和 RocketMQ 的機制一樣，也會在消息寫入的時候提醒這些延遲請求消息來了，具體代碼我不貼了， 在 ReplicaManager#appendRecords 方法內部再深入個兩方法可以看到。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不過雖說代碼不貼，圖還是要畫一下的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/40/4034607e2476e0a5a550df04d5d3743a.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"小結一下"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到 RocketMQ 和 Kafka 都是採用“長輪詢”的機制，具體的做法都是通過消費者等待消息，當有消息的時候 Broker 會直接返回消息，如果沒有消息都會採取延遲處理的策略，並且爲了保證消息的及時性，在對應隊列或者分區有新消息到來的時候都會提醒消息來了，及時返回消息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一句話說就是消費者和 Broker 相互配合，拉取消息請求不滿足條件的時候 hold 住，避免了多次頻繁的拉取動作，當消息一到就提醒返回。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"最後"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"總的而言推拉模式各有優劣，而我個人覺得一般情況下拉模式更適合於消息隊列。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"看了這篇文章相信之後面試官問你推還是拉？建議給他個歪嘴笑。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9c/9c486b96c3d0d9e0c23e5062c1f05ea7.png","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"我是 yes，從一點點到億點點，我們下篇見"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}