向消息延遲說bybye:閒魚消息及時到達方案(詳細)

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"IM消息作爲閒魚用戶重要的交易諮詢工具,核心目標有兩點,第一是保證用戶的消息不丟失,第二是保證用戶的消息及時送達接收方。IM消息根據消息的接收方設備是否在線,分爲離線和在線推送,數據顯示目前閒魚每天有超過一半以上的IM消息是走在線通道的,而在線消息的到達率、及時性是直接影響用戶體驗的,本文將着重分析優化在線通道的穩定性,保證用戶消息及時到達。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"面臨哪些問題"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"端內長連接中斷"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在IM場景中,用戶與雲端通信頻繁,且爲了實現用戶的消息及時到達,往往採用雲端下推消息的方式觸達用戶,所以用戶在線時設備與雲端會維持一條TCP長連接通道,可以更輕量級的與服務端進行交互,現代IM即時通訊的下行消息都是通過長連下發的,閒魚消息使用的是ACCS長連接,ACCS是淘寶無線提供的全雙工、低延時、高安全的通道服務。但是由於用戶設備網絡狀態的不確定性,可能會發生各種各樣的網絡異常情況導致長連接通道中斷,長連接一旦意外中斷,就會導致用戶無法及時收到在線消息,所以我們需要儘可能及時的感知到長連中斷並嘗試重連。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3e\/3ee2ffa869d1360c220ecf925bac3bb5.png","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"下推的消息未達"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"感知長連中斷並重連只能在大多數時間保證長連接的有效性,但是在長連接無效或不穩定期間下推的消息客戶端可能根本收不到,簡單說就是僅僅有重連機制無法保證下行消息必達,可能有以下場景導致下行消息失敗:"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務端發送下行消息時長連暢通,消息在傳輸路上通道斷掉,客戶端無法收到 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"設備的在線狀態存在延遲,服務端下行消息時認爲設備在線,實際上設備已經離線,無法收到 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"客戶端收到了下行消息,但端上後續處理失敗,比如落庫失敗,消息沒有成功展示給用戶"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們通過數據埋點統計得出,accs下行成功率在97%左右"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/fc\/fc8d59a58fb3fa0c4d8247c70269104b.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有心急的同學就要問了,丟了3%的消息嗎?並沒有,這3%的消息不會丟失,只是不保證及時觸達給用戶。我們的消息同步模型是推拉結合模式,在用戶拉取消息時會拉取到設備當前位點與服務端最新位點的所有消息,accs下行失敗的消息會通過主動拉模式獲取到,但客戶端主動拉取消息的觸發時機有限,主要有以下幾個:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用戶冷啓動app,主動同步消息 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用戶主動下拉刷新 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"app後臺切換前臺 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"收到一條推送消息,客戶端發現新消息的位點跟本地最新的位點有gap,觸發同步"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可見上述主動同步消息的觸發很大程度上依賴用戶行爲或者有沒有收到新消息,難以保證消息及時到達。如果是用戶高頻打開的IM軟件,這樣也不會有太大的問題,但是閒魚app的活躍度較低,有時候甚至依賴IM消息拉活,而且一條延遲的消息觸達可能導致用戶錯過一筆交易,閒魚消息不允許有這樣的延遲發生。基於上述分析,我們先描述一個數據指標來反映現狀,通過上面的描述可知,accs消息並不全都是推下來的,也可能是主動拉下來的,如果是推,必定可以及時到達,如果是拉,則受限於用戶行爲。拉的這部分消息,我們定義爲accs消息補償到達,然後計算accs消息補償到達耗時,消息範圍限定爲服務端accs成功下行但是客戶端通過主動拉取同步到的消息,以往的版本這個數據在60分鐘左右。需要注意這個數據並不是消息觸達到用戶的耗時,因爲如果在線轉離線觸達,拉取到消息的時間取決於用戶行爲(用戶何時打開了app),但這個數據也能大致反映在線消息的到達延遲狀況。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/79\/791ca1e55bc32df8e601b0e2445a3fbe.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來本文將從長連接的重連和未達消息重發兩個方面詳細講述我們是如何優化在線通道穩定性的。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"長連接重連"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"長連接爲什麼會中斷?"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"百因必有果,我們先來分析下有哪些原因會導致連接中斷,可能有以下原因: "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用戶設備斷網 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"設備發生了網絡切換 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"設備處於弱網環境,網絡不穩定 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"設備網絡正常,TCP連接由於NAT超時導致連接被運營商中斷"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果是用戶操作導致網絡狀態變化的情況,會有網絡狀態變化事件通知,這種情況可以監聽事件並主動嘗試重連,但現實中的大多數情況都是“意料之外”。那麼如何有效感知到各種異常狀況呢?"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"心跳檢測"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"像大多數探活場景一樣,最有效的檢測手段就是心跳檢測,客戶端通過定時發送心跳包,可以感知到連接中斷,從及時性效果來看,心跳間隔越短越好,而頻繁的心跳檢測勢必會帶來用戶流量以及電量的損耗,所以我們的目標是如何儘可能少的心跳檢測而又儘量及時地感知到長連中斷的意外情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"狀態機+消息心跳隊列:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/12\/129fecbcf6211007da940ce0596a23ce.png","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在心跳協議設計上,要注意心跳包的核心目標是檢測長連通道是否暢通,客戶端主動上行心跳包且能收到服務端回包,就認爲長連通道健康,所以上行消息以及回包的數據包應儘可能小,一般來說,通過協議頭標識心跳包及響應即可"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"心跳策略"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"心跳策略是實現我們上述目標的核心機制,但關於心跳策略的詳細設計甚至可以單獨寫一篇文章,本文僅簡單列舉幾種心跳策略,有興趣的同學可以閱讀文末推薦的文章繼續深入研究。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"短心跳檢測 初始狀態連續 ping 3次 收到 ack 後,可以認爲進入穩定狀態"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"常規固定時長心跳(根據app狀態不同,頻率可調Mid+,Mid-, Long)"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自適應心跳 根據設備網絡狀態變化自動適應的心跳間隔"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"冗餘心跳,app後臺切前臺,主動心跳一次"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"消息ack與重發"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了解決上面的問題,引入消息ack與重發機制,整體思路是客戶端在收到accs消息並處理成功後,給服務端回一個ack,服務端下行accs消息時將消息加入重試隊列,收到ack後更新消息到達狀態,並終止重試。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整體設計流程圖:"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/71\/711c51a9ba58dc14089f5dfcca171162.png","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該方案的難點即重試處理器的實現設計,接下來我們將重點講述這部分的詳細設計"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"重試隊列存儲設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們採用阿里雲表格存儲TimeLine模型來存儲下行消息的到達狀態,阿里雲表格存儲是阿里雲自研的多模型結構化數據存儲,提供海量結構化數據存儲以及快速的查詢和分析服務。表格存儲的分佈式存儲和強大的索引引擎能夠支持PB級存儲、千萬TPS以及毫秒級延遲的服務能力。而Timeline 模型是針對消息數據場景所設計的數據模型,它能滿足消息數據場景對消息保序、海量消息存儲、實時同步的特殊需求,在IM、Feed流等消息場景應用廣泛。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們給每個用戶設備定義一個TimeLine,timeline-id定義爲userId_deviceId,sequenceId自定義爲消息位點,存儲結構如下: "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/27\/27ac3fd17ae55c1af39c6c1394204c80.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每通過accs成功下行一條消息,則插入到接收用戶設備的TimeLine中,收到ack後根據消息id更新消息到達狀態,同時由於重試動作只發生在下行消息後較短的一段時間內,所以我們設置一個比較短的全局過期時間即可,避免數據膨脹。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"延遲重試設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/dd\/ddbab3e77d82e787bc54daebd862cd4d.png","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1)每通過accs下發一條消息,先插入到Timeline中,初始狀態爲未達,然後生產一條延遲N秒的延遲消息 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2)每次消費到延遲消息後,讀取tablestore中該消息的到達狀態,如到達則終止延遲,否則繼續 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3)每次重試先判斷設備是否在線,如果設備不在線,轉發離線通道並終止重試,如果設備在線,則重推未到達的消息,並再次延遲N秒消費 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4)每條消息的重試生命週期中用的同一條延遲消息,最多重試消費M次,超過次數不再重試並打日誌埋點,後續可以監控這種情況並基於這個數據進行優化"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"延遲重發策略"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"延遲重發策略是指在重發流程中,如何選擇合適的延遲時間來使得重發的效率最高。不同用戶在不同時間、地點所處的網絡環境差別較大,網絡恢復到穩定態所需要的時間也有差異,需要選用合適的延遲策略來保證重發效率,最優的延遲策略的目標是在最短的時間內,使用最少的重發次數將消息投遞成功。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"固定延遲時間"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要想找到最優的延遲策略,必須從數據中通過分析得到答案,天馬行空的想象往往離實際相差甚遠,我們先採用固定的延遲時間(10s)最大重試6次來分析一波數據"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f1\/f10014de08eaec59991b7fecaeb87c59.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們通過這組數據可以看到,有約85%的消息在40s內重發可以投遞成功,還有12%的消息在達到最大重試次數後依舊沒有收到ack,在4次重試之後,第5次成功只有2.03%,第6次只有0.92%,繼續重發的收益已經變得很低,6次以後還有部分消息沒有收到ack,這部分消息如果用固定延遲時間策略,性價比很低,頻繁重發浪費系統資源,我們繼續改進策略。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"固定延遲+固定步長遞增"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"考慮到部分用戶的網絡短時間無法恢復,頻繁的短間隔重發價值不大,我們採用4次固定短間隔延遲N秒後,之後每次延遲時間都是上一次延遲時間遞增固定步長M秒的策略,直到收到ack、用戶設備離線或者達到了最大延遲時間MAX(N)。這種策略一定程度上可以解決固定延遲時間重發策略的問題,但如果用戶短時間網絡無法恢復,每次重發都要重新遞增,也不是一種最優解。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"自適應延遲"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"設計流程圖:"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/90\/9031eb5457ad15c70d059a7f478f146e.png","alt":"圖片","title":"null","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖,我們最終衍生出了自適應延遲策略,自適應延遲是指根據用戶的網絡狀況,採取自動調整的延遲時間,以期望達到最高的重發效率,新消息先通過4次固定N秒的短延遲來探測設備的網絡狀況,一旦網絡恢復,我們將設備的N值清空,設備N值是指根據上幾次重發經驗,當前設備網絡能回覆ack所需要的最短時間,默認情況該值爲空,代表用戶設備網絡正常。4次重發後依舊收不到ack,我們嘗試讀取設備N值,如果爲空,則取初始值,以後每次延遲都遞增固定步長M,並在重發後更新當前設備的N值,直到消息收到ack或者達到了最大延遲時間MAX(N)。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"新老版本兼容性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要注意的是老版本的app是不會回ack的,如果下發給老版本設備的消息也加入重試隊列,那此類消息將一直重試到最大次數纔會終止,無端消耗資源,所以我們設計在accs長連建立之後,客戶端主動上行一條設備信息,其中包含app的版本號,服務端存儲一定時間,在將消息加入重試隊列之前,先校驗接收者設備app的版本號,符合要求再加入重試隊列。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"方案效果"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息重連重發方案上線後,我們上面定義的指標accs補償到達時間從60分鐘大幅降低至15分鐘,降幅達75%,從而印證了我們的技術分析,同時用戶有關消息延遲的輿情反饋每週不超過2個,可見消息重發機制對保證用戶消息及時到達成效顯著。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"未來展望"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息在線通道穩定性優化至此已告一段落,未來我們將繼續優化閒魚消息的使用體驗,包括基礎功能的完善以及基礎體驗的提升。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基礎功能方面,我們在近期的版本中已經支持了消息撤回、草稿功能,後續將逐步支持發送定位,會話分組、備註,消息搜索等功能;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基礎體驗方面,我們對閒魚消息的UI樣式做了優化升級,並優化了app消息tab頁的cpu及內存使用,後續將繼續從流量、電量、性能方面繼續優化消息的使用體驗。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"References"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"[1]"}]},{"type":"text","text":"現代IM系統中的消息系統架構 - 架構篇: "},{"type":"text","marks":[{"type":"italic"}],"text":"https:\/\/developer.aliyun.com\/article\/698301"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"[2]"}]},{"type":"text","text":"現代IM系統中的消息系統架構 - 模型篇: "},{"type":"text","marks":[{"type":"italic"}],"text":"https:\/\/developer.aliyun.com\/article\/701593"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"[3]"}]},{"type":"text","text":"高併發IM系統架構優化實踐: "},{"type":"text","marks":[{"type":"italic"}],"text":"https:\/\/developer.aliyun.com\/article\/66461"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自:閒魚技術(ID:XYtech_Alibaba)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接:"},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/-x0hKwKAsPfB9V6FoJWcLA","title":"xxx","type":null},"content":[{"type":"text","text":"向消息延遲說bybye:閒魚消息及時到達方案(詳細)"}]}]}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章