Aeron是如何實現的？—— Ipc Publication

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接上文","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Aeron 是什麼？","attrs":{}},{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/27422063a2cadcc054187135e","title":"","type":null},"content":[{"type":"text","text":"https://xie.infoq.cn/article/27422063a2cadcc054187135e","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Aeron 中這麼多空閒策略選哪個？","attrs":{}},{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/41d0885f46594e90cbdba4b2b","title":"","type":null},"content":[{"type":"text","text":"https://xie.infoq.cn/article/41d0885f46594e90cbdba4b2b","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Aeron是如何實現的？—— Conductor ","attrs":{}},{"type":"link","attrs":{"href":"https://xie.infoq.cn/article/b4953b06323cd26e3a1397874","title":"","type":null},"content":[{"type":"text","text":"https://xie.infoq.cn/article/b4953b06323cd26e3a1397874","attrs":{}}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"0. 簡介","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最近我們用 Aeron 實現了 Mesh agent 與 sdk 之間的共享內存通信，但是在使用過程中越來越感覺到 Aeron 框架太重了，其中很大部分功能完全用不到，有些想要自定義的邏輯很難在現有框架中實現。所以我們計劃深入到 Aeron 源碼中，看看它是如何實現的，最終嘗試實現一個輕量的 Mesh 共享內存通信類庫。上文分析了 Conductor 的邏輯，本文繼續分析 Ipc Publication 的邏輯。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2e/2ed9a71252850386fc0eda2672e2afab.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1. Driver Conductor - add[Exclusive]Publication","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在發送數據之前，需要先向 Driver Conductor 發送ADD_[EXCLUSIVE_]PUBLICATION命令，讓 Driver 初始化通信的共享內存結構。至於 Conductor 交互的邏輯不再贅述，直接看 Driver Conductor 處理ADD_[EXCLUSIVE_]PUBLICATION命令的邏輯。處理邏輯的入口在io.aeron.driver.ClientCommandAdapter：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7d/7df106202d7d92d03e7f2b5668291439.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏只關心IPC_CHANNEL，進入io.aeron.driver.DriverConductor：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1a/1aca6d34547a05d11ada531047ff4407.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"三個主要步驟：","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"創建IpcPublication，核心就是 \"${aeronDirectory}/${correlationId}.logbuffer\" 共享內存文件；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"返回ON_[EXCLUSIVE_]PUBLICATION_READY消息；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"向所有相關的 subscribers 發送ON_AVAILABLE_IMAGE消息。（subscribers 收到該消息就會讀取共享內存進行消費，具體邏輯下篇分析）","attrs":{}}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.1 ${correlationId}.logbuffer 共享內存文件","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"logbuffer 文件結構的定義在io.aeron.logbuffer.LogBufferDescriptor中：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/33/33ef4d3926480743bc384f5556eff2c6.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面三個 Term 用於傳輸數據，這三個 Term 輪轉使用，Aeron Cookbook 上有個動畫演示，看起來這種設計主要有利於 UDP 數據流重建。下面的 Meta Data 比較複雜，具體字段用到的時候再詳細看：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/05/051f080424852f7a38e452200bbd47bb.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2. Client Conductor - onNew[Exclusive]Publication","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"client 收到ON_[EXCLUSIVE_]PUBLICATION_READY消息後構建相應的[Exclusive|Concurrent]Publication，該類封裝了發送消息的邏輯。其中ConcurrentPublication支持併發發送數據，但是性能不如ExclusivePublication好。看一下父類io.aeron.Publication的構造方法：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/25/251d1a536f97562e4cb6a167e51ec771.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"再看一下io.aeron.ExclusivePublication的構造方法（ConcurrentPublication 的主幹邏輯類似，這裏就不詳細看了）：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/53/5323859c1258ba472711a5be2dca7a3c.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大部分參數比較直觀，termAppenders封裝了具體 Term 的寫入邏輯。上面的提到的 Term 輪轉用法具體到代碼中，就是activePartitionIndex和termBeginPosition這兩個變量的維護。首先看一下選哪個 Term：","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"從 Log Meta Data 中取出 Active Term Count (termCount)","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"termCount按照PARTITION_COUNT(3)取餘得到當前的 Term 索引 (index)","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然後看一下從哪個位置開始寫：","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"從 Log Meta Data 中獲取對應的的 Tail Counter #index (rawTail)","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"rawTail的高 32 位是 termId，低 32 位是termOffset","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"termBeginPosition = (termId - initialTermId) << positionBitsToShift，其中positionBitsToShift這步操作本質上就是乘 Term 的長度","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後需要解釋一下positionLimit，這個值讀取的是 cnc.dat 中的一個 Counters Buffer，用於表示可以寫入的位置限制，主要作用是傳遞 Subscriber 的消費能力，用於背壓。（這個值的更新在 Driver 中，下篇分析 Subscription 時再詳細看）這個信息可以通過io.aeron.samples.AeronStat工具查看。其 label 是 \"pub-lmt: ${registrationId} ${sessionId} ${streamId} ${channel}\"，其中的 registrationId 就是 logbuffer 文件名的 correlationId。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3. ExclusivePublication - offer","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"發送數據有兩個方法offer和tryClaim。看一下ExclusivePublication的offer方法（ConcurrentPublication的主幹邏輯與之類似，只是多了些併發控制）：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/05/05ab2d62d2c796a72f90e9c06eefdcc5.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先如果寫入的位置position大於等於limit，也就是消費的能力更不上寫入了，那麼產生背壓（ConcurrentPublication由於存在併發的情況，所以並不是嚴格限制，但是不會超過該 term）：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ad/ad68ac34db9a8dba47f70a15b5d9ae60.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏還有個細節maxPossiblePosition，主要是限制 termCount 這個 int 值不要溢出。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1 termAppender.append[Un]fragmentedMessage 寫入數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於正常可以寫入的情況，如果寫入的數據小於 MTU，那麼調用appendUnfragmentedMessage：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5c/5c81e7efa2fe2f23255e956bc2e8b352.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先更新 Tail Counter #index。如果寫入的長度大於本 term 剩餘可寫空間了，那麼在handleEndOfLogCondition方法中處理異常情況：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4e/4e55f86ed99a44844caf79534920972a.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"剩餘空間填充一個PADDING_FRAME_TYPE的消息，然後返回FAILED(-1)。如果本 term 剩餘可寫空間足夠，那麼依次寫入 header、reservedValue 和數據。關於長度字段的寫入，有個小細節很有意思，先寫入一個負值，最後再寫入一個正值。通過這種方式可以保證 subscriber 讀到“正值”時，數據已經全部寫入了。對於數據大於 MTU 的場景，就需要對數據分段了，其它的邏輯跟上面是一樣的，只是通過BEGIN_FRAG_FLAG和END_FRAG_FLAG標識了消息。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.2 newPosition 更新位置","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據發送成功後，通過newPosition方法更新位置信息：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e6/e6f3ee9f04278d6a3a06f8e4d30ad0d5.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果寫入成功了，那麼直接更新本地變量termOffset即可。如果寫入失敗了，又不是超過最大位置的場景，那麼就需要輪轉 term 了。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/66/663b59101e9bb464917ad444bb855f27.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更新 Log Meta Data 中的 Active Term Count 和 Tail Counter #next，以及相關的本地變量。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4. tryClaim","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後看一下tryClaim方法，主幹邏輯與offer一致。區別在於：offer 時，數據已經準備好，所以在處理邏輯中直接寫入；tryClaim 時，只是預先佔着位置，上層通過BufferClaim寫入數據，最後commit時，提交“正值”的長度字段。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/aa/aa9ca8a04fe6bf5fda3720294fdbc53e.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule","attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a1/a1952246cc74835b57e19e801c0d98ce.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/80/80d2cd7afbb368c29c287fef5b7366b7.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9d/9d3abba7b4cfe3e32b78cdb1b198b22d.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}