分佈式磁盤 KV 存儲 - Kvrocks

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kvrocks 是基於 RocksDB 之上兼容 Redis 協議的 NoSQL 存儲服務，設計目標是提供一個低成本以及大容量的 Redis 服務，作爲 Redis 在大數據量場景的互補服務，選擇兼容 Redis 協議是因爲簡單易用且業務遷移成本低。目前線上使用的公司包含: 美圖、攜程、百度以及白山雲等，在線上經過兩年多大規模實例的驗證。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"項目核心功能包含:","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"兼容 Redis 協議","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持主從複製","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持通過 Namespace 隔離不同業務的數據","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"高可用，支持 Redis Sentinel 自動主從切換","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"集羣模式 (進行中，預計在 7-8 月份完成)","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"GitHub地址：","attrs":{}},{"type":"link","attrs":{"href":"https://link.zhihu.com/?target=https%3A//github.com/kvrockslabs/kvrocks","title":null,"type":null},"content":[{"type":"text","text":"https://github.com/kvrockslabs/kvrocks","attrs":{}}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"實現方案對比","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了 Kvrocks 之外，社區也有一些類似的基於磁盤存儲兼容 Redis 協議的開源產品，從存儲設計來看可以分爲幾類:","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"基於磁盤 KV 存儲引擎(比如 RocksDB/LevelDB) 實現 Redis 協議","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"基於 Redis 存儲之上將冷數據交換到磁盤(類似早期 Redis VM 的方案)","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"基於分佈式 KV(比如 TiKV) 實現 Redis 協議代理，本地不做存儲","attrs":{}}]}]}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/19/1958eff31bab03eda4196ff0ac11beff.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"方案 1:","attrs":{}},{"type":"text","text":" 基於磁盤 KV 之上兼容 Redis 協議，絕大多數的本地磁盤 KV 只提供最簡單的 Get/Set/Delete 方法，對於 Hash/Set/ZSet/List/Bitmap 等數據結構需要基於磁盤 KV 之上去實現。優點是可以規避下面方案 2 裏提到的大 Key 問題，缺點是實現工作量大一些。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"方案 2:","attrs":{}},{"type":"text","text":" 基於 Redis 把冷數據交換磁盤是以 Key 作爲最小單元，在大 Key 的場景下會有比較大的挑戰。交換大 Key 到磁盤會有嚴重讀寫放大，甚至可能會導致整個服務不可用，所以這種實現只能限制 Value 大小。這種方案的優點在於實現簡單且可按照 Key 維度來做冷熱數據分離。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"方案 3:","attrs":{}},{"type":"text","text":" 基於分佈式 KV 之上實現 Redis 協議，最大的區別在於所以的操作都是通過網絡。這種實現方式最大優點是隻需要實現 Redis 協議的部分，服務本身是無狀態的，無須考慮數據複製以及擴展性的問題。缺點也比較明顯，因爲所有的命令都是通過網絡 IO，對於非 String 類型的讀寫一般都是需要多次網絡 IO 且需要通過事務來保證原子，從而在延時和性能上都會比方案 1 和 2 差不少。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Kvrocks 設計的初衷是作爲 Redis 場景的互補，低成本、低延時和高吞吐是最重要的設計目標","attrs":{}},{"type":"text","text":"。基於 Redis 實現冷熱數據交換的方式在大 Key 場景下可能導致不可用，從而需要限制單個 Key 大小，這個對於我們想實現一個通用的 NoSQL 存儲服務是無法接受的。而對於方案 3 這種遠程存儲的方案，延時和吞吐一定是無法滿足預期，所以我們最終選擇的方案 1 這種基於磁盤 KV 之上實現 Redis 協議以及複製。除了數據存儲方式之外， Kvrocks 並沒有淘汰策略，所以一般是作爲存儲而不是緩存，當達到實例最大容量或者磁盤容量時會無法寫入。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"性能","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要注意的是以下提供的性能數據是基於特定的配置進行壓測，不同配置會有比較大的差異。壓測的硬件以及 Kvrocks 配置說明可參考: ","attrs":{}},{"type":"link","attrs":{"href":"http://link.zhihu.com/?target=https%3A//github.com/kvrockslabs/kvrocks%23performance","title":null,"type":null},"content":[{"type":"text","text":"https://github.com/kvrockslabs/kvrocks#performance","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d6/d6157caa18acc11ef00f9e67a00a3c93.png","alt":"image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏提供性能數據只是爲了給讀者更加直觀瞭解 Kvrocks 的性能情況，大部分命令由於可多線程並行執行，從 QPS 的維度來看會比 Redis 更好一些，但延時肯定會比 Redis 略差。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"快速體驗","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以使用 Docker 的方式來啓動 Kvrocks:","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"docker run -it -p 6666:6666 kvrocks/kvrocks","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接着可以跟使用 Redis 一樣使用:","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"➜ ~ redis-cli -p 6666\n\n127.0.0.1:6666> set foo bar\nOK\n127.0.0.1:6666> get foo\n\"bar\"","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"整體設計","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/26/261d2c4593d37d683ddc5d32be0709c9.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kvrocks 主要有兩類線程:","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Worker 線程，主要負責收發請求，解析 Redis 協議以及請求轉爲 RocksDB 的讀寫","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"後臺線程，目前包含以下幾種後臺線程:Cron 線程，負責定期任務，比如自動根據寫入 KV 大小調整 Block Size、清理 Backup 等Compaction Checker 線程，如果開啓了增量 Compaction 檢查機制，那麼會定時檢查需要 Compaction 的 SST 文件Task Runner 線程，負責異步的任務執行，比如後臺全量 Compaction，Key/Value 數量掃描主從複製線程，每個 slave 都會對應一個線程用來做增量同步","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"下面以 Hash 爲例來說明 Kvrocks 是如何將複雜的數據結構轉爲 RocksDB 對應的 KV?","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最簡單的方式是將 Hash 所有的字段進行序列化之後寫到同一個 Key 裏面，每次修改都需要將整個 Value 讀出來之後修改再寫入，當 Value 比較大時會導致嚴重的讀寫方法問題。所以我們參考 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"blackwidow","attrs":{}}],"attrs":{}},{"type":"text","text":" 的實現，把 Hash 拆分成 Metadata 和 Subkey 兩個部分，Hash 裏面的每個字段都是獨立的 KV，再使用 Metadata 來找到這些 Subkey:","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" +----------+------------+-----------+-----------+\nkey => | flags | expire | version | size |\n | (1byte) | (4byte) | (8byte) | (8byte) |\n +----------+------------+-----------+-----------+\n (hash metadata)\n\n +---------------+\nkey|version|field => | value |\n +---------------+\n (hash subkey)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"裏面的 flags 目前是來標識當前 Value 的類型，比如是 Hash/Set/List 等。expire 是 Key 的過期時間，size 是這個 Key 包含的字段數量，這兩個比較好理解。version 是在 Key 創建時自動生成的單調遞增的 id，每個 Subkey 前綴會關聯上 version。當 Metadata 本刪除時，這個 version 就無法被找到，也意味着關聯這個 version 的全部 Subkey 也無法找到，從而實現快速刪除，而這些無法找到的 Subkey 會在後臺 Compact 的時候進行回收。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以 HSET 命令爲例，僞代碼如下:","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"HSET key, field, value:\n // 先根據 hash key 找到對應的 metadata 並判斷是否過期\n // 如果不存在或者過期則創建一個新的 metadata\n metadata = rocksdb.Get(key)\n if metadata == nil || metadata.Expired() {\n metadata = createNewMetadata();\n }\n // 根據 metadata 裏面的版本組成 subkey\n subkey = key + metadata.version+field\n\tif rocksdb.Get(subkey) == nil {\n metadata.size += 1\n }\n // 寫入 subkey 以及更新 metadata\n rocksdb.Set(subkey, value)\n rocksdb.Set(key, metadata)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更多的數據結構設計可以參考: ","attrs":{}},{"type":"link","attrs":{"href":"http://link.zhihu.com/?target=https%3A//github.com/kvrockslabs/kvrocks/blob/unstable/docs/metadata-design.md","title":null,"type":null},"content":[{"type":"text","text":"https://github.com/kvrockslabs/kvrocks/blob/unstable/docs/metadata-design.md","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Kvrocks 是如何進行數據複製?","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kvocks 的定位是作爲大數據量場景下 Redis 的替代方案，提供與 Redis 一樣的數據最終一致性保證，採用了類似 Redis 的主從異步複製模型。考慮到需要應對更多的業務場景，後續會支持半同步複製模型。在實現上，全量複製利用 RocksDB 的 CheckPoint 特性，增量複製採用直接發送 WAL 的方式，從庫接收到WAL直接操作後端引擎。相比於 Binlog 複製方式（回放從客戶端接收到的命令），省去了命令解析和處理的開銷，複製速度大幅提升，這樣也就解決了其它採用 Binlog 複製方式的存儲服務所存在的複製延遲問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Kvrocks 是如何實現分佈式集羣？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業界常用Redis集羣方案主要有兩類：類似 Codis 中心化的集羣架構和社區 Redis Cluster 去中心化的集羣架構。Kvrocks 集羣方案選擇了類似 Codis 中心化的架構，集羣元數據存儲在配置中心，但不依賴代理層，配置中心爲存儲節點推送元數據，對外提供 Redis Cluster 集羣協議支持，對於使用 Redis Cluster SDK 或者 Proxy 的用戶不需要做任何修改。同時也可以避免類似Redis Cluster 受限於 Gossip 通信的開銷而導致集羣規模不能太大的問題。另外，單機版的 Kvrocks 和 Redis 一樣可以直接支持 Twmeproxy，通過Sentinel實現高可用，對於 Codis 通過簡單的適配也能夠比較快的支持。目前集羣方案處在測試階段，預計7月份發佈，待正式發佈後會給大家詳細介紹，這裏不過多展開。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"對於分佈式集羣來說，彈性伸縮的能力是必不可少的，Kvrocks 是如何實現彈性伸縮的？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整個擴縮容拆分爲遷移全量數據、遷移增量數據、變更拓撲三個階段。遷移全量數據利用 RocksDB的 Snapshot 特性，生成 Snapshot 迭代數據發送到目標節點。同時，爲了加快迭代效率數據編碼上Key 增加 SlotID 前綴。遷移增量數據階段直接發送 WAL。當待遷移的增量 WAL 小於設定的閾值則開始阻寫，等發送完剩餘的 WAL 切換拓撲之後解除阻寫，這個過程通常是毫秒級的。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"優化點","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相比內存型服務來說，最常見的問題是磁盤的吞吐和延時帶來的毛刺點問題。除了通過慢日誌命令來確認是否有慢請求產生之外，還提供了 perflog 命令用來定位 RocksDB 訪問慢的問題，使用方式如下:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"# 第一條命令設定只對 SET 命令收集 profiling 日誌\n# 第二條命令設定隨機採樣的比例\n# 第三條命令設定超過多長時間的命令才寫到 perf 日誌裏面(如果是爲了驗證功能可以設置爲 0)\n127.0.0.1:6666> config set profiling-sample-commands set\nOK\n127.0.0.1:6666> config set profiling-sample-ratio 100\nOK\n127.0.0.1:6666> config set profiling-sample-record-threshold-ms 1\nOK\n\n# 執行 Set 命令，在去看 perflog 命令就可以看到對應的耗時點\n127.0.0.1:6666> set a 1\nOK\n127.0.0.1:6666> perflog get 2\n1) 1) (integer) 1\n 2) (integer) 1623123739\n 3) \"set\"\n 4) (integer) 411\n 5) \"user_key_comparison_count = 7, write_wal_time = 122300, write_pre_and_post_process_time = 91867, write_memtable_time = 47349, write_scheduling_flushes_compactions_time = 13028\"\n 6) \"thread_pool_id = 4, bytes_written = 45, write_nanos = 46030, prepare_write_nanos = 21605\"\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之前通過這種方式發現了一些 RocksDB 參數配置不合理的問題，比如之前 SST 文件大小默認是 256MiB，當業務的 KV 比較小的時候可能會導致一個 SST 文件裏面可能有百萬級別的 KV，從而導致 index 數據塊過大(幾十 MiB)，每次從磁盤讀取數據需要耗費幾十 ms。但線上不同業務的 KV 大小可能會差異比較大，通過 DBA 手動調整的方式肯定不合理，所以有了根據寫入 KV 大小在線自動調整 SST 和 Block Size 的功能，具體描述見: ","attrs":{}},{"type":"link","attrs":{"href":"http://link.zhihu.com/?target=https%3A//github.com/kvrockslabs/kvrocks/issues/118","title":null,"type":null},"content":[{"type":"text","text":"https://github.com/kvrockslabs/kvrocks/issues/118","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外一個就是 RocksDB 的全量 Compact 導致磁盤 IO 從而造成業務訪問的毛刺點問題，之前策略是每天凌晨低峯時段進行一次，過於頻繁會導致訪問毛刺點，頻率過低會導致磁盤空間回收不及時。所以增加另外一種部分 Compact 策略，優先對那些比較老以及無效 KV 比較多的 SST進行 Compact。開啓只需要在配置文件裏面增加一行，那麼則會在凌晨 0 到 7 點之間去檢查這些 SST 文件並做 Compact","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"compaction-checker-range 0-7\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在設計和實現上，Kvrocks 更注重簡潔高效、穩定可靠、易於使用和問題定位。目前 Kvrocks 已經在線上大規模運行兩年之久，基本功能已充分驗證，大家可以放心使用。如遇到問題，大家可以在微信羣，Slack(見 GitHub README)，Issue 上反饋和交流，我們也歡迎提 PR 來一起完善 Kvrocks。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在社區維護上，希望可以有更加開放的交流氛圍，而不只是把代碼放到 GitHub 的開源。不管是功能設計還是代碼開發，都會盡量把相關細節都在 GitHub 裏面公開去討論。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，2.0 版本預計在 7-8 月份會完成全部功能的開發，大家可以期待一下（具體進展見: ","attrs":{}},{"type":"link","attrs":{"href":"http://link.zhihu.com/?target=https%3A//github.com/kvrockslabs/kvrocks/projects/","title":null,"type":null},"content":[{"type":"text","text":"https://github.com/kvrockslabs/kvrocks/projects","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"歡迎大家掃碼關注 ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"「Kvrocks Community」","attrs":{}},{"type":"text","text":"公衆號並回復: ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"進羣","attrs":{}},{"type":"text","text":"，來加入我們的微信羣！","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/9c/9c702b9268d770d2ed26b60e5309de2f.png","alt":"Image","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}