RocketMQ 在使用上的一些排坑和優化

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"前言：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ 在我們的項目中使用非常廣泛，在使用的過程中，也遇到了很多的問題。比如沒有多環境的隔離，在多個版本同時開發送測的情況下，互相干擾嚴重。RocketMQ 的投遞可能會失敗，導致丟失消息。另外開源版本的 RocketMQ 不支持任意時間精度的延時消息，僅支持特定的 level。在使用的過程中，我們做了一些針對性的優化，整理出了這篇文章。通過閱讀這篇文章，你會了解到這些知識","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ 多環境隔離方案嘗試基於 RocksDB 的消息“可靠”投遞方案基於 RocksDB 和 RocketMQ 實現任意延時的時延消息","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"RocketMQ 多環境隔離","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲我們有很多功能需求會並行開發和送測，開發和測試的環境各有三四套之多，假設現在我們有三個版本在同時開發，對於同一個 topic，dev1 開發環境產生的消息可能會被 dev3 開發環境消費，這兩個環境消費端的代碼可能不一致，造成沒有辦法完成這部分功能的測試，這種情況下，開發人員苦不堪言，經常需要去下線掉其它環境的消費端才能繼續進行開發測試，如下圖所示。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5d/5d78f5f48914e7f284a6cc1271e86111.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了解決這個問題，一開始是想在 topic 上下功夫，通過修改 Producer 端，讓每個環境的 topic 統一加一個環境後綴，這樣 topic_ABC 在 dev1 環境就會變爲 topic_ABC_dev1。這種方式理論上也可以解決，只是需要創建較多 topic，代價比較高，改動量大。後面採用的方案是給每個環境分配獨立的 RocketMQ 隊列來實現，下面爲了講述的簡單起見，這裏只給每個環境分配一個隊列，如下所示。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/38/384974bf7f61e678774c039e401cdfd7.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"通過環境變量的區分","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在生產端：dev1 環境投遞到 RocketMQ 第 0 號隊列，dev2 環境投遞到第 1 號隊列，後面以此類推在消費端：dev1 環境只拉取 RocketMQ 第 0 號隊列的消息，dev2 環境只第 1 號隊列的消息，後面以此類推","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"生產端實現","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ 的消息投遞提供了 MessageQueueSelector 接口可以自定義消息隊列選擇器，指定消息要投遞的 queue，它的定義如下所示","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"public interface MessageQueueSelector {\n MessageQueue select(final List mqs, final Message msg, final Object arg);\n}\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中 mqs 參數是當前 topic 的所有可用隊列，返回值是此次要投遞的 queue。它有下面這個幾個實現類：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SelectMessageQueueByHash：使用 msg 參數的 hashcode 的絕對值與 queue 大小取模SelectMessageQueueByRandom：調用 Random.nextInt 方法獲取一個 0~mqs.size()-1 區間的隨機數SelectMessageQueueByMachineRoom：實現爲空","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於我們的場景，這裏簡化處理，根據環境的編號直接映射 queue，生產端的示例代碼如下所示","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"DefaultMQProducer producer = // ...;\n\nfinal int envIndex = getEnvIndex();\nSendResult sendResult = producer.send(message, new MessageQueueSelector() {\n @Override\n public MessageQueue select(List mqs, Message msg, Object arg) {\n return mqs.get(envIndex-1); \n }\n}, envIndex);\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣 dev1 環境映射到第 0 個隊列，dev3 環境映射到第 2 個隊列。消費端實現對於消費端，RocketMQ 定義了 AllocateMessageQueueStrategy 策略接口，可以自己實現當前消費者可以消費哪些 queue 隊列。AllocateMessageQueueStrategy 接口的定義如下所示","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"public interface AllocateMessageQueueStrategy {\n\n /**\n * Allocating by consumer id\n *\n * @param consumerGroup 當前 consumer group\n * @param currentCID 當前 consumer id\n * @param mqAll 當前 topic 的所有 queue 列表\n * @param cidAll 當前 consumer group 下所有的 consumer id set 集合\n * @return 根據策略給當前 consumer 分配的 queue 列表\n */\n List allocate(\n final String consumerGroup,\n final String currentCID,\n final List mqAll,\n final List cidAll\n );\n\n /**\n * 策略算法名\n */\n String getName();\n}\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ 內置提供了下面這些分配算法","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"AllocateMessageQueueAveragely：平均分配算法","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"AllocateMessageQueueAveragelyByCircle：按照 queue 隊列組成的環形逐個分配","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"AllocateMachineRoomNearby：基於機房臨近原則算法","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"AllocateMessageQueueByMachineRoom：基於機房分配算法","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"AllocateMessageQueueConsistentHash：基於一致性 hash 算法，將 consumer 消費者作爲 Node 節點 hash 到一個虛擬環上","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"AllocateMessageQueueByConfig：基於配置分配算法，沒有什麼作用，可以作爲 example 擴展","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於我們的場景，這裏簡化處理，根據環境的編號直接映射 queue，消費端的代碼如下所示","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(MQConstant.MQ_CONSUMER_GROUP_NAME, null,\n new AllocateMessageQueueStrategy() {\n @Override\n public List allocate(String consumerGroup, String currentCID, List mqAll, List cidAll) {\n List list = new ArrayList<>();\n list.add(mqAll.get(envIndex-1));\n return list;\n }\n\n @Override\n public String getName() {\n return \"env-based\";\n }\n });\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"利弊分析","attrs":{}},{"type":"text","text":"這種方式的實現非常簡單，客戶端改動量非常小，不用修改 topic。如果你的環境數量比較固定，可以修改上面的策略，讓一個環境可以使用固定的多個 queue，只要保證多個環境不使用同一個 queue 接口即可。如果開發測試環境的消息數量不多，用一個隊列也問題不大。線上生產環境多機房、多環境也可以用類似的思路去實現。到這裏多環境隔離的介紹就告一段落。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"消息丟失之傷","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocketMQ 本身是一個服務端，當然就會有服務不可用、服務繁忙等問題，尤其是我們的公司所有的業務共用一個 RocketMQ，時不時會出現 “system busy , start flow control for a while” 等投遞異常問題。爲了解決投遞可靠性的問題，一開始是想在投遞異常的時候將消息寫入到數據庫等持久化存儲中，然後有一個定時任務去補償消費。這種方案看起來是比較完美的，但是當 RocketMQ 整體不可用，大量的消息都投遞失敗時，數據庫的瞬間寫入壓力會非常大，這種方案沒有被採用。後面想到了使用 RocksDB 來曲線救國","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"主角 RocksDB","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a4/a4ddc6c7ca3e12ea9ec64ab81fb8e591.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"RocksDB 是 Facebook 基於 Google Jeff Dean 寫的 LevelDB 改進的一種嵌入式 key-value 存儲系統，做了許多優化，性能相對 LevelDB 有了很大的提升，大名鼎鼎的 TiDB 底層的存儲引擎就是使用的 RocksDB。RocksDB 是一個基於 LSM 樹的存儲引擎，LSM 是 Log-structured merge-tree 的縮寫，關於 RocksDB 的底層原理，這篇文章不展開說明，有機會我會詳細寫一下。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"基於 RocksDB 的重試機制","attrs":{}},{"type":"text","text":"核心的邏輯是投遞失敗以後，將消息寫入到本地 RocksDB 存儲中，然後有一個線程去輪詢是否有消息，如果有則進行重試，如果再次投遞失敗會重新將消息寫入到 RocksDB，過程如下圖所示","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/be/bee35d413e24d6afc39e041a45c788fa.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在實現上，寫入 RocksDB 的 key 採用瞭如下的格式：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"expireTime:retryCount:typeName:uuid\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中 expireTime 的生成邏輯爲當前時間戳（到秒）+ 投遞延遲時間，代碼如下所示：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"val RETRY_TIME_STEP_ARRAY = arrayOf(\n 3, 5, 30, 60, 120, 300, 480, 600, 900, 1800\n)\n\nval expire = System.currentTimeMillis() / 1000 + (RETRY_TIME_STEP_ARRAY.getOrNull(retryCount) ?: 10)\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當消息投遞到 MQ 失敗時，將其寫入到 RocksDB，這部分代碼如下所示","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"private fun insert(msg: ByteArray, retryCount: Int, typeName: String) {\n val key = genKey(retryCount, typeName)\n rocksDB.put(mqRetryCFHandler, WRITE_OPTIONS_SYNC, key.toByteArray(), msg)\n}\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"master 線程負責輪詢 RocksDB，如果有記錄將其查出來放入一個 blockingQueue 中，master 線程核心邏輯如下所示","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"private var lastSeekTime: Long = 0 // 單調遞增的值，初始值爲當前時間戳（到秒）\n\nprivate fun loop() {\n val now = // 當前時間戳，到秒\n if (lastSeekTime > now) { // 如果時鐘回撥或者還沒到處理時間片，睡眠一段時間\n TimeUnit.MILLISECONDS.sleep(400)\n return\n }\n\n rocksDB.newIterator(mqRetryCFHandler, READ_OPTIONS).use {\n it.seek(\"$lastSeekTime\".toByteArray()) // seek 到以 lastSeekTime 開頭的 key 的地方\n while (it.isValid) {\n val value = it.value()\n blockingQueue.put(String(it.key()) to value) // 放入一個固定大小的阻塞隊列中\n it.next()\n }\n }\n ++lastSeekTime\n}\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"worker 線程負責消息的重新投遞，代碼如下所示","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"private fun startConsume() {\n repeat(THREAD_NUM) {\n thread {\n while (true) {\n val list = drain() // 批量從 blockingQueue 中取數據\n list.forEach {\n try {\n val typeName = getTypeName(it.first)\n val handler = getHandler(typeName) ?: return@forEach\n val success = handler.handler(it.second)\n // 如果不成功，則重新寫入 RocksDB\n if (!success) {\n val currentRetryCount = getRetryCountFromKey(it.first) + 1\n val maxRetryCount = handler.retryCount\n if (currentRetryCount >= RETRY_TIME_STEP_ARRAY_SIZE || currentRetryCount >= maxRetryCount) {\n val msgString = getStringFromBytes(it.second)\n logger.info(\"send reach limit, retry count:$currentRetryCount,default count:$RETRY_TIME_STEP_ARRAY_SIZE,custom count:$maxRetryCount, msg: $msgString\")\n exceptionHandle.handler(\"retry $currentRetryCount fail,msg:$msgString\")\n return@forEach\n }\n insert(it.second, currentRetryCount, typeName)\n }\n } catch (ex: Throwable) {\n exceptionHandle.handler(\"key: $it.first ,error: ${ex.message}\")\n Thread.sleep(30)\n }\n }\n }\n }\n }\n}\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過上面的這幾步改造，在過去大半年內成功的躲過了好幾次 RocketMQ 的短時間故障，消息沒有丟失，全部重試成功，沒有造成數據的異常。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"利弊分析","attrs":{}},{"type":"text","text":"這個方案的優點是很輕量化，寫入讀取本地 RocksDB 速度都極快，在極端場景下性能幾乎沒有影響。但也有一個缺點需要考慮，因爲沒有落地到集中式存儲比如 MySQL，如果項目部署到 Docker 容器中，容器重啓以後，這部分重試的數據還是會丟失。使用這種方案沒有辦法保證百分百不丟數據，考慮到 mq 故障發生的並不頻繁，在性能和丟數據中取得一個平衡也是一種可行的措施。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"基於 RocksDB 的任意延時消息設計","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在做完上面的“可靠投遞”方案以後，衍生出另外一個解決方案，使用 RocksDB 來實現任意時延的延時消息隊列，它的設計目標有三個：支持任意時延充分利用現有的基礎設施需要能無限堆積，寫入查詢效率要求要高","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"於是基於 RocksDB，我們實現了一個內部稱爲 Rock-DMQ 的項目，名字來源是 RocksDB for Delay MQ。它的實現原理也非常簡單，如下圖所示。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a8/a8aad4b9c460a8ceda17d2a927058484.png","alt":"image.png","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在投遞一個延時消息時，以 topic 爲 “cancel_order” 爲例，整個延時消息的實現邏輯如下所示。1、通過修改 Producer 端，實際投遞到 RocketMQ 的 topic 不是這個，而是替換爲了一個統一的 topic，名爲 dmq_inner_topic，原始 topic 被轉爲 body 的一部分。2、Rock-DMQ 項目會消費 dmq_inner_topic 這個特殊的 topic3、消費 dmq_inner_topic 的消息後，Rock-DMQ 項目會將其寫入到本地的 RocksDB 中，key 爲到期時間爲前綴（這一點比較重要）4、Rock-DMQ 項目採用文中第二部分的內容相似的實現方式，隔一段時間去輪詢 RocksDB ，看有無到期的消息5、如果有到期消息，Rock-DMQ 項目將這個消息投遞到 RocketMQ 中6、訂閱了這個 topic 的原有消費端就可以消費到這條消息了通過這種實現，可以實現支持任意秒數的時延消息，也比較好的複用了現有的技術組件，對 RocketMQ 本身無任何改動，在水平擴展性上也得到了比較好的支持。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"核心代碼在第二部分已經介紹，這裏不再贅述。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"小結","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上就是 RocketMQ 在我們這邊的落地實踐和填坑記錄，這些方案都還在快速迭代優化中，如果你有更好的想法，可以一起溝通交流~","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"看完三件事❤️","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你覺得這篇內容對你還蠻有幫助，我想邀請你幫我三個小忙：","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"點贊，轉發，有你們的 『點贊和評論』，纔是我創造的動力。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關注公衆號 『 java 爛豬皮 』，不定期分享原創知識。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時可以期待後續文章 ing🚀","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關注後回覆【666】掃碼即可獲取學習資料包","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/68/68a35532863af30e801cac6e061f5853.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文作者：挖坑的張師傅","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"文章出處：https://club.perfma.com/article/2331590","attrs":{}}]}],"attrs":{}}]}