ClickHouse在京東能源管理平臺的應用實踐

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClickHouse是一款面向大數據場景下的OLAP數據庫，相比於傳統的基於Hadoop生態圈的OLAP大數據分析系統，ClickHouse具有極致的查詢性能、輕量級的架構設計及維護簡單等優勢。目前社區活躍度高，業界應用實踐日趨廣泛。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、業務介紹"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"京東能源管理平臺是京東科技IoT產品部面向政企客戶推出的一款利用物聯網、大數據和AI技術實現用能企事業單位對能源大數據進行採集、監測、分析和告警的能耗分析產品，旨在幫助客戶實現節能減排，降低單位產品能耗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/28\/28ffb8e0dfff47e193213ff7740affa5.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"能源指標包括用電量、用水量和用天然氣量，維度有時間維度（年、月、周、日、時）、廠家、車間、生產線類型、生產線、設備。針對這些指標和維度，提供了實時的數據多維分析與診斷服務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、技術選型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於數據指標的多維度分析場景，上世紀業界就提出了BI（商業智能）的概念。相較於OLTP（聯機事務）系統，業界把此類面向BI的系統統稱爲OLAP(聯機分析)系統。伴隨着計算機軟件技術的發展、從單機工具的少量數據分析（如Excel），到中等規模數據通過分析型關係數據庫構建（如微軟的SSAS）的OLAP，再到今日的大數據時代，海量數據的實時OLAP分析引擎，技術上的推陳出新，工具系統上百花齊放百家爭鳴，各有優勢，但大體上可以將它們從架構模式上劃分爲兩大類："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"strong"}],"text":"1. MPP架構。"},{"type":"text","text":"MPP架構特點是服務將接收到的查詢請求發送到每個計算節點，待計算節點計算完成後，通過一個節點將最終結果彙總在一起得到最終結果。典型實現如Presto、Impala、SparkSQL、Drill等。MPP架構的特點是支持靈活的數據模型，要達到較高性能對內存開銷大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"strong"}],"text":"2. 預計算系統。"},{"type":"text","text":"預計算的核心思想是利用空間換時間，通過深入業務理解，將需要查詢的數據指標和維度組合進行預處理，將計算好的結果存入數據庫並建立對應索引，實現查詢加速。典型實現如Kylin、Druid。預計算系統特點是性能較高，但靈活性較差，一般對數據模型調整會涉及到歷史數據的重跑，維護困難。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/42\/4296041f736d8ded56e514033f047c88.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上表可知，目前業界還沒有一個OLAP引擎能夠同時兼顧性能和靈活性的要求，京東能源管理平臺在做技術選型的時候，綜合考慮了模型的靈活性、部署的難易程度、開發成本、可維護性以及是否適合雲端部署等因素，最終決定使用基於MPP架構的ClickHouse作爲我們的OLAP引擎。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、ClickHouse的應用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1、系統架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"京東能源管理平臺主要是對各種表計（水錶、電錶、天然氣表等）設備上報的計數進行多維度分析統計、AI診斷和出具能耗報表等。表計的原始數據通常都是累計值，如電量度數就是一個從電錶安裝以來，所有耗電量的一個累計。因此，我們在數據接入前會引入一個差分器對數據進行預處理，使得進入ClickHouse的指標數據變成可直接累加的指標，方便利用SQL對接ClickHouse實現多維的查詢服務。架構圖如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/1a\/32\/1a5122d96f4648fe8847374ba063a632.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"說明："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"物管平臺："},{"type":"text","text":"對設備的管理，管理物模型及設備狀態、採集設備數據。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"消息總線："},{"type":"text","text":"kafka消息隊列，利用JSON格式數據實現物管平臺和能平臺的數據交互。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"差分器："},{"type":"text","text":"對每次上報的累計值同上一次上報的累計值做差值計算，得到可累加指標。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"異常規則鏈："},{"type":"text","text":"提供一個異常規則集，用於差分器判定上報數據是否異常，如異常則進行記錄，數據不作處理。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"OLAP引擎："},{"type":"text","text":"基於ClickHouse實現的OLAP引擎。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"多維分析服務："},{"type":"text","text":"提供通用的數據多維分析查詢服務，能夠通過統一的API實現各種維度和指標的組合查詢。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"政府和企業界面："},{"type":"text","text":"政企客戶的WEB界面。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2、ClickHouse應用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過上面的架構圖可以看出，能源平臺採用ClickHouse作爲OLAP引擎提供多維查詢服務。下面重點從數據的接入、存儲以及通用化接口設計方面談一談"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClickHouse的應用："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"數據接入"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClickHouse基於kafka引擎表的數據接入可以看做是一個典型的ETL過程，數據的抽取（Extract）是通過建立一張kafka引擎表，產生消費端訂閱kafka topic實現；數據的轉換（Transform）通過物化視圖實現；數據最終加載（Load）進MergeTree表，實現實際數據存儲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/56\/561e39285a93601db729da7a4afbbf2c.webp","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"創建Kafka表示例："}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":" 1CREATE TABLE statistics_kafka ON CLUSTER '{cluster}' (\n 2  timestamp UInt64,\n 3  level String,\n 4  message String\n 5 ) ENGINE = Kafka SETTINGS kafka_broker_list = 'kafka.jd.com:9092',\n 6                         kafka_topic_list = 'statistics',\n 7                         kafka_group_name = 'gp-st',\n 8                         kafka_format = 'JSONEachRow',\n 9                         kafka_skip_broken_messages = 1, \n10                         kafka_num_consumers = 3;\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":""}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kafka_broker_list: kafka broker地址。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kafka_topic_list：消費的topic。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kafka_group_name：消費groupId。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kafka_format：數據格式JSONEachRow表示消息體爲JSON格式。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kafka_skip_broken_messages：表示忽略的kafka異常消息條數，默認爲0。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kafka_num_consumers：消費者個數，默認值爲1，建議同kafka分區數對應。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"創建物化視圖示例："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"1CREATE MATERIALIZED VIEW statistics_view ON CLUSTER '{cluster}' TO statistics_replica AS\n2SELECT timestamp,\n3       level,\n4       message\n5FROM statistics_kafka;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":""}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"創建MergeTree引擎表示例："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"1CREATE TABLE statistics_replica ON CLUSTER '{cluster}'{\n2 timestamp UInt64,\n3 dt String,\n4 deviceId String,\n5 level String,\n6 message String\n7} ENGINE = ReplicatedMergeTree('\/clickhouse\/tables\/{shard}\/statistics_replica','{replica}')\n8PARTITION BY dt\n9ORDER BY (dt,deviceId,level);"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"存儲"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"a. ClickHouse表類型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本地表：實際數據存儲的表，如上示例表statistics_replica。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分佈式表：一個邏輯上的表, 可以理解爲數據庫中的視圖, 一般查詢都查詢分佈式表. 分佈式表引擎會將我們的查詢請求路由本地表進行查詢, 然後進行彙總最終返回給用戶。創建分佈式表示例："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"1CREATE TABLE statistics ON CLUSTER '{cluster}' AS statistics_replica 2ENGINE = Distributed(ck_cluster_1,test,events_local,rand());"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"b. Replication和Sharding"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Replication是ClickHouse提供的副本機制，對於Replicated MergeTree 系列複製表，可以設置每個表有多份完全一樣的數據存放在不同的計算節點上，每一份數據都是完整的，並且稱爲一個副本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Shard：將表中的數據按照一定的規則拆分爲多個部分，每個部分的數據均存儲在不同的計算節點上，每個計算節點上的數據稱爲一個分片。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/5b\/5b202545a802d19219190a4909713926.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClickHouse基於Replicated MergeTree引擎與Zookeeper實現了複製表機制，在創建表時，可以決定表是否高可用。上一節的statistics_replica表，其中\/clickhouse\/tables\/{shard}\/statistics_replica表示Zookeeper中對應副本表的node。當數據寫入ReplicatedMergeTree表時，過程如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/36\/369e96e594cccd6e7692f617879ceb3f.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"某一個ClickHouse節點接收到數據寫入請求。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"通過interserver HTTP port端口同步到其他實例。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"更新Zookeeper集羣上的node信息。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3、OLAP通用接口設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClickHouse提供標準的SQL查詢引擎，通過JDBC引用程序可以實現多ClickHouse的基本操作。OLAP的常規操作如上卷、下鑽和切片會涉及到多種維度自由組合、多種指標交叉剖析的過程，如果服務端採用Mybatis或JPA等常規ORM操作，工程師很容易根據不同的查詢場景要求設計出對應的接口，亦或是根據大量的分支操作設計出複雜的判定性接口，鑑於此，作者從mdx思想獲得啓示，設計一套對OLAP優化的通用多維服務查詢接口。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，一個典型的分析類SQL語句如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":" 1SELECT day_str,\n 2       factory_name,\n 3       workshop_name,\n 4       prodline_name,\n 5       device_id,\n 6       SUM(w_total) AS total\n 7FROM statistics\n 8WHERE day_str BETWEEN '2020-10-01' AND '2020-12-31'\n 9GROUP BY day_str,factory_name,workshop_name,prodline_name,device_id\n10ORDER BY day_str ASC;\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":""}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上語句，我們翻譯成業務語言爲『分別查詢2020年4季度全廠所有設備的耗電量』，從這裏我們可以清楚的知道這裏的維度是指『設備名稱』，指標爲『耗電量』，基於此，可以進一步歸類，維度通常出現在SQL語句的SELECT、WHERE、GROUP BY和ORDER BY後面，指標則通常出現在SELECT後面，也就是可以總結如下模式："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"1SELECT {維度},{指標}\n2FROM table_name\n3WHERE {維度}='xxx'\n4GROUP BY {維度}\n5ORDER BY {維度};\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，我們可以設計如下通用接口方法："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"1\/\/通用方法\n2List