ClickHouse在京东能源管理平台的应用实践

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClickHouse是一款面向大数据场景下的OLAP数据库，相比于传统的基于Hadoop生态圈的OLAP大数据分析系统，ClickHouse具有极致的查询性能、轻量级的架构设计及维护简单等优势。目前社区活跃度高，业界应用实践日趋广泛。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、业务介绍"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"京东能源管理平台是京东科技IoT产品部面向政企客户推出的一款利用物联网、大数据和AI技术实现用能企事业单位对能源大数据进行采集、监测、分析和告警的能耗分析产品，旨在帮助客户实现节能减排，降低单位产品能耗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/28\/28ffb8e0dfff47e193213ff7740affa5.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"能源指标包括用电量、用水量和用天然气量，维度有时间维度（年、月、周、日、时）、厂家、车间、生产线类型、生产线、设备。针对这些指标和维度，提供了实时的数据多维分析与诊断服务。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、技术选型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于数据指标的多维度分析场景，上世纪业界就提出了BI（商业智能）的概念。相较于OLTP（联机事务）系统，业界把此类面向BI的系统统称为OLAP(联机分析)系统。伴随着计算机软件技术的发展、从单机工具的少量数据分析（如Excel），到中等规模数据通过分析型关系数据库构建（如微软的SSAS）的OLAP，再到今日的大数据时代，海量数据的实时OLAP分析引擎，技术上的推陈出新，工具系统上百花齐放百家争鸣，各有优势，但大体上可以将它们从架构模式上划分为两大类："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"strong"}],"text":"1. MPP架构。"},{"type":"text","text":"MPP架构特点是服务将接收到的查询请求发送到每个计算节点，待计算节点计算完成后，通过一个节点将最终结果汇总在一起得到最终结果。典型实现如Presto、Impala、SparkSQL、Drill等。MPP架构的特点是支持灵活的数据模型，要达到较高性能对内存开销大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"strong"}],"text":"2. 预计算系统。"},{"type":"text","text":"预计算的核心思想是利用空间换时间，通过深入业务理解，将需要查询的数据指标和维度组合进行预处理，将计算好的结果存入数据库并建立对应索引，实现查询加速。典型实现如Kylin、Druid。预计算系统特点是性能较高，但灵活性较差，一般对数据模型调整会涉及到历史数据的重跑，维护困难。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/42\/4296041f736d8ded56e514033f047c88.webp","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从上表可知，目前业界还没有一个OLAP引擎能够同时兼顾性能和灵活性的要求，京东能源管理平台在做技术选型的时候，综合考虑了模型的灵活性、部署的难易程度、开发成本、可维护性以及是否适合云端部署等因素，最终决定使用基于MPP架构的ClickHouse作为我们的OLAP引擎。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、ClickHouse的应用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1、系统架构"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"京东能源管理平台主要是对各种表计（水表、电表、天然气表等）设备上报的计数进行多维度分析统计、AI诊断和出具能耗报表等。表计的原始数据通常都是累计值，如电量度数就是一个从电表安装以来，所有耗电量的一个累计。因此，我们在数据接入前会引入一个差分器对数据进行预处理，使得进入ClickHouse的指标数据变成可直接累加的指标，方便利用SQL对接ClickHouse实现多维的查询服务。架构图如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/1a\/32\/1a5122d96f4648fe8847374ba063a632.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"说明："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"物管平台："},{"type":"text","text":"对设备的管理，管理物模型及设备状态、采集设备数据。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"消息总线："},{"type":"text","text":"kafka消息队列，利用JSON格式数据实现物管平台和能平台的数据交互。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"差分器："},{"type":"text","text":"对每次上报的累计值同上一次上报的累计值做差值计算，得到可累加指标。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"异常规则链："},{"type":"text","text":"提供一个异常规则集，用于差分器判定上报数据是否异常，如异常则进行记录，数据不作处理。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"OLAP引擎："},{"type":"text","text":"基于ClickHouse实现的OLAP引擎。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"多维分析服务："},{"type":"text","text":"提供通用的数据多维分析查询服务，能够通过统一的API实现各种维度和指标的组合查询。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"政府和企业界面："},{"type":"text","text":"政企客户的WEB界面。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2、ClickHouse应用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过上面的架构图可以看出，能源平台采用ClickHouse作为OLAP引擎提供多维查询服务。下面重点从数据的接入、存储以及通用化接口设计方面谈一谈"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClickHouse的应用："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"数据接入"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClickHouse基于kafka引擎表的数据接入可以看做是一个典型的ETL过程，数据的抽取（Extract）是通过建立一张kafka引擎表，产生消费端订阅kafka topic实现；数据的转换（Transform）通过物化视图实现；数据最终加载（Load）进MergeTree表，实现实际数据存储。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/56\/561e39285a93601db729da7a4afbbf2c.webp","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"创建Kafka表示例："}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":" 1CREATE TABLE statistics_kafka ON CLUSTER '{cluster}' (\n 2  timestamp UInt64,\n 3  level String,\n 4  message String\n 5 ) ENGINE = Kafka SETTINGS kafka_broker_list = 'kafka.jd.com:9092',\n 6                         kafka_topic_list = 'statistics',\n 7                         kafka_group_name = 'gp-st',\n 8                         kafka_format = 'JSONEachRow',\n 9                         kafka_skip_broken_messages = 1, \n10                         kafka_num_consumers = 3;\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":""}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kafka_broker_list: kafka broker地址。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kafka_topic_list：消费的topic。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kafka_group_name：消费groupId。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kafka_format：数据格式JSONEachRow表示消息体为JSON格式。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kafka_skip_broken_messages：表示忽略的kafka异常消息条数，默认为0。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"kafka_num_consumers：消费者个数，默认值为1，建议同kafka分区数对应。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"创建物化视图示例："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"1CREATE MATERIALIZED VIEW statistics_view ON CLUSTER '{cluster}' TO statistics_replica AS\n2SELECT timestamp,\n3       level,\n4       message\n5FROM statistics_kafka;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":""}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"创建MergeTree引擎表示例："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"1CREATE TABLE statistics_replica ON CLUSTER '{cluster}'{\n2 timestamp UInt64,\n3 dt String,\n4 deviceId String,\n5 level String,\n6 message String\n7} ENGINE = ReplicatedMergeTree('\/clickhouse\/tables\/{shard}\/statistics_replica','{replica}')\n8PARTITION BY dt\n9ORDER BY (dt,deviceId,level);"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"存储"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"a. ClickHouse表类型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本地表：实际数据存储的表，如上示例表statistics_replica。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分布式表：一个逻辑上的表, 可以理解为数据库中的视图, 一般查询都查询分布式表. 分布式表引擎会将我们的查询请求路由本地表进行查询, 然后进行汇总最终返回给用户。创建分布式表示例："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":"1CREATE TABLE statistics ON CLUSTER '{cluster}' AS statistics_replica 2ENGINE = Distributed(ck_cluster_1,test,events_local,rand());"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"b. Replication和Sharding"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Replication是ClickHouse提供的副本机制，对于Replicated MergeTree 系列复制表，可以设置每个表有多份完全一样的数据存放在不同的计算节点上，每一份数据都是完整的，并且称为一个副本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Shard：将表中的数据按照一定的规则拆分为多个部分，每个部分的数据均存储在不同的计算节点上，每个计算节点上的数据称为一个分片。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/5b\/5b202545a802d19219190a4909713926.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClickHouse基于Replicated MergeTree引擎与Zookeeper实现了复制表机制，在创建表时，可以决定表是否高可用。上一节的statistics_replica表，其中\/clickhouse\/tables\/{shard}\/statistics_replica表示Zookeeper中对应副本表的node。当数据写入ReplicatedMergeTree表时，过程如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/36\/369e96e594cccd6e7692f617879ceb3f.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"某一个ClickHouse节点接收到数据写入请求。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"通过interserver HTTP port端口同步到其他实例。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"更新Zookeeper集群上的node信息。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3、OLAP通用接口设计"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClickHouse提供标准的SQL查询引擎，通过JDBC引用程序可以实现多ClickHouse的基本操作。OLAP的常规操作如上卷、下钻和切片会涉及到多种维度自由组合、多种指标交叉剖析的过程，如果服务端采用Mybatis或JPA等常规ORM操作，工程师很容易根据不同的查询场景要求设计出对应的接口，亦或是根据大量的分支操作设计出复杂的判定性接口，鉴于此，作者从mdx思想获得启示，设计一套对OLAP优化的通用多维服务查询接口。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，一个典型的分析类SQL语句如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":" 1SELECT day_str,\n 2       factory_name,\n 3       workshop_name,\n 4       prodline_name,\n 5       device_id,\n 6       SUM(w_total) AS total\n 7FROM statistics\n 8WHERE day_str BETWEEN '2020-10-01' AND '2020-12-31'\n 9GROUP BY day_str,factory_name,workshop_name,prodline_name,device_id\n10ORDER BY day_str ASC;\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":""}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上语句，我们翻译成业务语言为『分别查询2020年4季度全厂所有设备的耗电量』，从这里我们可以清楚的知道这里的维度是指『设备名称』，指标为『耗电量』，基于此，可以进一步归类，维度通常出现在SQL语句的SELECT、WHERE、GROUP BY和ORDER BY后面，指标则通常出现在SELECT后面，也就是可以总结如下模式："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"1SELECT {维度},{指标}\n2FROM table_name\n3WHERE {维度}='xxx'\n4GROUP BY {维度}\n5ORDER BY {维度};\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，我们可以设计如下通用接口方法："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"1\/\/通用方法\n2List