Clickhouse Projection 特性探索

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"年初的clickhouse meetup上快手团队分享了clickhouse projection在其公司内部的实践。分享包括了projection原理、使用、性能测试等内容。从性能测试的数据上看,projeciton对查询性能有着百倍级别的提升,意味着之前分钟级的查询响应延迟,将会提升到秒级响应。秒级的查询响应延迟,将会提升到毫秒级的响应,对于使用者将会有更加完美的体验。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"看完了快手同学的clickhouse projeciton的分享,在我脑中也产生了几个问题?","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"没有projection功能之前,clickhouse还存在什么问题?","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"clickhouse projection如何解决的问题?","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"clickhouse projection适用于哪些场景?","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"clickhouse proejction有什么要注意的吗?","attrs":{}}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"没有projection功能之前,clickhouse还存在什么问题?","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"clickhosue作为一款olap引擎,处于数据平台中的最顶层,直接对接平台用户。查询性能的好坏,直接决定着用户的使用体验。","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"clickhouse的查询性能虽然已经非常完美,但是面对超大数据量的场景还是会存在一定的问题,原因是clickhouse是基于内存计算的MPP架构分析型数据库,与Spark, Hive, MR等计算框架不同,计算 过程中的临时数据没有磁盘选项。查询过程中,数据会加载到内存中。如果内存配置不够,将会导致查询失败,对clickhouse集群的稳定也会有一定的影响。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"用户在数据查询的场景中,会有着一定的使用习惯。比如,每天定时都会查看一些特定的图表。这些图表中包含全量的数据统计,复杂的数据查询逻辑等。这些查询相较于其他查询,可以归属于异常查询。这些查询可能因为内存问题导致查询失败,也可能因为复杂的计算逻辑导致查询时间过长,影响平台上其他用户的查询。","attrs":{}}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"clickhouse projection如何解决的问题?","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在OLAP领域中,根据数据模型主要分为ROLAP(Relational OLAP) 关系OLAP,MOLAP(Multidimension OLAP) 多维OLAP 两种。ROLAP将数据表达为二维关系模型,类似关系型数据库模型,数据表达能力较好,对外提供SQL接口。MOLAP将OLAP分析所用到的多维数据物理上存储为多维数组的形式,形成“立方体”的结构。维的属性值被映射成多维数组的下标值或下标的范围,而汇总数据作为多维数组的值存储在数组的单元中,采用预聚合的思想,加速数据查询,但是数据模型不够灵活。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"clickhouse作为ROLAP典型代表之一,纯列式存储单表查询性能几乎没有对手。projection 名字起源于vertica,相当于传统意义上的物化视图。它借鉴 MOLAP 预聚合的思想,在数据写入的时候,根据 projection 定义的表达式,计算写入数据的聚合数据同原始数据一并写入。数据查询的过程中,如果查询SQL通过分析可以通过聚合数据得出,直接查询聚合数据减少计算的开销,解决了由于数据量导致的内存问题。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"projeciton 底层存储上属于part目录下数据的扩充,可以理解为查询索引的一种形式。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从数据写入逻辑的核心代码上看(clickhouse version 21.7),多个projection在part目录下以多个子目录存储,projection目录下存储基于原始数据聚合的数据。所以,projection写入与原始数据写入同步,只有创建projection之后写入的数据才会被物化,保证数据的一致性。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"cpp"},"content":[{"type":"text","text":"MergeTreeDataWrite.cpp.390\n\n如果存在projection配置,将projection part添加new_data_part中。\nif (metadata_snapshot->hasProjections())\n{\n for (const auto & projection : metadata_snapshot->getProjections())\n {\n /// 1. 获取projection query的执行计划。\n /// 2. 当前Block作为输入,计算聚合结果\n /// 3. 获取数据流\n auto in = InterpreterSelectQuery(\n projection.query_ast,\n context,\n Pipe(std::make_shared(block, Chunk(block.getColumns(), block.rows()))),\n SelectQueryOptions{\n projection.type == ProjectionDescription::Type::Normal ? QueryProcessingStage::FetchColumns : QueryProcessingStage::WithMergeableState})\n .execute()\n .getInputStream();\n in = std::make_shared(in, block.rows(), std::numeric_limits::max());\n in->readPrefix();\n // 4. 读取prjeciton计算的数据块\n auto projection_block = in->read();\n if (in->read())\n throw Exception(\"Projection cannot grow block rows\", ErrorCodes::LOGICAL_ERROR);\n in->readSuffix();\n if (projection_block.rows())\n {\n // 5. 将聚合的数据(.proj)添加到new_data_part中\n new_data_part->addProjectionPart(projection.name, writeProjectionPart(projection_block, projection, new_data_part.get()));\n }\n }\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从文件系统目录上看,p2.proj 为data part下p2 projection的数据目录,目录下聚合列,聚合函数作为单独的列存文件存储。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"\n├── dim1.bin \n├── dim1.mrk2 \n├── dim2.bin \n├── dim2.mrk2 \n├── dim3.bin \n├── dim3.mrk2 \n├── event_key.bin \n├── event_key.mrk2 \n├── event_time.bin \n├── event_time.mrk2 \n├── p2.proj \n│ ├── checksums.txt \n│ ├── columns.txt \n│ ├── count%28%29.bin \n│ ├── count%28%29.mrk2 \n│ ├── count.txt \n│ ├── default_compression_codec.txt \n│ ├── dim3.bin \n│ ├── dim3.mrk2 \n│ ├── groupBitmap%28user%29.bin \n│ ├── groupBitmap%28user%29.mrk2 \n│ └── primary.idx ","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"clickhouse projection适用于哪些场景?","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了探索projection适用于哪些场景,准备了典型的用户行为数据集,数据量为1亿条, 数据模型选择事件模型,模型中包含了用户做过什么事件,以及事件对应的维度。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"维度选择上,dim1,dim2为普通维度值,维度值种类有10种。dim3为高基维维度,维度值种类有100000种。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4a/4a16e1e8d3c639e5597218b251fa72e2.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"
user
唯一用户标识
event_key
事件时间
event_time
事件时间
dim1
普通维度
dim2
普通维度
dim3
高基维度
"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"如何为数据表构建projection?","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"建表的时候指定多个projection 定义,projection中为基本的select语句,可以省略from table子句,默认与源表保持一致。","attrs":{}}]}]}]},{"type":"codeblock","attrs":{"lang":"sql"},"content":[{"type":"text","text":"CREATE TABLE event_projection1 \n( \n `event_key` String, \n `user` UInt32, \n `event_time` DateTime64(3, 'Asia/Shanghai'), \n `dim1` String, \n `dim2` String, \n `dim3` String, \n PROJECTION p1 \n ( \n SELECT \n groupBitmap(user), \n count(1) \n GROUP BY dim1 \n ) \n) \nENGINE = MergeTree() \nORDER BY (event_key, user, event_time) ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. alter table 语句补充projection定义","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"sql"},"content":[{"type":"text","text":"ALTER TABLE event_projection1 \n ADD PROJECTION p2 \n ( \n SELECT \n count(1), \n groupBitmap(user) \n GROUP BY dim1, dim3 \n ) ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"怎么查询才能命中projection?","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"select表达式必须为projection定义中select 表达式的子集。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"group by clause必须为projection定义中group by clause的子集。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"where clause key必须为projeciton定义中的group by column的子集。","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"如何知道是否命中了projection?","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"explain查看执行计划,ReadFromStorage (MergeTree(with projection)) 表示命中projection","attrs":{}}]}]}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"EXPLAIN SQL \nexpain actions=1 select dim, count(1) from event_projection group by dim1 \n \n \n执行计划: \nExpression ((Projection + Before ORDER BY)) \nActions: INPUT :: 0 -> dim1 String : 0 \n INPUT :: 1 -> count() UInt64 : 1 \nPositions: 0 1 \n SettingQuotaAndLimits (Set limits and quota after reading from storage) \n ReadFromStorage (MergeTree(with projection)) ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2. clickouse 查询关键日志","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"查询命中了projection p \n(SelectExecutor): Choose aggregate projection p \n(SelectExecutor): projection required columns: dim1, count() \n(SelectExecutor): Reading approx. 63 rows with 4 streams ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"查询效果如何?","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/db/dba6396d038dcc95068030b160a618fe.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"
projection定义
查询耗时
存储
插入时间
无projection
5.347s
650M
7min
dim1聚合
0.018s
654M
12min
(dim1 + dim3) 聚合
0.319s
923M
20min
"}}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"命中projection相比没有命中projection对于查询性能的提升非常明显。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"构建projection对于存储,数据插入有一定的额外开销。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"如果构建projection的时候混入了高基维度,查询耗时相比没有混入高基维度,查询性能同比降低了近200倍,存储与插入时间也付出了更多的额外开销。","attrs":{}}]}]}]},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"
场景: 不同聚合函数性能提升效果对比
"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b0/b0143afca11ac714832d7f0db46d2d66.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"
聚合函数
没有projection
普通维度聚合
高基维度聚合
count(1)
5.347s
0.018s
0.319s
groupBitmap(user)
7.936s
0.040s
5.840s
"}}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"相同的条件下groupBitmap没有count聚合函数的性能提升效果好,","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"高基维的场景下,即使命中了projection与没有命中projection,查询效果几乎相同, 而且付出了额外的存储计算开销。","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"综上以上测试可以得出,高基维度对于projection特性并不友好,查询性能提升有限,并且还有付出不小的额外开销,不建议projection构建的时候应用高基维度。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"clickhouse projection有什么要注意的吗?","attrs":{}}]},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"额外的存储开销","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面有提到,每个projection在part目录下存到单独的目录独立存储,projection目录下存储基于原始数据计算的聚合数据。projection数据可以抽象理解为一张聚合表,按照不同的维度聚合,聚合度不同,projection的存储开销也会同。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"2. 影响数据写入速度","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过源代码分析可以发现, projection写入与原始数据写入过程保持一致。每一份数据part写入都会基于原始数据Block结合projection定义计算聚合数据,增加了数据写入的额外开销,也增加了数据写入的时间,降低了数据的时效性。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"3. 历史数据不会自动物化","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"projection基于part粒度存储,并且与数据写入保证一致,创建了projection之后插入的数据才会被物化。同时,part之间的merge包含projection之间的merge,如果part之间的projeciton定义不一致,将会导致part merge失败,可以通过projection materialization操作将part中projection数据拉齐。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"projection materialization: projection计算基于原始数据block,对于比较大part计算的过程中很容易出现内存问题。可以构建insert select pipeline模拟新数据产生的过程,中间会生成多个临时小part,小part中的proejction进行多段merge。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"4. part过多导致projection不能命中","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"数据查询命中projeciton其中的一个条件为50%以上的part 覆盖projection。存在部分场景,由于数据频繁写入,导致生成很多小part,part数量增加增大了计算覆盖率的分母,导致没有达成命中projection的条件。但是,伴随着part的合并part数量的减少,之后的查询有可能命中projection。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本篇文章只是针对clickhouse proejction特性进行了简单的介绍,并进行了基础的性能测试。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在性能测试中,也发现了高基维度对于clickhouse projection的影响。后续将会其他文章对 clickhouse 的查询流程,底层存储进行细致的详解,分析其影响的内部原因。","attrs":{}}]}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章