Apache Kylin 在汽车之家的实时多维分析实践

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"近期，Apache Kylin 5 周年在线庆典顺利结束，来自汽车之家的实时计算平台负责人 邸星星 老师为大家介绍了 Apache Kylin 在汽车之家的升级历程，以及在实时多维分析方面的实践，最后也展望了对 Kylin 4.0 版本的期待。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/49\/1f\/49879e7f1630d05e8d182bcd517dba1f.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/www.bilibili.com\/video\/BV1EV411t7eP?from=search&seid=7421771643899774600","title":"","type":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"会议完整视频"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以下是邸星星老师的现场分享实录。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Apache Kylin 在汽车之家的升级演进"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"近年来，面对大数据量与终端用户的不断增长，2016年的时候我们梳理了针对多维分析引擎的选型需求："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"面临千亿，乃至万亿数据量的情况下，要达到秒级或者亚秒级的查询响应。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要满足相对较高的查询吞吐量以及较高的可用性，当时汽车之家的数据主要是基于离线数据来做多维分析。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Kylin 在汽车之家的升级历程"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/a2\/8c\/a25ff8d7d89e1227637ce6fff3af618c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"Kylin 过往 5年里主要的发展里程碑"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2016年初，汽车之家正式启用 Kylin 1.5 版本"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此时共有10+ Cube，主要支持部门内的少量数据分析，同时用作技术验证。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2017年，内部正式上线 Kylin 1.6 版本，并逐步承接线上业务"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"累计有100+ Cube，2017年底使用 Kylin 支持了汽车之家的战略级商业化数据产品。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2018年，升级到 Kylin 2.2 版本 ，支持 Spark 引擎做构建，同时对 HBase 集群做了 T+1 备份"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在极端情况下，如果 HBase 集群发生故障，可以把用户的查询请求路由到备用集群里来保障业务正常运行。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2019年，升级到 Kylin 2.6 版本，并在内部多个 BU 推广使用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cube 数量达到了 400+ ，另外公司内部也研发了BI 产品，也可以支持 Kylin 作为它的查询引擎。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2020年，将 Kylin 升级到 3.1 版本，主要来做实时多维分析的应用"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/yy\/f6\/yy5ef78005520f15df3c35709b3238f6.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"可以看到目前集群规模大概有 500+Cube，20000+个Segment，有15万左右的 HBase Region，存储在 300T 左右，每天的查询请求在 20 万以上，使用 Kylin 95% 的响应时间会在 2 秒以内返回结果。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Kylin 在实时多维分析方面的应用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1）Kylin Real-time OLAP 架构"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kylin 3.0 版本开始提供实时分析能力，主要是引入了两个角色，一个是 Coordinator，另一个是 Receiver。Coordinator 主要是负责协调，做一些分配，启动、停止这样的工作。Coordinator 是一个真实的一个计算节点。同时，Kylin 会抽象 Replica Set 的概念，就是把两个 Receiver 看作一组，当作一组 Replica Set，然后这两个 Receiver 做的是一样的事情，他们计算的是一样的数据，也提供同样的查询，当一个 Receiver 挂掉的时候，整体的查询是不会受影响的，相当于是做了一个 Receiver 层面的高可用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/75\/0e\/75596259cd4e8b7d0ed7eace2e85f90e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2）Kylin 如何支持批流一体？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"接下来谈谈流批一体这个概念，目前流批一体的概念比较流行。"},{"type":"text","text":" Kylin 因为支持实时 Cube 的 Lambda 模式，我们在页面上做一次建模，然后把它定义为 Lambda 模式，那它实时的数据会从 Kafka 里面实时消费出来，然后去支持实时的查询。 "},{"type":"text","marks":[{"type":"strong"}],"text":"当离线的数据准备好之后，可以通过构建当天的离线数据去覆盖实时结果，也就是说 Kylin 能自动纠正实时计算的结果，这样就很好的解决了离线和实时的开发体系不统一的问题。"},{"type":"text","text":" 比如说用 Flink 做实时计算，然后用 Hive 做离线计算，其实是两套开发体系，而且有可能是两个不同的开发人员；使用 Kylin 的时候完全可以由一个开发人员完成，在 Kylin 的页面上来做正常的建模，就可以做到批流一体的多维分析， "},{"type":"text","marks":[{"type":"strong"}],"text":"这其实也降低了人力的浪费，同时也能保证数据口径是完全一致的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3）汽车之家在实时方面遇到的挑战"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"再来说下汽车之家在实时方面现在面临的 3 个问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先 Kylin 的 Receiver 节点和之前的 Job 以及 Query 节点都不太一样，因为它既负责实时的计算，又要负责查询，而且是查询本地磁盘上的数据，所以负担比较重。当面临越来越多实时业务的时候，可能需要维护一个比较大的 Receiver 集群，这样对我们来说的这个维护成本会是一个大问题。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外， Receiver 本身不只是服务于某一个 Cube，一个 Receiver 进程里面可能会支持好几个 Cube 的计算，假如某一个 Cube 的数据量突然激增，或者某个 Cube 的数据有一些问题，就可能会影响到这个 Receiver 进程本身的稳定性，所以隔离方面其实也不是特别好。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最后说下弹性伸缩这块，如果单个 topic 的数据量激增的时候，要怎么去快速的做扩容，然后等它的访问量下去之后，是不是可以快速的做缩容？"}]}]},{"type":"listitem","attrs":{"listStyle":"none"},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之前的方式都是需要手动去做调整，维护成本就会相应增加。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"4）Receiver on Kubernetes 解决方案介绍"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"目前 Kylin 在做的云原生，我们就自然地想到了可以和 K8s 做集成来解决这些问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前我们是在内部引入了一个 Kubernetes Resource Manager 的角色，然后由它去和 K8s 集群做交互，来动态申请相应的资源。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下图是目前我们正在做的和 K8s 集成的整体架构，我们做的改造主要集中在前面几步，后面这几步，包括查询这些基本没有改动。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/00\/d9\/007ec4a72557f177b9a4db7byyf857d9.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们在 Coordinator 内部引入了一个组件，就是Kubernetes Resource Manager，当用户完成建模后，在Enable Cube 的时候，会根据配置判断这个 Cube 是不是需要去创建一个独立的 Receiver 集群（Streaming Cluster），如果需要独立的 Receiver 集群，Coordinator 会根据配置好的并行度、副本数、CPU、内存等基本参数，通过 Kubernetes Resource Manager 和 K8s 去做交互，动态创建 Cube 所需要的 Reciever 集群。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这中间有一个映射关系，Kylin 里面的一个 Replica Set 会对应 K8s 里面的一个 StatefulSet，每个 StatefulSet 里面会有对应副本个数的 Receiver 实例，这个例子中我们设置的并行度为 2，副本数为 2，那么就会有 2 个StatefulSet，每个 StatefulSet 里会有 2 个 Receiver 实例。当 Kubernetes Resource Manager 把 Receiver 实例都启动成功后，Receiver 会自动注册到 ZK 中，同时会写入 Cube 名称，声明这个 Receiver 是属于哪一个 Cube 的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当 Coordinator 发现 Cube 所需的 Receiver 都启动完毕以后，会创建 Replica Set，并进行 assign 操作，同时把Cluster state 置为 ready 状态，最后向 Cube 对应的所有 Receiver 发送 REST 请求，通知这些 Receiver 开始消费数据，至此整个 Enable Cube 的过程就结束了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"简单回顾一下，通过和 K8s 集成已经解决了前面提到弹性伸缩的问题，我们不再需要维护一个很大的 Streaming Cluster，Streaming Cluster 可以是 Cube 级别的，每个 Cube 会对应一个独立的 Streaming Cluster，资源在不同的 Cube 间是相互隔离的，并且资源可以方便的动态伸缩。比如说用户想要增加资源，调大了并行度，那么 Coordinator 会识别到这个扩容事件，去动态的再创建一组 StatefulSet 和 Kylin Replica Set，进行重新分配，完成扩容。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"目前这一套架构已经在我们的准生产环境跑通了，然后接下来会上线支持一些我们内部的业务，运行稳定之后会逐步在各条业务线推广。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"对 Kylin 4.0 的展望"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最后来说下对 Kylin4.0 的展望，Kylin 4.0 主要是往云原生方向发展，主要有两块大的改进："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"全栈化 Spark，脱离 Hadoop 组件做查询"},{"type":"text","text":" ，这是云原生的基础；同时使用 Spark 引擎做查询还有一个额外的好处，就是之前用 Calcite 会有查询单点的问题，用 Spark 的话就可以很好地解决这个难点。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Kylin on Parquet，基于列式存储有很高的 IO 效率，也能一定程度上提供查询的稳定性"},{"type":"text","text":" ，对比 HBase是一个有状态的存储，但是 Parquet 只是一个文件格式，所以查询的链路上也会更加轻量级。去 HBase 后也不再需要额外的运维成本来维护 HBase。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"综上，我们非常期待 Kylin 4.0 GA 的发布。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介绍"},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"邸星星，汽车之家实时计算平台负责人，长期从事实时计算与数据分析领域的平台建设工作，致力于为公司提供大规模、高效、稳定的计算与查询服务。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"本文转载自公众号apachekylin（ID：ApacheKylin）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文链接"},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s?__biz=MzAwODE3ODU5MA==&mid=2653082037&idx=1&sn=26f7afa97b1351c6bf364c829e1360b6&chksm=80a4af44b7d32652d331914023e3e6e1b655dfe744bb5c5fa6aa36baed046b7eed13b73ef7c2&token=1845978438&lang=zh_CN#rd","title":"","type":null},"content":[{"type":"text","text":"Apache Kylin 在汽车之家的实时多维分析实践"}]}]}]}