Kylin 在携程的实践（下）

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"案例分享"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"离线分析案例"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/00\/0e\/0002yydc075eca19f172cc15758be50e.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"携程之前使用的是 OpenTSDB+Hive。采用 Kylin 前，先从 Hive 先生成聚合表，然后导入 HBase，通过 OpenTSDB 去分析，现在积累了接近百亿的数据，随着数据的增长，老的方案已经无法满足业务需求了，而且同步数据成本高，OpenTSDB 没办法支持精准去重响应时间也很差。"},{"type":"text","marks":[{"type":"strong"}],"text":"用了 Kylin 之后，现在的业务规模已经可以支撑上百亿了，目前已经配有 200 个左右的线上活跃的 Cube。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"实时分析案例"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/ea\/c2\/ea9c60b75aa83d02yy277f41721029c2.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这个是去年 3、4 月份用户提的新需求。Kylin 现在是上图所示的 Streaming-Cube 的架构，Kylin 接入的是携程的 Hermes，Hermes 是 Kafka 的一个封装。我们现在支持原生 Kafka 接入和 Hermes 接入，底层沿用 MR，因为我们测试过Spark，其实很多的场景上和 MR 相当，效果不是特别明显。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/db\/b3\/dbd5fef7fb6d498fb2cd576dfd5caab3.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这部分主要是用于度假预订状态告警，度假团队需要去分析用户预订的情况，准确实时地发送给客服人员任何预订失败等错误状况，所以这块对于数据构建落地的时间敏感度比较高。目前，通过一系列优化，Streaming 的构建基本保持在5分钟左右，可以满足一部分业务的需求。但是，"},{"type":"text","marks":[{"type":"strong"}],"text":"更大的挑战是达到一分钟以内，也就是说秒级构建，所以对于我们来说Streaming-realtime 会是一个值得尝试的方向。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"展望"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"携程针对 Kylin 主要有两方面的展望。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1 支持自动构建Cube"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这块我们目前在调研，通过分析应用采集的元数据、SQL特征，可以自动地为用户构建 Cube，为用户节约 Kylin 的学习成本，同时减少重复查询对于 MPP 的压力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2 Real-time Streaming的调研和落地"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了能够更加丰富 Kylin 的使用场景，我们打算对 eBay 为 Kylin 贡献的"},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s?__biz=MzAwODE3ODU5MA==&mid=2653079081&idx=1&sn=7a5088406e01f6bfa62ffb02d7254d2f&scene=21#wechat_redirect","title":"","type":null},"content":[{"type":"text","text":"实时流处理技术"}]},{"type":"text","text":"做进一步调研和落地工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Q&A"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Q：演讲中提到的构建的 Cube 有 20 个指标，这种情况下去重，是精准去重还是近似去重？有多少个指标呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：用户配的是精确。精确去重指标不会太多。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Q：演讲中提到20个维度的响应时间是亚秒级，有 20 个维度。请问你们做了哪些优化的工作来达到如此快的响应时间？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：我们构建的时候，对于这种维度多的情况，"},{"type":"text","marks":[{"type":"strong"}],"text":"建议当用户采取了以下 3 种措施来优化查询："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"使用Mandatory Dimension；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"实现分布式缓存；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"配置高基维度的时候，会建议他们把高基维度往前移，这样会更高效地命中 Cube，并减小扫描的数据范围）。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Q：配了 20 个维度，最终产生的 Cube 单日有多大？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：最大的 Cube 日产生 13 T 的数据。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Q：刚刚提到的监控方案是你们自主研发的，还是有开源的方案可以用？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：监控是我们自主研发的。我们接入了公司已经成熟的监控平台，避免反复造轮子。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Q：分享里提到的实时5 分钟构建一次，我理解是采用批操作，并不是真正的流，而是把流几分钟拆成一个批次。是吗？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：对的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Q：前面讲到底层用的 MR，没用 Spark，因为觉得时间上并没有什么节省。这个是 Spark 本身的原因，还是因为你们的任务还不是很大的量？因为每次 Spark 启任务的时间和MR相比有差别？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：离线这块目前可以达到要求，所以还没有转成Spark。我们在实时这块用 Spark 的过程中，就是像你说的，每次提交任务就很慢，达不到要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Q：是因为频繁提交的问题？不是因为它本身？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：对，不是因为它本身。我们也在调研如何避免每个构建过程都启动一次 driver。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Q：在我之前的应用场景里，有一个维度特别的高基维，每天增量就很大，我们查询机制里这个维度是必选的。比如说是人的工号，里面放了很多人，然后我们要去预计算，如果说这个维度非常高，数据量会非常大，这种情况下你们会采取什么办法呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：高基字段可以设置下 shard by。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Q：携程每天预计算的集群大概是有多大？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：离线集群是 2 台物理机，每台 100 多 G 的物理机，查询节点放了 4 台虚机。实时这块，因为用户量目前不多，所以都是建在虚机上，所以内存也不大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Q：在维度特别大，数据量又很大的情况下，剪枝的话，Cuboid 大概会控制在多少？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"A：维度特别大的情况，我们最多是 4096 个 Cuboid。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"本文转载自公众号apachekylin（ID：ApacheKylin）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文链接"},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s?__biz=MzAwODE3ODU5MA==&mid=2653079116&idx=1&sn=cd1e06a78d4a07955d85dbba4089c542&chksm=80a4b8bdb7d331aba0fdb00d16168e6c74098a14b153c1a5d78b2000c47625cb2091714f1e4d&scene=21#wechat_redirect","title":"","type":null},"content":[{"type":"text","text":"Kylin 在携程的实践（下）"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}