贝壳OLAP平台架构演进

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文根据贝壳找房资深工程师肖赞老师在2020年\"面向AI技术的工程架构实践\"大会上的演讲速记整理而成。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1 贝壳OLAP平台架的构演化历程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/9f\/28\/9fc134cee7b2aa82eef2866a5ea3bf28.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上图所示，贝壳OLAP平台架构的演化历程大致可以分成三个阶段："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一个阶段是从 2015 年到 2016 年，Hive to MySQL的初期阶段，这是一个无到有的阶段；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二个阶段是从 2016 年到 2019 年初，基于Kylin的OLAP平台建设阶段，这个阶段围绕着 Apache Kylin引擎构建OLAP 平台；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三个阶段是从 2019 年初到现在，灵活支持多种OLAP引擎的OLAP平台建设阶段，这个阶段解耦了OLAP平台与Kylin的强绑定，具备灵活支持Kylin之外多种不同OLAP引擎的能力；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/dc\/90\/dc85c2ffc332740dc08928823015b590.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先是第0阶段 - Hive2MySQL初期阶段"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在这个阶段，数据处理流程比较简单，数据包括日志、Dblog等，经过Sqoop批量或Kafka实时接入大数据平台HDFS里，在大数据平台完成ETL处理后，通过大数据调度系统Ooize每天定时写入到关系型数据库MySQL中，然后再以MySQL中的数据为基础产出各种报表。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这是一个从无到有的过程，很多公司初期也都是这么做的。这种方式的优点是简单，很快能够落地跑通，但是问题也很明显："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"整个平台受限于MySQL的能力，MySQL无法支持大数据量的快速查询和分析，一般几百万上千万后，MySQL就难以支撑；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"这种方式缺少共性能力的沉淀，Case by Case的处理用户需求，需求开发时间长；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这也是我们为什么称为第0阶段，因为这个阶段平台化工作比较少，严格说还称不上一个平台。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/8e\/d4\/8e97fe996fc865533d15649e08cdd6d4.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"随着公司业务的迅速发展，数据应用需求的急剧增加，Hive2MySQL的问题逐步突显，对这种原始架构进行升级改造是一个必然的选择。改造的目标很直接，首先解决MySQL无法支持海量数据分析查询的问题；第二就是要平台化，沉淀共性能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}