数仓实时化改造：Hudi on Flink 在顺丰的实践应用

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文主要介绍顺丰在数据仓库的数据实时化、数据库 CDC、Hudi on Flink 上的实践应用及产品化经验。文章主要分为以下几部分："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"顺丰业务介绍"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"Hudi on Flink"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"产品化支持"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"后续计划"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1 顺丰业务"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.1 顺丰大数据的应用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先来看一下顺丰大数据业务的全景图。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/03\/0396c6ef42d17ef45ce6f9ee4a32c282.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大数据平台，中间的基础部分是大数据平台，这块是顺丰结合开源组件自行搭建的。与之相关的是大数据分析与人工智能，顺丰有一个非常强的地面部队，就是线下的快递小哥以及运输车辆，需要使用 AI 以及大数据分析来辅助管理，提升整体效率。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"区块链，顺丰对接了很多客户与商家，对于商家来说，首先需要确保快件是可信的能够做货物的交易与交换。这块涉及的基本上都是品牌商家，溯源与存证的业务顺丰也有涉及。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"IoT，就像之前提及到的，因为顺丰地面部队较多，相应需要采集的数据也会比较多。我们的部分包裹中是有传感器的，车辆也有相关的传感器，如车辆的摄像头，以及快递小哥的手环（包含地理位置、员工的健康状态，对应做一些关怀的举动）。同时，还有一些工作场景既有叉车，也有分拣设备，这些就需要大数据平台来做一些联动，因此 IoT 的应用相对较多。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"智慧供应链和智慧物流，这两块更多的是指如何用大数据的手段辅助业务做一些经营上的决策。比如我们有很多 B 端客户，对于他们来说如何在每个仓库里备货，如何协调以及互相调拨，这部分就由智慧物流来完成。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面这块就是 IOT 实践中的一部分："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/49\/49d334b9a87992efb3b91cb18417019c.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从上面可以看出物流本身的环节是非常多的，下单、小哥收件、分拣、陆运中转等整个过程，红色解释部分是指我们会做的一些 IoT 与大数据结合的应用，这里其实大部分都是基于 Flink 来完成的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.2 顺丰大数据技术矩阵"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面这张图是顺丰目前大数据整体的架构概览："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/f5\/f5e1adea99c092d854057a5737a78f40.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"数据集成层：最下面为数据集成层，因为顺丰的历史原因，所以包含了很多数据存储引擎，如 Oracle、MySQL、MongoDB 等，并且部分引擎仍会继续支持。右下物联网设备相对较新，主要是进行包含普通文本、网络数据库、图像、音频、视频等的数据采集。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"数据存储计算：实时这块顺丰目前用的最多的还是 Flink，Storm 没有标示出来，目前我们在做迁移。消息中间件处理目前主要使用 Kafka。然后右边存储结构的种类就相对丰富，因为不同的场景有不同的处理方式，比如数据分析需要性能比较强的 Clickhouse；数仓和离线计算这块还是比较传统，以 Hive 为主结合 Spark，目前我们是结合 Flink 与 Hudi 去实现离线实时化。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"数据产品，我们倾向的还是首先降门槛，让内部开发与用户更容易上手。内部同学如果要掌握如此多的组件，成本是非常高的，再加上规范化会导致沟通、维护以及运维的高额成本，所以我们一定要去做一些产品化、规范化的事情。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.3 顺丰科技数据采集组成"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/6f\/6fd29b0cbe1f75e2c1846f623627214e.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上图就是我们大数据整体数据采集的概览，数据采集当前包括微服务的应用，部分数据直发到 Kafka，还有些会落成日志，然后我们自己做了一个日志采集工具，类似于 Flume，更加的轻量化，达到不丢、不重、以及远程的更新、限速。另外我们也会将 Kafka 中的数据通过 Flink 放到 HDFS，以 Hudi 的形式去做。下面会详细介绍。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.4 顺丰数据应用架构"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/wechat\/images\/9b\/9b382f2048fd362f15dfd8122725bbd2.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}