Flink+Hologres助力伊的家电商平台建设新一代实时数仓

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"广州伊的家网络科技有限公司是一家专注于服务女性的B2B2C电商平台，业务范围包括护肤、彩妆、营养美容食品、私人定制服装、跨境电商等领域。自2008年孵化项目，2011年5月上线天猫商城，全国8大配送中心，妍诗美、妍膳等品牌陆续成立，并于2013年上线了伊的家自主电商平台，2020年全面启动品牌升级。伊的家以互联网主动式服务营销，打造护肤老师与客户强连接关系，从上到下严格贯彻以品质及专业为基础，以社交信任做连接，以服务获得认可的经营思路，经过持续的创新和积累，成为社交电商翘楚。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"业务场景与痛点分析","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"伊的家是一家集开发、设计、运营、销售于一体一个B2B2B的电商平台，服务百万级会员之外，还同时支持上千级别经销商和代理商，业务应用多、数据量大、数据查询并发要求高。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"伊的家技术部门在近3年经历了高速发展，在发展过程中，始终坚持业务优先，为此也进行了应用整合、拆分微服务、聚合分布式应用的多种技术升级改造，目前整个部门现状分析如下：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"架构方面：","attrs":{}},{"type":"text","text":"多语言、多数据源、技术升级的业务入侵问题明显；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"数据方面：","attrs":{}},{"type":"text","text":"应用拆分引发的数据孤岛问题，继而造成大量的数据复制、重新建设问题；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"应用方面：","attrs":{}},{"type":"text","text":"从业绩的角度出发，业务方希望及时准确地看见业绩数据，对实时性有了较高需求；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"效率方面：","attrs":{}},{"type":"text","text":"体系化的流程与工具诉求愈发强烈；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"成本方面：","attrs":{}},{"type":"text","text":"主要问题是既懂大数据又懂业务的人才招聘难，团队建设成本高","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1a/1adc30b616ae3aa811a549b26158bcdb.png","alt":"伊1.png","title":"伊1.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"伊的家近几年业务高速增长，数据量激增，业务复杂度也随之增大，解决在当前大数据架构之下，“人才储备难”、“业务升级受限于已有技术”、“双11活动压力大”等痛点问题已迫在眉睫。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8c/8c67b3bfff8ddd644ab098a7f1835422.png","alt":"伊2.png","title":"伊2.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"产品选型","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"伊的家技术部门对于技术升级改造的需求有非常明确清晰的定义，","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"主要围绕关存储弹性扩缩容、查询性能优化、OLAP、学习成本、查询响应、可扩展等角度进行展开","attrs":{}},{"type":"text","text":"，核心关注以下3个问题：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）如何快速完成数据清洗","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）如何快速精准完成数据校验","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）如何快速进行故障恢复处理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/96/962ab924e8e7ced81047410eff2b73bc.png","alt":"伊3.png","title":"伊3.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在技术选型时始终坚持“技术选型是第一生产力”的原则，坚信技术储备没有最好只有更好，坚信技术选型是决定能力差异化所在，坚持提高一次性把事情做对的能力，坚信开放分享、认知升级的重要性。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e3/e33df540c673d2875ae54100b9f1c277.png","alt":"伊4.png","title":"伊4.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"早期耶基于Hadoop、HBase、Kafaka、Azkaban、Spark、Greenplum等开源大数据产品进行了许多摸索尝试，通过性能对比最终采用了Greenplum，但最终发现Greenplum并发能力差，只适合分析场景，并不适合高并发的查询服务。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"后来，在阿里云大数据计算平台团队的建议下，伊的家技术部进行了全面架构升级，","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"整个架构由DataWorks、实时计算Flink和Hologres组成，架构简单、学习成本非常低，仅通过SQL即可轻松跑通全链路","attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面将会给大家介绍，阿里云技术产品在伊的家落地的场景最佳实践","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"最佳实践","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"一、客户系统实践","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"伊的家原客户关系管理系统（CRM）主要基于MySQL、MQ、Canal以及自研应用组成，为支持业务系统切断式升级，技术部门自主研发了一套消息中间件，维护成本较高；基于Binlog、MQ、OLAP等产品自定义的数据开发流程过程繁琐复杂、维护成本极高，且因为系统要求数据有序对清洗的并发产生了一定的限制。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5d/5d38193577b1f12b970a9568f8a88d89.png","alt":"伊5.png","title":"伊5.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基于Hologres+DataWorks+实时计算Flink进行架构升级后，直接通过DataWorks数据集成将数据库数据实时写入Hologres，然后通过实时计算Flink订阅Hologres做进一步实时清洗，把结果表更新到数据库，即可直接服务业务。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"整体架构清晰简单、数据精准、端到端纯实时、存储分析一体化、托管式运维、全自动工具作业，原系统15人花了3个月才完成项目上线，当前架构仅需2天即部署完成。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/de/deecb3f1ed78fd485374f1d75b891469.png","alt":"伊6.png","title":"伊6.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"二、BI业绩系统实践","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"BI业绩系统也可以理解为实时GMV大屏，业务数据主要有两方面的要求：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"实时","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"精准，业绩计算绝不允许出错。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原架构如下图图所示，原始数据层通过Binlog，再经过Canal套件实时写入MQ，之后根据业务域进行业务数据分层和清洗。任务调度系统更新业绩的顺序为“日-月-季度-年”，这个看似完美的方案实际存在着几个问题：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"实时性问题：看似实时，其实过程中可能存在5~10分钟的延迟；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"并发问题：消费的并发有一定限度。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"运维问题：如果图中的某个环节出现问题，可能会导致系统也跟着出现问题。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"数据清洗时效问题：清洗脚本运行一次可能需要数分钟，这期间可能会发生许多其他事情。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f8/f8ea226a8ab2e545b5a28ba482f93b7c.png","alt":"伊7.png","title":"伊7.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下图为升级后的BI业绩系统新架构。通过DataWorks实时同步明细数据至Hologres，基于Hologres数据再增加一份实时计算Flink的实时ETL作业，即可完成“日-月-季度-年”数据的加工，最后基于Hologres对上层应用提供分析查询服务。","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"整个系统纯实时调度、实时性高、秒级延迟、全SQL开发、数据校验高效。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/53/53ed48785b6866b328d50cb41fcaa24f.png","alt":"伊8.png","title":"伊8.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"三、实时应用数仓架构实践","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"伊的家的技术部门也一直在思考如何让应用开发人员也具备大数据开发能力，如何让大数据不仅仅为大数据团队所用，还同时为应用开发团队所用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"基于实时计算FLink+Hologres+DataWorks实时数仓架构的落地，提升了数据底盘的可复用性，提高了应对业务变化的数据动态调整的灵活性，与应用团队共同构建起带数据的应用系统。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f8/f86e4759936ef78070e0277fbf84dc7b.png","alt":"伊9.png","title":"伊9.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"四、集团数仓架构实践","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"伊的家数仓团队服务在电商业务的同时，还需要支持集团内部业务。集团数仓平台如市场主流数仓架构、基于开源大数据体系构建，目前也已经全面升级为Hologres+实时计算Flink+DataWorks实时数仓架构。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fa/fac95e74577e098e1394b3bac3084449.png","alt":"伊10.png","title":"伊10.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3f/3f77e8ffd40edb8cdb3be9b5a8e90aec.png","alt":"伊11.png","title":"伊11.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"业务价值与赋能","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hologres+实时计算Flink+DataWorks实时数仓新方案为业务上带来的价值主要如下：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"统一数据：一套方案就能支持完整流程，明细表、维度表等数据统一、有序","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"统一服务：由Hologers直接提供各种线上服务，包括数据分析，数据服务等，减少接口建设。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"统一存储：以Hologres为统一存储，多数据源都能直接写入到Hologres，无冗余存储，节约成本","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"统一治理：DataWorks提供统一标准、统一作业和统一监控等，为大数据开发平台提供统一治理。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从业务上来说，新的大数据方案真的做到了","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"开箱即用，所见即所得","attrs":{}},{"type":"text","text":"。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b6/b60f26c235fef738828430ca17b7f5db.png","alt":"伊12.png","title":"伊12.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"展望未来","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在大数据领域，数据规模和业务复杂性是同时制约查询性能的关键因素，在这个过程中，唯有我们的开发人员不断打磨自己的数据模型，当数据模型达到一定成熟度，性能问题即可迎刃而解。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最后，希望大家拥抱技术、拥抱变化、赢在模型，数据服务业务，数据服务应用，让我们为应用而生，为应用而战。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2e/2e61c2c4fadb08297d312d4d6268e59f.png","alt":"伊13.png","title":"伊13.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/36/36e46415527c31ecdc5aaa1010a20066.png","alt":"伊14.png","title":"伊14.png","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"作者：刘松森 ，伊的家CTO，高级工程师，副教授职称，国内多所高校客座教授","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Flink 中文学习网站：https://flink-learning.org.cn/","attrs":{}}]}]}