1 |
伴鱼实时计算平台 Palink 的设计与实现
{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在"},{"type":"link","attrs":{"href":"https:\/\/tech.ipalfish.com\/blog\/2021\/06\/01\/palink\/","title":"xxx","type":null},"content":[{"type":"text","text":"伴鱼"}]},{"type":"text","text":"发展早期,出现了一系列实时性相关的需求,比如算法工程师期望可以拿到用户的实时特征数据做实时推荐,产品经理希望数据方可以提供实时指标看板做实时运营分析。这个阶段中台数据开发工程师主要是基于「Spark」实时计算引擎开发作业来满足业务方提出的需求。然而,这类作业并没有统一的平台进行管理,任务的开发形式、提交方式、可用性保障等也完全因人而异。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"伴随着业务的加速发展,越来越多的实时场景涌现出来,对实时作业的开发效率和质量保障提出了更高的要求。为此,我们从去年开始着手打造伴鱼公司级的实时计算平台,平台代号「Palink」,由「Palfish」 + 「Flink」组合而来。之所以选择"},{"type":"link","attrs":{"href":"https:\/\/s.geekbang.org\/search\/c=0\/k=flink\/t=","title":"xxx","type":null},"content":[{"type":"text","text":"「Flink」"}]},{"type":"text","text":"作为平台唯一的实时计算引擎,是因为近些年来其在实时领域的优秀表现和主导地位,同时活跃的社区氛围也提供了非常多不错的实践经验可供借鉴。目前「Palink」项目已经落地并投入使用,很好地满足了伴鱼业务在实时场景的需求。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"核心原则"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过调研阿里云、网易等各大厂商提供的实时计算服务,我们基本确定了「Palink」的整个产品形态。同时,在系统设计过程中紧紧围绕以下几个核心原则:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"极简性:保持简易设计,快速落地,不过度追求功能的完整性,满足核心需求为主。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"高质量:保持项目质量严要求,核心模块思虑周全。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可扩展:保持较高的可扩展性,便于后续方案的迭代升级。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"系统设计"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"平台整体架构"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以下是平台整体的架构示意图:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/8c\/8ca26dab825b3e04022b48c3d11d42a6.png","alt":"palink","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整个平台由四部分组成:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Web UI:前端操作页面。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Palink(GO) 服务:实时作业管理服务,负责作业元信息及作业生命周期内全部状态的管理,承接全部的前端流量。包括作业调度、作业提交、作业状态同步及作业 HA 管理几个核心模块。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"PalinkProxy(JAVA) 服务:SQL 化服务,Flink SQL 作业将由此模块编译、提交至远端集群。包括 SQL 语法校验、SQL 作业调试及 SQL 作业编译和提交几个核心模块。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Flink On Yarn:基于 Hadoop Yarn 做集群的资源管理。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这里之所以将后台服务拆分成两块,并且分别使用 GO 和 JAVA 语言实现,原因主要有三个方面:一是伴鱼拥有一套非常完善的基于 GO 语言实现的"},{"type":"link","attrs":{"href":"https:\/\/s.geekbang.org\/search\/c=0\/k=%E5%BE%AE%E6%9C%8D%E5%8A%A1\/t=","title":"xxx","type":null},"content":[{"type":"text","text":"微服务"}]},{"type":"text","text":"基础框架,基于它可以快速构建服务并拥有包括服务监控在内的一系列周边配套,公司目前 95% 以上的服务是基于此服务框架构建的;二是 SQL 化模块是基于开源项目二次开发实现的(这个在后文会做详细介绍),而该开源项目使用的是 JAVA 语言;三是内部服务增加一次远程调用的成本是可以接受的。这里也体现了我们极简性原则中对快速落地的要求。事实上,以 GO 为核心开发语言是非常具有「Palfish」特色的,在接下来伴鱼大数据系列的相关文章中也会有所体现。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下来本文将着重介绍「Palink」几个核心模块的设计。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"作业调度&执行"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"后端服务接收到前端创建作业的请求后,将生成一条 PalinkJob 记录和 一条 PalinkJobCommand 记录并持久化到 DB,PalinkJobCommand 为作业提交执行阶段抽象出的一个实体,整个作业调度过程将围绕该实体的状态变更向前推进。其结构如下:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.