利用 JuiceFS 给 Flink 容器启动加速

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Flink 因为其可靠性和易用性，已经成为当前最流行的流处理框架之一，在流计算领域占据了主导地位。早在 18 年知乎就引入了 Flink，发展到现在，Flink 已经成为知乎内部最重要的组件之一，积累了 4000 多个 Flink 实时任务，每天处理 PB 级的数据。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Flink 的部署方式有多种，根据资源调度器来分类，大致可分为 standalone、Flink on YARN、Flink on Kubernetes 等。目前知乎内部使用的部署方式是 Flink 官方提供的 native Kubernetes。谈到 Kubernetes，就不得不说容器镜像的问题，因为 Flink 任务的依赖多种多样，如何给 Flink 打镜像也是一个比较头疼的问题。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Flink 镜像及依赖处理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Flink 的任务大致可分为两类，第一类是 Flink SQL 任务，Flink SQL 任务的依赖大致有以下几种：、"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.官方的 connector JAR 包，如 flink-hive-connector、flink-jdbc-connector、flink-kafka-connector 等;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.非官方或者是内部实现的 connector JAR 包；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3.用户的 UDF JAR 包，一些复杂的计算逻辑，用户可能会自己实现 UDF。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二类 Flink 任务是 Flink 的 jar 包任务，除了以上三种依赖，还需要依赖用户自己写的 Flink jar 程序包。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"显然，对于每一个 Flink 任务，它的依赖不尽相同，我们也不可能为每一个 Flink 任务单独打一个镜像，我们目前的处理如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.将依赖进行分类，分为稳定依赖和非稳定依赖；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.稳定依赖包括组件（如 Flink、JDK 等）以及官方的 connector 包，这类依赖十分稳定，只会在 Flink 版本升级和 bug 修复这两种情况下进行改动，因此我们会在构建镜像时，将这类依赖打入镜像；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3.非稳定依赖包括第三方的 connector 和用户自己的 JAR 包。第三方的 connector 因为不是 Flink 官方维护，所以出问题需要修复的概率相对更大；用户自己的 JAR 包对于每个任务来说都不相同，而且用户会经常改动重新提交。对于这类不稳定的依赖，我们会动态注入，注入的方式是将依赖存入分布式文件系统，在容器启动的时候，利用 pre command 下载进容器里。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"经过以上处理，Flink 镜像具备了一定的动态加载依赖的能力，Flink Job 的启动流程大致如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/4a\/6f\/4acb9d0ece2f55545a6529cdc8ff936f.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"文件系统选取"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"HDFS 存放依赖的痛点"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"存放 Flink 依赖的文件系统在之前我们一直都是选用的 HDFS, 但是在使用过程中我们遇到了以下痛点："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.NameNode 在任务高峰期压力过大，容器在下载依赖时向 NameNode 请求文件元数据会存在卡顿的情况，有些小的批任务，任务本身可能只需要运行十几秒，但是因为 NameNode 压力过大，导致下载依赖可能需要几分钟；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.目前 Flink 集群我们是多数据中心部署，但是 HDFS 只有一个离线机房大集群，这样会存在跨数据中心拉文件的情况，消耗专线带宽；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3.有一些特殊的 Flink 任务完全不依赖 HDFS，换句话说它既不使用 checkpoint 也不读写 HDFS，但是因为 Flink 容器的依赖存放在 HDFS 上，导致这类任务依然离不开 HDFS。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"使用对象存储的痛点"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"后面我们将 HDFS 换成了对象存储，解决了 HDFS 的一些痛点，但是很快我们发现了新的问题 — 对象存储单线程下载的速度慢。对象存储下载加速可选的方案一般有以下几种："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.使用多线程下载进行分段下载，但是容器的 pre command 其实只适合执行一些比较简单的 shell 命令，如果采用分段下载，就必须对这一块进行比较大的改造，这是一个比较大的痛点；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.给对象存储加代理层做缓存，加速的事情由代理来做，客户端依然可以单线程读取。这种办法的缺点是需要额外维护一个对象存储的代理组件，组件的稳定性也需要有保障。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"尝试 JuiceFS"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"比较凑巧的是公司内部正在做 JuiceFS 的 POC, 有现成的对象存储代理层可用，我们对其进行了一系列测试，发现 JuiceFS 完全满足我们这个场景的需求，让我们比较惊喜的地方有以下几点："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.JuiceFS 自带 S3 gateway 完美兼容 S3 对象存储协议，能够让我们很快上线，无需任何改动，并且 S3 gateway 本身无状态，扩缩容非常方便；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.JuiceFS 自带缓存加速功能，经过测试，用 JuiceFS 代理对象存储后，单线程读取文件的速度是原来的 4 倍；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3.JuiceFS 提供本地文件系统挂载的方式，后面可以尝试依赖直接挂载进容器目录；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4.JuiceFS 可选用元数据与存储分离部署的方式，存储我们选用原来的对象存储，云厂商保证 11 个 9 的可用性；元数据我们选用分布式 KV 系统—TiKV，选用 TiKV 的原因是我们在线架构组的同事对 TiKV 有着丰富的开发和运维经验，SLA 能够得到极大的保障。这样 JuiceFS 的可用性和扩展性是非常强的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"JuiceFS 上线"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"JuiceFS 的上线过程分为以下阶段："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.数据迁移，我们需要将原先存储在 HDFS 和对象存储上的数据同步到 JuiceFS 上，因为 JuiceFS 提供了数据同步的工具，并且 Flink 的依赖也不是特别大，所以这部分工作我们很快就完成了；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.修改 Flink 镜像拉取依赖的地址，因为 JuiceFS 兼容对象存储协议，我们只需要在平台侧修改原来的对象存储的 endpoint 为 JuiceFS S3 gateway 的地址即可。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"JuiceFS 上线后，我们 Flink 任务启动的流程图大致如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/64\/40\/64b78f4133d06c2c2020a8b7dc8a9f40.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相比于使用 HDFS 的方式，我们能得到一个可预期的容器启动时间，容器下载依赖的速度不会受业务高峰期的影响；相比于原生的对象存储，容器下载依赖的速度提高约 4 倍。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"展望"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从开始调研 JuiceFS 到 JuiceFS 上线花费时间不到半个月，主要是因为 JuiceFS 的文档十分完备，让我们少走了很多弯路，其次是 JuiceFS 社区的伙伴也有问必答，因此我们的上线过程十分顺利。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"初步尝试 JuiceFS 给我们带来的收益还是比较明显的，后续我们会考虑将 JuiceFS 应用在数据湖场景和算法模型加载的场景，让我们数据的使用更加灵活和高效。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介绍："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"胡梦宇，知乎大数据架构开发工程师，主要负责知乎内部大数据组件的二次开发和数据平台建设。"}]}]}