京东Flink优化与技术实践

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"导读：Flink是目前流式处理领域的热门引擎，具备高吞吐、低延迟的特点，在实时数仓、实时风控、实时推荐等多个场景有着广泛的应用。京东于2018年开始基于Flink+K8s深入打造高性能、稳定、可靠、易用的实时计算平台，支撑了京东内部多条业务线平稳度过618、双11多次大促。本次讲演将分享京东Flink计算平台在容器化实践过程中遇到的问题和方案，在性能、稳定性、易用性等方面对社区版Flink所做的深入的定制和优化，以及未来的展望和规划。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"实时计算引进"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1.发展历程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/99\/00\/998ef28f6464eb802f16d28d875f1100.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最初大数据的模式基本都是T+1，但是随着业务发展，对数据实时性的要求越来越高，比如对于一个数据，希望能够在分钟级甚至秒级得到计算结果。京东是在2014年开始基于Storm打造第一代流式计算平台，并在Storm的基础上，做了很多优化改进，比如基于cgroup实现对worker使用资源的隔离、网络传输压缩优化、引入任务粒度toplogy master分担zk压力等。到2016年，Storm已经成为京东内部流式处理的最主要的计算引擎，服务于各个业务线，可以达到比较高的实时性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"随着业务规模的不断扩大，Storm也暴露出许多问题，特别是对于吞吐量巨大、但是对于延迟不是那么敏感的业务场景显得力不从心。于是，京东在2017年引入了Spark Streaming流式计算引擎，用于满足此类场景业务需要。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}