京東Flink優化與技術實踐

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"導讀：Flink是目前流式處理領域的熱門引擎，具備高吞吐、低延遲的特點，在實時數倉、實時風控、實時推薦等多個場景有着廣泛的應用。京東於2018年開始基於Flink+K8s深入打造高性能、穩定、可靠、易用的實時計算平臺，支撐了京東內部多條業務線平穩度過618、雙11多次大促。本次講演將分享京東Flink計算平臺在容器化實踐過程中遇到的問題和方案，在性能、穩定性、易用性等方面對社區版Flink所做的深入的定製和優化，以及未來的展望和規劃。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"實時計算引進"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"1.發展歷程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/99\/00\/998ef28f6464eb802f16d28d875f1100.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最初大數據的模式基本都是T+1，但是隨着業務發展，對數據實時性的要求越來越高，比如對於一個數據，希望能夠在分鐘級甚至秒級得到計算結果。京東是在2014年開始基於Storm打造第一代流式計算平臺，並在Storm的基礎上，做了很多優化改進，比如基於cgroup實現對worker使用資源的隔離、網絡傳輸壓縮優化、引入任務粒度toplogy master分擔zk壓力等。到2016年，Storm已經成爲京東內部流式處理的最主要的計算引擎，服務於各個業務線，可以達到比較高的實時性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着業務規模的不斷擴大，Storm也暴露出許多問題，特別是對於吞吐量巨大、但是對於延遲不是那麼敏感的業務場景顯得力不從心。於是，京東在2017年引入了Spark Streaming流式計算引擎，用於滿足此類場景業務需要。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}