新老手都值得看的Flink關鍵技術解析與優化實戰

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"本文由 dbaplus 社羣授權轉載。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本次分享主要分爲三部分。首先介紹流式計算的基本概念， 然後介紹Flink的關鍵技術，最後講講Flink在快手生產實踐中的一些應用，包括實時指標計算和快速failover。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、流式計算的介紹"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"流式計算的定義： "},{"type":"text","marks":[{"type":"strong"}],"text":"流式計算主要針對unbounded data（無界數據流）進行實時的計算，將計算結果快速的輸出或者修正。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這部分將分爲三個小節來介紹。第一，介紹大數據系統發展史，包括初始的批處理到現在比較成熟的流計算；第二，爲大家簡單對比下批處理和流處理的區別；第三，介紹流式計算裏面的關鍵問題，這是每個優秀的流式計算引擎所必須面臨的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1、大數據系統發展史"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/a6\/1c\/a615be46804c09d4ac58b11d190b791c.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是2003年到2018年大數據系統的發展史，看看是怎麼一步步走到流式計算的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2003年，Google的MapReduce橫空出世，通過經典的Map&Reduce定義和系統容錯等保障來方便處理各種大數據。很快就到了Hadoop，被認爲是開源版的MapReduce， 帶動了整個apache開源社區的繁榮。再往後是谷歌的Flume，通過算子連接等pipeline的方式解決了多個MapReduce作業連接處理低效的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}