扎心！天天寫代碼，方向真的對嗎？

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“每個人的時間都是有限的，在有限的時間裏選擇一項值得投入的技術會變得尤爲重要。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"筆者從 2008 年開始工作到現在也有 12 個年頭了，一路走來都在和數據打交道，做過很多大數據底層框架內核的開發（Hadoop，Pig，Hive，Tez，Spark），也做過多年上層數據計算框架（Livy,  Zeppelin）以及數據應用開發，包括數據處理，數據分析以及機器學習。現在是 Apache Member 以及多個 Apache 項目的 PMC 。2018 年加入阿里巴巴實時計算團隊專注在 Flink 的研發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"        "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天我想結合自己過去的職業經歷來聊聊如何評估一項技術是否值得學習。我一直在大數據這個圈子，從最初的 Hadoop 到後來的 Hadoop 生態項目 Pig，Hive，Tez，然後又到新一代的計算引擎 Spark ，再到最近在做的 Flink ，大數據計算引擎貫穿我的整個職業生涯。我個人來說是比較幸運的，在每個階段都在做比較火的技術，當時更多的是憑着自己的興趣和直覺在選擇技術類型。現在回過頭來看我覺得需要從下面 3 個大的緯度來評估一項技術是否值得學習。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、技術深度"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、生態廣度"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、進化能力"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4f/4fb290bab8d379ad09924873f228f5c6.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":2}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"技術深度"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"技術深度是指這項技術的根基是否紮實，護城河是否夠寬夠深，是否很容易被其他技術所替代。通俗的來說就是這項技術是否解決了其他技術所不能解決的有重要價值的問題。這裏有兩個要點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、這個問題沒有人能解，是這項技術首先解決了這個問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、解決這個問題能夠帶來重大價值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"拿我職業生涯開始階段學習的 Hadoop 爲例。當時 Hadoop 剛出來的時候是一項革命性的技術，因爲當時除了 Google 宣稱自己內部有一套 GFS 和 MapReduce 系統外，業界其他公司都沒有一套完整的海量數據解決方案。而隨着互聯網技術的發展，數據量與日俱增，處理海量數據的能力迫在眉睫。Hadoop 的誕生正好解決了這一燃眉之急。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着技術的發展， Hadoop 的處理海量數據能力的優勢慢慢被人習慣，相反 Hadoop 存在的缺陷被人不斷詬病（性能差，MapReduce 編寫複雜等等）。而這時候Spark應運而生，解決了 Hadoop MapReduce 計算引擎的頑疾。Spark 遠超過 Hadoop 的計算性能以及極其優雅簡單的 API 迎合了當時用戶的需求，受到了廣大大數據工程師的熱捧。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在我在阿里巴巴從事的是關於 Flink 的研發工作，主要原因是我看到了工業界對實時性的需求以及 Flink 在實時計算這個領域的霸主地位。之前大數據遇到的最大挑戰在於數據規模大（所以大家會稱之爲“大數據”），經過工業界多年的努力和實踐，規模大這個問題基本已經解決了。接下來幾年，更大的挑戰在於速度，也就是實時性。而大數據的實時性並不是指簡單的傳輸數據或者處理數據的實時性，而是從端到端的實時，任何一個步驟速度慢了，就影響整個大數據系統的實時性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Flink 看來， Everything is stream 。Flink 的以 Stream 爲核心的架構是業界獨一無二的，由此而產生的性能優越，高擴展性，端到端 Exactly Once 等特性，更是使得 Flink 在流計算領域是當之無愧的王者。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前主流的流計算引擎有 3 個：Flink、Storm 和 SparkStreaming 。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a6/a6894332d22ecf21e5e42838ded91ecd.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注：Spark Streaming 只能選擇搜索字詞，理論上這樣的對比是不嚴謹的。但作爲趨勢，我們更關注的是其變化曲線，實際影響應該不大。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上面的 Google trends 曲線可以看出，Flink 處在一個快速增長期， Storm 的熱度在逐年下降，而 Spark Streaming 幾乎進入了平臺期。這就證明了 Flink 在流計算領域的根基之深，目前來看還沒有誰可以超越 Flink 在流計算領域的霸主地位。"}]},{"type":"heading","attrs":{"align":null,"level":1}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"生態廣度"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一項技術只有技術深度是不夠的，因爲一項技術只能專注於做好一件事情，如果要解決實際生活中的複雜問題，必定要和其他技術整合聯動，這就要求這項技術具有足夠寬的生態廣度。生態的廣度有 2 個緯度可以衡量："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、上下游生態。上下游生態指從數據流的角度來說的數據上下游。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、垂直領域生態。垂直領域生態是指某個細分領域或者應用場景的整合。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a7/a71d07a090a1999105f02e9762768cd6.png","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當 Hadoop 剛出來的時候只有 2 個基本的組件：HDFS 和 MapReduce ，分別解決了海量存儲和分佈式計算的問題。但隨着發展，需要解決的問題越來越複雜，HDFS 和 MapReduce 已經不能很方便的解決一些複雜問題，這時候 Hadoop 的其他生態項目應運而生，比如 Pig，Hive，HBase 等等從垂直領域生態這個角度解決了 Hadoop 不容易或者不能解決的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Spark 亦是如此，一開始的 Spark 是要替換原來的 MapReduce 計算引擎，後來 Spark 發展了各種語言接口，各種上層框架，比如 Spark SQL，Spark Structured Streaming，MLlib，GraphX 等等，大大豐富了 Spark 的使用場景，擴展了Spark的垂直領域生態。Spark 對各種 Data Source 的支持，更是讓 Spark 這個計算引擎和存儲結成了聯盟，建立了強大的上下游生態系統，爲端到端的解決方案奠定了基礎。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我現在做的 Flink 項目的生態仍然處於起步階段，當時我加入阿里巴巴正不僅僅是看到了 Flink 作爲流計算引擎的霸主地位，更是因爲看到了 Flink 生態的機會。大家如果從我的職業生涯來看，會發現些許變化，我在從一開始專注於大數據的核心框架層慢慢在往周邊生態項目發展。一個主要的原因是我對整個大數據行業的判斷：大數據上半場戰鬥集中在底層框架，目前已經接近尾聲，未來的底層大數據生態圈中將不再有那麼多的新的技術和框架，每個細分領域都將優勝劣汰，走向成熟，更加集中化。下半場戰鬥的重點講從底層走向上層，走向生態。之前的大數據創新更偏向於 IAAS 和 PAAS ，未來你將看到更多 SAAS 類型的大數據產品和創新。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/35/3576480ad3092c0e21352a6edb04a1ab.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每次談到大數據的生態，我都拿出上面這張圖。這張圖基本上把你日常需要處理的大數據場景都包括進來。從最左邊的數據生產者，到數據收集，數據處理，然後再到數據應用（BI + AI）。你會發現 Flink 可以應用在每一個步驟。不僅涉及到大數據，也涉及到 AI ，但是 Flink 的強項在於流計算處理，在其他領域的生態仍在起步階段，我個人正在做的工作就是完善 Flink 在上面這張圖上端到端的能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"進化能力"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 一項技術如果技術深度和生態廣度都沒有問題，那麼至少說明這項技術在當下是值得學習的。但是投資一項技術還需要從時間這個緯度上考量。你肯定不希望自己學習的技術很快就被淘汰，每年都要去學習一項新技術。所以一項值得投資學習的技術必定需要具有持久的進化能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我最初學的 Hadoop 到現在已經 10 多年了，現在仍然被廣泛使用着。雖然現在有很多公有云廠商在搶佔 Hadoop 的市場，但你不得不承認如果一家公司要成立一個大數據部門，第一件事恐怕就是建一個 Hadoop 集羣吧。當我們現在談論 Hadoop 的時候，他已經不是當初的 Hadoop 了，他更多的是 Hadoop 生態圈的統稱。大家有空可以看看 Cloudera CPO Arun 的這篇文章"},{"type":"text","marks":[{"type":"strong"}],"text":"【1】"},{"type":"text","text":"，我對其中的觀點非常認同。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"【1】："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"},{"type":"underline"}],"text":"https://medium.com/@acmurthy/hadoop-is-dead-long-live-hadoop-f22069b264ac"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Spark 項目就更不用多說了。Spark 經過 14，15 年爆發，現在已經進入平穩期。但是 Spark 仍在進化，仍在擁抱變化。Spark on K8s 就是 Spark 擁抱雲原生的最好佐證。現在 Spark 社區炙手可熱的Delta，MLFlow 更是 Spark 的強大的進化能力的佐證。現在的 Spark 也不僅僅是當年要取代 MapReduce 的那個 Spark ，更多是一個適用於多種場景的通用計算引擎。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我從 18 年加入阿里巴巴到現在差不多 1 年半時間，在這一年半的時間了，我正好見證了 Flink 的進化能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先 Flink 經過幾個大版本的發佈，融入了 Blink 的大部分功能，將 Flink SQL 的能力提升了一大截。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其次 Flink 對 K8s 的支持，對 Python 的支持，對 AI 的支持都在向人們證明這Flink自身強大的進化能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"小 Tips"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了以上的 3 大維度，在這裏我還想分享下我在評估一項新技術時候的一些小技巧。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、利用 Google trends 。Google trends 能很好的反映一項技術的發展勢頭，上面提到的趨勢圖很好的比較了 3 大流計算引擎 Flink ， Spark Streaming 和 Storm ，我們不難得出結論：Flink 是流計算領域的王者。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、查看 GitHub 上的awesome。一項技術受歡迎的一個指標是 GitHub 上的 awesome list，你可以看看這個 awesome list 的 GitHub star 數。此外你可以抽一個週末的時間看看這個 awesome list 上的內容，因爲上面基本上是關於這項技術的精華內容，通過這些內容你大致可以判斷出這項技術的價值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、看看技術網站上是否有一些技術佈道者爲這項技術背書（我個人經常會看medium.com）。技術圈裏通常有這樣一羣人，他們對技術很執着，也很有品位。如果一項技術真的很好，那麼就會有技術佈道者無償的爲這項技術背書，分享如何這項技術的使用心得。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每個人的時間都是有限的，在有限的時間裏選擇一項值得投入的技術會變得尤爲重要。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上是我對如何評估一項技術是否值得學習的一些思考，也算是對我自己事業生涯在技術選型方面的一個小小的總結和回顧，希望我的這些思考能對大家的職業生涯有所幫助。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"章劍鋒（簡鋒），開源界老兵，Github ID：@zjffdu，Apache Member，曾就職於 Hortonworks，目前在阿里巴巴計算平臺事業部任高級技術專家，並同時擔任 Apache Tez、Livy 、Zeppelin 三個開源項目的 PMC ，以及 Apache Pig 的 Committer。有幸很早就接觸了大數據和開源，希望可以在開源領域爲大數據和數據科學做點貢獻。"}]},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關注 Flink 中文社區，獲取更多技術乾貨"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2c/2c0578a2f5f12d22e30673c9b6bc4e86.jpeg","alt":null,"title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}