爲什麼Netflix“永不宕機”？

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上個月，Netflix出品的《魷魚遊戲》劇集再次引爆全球無數觀衆。雖然“身經百戰”的觀衆早已對這類大逃殺題材的作品建立起充分的心理預期，但Netflix憑藉着對反烏托邦題材的極致挖掘再次證明，他們有決心打造一出迄今爲止最成功的原創劇集，同時也向其他競爭對手證明自己的業務高度已然遙不可及。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與此同時，很多同類流媒體平臺還深陷老大難問題而無法自拔：視頻卡頓、控件失效或者服務崩潰等等。而Netflix卻鮮少出現問題，這離不開其優秀的基礎設施建設。近日，《The Verge》撰文談Netflix龐大的服務器網絡"},{"type":"text","marks":[{"type":"strong"}],"text":"Open Connect"},{"type":"text","text":"，其指出，正是這個不被太多人在意的技術細節，從根本上解決了大部分現代流媒體面臨的共同難題。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Open Connect的奧祕"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"過去十多年來，Netflix流媒體服務出過多少次技術問題？不能說沒有，但肯定屈指可數。服務的核心，也就是應用程序底層的工程方案，纔是決定一切流媒體成功與否的關鍵。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Netflix用了十年時間搭建起一套名爲"},{"type":"text","marks":[{"type":"strong"}],"text":"Open Connect"},{"type":"text","text":"的服務器網絡。Netflix公司Open Connect副總裁Gina Haspilaire表示，當初之所以要構建Open Connect，是因爲Netflix“清醒地意識到，我們需要構建一定水平的基礎設施技術才能維持起我們想要獲得的預期流量。我們覺得自己能成功，也知道當時的互聯網還支撐不起預期中的全球流量規模。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"沒人能在應用程序不斷崩潰、視頻老在緩衝的情況下堅持看完一部電影。Netflix對這個問題很有先見之明，他們知道要想保持穩定的質量水平，首先需要建立自己的內容分發系統。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Open Connect是Netflix的內容交付網絡，專門用於向全球用戶分發電視節目與電影。項目始於2012年，包括由Netflix向互聯網服務商提供物理設備，藉此實現媒體流量本地化。這些設備存儲着Netflix的內容副本，負責減少內容在抵達播放用戶之前需要經過的通道數量，藉此減輕網絡壓力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大多數主要流媒體服務都依賴於第三方內容交付網絡（即CDN）實現視頻傳遞，正是這種通行實踐讓Netflix的自有服務器網絡變得鶴立雞羣。如果沒有像Open Connect這樣的系統或者其他第三方CDN，互聯網服務商所請求的內容必須“通過一個對等點、再經由四到五個其他網絡，才能抵達數據源頭或者內容存儲位置。”內容交付網絡廠商Akamai媒體工程首席架構師Will Law強調，這不僅會拖慢內容交付速度，而且考慮到互聯網服務商可能需要付費才能完成內容訪問，所以整個實現成本也更爲高昂。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了迴避流量和成本問題，Netflix會提前將內容副本發送至自有服務器，從而防止Netflix流量在流媒體峯值時段被播放端網絡所阻塞。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“"},{"type":"text","marks":[{"type":"strong"}],"text":"我們的Open Connect會在離用戶互聯網服務商最近的位置部署一份Bridgerton副本——有時候甚至會直接部署在互聯網服務商的網絡之內。這就基本避免了互聯網服務商先通過網絡對接服務器、再把提取到的內容傳輸給用戶的過程。"},{"type":"text","text":"”Gina Haspilaire說道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/2e\/2e4f70998dffcf96bd99f80ac4e5a776.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而且，Netflix的服務器幾乎無處不在。目前，Netflix表示已經在158個國家\/地區部署有17000臺服務器，而且未來還計劃進一步擴展其內容交付網絡。根據介紹，Netflix會根據哪些位置周邊的用戶更多、哪些區域的互聯網服務器商合作門檻更低對服務器部署做出優先級排序。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Law解釋道，“"},{"type":"text","marks":[{"type":"strong"}],"text":"爲了提高性能，流媒體服務商紛紛將服務器儘可能靠近最終用戶，再由這些服務器提供最後一英里區域內的網絡內容。"},{"type":"text","text":"這就避免了把所有內容請求都返回源頭，再將內容經由互聯網整體傳遞的過程。這不僅減輕了互聯網的傳輸負擔，也緩解了對等點的承載強度。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"十年前Open Connect剛剛立項時，Netflix也確定了與互聯網服務商全面合作的基本部署思路。Netflix免費爲互聯網服務商提供服務器，再由內部可靠性團隊與這些服務商一道維護這些服務器設備。根據Netflix與Akamai的介紹，互聯網服務商也能從中受益，因爲就近獲取內容副本可以降低他們的網絡基礎設施運營成本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Law稱，“由此帶來的運營職責不算特別繁重，但卻帶來了令人釋懷的解脫。而且這些服務器的基本作用跟Akamai乃至其他CDN完全相同，唯一的區別就是它們只服務於Netflix，屬於Netflix內容的專屬CDN。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"包括Akamai在內大多數主要第三方CDN服務商都需要服務成千上萬客戶並應對企業提出的種種要求，而"},{"type":"text","marks":[{"type":"strong"}],"text":"Netflix的內部CDN只着眼於一個目標：分發Netflix內容。"},{"type":"text","text":"Law表示，爲了順利將電影或電視節目交付到每一位點播觀衆面前，內容分發商必須建立起這樣一套全面的CDN合作關係或者服務器網絡。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"雖然Netflix沒有透露他們搭建並維護這些服務器的具體成本，但也提到"},{"type":"text","marks":[{"type":"strong"}],"text":"自十年前立項以來，Open Connect的總投資已經達到10億美元上下。"},{"type":"text","text":"他們還在不斷向CDN投入大量資金，繼續堅定推進這一以良好觀看體驗與用戶高參與度爲核心的Netflix業務戰略。很明顯，Netflix的整個訂閱商業模式之所以成立，依靠的就是爲用戶提供符合預期的視頻流質量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Netflix還需要考慮到一個殘酷的現實：美國的互聯網基礎設施在本質上可謂支離破碎、頭尾難以相顧。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"電子前沿基金會的Katharine Trendacosta在採訪中表示，“"},{"type":"text","marks":[{"type":"strong"}],"text":"Netflix之所以必須建立自有CDN，是因爲美國的互聯網服務商實在垃圾得不行。"},{"type":"text","text":"他們很清楚，只要寄希望於互聯網服務商，就必然會有相當一部分用戶始終受到視頻緩衝或者低畫質圖像的折磨。”"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"爲什麼有效"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Netflix最關注的目標，就是無論互聯網服務商有多差勁、都能保證用戶享受到良好的內容觀看體驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲此，"},{"type":"text","marks":[{"type":"strong"}],"text":"Netflix會在服務器上爲每部劇集或影片保留3個副本，這些副本的圖像質量各不相同。如果您的互聯網基礎設施不堪重負甚至暫時中斷，系統會導入碼率較低的版本，保證用戶始終有內容可看。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Haspilaire解釋道，“我們會根據網絡質量調整內容，而不是根據內容調整網絡質量。正因爲如此，觀衆在觀看時感受不到網絡條件發生了哪些波動——流媒體內容一直在穩定呈現，而我們會隨時間推移隨時調整播放的版本……這樣即使短時間內斷開網絡又重新連入，用戶的屏幕上也不會出現緩衝提示。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼，爲什麼是3個副本？Trendacosta指出，美國本地的互聯網服務體驗毫無可靠性可言。設施宕機、WiFi連接不良乃至其他網絡故障都有可能影響用戶的互聯網訪問能力，而Netflix則儘可能與互聯網服務商合作以解決大部分網絡傳輸問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Haspilaire提到，Netflix會在非峯值時段預先部署好影音內容，保證這部分傳輸任務不會跟其他流量競爭互聯網傳輸帶寬。至於部署的具體內容和位置，Netflix會提前預測高人氣影片\/劇集，並將數據相應發送至目標服務器。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Haspilaire表示，“我們不只是把內容部署在世界各地的內容交付服務器上，同時也會根據預先做出的流行度預測設計部署方案。我們會預測哪些內容更受歡迎，並把影音儘可能部署在正確的服務器中。通過提前準備，我們的電影和劇集就能在黃金時段爲幾乎100%的觀衆提供本地副本，基本消除了由網絡服務中斷引發的糟糕觀感。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之後，Netflix會根據內容的受歡迎度在服務器上轉移視頻。"},{"type":"text","marks":[{"type":"strong"}],"text":"Open Connect擁有兩類服務器：閃存型（提供更快的交付速度）和存儲型（最多可容納350 TB數據）。"},{"type":"text","text":"如果存儲內容的人氣不斷提升，Netflix就會把相應的副本轉移到閃存服務器當中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Netflix公司解釋道，“閃存服務器的基本定位就是支撐大部分流量服務，所以隨着某些節目或影片的需求量提升，我們的OCA服務器就會把內容從存儲型轉移到閃存型、保障觀衆獲得穩定的觀看體驗。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新冠疫情的突然爆發，讓全球數億人窩在家中持續觀看流媒體內容。而這一突發狀況，也成爲Netflix這項十年規劃的收官之戰與立威之路。Haspilaire表示，“這場疫情以一種我們不願看到的方式測試了Netflix基礎設施與技術體系。”而Open Connect頂住壓力，證明自己有能力幫助Netflix應對不斷增長的未來需求。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"流媒體之王"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Open Connect是Netflix在疫情期間贏得良好市場表現的一大支柱，但Netflix力壓競爭對手靠的顯然不只這一招。Rayburn也向《The Verge》列舉了Netflix視頻與音頻編碼項目的例子，強調雖然看起來Netflix當下已經建立起“巨大的優勢”，但作爲戰場中的後來者，他們還需要憑藉服務品質與服務態度去吸引和留存用戶。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Rayburn指出，“每個人都承認，Netflix建立起了一套能夠大規模穩定運行的系統，而這也是整個流媒體行業最不關注的方向。"},{"type":"text","marks":[{"type":"strong"}],"text":"只有在規模化運營中始終提供良好、優質的消費者體驗，我們才能獲得如此龐大的訂閱者羣體。"},{"type":"text","text":"沒人擁有Netflix這樣的基礎設施規模，也沒人擁有同等程度的專業積累。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至於Netflix爲什麼這麼成功，已經有不少人從各個角度進行剖析，其技術和企業文化也一直廣受好評。資深架構師、微服務技術專家楊波曾向我們"},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s?__biz=MjM5MDE0Mjc4MA==&mid=2651087494&idx=2&sn=f86c9b3b4e1f5eeb73f41a2262f9c3e9&chksm=bdb99ad58ace13c30d3d027dba758a5d922cb8c8908777d9f6d5e37872b82173668ea3f26e69&scene=27#wechat_redirect","title":"","type":null},"content":[{"type":"text","text":"總結"}]},{"type":"text","text":"了一些Netflix的厲害之處，其中有幾點尤其能體現Netflix在技術創新上的大膽和前沿。比如，在 2009 年左右， Netflix 就開始啓動上公有云戰略（雖然背後也是吃了自建數據中心的苦頭），應用逐步向 AWS 遷移，這個過程一共持續了將近 5 年，到2015年的時候遷移完畢。而現在回想，2009 年左右的 AWS 其實還不太成熟，別說當時，即使現在也沒有幾個大廠敢提全部遷移公有云的戰略。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"又比如，Netflix早在 2013 年就提出了 Cloud Native 理念，也就是現在常說的雲原生，而且它的整個平臺 (AWS IaaS + PaaS) 都是雲原生的。近年容器技術大火，大家都在談不可變基礎設施的理念，本質就是鏡像部署。Netflix 在 2012 年前就已經實現鏡像部署，發佈的單位是 AMI(Amazon Machine Image) 鏡像。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面，開放開源也是 Netflix 的一種競爭戰略。在進行大規模生產級微服務架構實踐的同時， Netflix 也開源了整個微服務技術棧，比如現在大家耳熟能詳的 Zuul 網關、Eureka 服務發現註冊中心、Hystrix 熔斷限流、Archaius 配置等組件。對Netflix來說，開源有望能將自己的解決方案建立爲行業標準和最佳實踐。其次，開源亦可以幫助建立 Netflix 技術品牌，有利於僱傭、留住和吸引頂級工程師。第三，Netflix 可以從共享生態中獲得反饋輸入並受益。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着其服務體系的不斷擴張，也許終有一天各大老牌流媒體廠商反過來要從Netflix的基礎設施與自上而下的業務戰略中取經。甚至，Netflix的方案可能最終會成爲行業標準。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"參考鏈接："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/www.theverge.com\/22787426\/netflix-cdn-open-connect","title":"","type":null},"content":[{"type":"text","text":"https:\/\/www.theverge.com\/22787426\/netflix-cdn-open-connect"}]}]}]}