为什么Netflix“永不宕机”？

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上个月，Netflix出品的《鱿鱼游戏》剧集再次引爆全球无数观众。虽然“身经百战”的观众早已对这类大逃杀题材的作品建立起充分的心理预期，但Netflix凭借着对反乌托邦题材的极致挖掘再次证明，他们有决心打造一出迄今为止最成功的原创剧集，同时也向其他竞争对手证明自己的业务高度已然遥不可及。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"与此同时，很多同类流媒体平台还深陷老大难问题而无法自拔：视频卡顿、控件失效或者服务崩溃等等。而Netflix却鲜少出现问题，这离不开其优秀的基础设施建设。近日，《The Verge》撰文谈Netflix庞大的服务器网络"},{"type":"text","marks":[{"type":"strong"}],"text":"Open Connect"},{"type":"text","text":"，其指出，正是这个不被太多人在意的技术细节，从根本上解决了大部分现代流媒体面临的共同难题。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Open Connect的奥秘"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"过去十多年来，Netflix流媒体服务出过多少次技术问题？不能说没有，但肯定屈指可数。服务的核心，也就是应用程序底层的工程方案，才是决定一切流媒体成功与否的关键。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Netflix用了十年时间搭建起一套名为"},{"type":"text","marks":[{"type":"strong"}],"text":"Open Connect"},{"type":"text","text":"的服务器网络。Netflix公司Open Connect副总裁Gina Haspilaire表示，当初之所以要构建Open Connect，是因为Netflix“清醒地意识到，我们需要构建一定水平的基础设施技术才能维持起我们想要获得的预期流量。我们觉得自己能成功，也知道当时的互联网还支撑不起预期中的全球流量规模。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"没人能在应用程序不断崩溃、视频老在缓冲的情况下坚持看完一部电影。Netflix对这个问题很有先见之明，他们知道要想保持稳定的质量水平，首先需要建立自己的内容分发系统。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Open Connect是Netflix的内容交付网络，专门用于向全球用户分发电视节目与电影。项目始于2012年，包括由Netflix向互联网服务商提供物理设备，借此实现媒体流量本地化。这些设备存储着Netflix的内容副本，负责减少内容在抵达播放用户之前需要经过的通道数量，借此减轻网络压力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大多数主要流媒体服务都依赖于第三方内容交付网络（即CDN）实现视频传递，正是这种通行实践让Netflix的自有服务器网络变得鹤立鸡群。如果没有像Open Connect这样的系统或者其他第三方CDN，互联网服务商所请求的内容必须“通过一个对等点、再经由四到五个其他网络，才能抵达数据源头或者内容存储位置。”内容交付网络厂商Akamai媒体工程首席架构师Will Law强调，这不仅会拖慢内容交付速度，而且考虑到互联网服务商可能需要付费才能完成内容访问，所以整个实现成本也更为高昂。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为了回避流量和成本问题，Netflix会提前将内容副本发送至自有服务器，从而防止Netflix流量在流媒体峰值时段被播放端网络所阻塞。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“"},{"type":"text","marks":[{"type":"strong"}],"text":"我们的Open Connect会在离用户互联网服务商最近的位置部署一份Bridgerton副本——有时候甚至会直接部署在互联网服务商的网络之内。这就基本避免了互联网服务商先通过网络对接服务器、再把提取到的内容传输给用户的过程。"},{"type":"text","text":"”Gina Haspilaire说道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/2e\/2e4f70998dffcf96bd99f80ac4e5a776.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而且，Netflix的服务器几乎无处不在。目前，Netflix表示已经在158个国家\/地区部署有17000台服务器，而且未来还计划进一步扩展其内容交付网络。根据介绍，Netflix会根据哪些位置周边的用户更多、哪些区域的互联网服务器商合作门槛更低对服务器部署做出优先级排序。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Law解释道，“"},{"type":"text","marks":[{"type":"strong"}],"text":"为了提高性能，流媒体服务商纷纷将服务器尽可能靠近最终用户，再由这些服务器提供最后一英里区域内的网络内容。"},{"type":"text","text":"这就避免了把所有内容请求都返回源头，再将内容经由互联网整体传递的过程。这不仅减轻了互联网的传输负担，也缓解了对等点的承载强度。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"十年前Open Connect刚刚立项时，Netflix也确定了与互联网服务商全面合作的基本部署思路。Netflix免费为互联网服务商提供服务器，再由内部可靠性团队与这些服务商一道维护这些服务器设备。根据Netflix与Akamai的介绍，互联网服务商也能从中受益，因为就近获取内容副本可以降低他们的网络基础设施运营成本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Law称，“由此带来的运营职责不算特别繁重，但却带来了令人释怀的解脱。而且这些服务器的基本作用跟Akamai乃至其他CDN完全相同，唯一的区别就是它们只服务于Netflix，属于Netflix内容的专属CDN。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"包括Akamai在内大多数主要第三方CDN服务商都需要服务成千上万客户并应对企业提出的种种要求，而"},{"type":"text","marks":[{"type":"strong"}],"text":"Netflix的内部CDN只着眼于一个目标：分发Netflix内容。"},{"type":"text","text":"Law表示，为了顺利将电影或电视节目交付到每一位点播观众面前，内容分发商必须建立起这样一套全面的CDN合作关系或者服务器网络。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"虽然Netflix没有透露他们搭建并维护这些服务器的具体成本，但也提到"},{"type":"text","marks":[{"type":"strong"}],"text":"自十年前立项以来，Open Connect的总投资已经达到10亿美元上下。"},{"type":"text","text":"他们还在不断向CDN投入大量资金，继续坚定推进这一以良好观看体验与用户高参与度为核心的Netflix业务战略。很明显，Netflix的整个订阅商业模式之所以成立，依靠的就是为用户提供符合预期的视频流质量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Netflix还需要考虑到一个残酷的现实：美国的互联网基础设施在本质上可谓支离破碎、头尾难以相顾。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"电子前沿基金会的Katharine Trendacosta在采访中表示，“"},{"type":"text","marks":[{"type":"strong"}],"text":"Netflix之所以必须建立自有CDN，是因为美国的互联网服务商实在垃圾得不行。"},{"type":"text","text":"他们很清楚，只要寄希望于互联网服务商，就必然会有相当一部分用户始终受到视频缓冲或者低画质图像的折磨。”"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"为什么有效"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Netflix最关注的目标，就是无论互联网服务商有多差劲、都能保证用户享受到良好的内容观看体验。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"为此，"},{"type":"text","marks":[{"type":"strong"}],"text":"Netflix会在服务器上为每部剧集或影片保留3个副本，这些副本的图像质量各不相同。如果您的互联网基础设施不堪重负甚至暂时中断，系统会导入码率较低的版本，保证用户始终有内容可看。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Haspilaire解释道，“我们会根据网络质量调整内容，而不是根据内容调整网络质量。正因为如此，观众在观看时感受不到网络条件发生了哪些波动——流媒体内容一直在稳定呈现，而我们会随时间推移随时调整播放的版本……这样即使短时间内断开网络又重新连入，用户的屏幕上也不会出现缓冲提示。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那么，为什么是3个副本？Trendacosta指出，美国本地的互联网服务体验毫无可靠性可言。设施宕机、WiFi连接不良乃至其他网络故障都有可能影响用户的互联网访问能力，而Netflix则尽可能与互联网服务商合作以解决大部分网络传输问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Haspilaire提到，Netflix会在非峰值时段预先部署好影音内容，保证这部分传输任务不会跟其他流量竞争互联网传输带宽。至于部署的具体内容和位置，Netflix会提前预测高人气影片\/剧集，并将数据相应发送至目标服务器。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Haspilaire表示，“我们不只是把内容部署在世界各地的内容交付服务器上，同时也会根据预先做出的流行度预测设计部署方案。我们会预测哪些内容更受欢迎，并把影音尽可能部署在正确的服务器中。通过提前准备，我们的电影和剧集就能在黄金时段为几乎100%的观众提供本地副本，基本消除了由网络服务中断引发的糟糕观感。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之后，Netflix会根据内容的受欢迎度在服务器上转移视频。"},{"type":"text","marks":[{"type":"strong"}],"text":"Open Connect拥有两类服务器：闪存型（提供更快的交付速度）和存储型（最多可容纳350 TB数据）。"},{"type":"text","text":"如果存储内容的人气不断提升，Netflix就会把相应的副本转移到闪存服务器当中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Netflix公司解释道，“闪存服务器的基本定位就是支撑大部分流量服务，所以随着某些节目或影片的需求量提升，我们的OCA服务器就会把内容从存储型转移到闪存型、保障观众获得稳定的观看体验。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"新冠疫情的突然爆发，让全球数亿人窝在家中持续观看流媒体内容。而这一突发状况，也成为Netflix这项十年规划的收官之战与立威之路。Haspilaire表示，“这场疫情以一种我们不愿看到的方式测试了Netflix基础设施与技术体系。”而Open Connect顶住压力，证明自己有能力帮助Netflix应对不断增长的未来需求。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"流媒体之王"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Open Connect是Netflix在疫情期间赢得良好市场表现的一大支柱，但Netflix力压竞争对手靠的显然不只这一招。Rayburn也向《The Verge》列举了Netflix视频与音频编码项目的例子，强调虽然看起来Netflix当下已经建立起“巨大的优势”，但作为战场中的后来者，他们还需要凭借服务品质与服务态度去吸引和留存用户。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Rayburn指出，“每个人都承认，Netflix建立起了一套能够大规模稳定运行的系统，而这也是整个流媒体行业最不关注的方向。"},{"type":"text","marks":[{"type":"strong"}],"text":"只有在规模化运营中始终提供良好、优质的消费者体验，我们才能获得如此庞大的订阅者群体。"},{"type":"text","text":"没人拥有Netflix这样的基础设施规模，也没人拥有同等程度的专业积累。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至于Netflix为什么这么成功，已经有不少人从各个角度进行剖析，其技术和企业文化也一直广受好评。资深架构师、微服务技术专家杨波曾向我们"},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s?__biz=MjM5MDE0Mjc4MA==&mid=2651087494&idx=2&sn=f86c9b3b4e1f5eeb73f41a2262f9c3e9&chksm=bdb99ad58ace13c30d3d027dba758a5d922cb8c8908777d9f6d5e37872b82173668ea3f26e69&scene=27#wechat_redirect","title":"","type":null},"content":[{"type":"text","text":"总结"}]},{"type":"text","text":"了一些Netflix的厉害之处，其中有几点尤其能体现Netflix在技术创新上的大胆和前沿。比如，在 2009 年左右， Netflix 就开始启动上公有云战略（虽然背后也是吃了自建数据中心的苦头），应用逐步向 AWS 迁移，这个过程一共持续了将近 5 年，到2015年的时候迁移完毕。而现在回想，2009 年左右的 AWS 其实还不太成熟，别说当时，即使现在也没有几个大厂敢提全部迁移公有云的战略。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"又比如，Netflix早在 2013 年就提出了 Cloud Native 理念，也就是现在常说的云原生，而且它的整个平台 (AWS IaaS + PaaS) 都是云原生的。近年容器技术大火，大家都在谈不可变基础设施的理念，本质就是镜像部署。Netflix 在 2012 年前就已经实现镜像部署，发布的单位是 AMI(Amazon Machine Image) 镜像。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面，开放开源也是 Netflix 的一种竞争战略。在进行大规模生产级微服务架构实践的同时， Netflix 也开源了整个微服务技术栈，比如现在大家耳熟能详的 Zuul 网关、Eureka 服务发现注册中心、Hystrix 熔断限流、Archaius 配置等组件。对Netflix来说，开源有望能将自己的解决方案建立为行业标准和最佳实践。其次，开源亦可以帮助建立 Netflix 技术品牌，有利于雇佣、留住和吸引顶级工程师。第三，Netflix 可以从共享生态中获得反馈输入并受益。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"随着其服务体系的不断扩张，也许终有一天各大老牌流媒体厂商反过来要从Netflix的基础设施与自上而下的业务战略中取经。甚至，Netflix的方案可能最终会成为行业标准。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"参考链接："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/www.theverge.com\/22787426\/netflix-cdn-open-connect","title":"","type":null},"content":[{"type":"text","text":"https:\/\/www.theverge.com\/22787426\/netflix-cdn-open-connect"}]}]}]}