Facebook史上最严重宕机：互联网企业是时候重新审视架构了？

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"刚被指责“利用放大仇恨言论的算法谋取利益”没多久，Facebook再次陷入危机。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"美国东部时间10月4日上午 11 点 39 分左右，美国社交媒体Facebook、Instagram和即时通讯软件WhatsApp出现大规模宕机，此次宕机长达近7个小时，刷新了Facebook自 2008年以来的最长宕机时长。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"美国互联网监控网站DownDectors的监控情况显示，Facebook在欧洲、美洲、大洋洲几乎是完全下线，在亚洲的日本、韩国、印度等国也无法访问。据悉，WhatsApp 和 Facebook Messenger 两款“微信”类即时通信产品，分别在全球范围拥有 20 亿用户和 13 亿用户，社交平台 Instagram 用户数也达到了10亿用户。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"除了让数十亿用户陷入困境之外， Facebook 服务中断还使其员工无法使用内部工具相互交流。Facebook 的电子邮件和工具都是企业内部管理的，Facebook 很多员工也无法正常工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/10\/100e8ad14861221ff45438cb24e90604.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Facebook首席技术官Mike Schroepfer在推特上道歉"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一条指令引发的“血案”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Facebook 表示，这次故障的根本原因是例行维护工作发出了一条糟糕的指令，结果导致其DNS服务器不可使用，切断了Facebook 整个骨干网络与数据中心之间的连接。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"所谓骨干网，是Facebook为一切计算设施构建而成的全局连接网络，由长达数万英里的光纤线缆组成，跨越全球并将各地的数据中心连接了起来。Facebook基础设施副总裁 Santosh Janardhan在"},{"type":"link","attrs":{"href":"https:\/\/engineering.fb.com\/2021\/10\/05\/networking-traffic\/outage-details\/","title":null,"type":null},"content":[{"type":"text","text":"文章中"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"解释道，数据中心主要有两种形式，一种是存放着数百万台数据存储与高强度计算负载运行设备的“巨大的建筑”，另一种则属于小型设备，通过骨干网络接入整体互联网并构建起Facebook社交平台的方方面面。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"当用户打开应用并加载摘要或消息时，应用提出的数据请求会由当前设备传输至最近的设施，之后再直接通过骨干网络与更大的数据中心进行通信。应用所需要的信息将在这些数据中心内进行检索与处理，再把结果通过网络发送回用户手机上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"维护基础设施的日常工作非常繁重。工程师们还经常需要让部分骨干网络离线以实施维护——包括修复光纤线路、扩大容量或者更新路由器自身软件等等。而这也是此次宕机事件的原因所在。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Janardhan表示，在一项日常维护工作中，工程师们发出一条用于评估全球骨干网容量可用性的指令，但意外切断了骨干网络中的所有连接，这实质上就是断开了Facebook全球数据中心之间的连接。不幸的是，Facebook的系统在设计上能够审查此类指令以防止出现错误，但其功能只是发出错误提示，并不能真正阻止指令执行。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"这次意外，导致Facebook的数据中心与互联网之间的服务器连接完全断开，由此带来了一系列连锁效应让情况进一步恶化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"在此次宕机事件中，由于整个骨干网都已陷入瘫痪，因此各DNS服务器位置均上报连接状态问题并撤回边界网关协议（BGP）通告。最终结果是，Facebook的DNS服务器虽然仍在运行但却无法正常访问，导致其他互联网用户也无法正常接入其服务器。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"响应DNS查询是小型设施执行的一项重要任务。DNS可以称之为互联网的地址簿，能够将用户在浏览器中输入的简单网络名称转换为特定的服务器IP地址。这些转换查询由Facebook的权威名称服务器给出应答，而这些服务器本身就占用着最众所周知的IP地址。接下来，这些服务器再通过边界网关协议（BGP）向互联网的其余部分发布通告。为了确保运行可靠性，如果DNS服务器自身无法与数据中心通信，则所有BGP通告都将被禁用，表示当前网络连接状态不正确。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"简单来说，Facebook 拿走了告诉世界计算机如何找到其各种在线资产的地图。结果，当在 Web 浏览器中键入 Facebook.com 时，浏览器不知道在哪里可以找到 Facebook.com，因此返回到了错误页面。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"为什么无法及时修复"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"为什么这次故障持续了近7个小时之久呢？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Janardhan表示，工程师们在修复这一故障时，面临着两个巨大的障碍：首先，Facebook的工程师们无法通过正常方式访问自己的Facebook数据中心，因为这时候骨干网已经出现了故障；其次，DNS没有响应致使Facebook无法使用调查及解决宕机问题的常规内部工具。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"骨干网与带外网络访问均出现故障，这意味着工程师只能亲自前往现场进行调试并尝试重启系统。但这需要时间，因为各处设施都遵循高水平的物理与系统安全保护政策。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#555555","name":"user"}}],"text":"错误的更新阻止了 Facebook 员工（其中大多数是远程工作）恢复和更改系统。与此同时，那些可以物理访问 Facebook 大楼的人无法访问 Facebook 的内部工具。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"“任何人员都很难进入，而且一旦进入并获得物理访问能力，这些硬件与路由器的设计也很难得到修改。因此，需要更多的时间将工程师们引导进机房，并为他们提供在服务器上工作所需要的安全访问协议。只有这样，我们才能确认问题并让骨干网重新上线。”Janardhan写道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/83\/834b69bc4af3aefccd42b96f25c1da41.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"有专家估计，Facebook、Instagram、WhatsApp 全球服务中断一小时将给全球经济造成 1.6 亿美元的损失。同时，Facebook当日股价盘中暴跌6%，扎克伯格个人财富一日蒸发逾60亿美元。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"屋漏偏逢连夜雨。在 Facebook 全球网络服务中断期间，据称在黑客论坛上有超过 15 亿 Facebook 用户的数据被出售。但Facebook 方面否认了这次用户数据泄露与服务中断有关。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"“我们要明确表示，这次宕机背后没有恶意活动，其根本原因是我们端的错误配置更改。我们也没有证据表明用户数据因此次停机而受到损害。”Janardhan"},{"type":"link","attrs":{"href":"https:\/\/engineering.fb.com\/2021\/10\/04\/networking-traffic\/outage\/","title":null,"type":null},"content":[{"type":"text","text":"说道"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"架构缺陷"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"美国东部时间下午 6 点 33 分，"},{"type":"link","attrs":{"href":"https:\/\/twitter.com\/Facebook\/status\/1445155265360416773?s=20","title":null,"type":null},"content":[{"type":"text","text":"Facebook 发推文"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"称其应用程序和服务已开始恢复运行。随着各数据中心区域中的骨干网连接的恢复，一切都随之复原。但问题还没有真正结束。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"一次性对所有服务全部重启会带来新的隐患，因为流量激增很可能导致新一轮崩溃。个别数据中心还上报称宕机导致设施耗电量下降了几十兆瓦，而突然上线带来的用电量暴增很可能给电气系统、缓存等各类装置带来意外冲击。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Janardhan表示，虽然Facebook 一直在做“风暴”演习，对服务、数据中心乃至整个区域进行脱机，并针对一切相关基础设施与软件开展压力测试以模拟主要系统故障，但并未演练过全球骨干网络脱机的状况，后续会找可行性方法作出应对。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"据监测互联网流量和故障的思科ThousandEyes的产品营销主管Angelique Medina表示，这起事件暴露了Facebook 架构的一个缺点：如果本身出现DNS故障，又没有后备DNS，就可能会出现长时间的故障，“所以我认为，这件事带来的一大经验教训就是要有冗余DNS。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Medina表示，一套更稳健的架构将拥有双DNS服务，那样一个DNS服务可以支援另一个。据Medina声称，比如说，亚马逊（其AWS提供DNS服务）为其DNS使用两项外部服务：Dyn和UltraDNS。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同时，这次宕机事件也让身处反垄断调查的 Facebook 雪上加霜。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"美国国会众议院成员Alexandria Ocasio-Cortez表示，Facebook爆发大规模宕机事故，这凸显出该公司在全球通信和其他服务领域的垄断地位。其在推特上表示，Facebook 周一发生的大规模宕机事故是对该公司垄断全球通讯和其他服务的一次提醒，再次表明 Facebook 应该被分拆。"}]}]}