字節跳動微服務架構體系演進

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文整理自字節跳動（火山引擎）基礎架構\/服務框架團隊負責人成國柱在 QCon 2021 的分享，主要介紹了 2018-2021 年間，服務框架團隊在 Golang 服務框架和 Service Mesh 上的技術實踐和經驗總結。"}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"字節跳動微服務架構概述"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在字節跳動，微服務架構的特徵可以被歸納爲 4 點，如下圖所示："}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/2d\/2d96bc7b946254b1562c82bb4aa6cf04.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先是"},{"type":"text","marks":[{"type":"strong"}],"text":"規模大、增長快"},{"type":"text","text":"。近三年來，字節跳動的微服務數量和規模迎來快速發展。2018 年，我們的在線微服務數大約是 7000-8000，到今年五月份，這一數字已經突破 5 萬。伴隨快速增長，服務框架團隊也遇到了非常多挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其次是"},{"type":"text","marks":[{"type":"strong"}],"text":"全面容器化、PaaS 化"},{"type":"text","text":"。字節跳動的在線微服務，超過 90％ 都運行在容器裏。所有上線都通過 PaaS 化平臺進行，這意味着線上不會存在物理機部署這種模式。這種做法既有一些挑戰：增加調度複雜性；也帶來了一些便利性：有利於新功能的推廣。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三，字節跳動的技術體系"},{"type":"text","marks":[{"type":"strong"}],"text":"以 Golang 語言爲主"},{"type":"text","text":"。根據最新的調查統計，公司裏有超過 55％ 的服務是採用 Golang 的，排名第二的語言是前端的 NodeJS，之後是 Python、JAVA、C++，Rust 也有一些使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後是 "},{"type":"text","marks":[{"type":"strong"}],"text":"Service Mesh 在字節跳動目前已經是全面落地狀態"},{"type":"text","text":"，這塊的詳細內容會在本文第三節展開。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於以上 4 個特點，當前字節跳動微服務架構遇到的主要挑戰還是圍繞研發效率、運行效率和穩定性。其中研發效率和穩定性是幾乎所有互聯網公司都會遇到的：多語言、易用性、性能、成本……在這些問題中，字節跳動服務框架團隊和火山引擎雲原生團隊最關注的是以下三個："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"快速迭代。研發和上線一定要快；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對多語言的支持要足夠好。配合員工規模增速，要對多語言保持非常包容的態度；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"運行時的穩定性。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Golang 微服務框架的演進"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2014 年，Golang 被引入字節跳動，以快速解決長連接推送業務所面臨的高併發問題。到 2016 年，技術團隊基於 Golang 推出了一個名爲 Kite 的框架，同時對開源項目 Gin 做了一層很薄的封裝，推出了 Ginex。這兩個原始框架的推出，極大推動了 Golang 在公司內部的應用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a6\/a6aa9d8cf35f5473522107fbf7a4d2d6.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這種情況一直持續到 2019 年年中。在 Kite 和 Ginex 發佈之初，由於很多功能版本過低，包括 Thrift 當時只有 v0.9.2，它們其實存在很多問題，再加上 Golang 迎來數輪大版本迭代，Kite 甚至連 golang context 參數都沒有。綜上種種原因，Kite 已經滿足不了內部使用需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 2019 年中，服務框架團隊正式啓動了 Kite 這個字節自有 RPC 框架的重構。這是一個自下而上的整體升級重構，圍繞"},{"type":"text","marks":[{"type":"strong"}],"text":"性能和可擴展性"},{"type":"text","text":"的訴求展開設計。2020 年 10 月，團隊完成了 KiteX 發佈，僅僅兩個月後，KiteX 就已經接入超過 1000 個服務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"類似的設計思路和底層模塊也被應用在字節跳動自研 Golang HTTP 框架 Hertz 上，該項目在 2021 年春晚當天承載的服務峯值 QPS 超過 1000w（未統計物理機部署服務），線上無一例異常反饋。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"RPC 框架 KiteX"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"KiteX 是字節跳動研發的下一代高性能、強可擴展的 Golang RPC 框架。除了具備豐富的服務治理特性，它還集成了自研的網絡庫 Netpoll，支持多消息協議（Thrift\/Protobuf）和多交互方式（Ping-Pong\/Oneway\/ Streaming），提供自研的、更加靈活可擴展的代碼生成器。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖是 KiteX 的架構圖，左側是它的一些核心特點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/2a\/2a6139b03fce2e9ce50d054b9ca4831d.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kite 在發佈之初使用的是原生的庫，所以存在兩個最主要的問題：一是無法直接感知對端連接狀態；二是原生 net 網絡庫在面對長連接時，容易產生 goroutine 爆炸的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這在 HTTP 場景下其實很常見，開發人員經常會遇到一個 HTTP 服務（包括使用 Gin 這種框架的時候）的內存出現持續增長。爲什麼？因爲實際場景中客戶端經常忘記關連接。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Netpoll 可以通過一個 React 模型去解決連接爆炸的問題，同時也帶來一些 zero-copy buffer 的使用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"關於多協議，字節跳動的大多數服務採用的都是 Thrift 協議。而這裏的多協議是應對一些業務提出的支持其他協議的需求。例如後端服務和端上的 Protobuf 打通。通過多協議支持實現了它和 Thrift 的解耦。現在 KiteX "},{"type":"text","marks":[{"type":"strong"}],"text":"同時支持 PB 和 Thrift 協議，也支持靈活的自定義協議擴展"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"高性能是所有做框架的開發者共同追求的目標，這裏不再贅述。最後，"},{"type":"text","marks":[{"type":"strong"}],"text":"KiteX 是面向開源社區設計的"},{"type":"text","text":"。如上圖所示，KiteX Core 是可以直接開源的部分，右側的 KiteX Byted 涉及字節跳動內部工具，這塊在對外時可以被替換成一些開源的日誌庫和方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"截至今年 5 月，公司內已經有超過 20000 個服務正在使用 KiteX，同時它也支持了 streaming\/泛化調用等需求。在開源上，技術團隊也提供了一些監控和日誌可插拔的 feature。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/2d\/2d94b82cd92d163edbb05e00f613d963.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在性能上，相比當前社區比較認可的 gRPC 框架，kitex\/thrift 實現了 2.5 倍吞吐量的提升，kitex\/protobuf 也有近 2 倍的提升。在 TP99 延遲上，KiteX 也有不錯的表現。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"HTTP 框架 Hertz"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2020 年 10 月，字節跳動內部推出 Hertz，作爲代替 Ginex（Gin 封裝版）的 Golang HTTP 首選框架。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hertz 的設計思路和 KiteX 大致相同。它借鑑了開源項目 Gin、FastHTTP 的優勢，引入了 Netpoll、內存池、零拷貝等技術，因此有着較高的性能。同等配置下，Hertz 的極限 QPS 是 Ginex 的兩倍，平均延時則只有 Ginex 的二分之一。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖是技術團隊基於最新版本製作的 Benchmark。在小包和中包場景下，Hertz 相比 FastHTTP 有 30％ 到 50％ 以上的 QPS 提升；在大包場景，雙方差距不大，Hertz 在平均時延上佔優。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/e6\/e625da24e8765960138c333045ed5c6d.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"json 庫 Sonic"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2021 年春節之前，技術團隊對線上容器數量排名前 50 的服務進行了性能瓶頸分析。通過調研，團隊發現 json 在整體上佔了 9.5% 的資源，在某些服務上的資源佔用甚至達到 40%！"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"既然 json 性能這麼差，爲什麼不換成 Protobuf？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在當時，技術團隊確實將部分服務切成了 Protobuf，但考慮到 json 的易用性、受歡迎程度，以及更換協議導致的遷移成本和遷移風險，他們也開始嘗試進行一些優化。他們調研了所有可獲取的 Golang json 庫，並在業務場景下都做了比較詳細的對比和分析，發現了一些問題並做了優化："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"JIT：有些 json 庫已經開始展現出 JIT 特徵，但還停留在把常見代碼片段通過聚合的方式組織起來，沒有做到非常極致的 just in time 編譯，因此需要推進一步；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前置篩選+SIMD：simdjson 在一些大包場景下的表現很不錯，但在一些小包場景下的表現其實不太好，因此需要做前置篩選和 SIMD 優化；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"asm2asm: C++ -> Go——“優化 Go 最好的方法就是不要用 Go”，技術團隊發現用 C++ 寫 json 常用函數，把它們編譯成 x86 彙編，再通過內部 asm2asm 這個工具轉成 Go 的彙編，可以獲得非常大的性能提升；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"lazy-load 解析器：針對多 key 查找的場景，做了一個解析器。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/00\/00332cdb03fba71841b90753a5739b35.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於上述優化，在一個 110KB 的真實業務請求場景下，相比其他 json 庫，Sonic 在每個場景下的表現都有成倍的提升。目前 Sonic 已經開源，但經過內部服務測試，它還處於比較初級的階段，有一些穩定性問題，後續會有相關團隊持續跟進。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"字節跳動的 Service Mesh"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先是幾個數字。從 2018 年 6 月至今，字節跳動在三年間共上線了約 30000 個服務，Service Mesh 管理的容器數已經超過 300 萬個。所有業務場景，以及 ToB 和邊緣計算場景，現在都處於 Mesh 全覆蓋的狀態。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖是公司內部 Service Mesh 的架構示意圖。除數據面和控制面外，它還有一個運維控制面（Operation Plane），且具備兩個突出特點："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在數據面，字節跳動的 Service Mesh 實現了中間件的能力 sidecar 化，形成一個標準模式，下圖中的通用 sidecar 即標準技術方案；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該 Service Mesh 的運維控制面可以發佈多種不同類型的資源。但凡需要集中發佈的資源，例如 Mesh 的 sidecar，例如 WebAssembly 的資源、動態庫，都可以通過運維控制面進行發佈。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/68\/68f128a836feb004c7a7257fec850c39.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"字節跳動內部 Service Mesh 的主要特徵"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當前字節跳動 Service Mesh 的特點可以用 4 個關鍵詞概括："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"全功能"},{"type":"text","text":"。除了前文提到的 RPC 框架、HTTP 框架，字節跳動的 Service Mesh 已經對中間件、MySQL、MongoDB、Redis、RocketMQ 等提供全面支持，在安全能力、服務治理能力，包括流量複製、mock、容災等方面，它均可提供完整功能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"多場景"},{"type":"text","text":"。字節跳動的 Service Mesh 適用於內網環境、邊緣，也可被用於把兩個 IDC 串聯起來。IDC 串聯爲什麼要通過 Mesh？因爲跨 IDC 的網絡是不穩定的，考慮到高昂的成本和嚴格的服務訪問控制，需要通過 Mesh 提供較強的邊緣管控能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"穩定性"},{"type":"text","text":"。這是一個比較常見的話題，此處不做展開。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"高性能"},{"type":"text","text":"。字節跳動 Service Mesh 的性能優化基於技術團隊的理念：如果目標是做一個真正 zero copy 的 proxy，而我們做不到，那麼它的原因是什麼？"},{"type":"text","marks":[{"type":"strong"}],"text":"網絡和內核、基礎庫、組件架構、編譯"},{"type":"text","text":"——如果阻礙來自內核，就去改內核的 API，例如降低 sendfile 的開銷。圍繞這一理念，技術團隊通過採用 Facebook 的 hashmap，帶來了 1％-2％ 的性能提升；通過重寫抽象層，實現 35％-50％ 的吞吐量提升；通過全靜態編譯，無需修改任何代碼，就獲得了 2％ 左右的性能提升……"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖展示了技術團隊在性能優化方面的主要思路和措施："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a3\/a3a7821169e11b8c17568fcd3e50847e.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"性能：基於共享內存的 IPC"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如前文所述，技術團隊想實現的是真正 zero copy 的 proxy。那麼拷貝發生在哪裏？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a8\/a8f58a2fbeba07df79ffd274b1dff589.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是一個最原始的狀態，mesh proxy 帶來了兩層 copy，一次是把業務進程 copy 到 Unix Domain Socket 的 Buffer 裏，另一次是讀取出去。此外它還多了一個進程，而多一個進程，就意味着增加了調度開銷，同時也會產生一些複製成本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/13\/13ca14fd7fdadfc412126ee6ce2ce4f2.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對這個問題，技術團隊採用的做法是"},{"type":"text","marks":[{"type":"strong"}],"text":"把它只寫進內核一次"},{"type":"text","text":"。通過一個共享內存的 IPC，業務進程把準備發送的數據寫進去後，mesh proxy 只需根據 meta header 就可以決定任何調度策略和治理策略，並通過調用下游發送函數把數據發出去。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但現在 TCP Socket 是不能直接把數據發出去的，怎麼辦？請負責內核的團隊寫一個 API。這樣，整個 overhead 就可以降到最低值。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/5b\/5bad481562d54da2358f373e34e2213e.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是採用共享內存 IPC 的性能測試結果。從 2020 年初開始，字節跳動內部就已經開始走上這種"},{"type":"text","marks":[{"type":"strong"}],"text":"數據走共享內存、控制信號走 Unix Domain Socket"},{"type":"text","text":" 的路，同時做好控制協議。確實這條路也是可以走下去的，當前已經有 500+ 服務在線上進行灰度，整體穩定性不錯。未來技術團隊也計劃把這種優化方式擴展到所有涉及"},{"type":"text","marks":[{"type":"strong"}],"text":"同機通訊"},{"type":"text","text":"的場景。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，定義一個通用的協議，可以實現這種能力的複用。隨着容器與容器之間的通信變得越來越頻繁，這種基於共享內存的通訊的重要性也會日益凸顯出來，字節跳動內部 Service Mesh 的通用 sidecar 也是基於相同的思考。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"可觀測性：Service Mesh 之痛"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基礎架構部門經常會被問到兩個問題："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Q1：上下游延遲不一致——爲什麼我在客戶端看到的延遲是 100 毫秒，而 Server 端處理只花了 50 毫秒，中間這 50 毫秒去哪裏了？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Q2：請求超時——接了 Mesh（不接 Mesh 其實也一樣），我的請求爲什麼就超時了？"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果深入地去思考這個問題，可以發現，它的根本原因在於一次請求的時間可以分爲業務可見的部分和業務不可見的部分。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"比如開發人員做一次加密耗時 1 毫秒，但調用一個服務做一次加密可能需要 10 毫秒。這 9 毫秒到底去哪裏了？很多開發人員會認爲這是環境的問題，或是基礎架構的問題。這個問題一直存在，而且相當普遍。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"那麼，延遲來自哪裏？"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"計算"},{"type":"text","text":"。比如序列化，上游其實不會關心請求的序列化和反序列化，以及從用戶態拷貝到內核態，這些都可以被歸類到計算裏，而這些延時業務側是感知不到的（此處需要區分感知與可觀測，通過監控可以看到這部分耗時，但一般業務不理解這兒的計算耗時，認爲處理時間僅僅是自己的業務邏輯）；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"調度"},{"type":"text","text":"。比如要異步發送一個東西，數據寫入後，上游並不知道會什麼時候發。如果機器負載非常高，可能需要等待 100 毫秒，甚至任務會在容器裏被 CPU throttle 掉，這時從上游的角度看就是請求超時；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"網絡"},{"type":"text","text":"。出現超時後，網絡永遠是第一個排查項：是不是掛了？是不是變慢了？"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分析清楚這三個問題後，那麼應當如何解決呢？服務框架團隊的做法是推進 LLT（Low Level Tracing）。即把上游看不見的地方都做可視化處理：如果計算耗時看不到，就把計算時間表現出來；如果調度情況無感知，就把調度的用時算出來；如果網絡情況不清楚，就把網絡的用時也算出來。完成內核網絡及調度事件收集後，這些時間最終都會匯入 OpenTracing 集中展示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d7\/d726217e647990cea5e58650879d1385.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖所示，當上遊開始調用 RPC Client write() 時，事實上就已經進入業務不可見範圍。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"字節跳動內部有一套基於 eBPF 做的 tracing 體系，它能根據提供的 log ID 展開追蹤，把請求什麼時候進入內核、什麼時候從網卡發出等時間戳記錄下來。有了這些信息，技術團隊就能通過 OpenTracing 看到請求什麼時候從本機發出、什麼時候對端收到、什麼時候開始進入 Mesh 處理、什麼時候開始進入業務層處理……整個鏈路變得十分清晰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上述方案的 CPU overhead 約爲 8％- 10％，整體開銷稍重，所以目前是按需開啓狀態。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"字節內部 Mesh：通用 sidecar 的出現"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Service Mesh 的出現源於技術社區希望用它解決多語言、運維、迭代等問題，這種概念可以被泛化，"},{"type":"text","marks":[{"type":"strong"}],"text":"即是否可以爲解決多元問題和運維問題同樣打造一套標準技術方案"},{"type":"text","text":"。毫無疑問，所有中間件都在這個範疇內，無論是 MySQL、Redis Client，還是 API Gateway、登錄組件、風控組件，它們都屬於這一領域。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時，Service Mesh 背後的能力也可以被複用：既然可以把 Mesh 的 sidecar 分發到線上任意一個容器，並完成安全的熱升級，那麼同樣的方法也可以被用於中間件的 sidecar。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"字節跳動的 Service Mesh 提供通用 sidecar，把諸如 API Gateway sidecar、風控 sidecar、登錄及 Session sidecar 等以 Mesh 的方式分發到容器裏。這樣做並不會對業務側造成什麼影響，但後果是它們也需要應對性能、可觀測性、穩定性等問題。由於 Service Mesh 在字節跳動已經大規模落地，這些問題的解決方案其實和 mesh proxy 沒有太大差別。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結與展望"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"談到微服務，由於缺乏直觀感受，很多人往往只關注服務的功能層面，而忽略底層的運維屬性。但其實現在爲什麼社區還在不斷討論微服務，因爲它部署在公有云、私有云、混合雲等充滿不確定性的環境下，解決的是由此帶來的各種複雜性問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但換個角度看，如果開發人員都使用同一種語言，不做跨機器和跨進程的通訊。當所有服務都部署在同一臺宿主機上，類似跨網絡通訊、請求超時等問題會大大減少，通過共享內存或是無序列化等手段，整體性能似乎可以獲得最極致的提升。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這是不是破壞了微服務的概念？其實不然。這樣的部署對於業務來說是透明的，只要宿主機規格足夠，開發人員可以通過調度按服務域進行聚合，框架層和流量調度層感知情況後，很自然地就會把遠程通訊切換成本地通訊，使得性能得到大幅優化、延時降低。如果機器出現宿主機級別的故障，所有服務確實會一起掛掉，但如果只是某一個模塊出現故障，則可以把它輕鬆 fallback 到其他服務上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這也是字節跳動內部正在進行的探索，爲解決由微服務數量不斷增長、複雜性不斷提高導致的非業務消耗比業務消耗還要高的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在微服務領域，除了內部探索，字節跳動也在嘗試通過服務框架團隊和火山引擎雲原生團隊做技術輸出，通過 ToB 品牌火山引擎爲更多企業提供微服務和 Service Mesh 方面的解決方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：字節跳動技術團隊（ID：toutiaotechblog）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/1dgCQXpeufgMTMq_32YKuQ","title":"xxx","type":null},"content":[{"type":"text","text":"字節跳動微服務架構體系演進"}]}]}]}