長連接網關技術專題(五)：喜馬拉雅自研億級API網關技術實踐

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文由喜馬拉雅技術團隊原創分享，原題《喜馬拉雅自研網關架構實踐》，有改動。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1、引言","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網關是一個比較成熟的產品，基本上各大互聯網公司都會有網關這個中間件，來解決一些公有業務的上浮，而且能快速的更新迭代。如果沒有網關，要更新一個公有特性，就要推動所有業務方都更新和發佈，那是效率極低的事，有網關後，這一切都變得不是問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"喜馬拉雅也是一樣，用戶數增長達到 6 億多的級別，Web 服務個數達到500+，目前我們網關日處理 200 億+次調用，單機 QPS 高峯達到 4w+。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網關除了要實現最基本的功能反向代理外，還有公有特性，比如黑白名單，流控，鑑權，熔斷，API 發佈，監控和報警等。我們還根據業務方的需求實現了流量調度，流量 Copy，預發佈，智能化升降級，流量預熱等相關功能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從技術上來說，喜馬拉雅API網關的技術演進路線圖大致如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/3c/3cc74c84cbf7f40062c45200d3a10bed.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文將分享在喜馬拉雅API網關在億級流量前提下，進行的技術演進發展歷程和實踐經驗總結。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"學習交流：","attrs":{}}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"- 移動端IM開發入門文章：《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-464-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"新手入門一篇就夠：從零開發移動端IM","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"- 開源IM框架源碼：","attrs":{}},{"type":"link","attrs":{"href":"https://github.com/JackJiang2011/MobileIMSDK","title":null,"type":null},"content":[{"type":"text","text":"https://github.com/JackJiang2011/MobileIMSDK","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"（本文同步發佈於：","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-3564-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"http://www.52im.net/thread-3564-1-1.html","attrs":{}}]},{"type":"text","text":"）","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2、專題目錄","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文是系列文章的第5篇，總目錄如下：","attrs":{}}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-1243-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"長連接網關技術專題(一)：京東京麥的生產級TCP網關技術實踐總結","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-2737-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"長連接網關技術專題(二)：知乎千萬級併發的高性能長連接網關技術實踐","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-3110-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"長連接網關技術專題(三)：手淘億級移動端接入層網關的技術演進之路","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-3539-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"長連接網關技術專題(四)：愛奇藝WebSocket實時推送網關技術實踐","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-3564-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"長連接網關技術專題(五)：喜馬拉雅自研億級API網關技術實踐","attrs":{}}]},{"type":"text","text":"》（* 本文）","attrs":{}}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3、第1版：Tomcat NIO+Async Servlet","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網關在架構設計時最爲關鍵點，就是網關在接收到請求，調用後端服務時不能阻塞 Block，否則網關的吞吐量很難上去，因爲最耗時的就是調用後端服務這個遠程調用過程。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果這裏是阻塞的，Tomcat 的工作線程都 block 住了，在等待後端服務響應的過程中，不能去處理其他的請求，這個地方一定要異步。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"架構圖如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b2/b2d5b3a9827919c1711b96a0d15cb2c2.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這版我們實現單獨的 Push 層，作爲網關收到響應後，響應客戶端時，通過這層實現，和後端服務的通信是 HttpNioClient，對業務的支持黑白名單，流控，鑑權，API 發佈等功能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是這版只是功能上達到網關的要求，處理能力很快就成了瓶頸，單機 QPS 到 5K 的時候，就會不停的 Full GC。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"後面通過 Dump 線上的堆分析，發現全是 Tomcat 緩存了很多 HTTP 的請求，因爲 Tomcat 默認會緩存 200 個 requestProcessor，每個 prcessor 都關聯了一個 request。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"還有就是 Servlet 3.0 Tomcat 的異步實現會出現內存泄漏，後面通過減少這個配置，效果明顯。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但性能肯定就下降了，總結了下，基於 Tomcat 做爲接入端，有如下幾個問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Tomcat 自身的問題：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）緩存太多，Tomcat 用了很多對象池技術，內存有限的情況下，流量一高很容易觸發 GC；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）內存 Copy，Tomcat 的默認是用堆內存，所以數據需要讀到堆內，而我們後端服務是 Netty，有堆外內存，需要通過數次 Copy；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）Tomcat 還有個問題是讀 body 是阻塞的, Tomcat 的 NIO 模型和 reactor 模型不一樣，讀 body 是 block 的。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"這裏再分享一張 Tomcat buffer 的關係圖：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/6f/6fad2776f7d8c6b8f4fda4e728869c15.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過上面的圖，我們可以看出，Tomcat 對外封裝的很好，內部默認的情況下會有三次 copy。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"HttpNioClient 的問題：","attrs":{}},{"type":"text","text":"獲取和釋放連接都需要加鎖，對應網關這樣的代理服務場景，會頻繁的建連和關閉連接，勢必會影響性能。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於 Tomcat 的存在的這些問題，我們後面對接入端做改造，用 Netty 做接入層和服務調用層，也就是我們的第二版，能徹底解決上面的問題，達到理想的性能。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4、第2版：Netty+全異步","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於 Netty 的優勢，我們實現了全異步，無鎖，分層的架構。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"先看下我們基於 Netty 做接入端的架構圖：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7b/7bb9e0b805015eec6e63347d64409f04.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"PS：如果你對Netty和Java NIO瞭解太少，下面幾篇資料請務必閱讀：","attrs":{}}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-2635-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"少囉嗦！一分鐘帶你讀懂Java的NIO和經典IO的區別","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-2846-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"Java的BIO和NIO很難懂？用代碼實踐給你看，再不懂我轉行！","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-2640-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"史上最強Java NIO入門：擔心從入門到放棄的，請讀這篇！","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-2190-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"寫給初學者：Java高性能NIO框架Netty的學習方法和進階策略","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-2043-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"新手入門：目前爲止最透徹的的Netty高性能原理和框架架構解析","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-3207-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"史上最通俗Netty框架入門長文：基本介紹、環境搭建、動手實戰","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.1 接入層","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Netty 的 IO 線程，負責 HTTP 協議的編解碼工作，同時對協議層面的異常做監控報警。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對 HTTP 協議的編解碼做了優化，對異常，攻擊性請求監控可視化。比如我們對 HTTP 的請求行和請求頭大小是有限制的，Tomcat 是請求行和請求加在一起，不超過 8K，Netty 是分別有大小限制。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"假如客戶端發送了超過閥值的請求，帶 cookie 的請求很容易超過，正常情況下，Netty 就直接響應 400 給客戶端。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過改造後，我們只取正常大小的部分，同時標記協議解析失敗，到業務層後，就可以判斷出是那個服務出現這類問題，其他的一些攻擊性的請求，比如只發請求頭，不發 body 或者發部分這些都需要監控和報警。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.2 業務邏輯層","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"負責對 API 路由，流量調度等一序列的支持業務的公有邏輯，都在這層實現，採樣責任鏈模式，這層不會有 IO 操作。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在業界和一些大廠的網關設計中，業務邏輯層基本都是設計成責任鏈模式，公有的業務邏輯也在這層實現。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"我們在這層也是相同的套路，支持了：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）用戶鑑權和登陸校驗，支持接口級別配置；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）黑白名單：分全局和應用，以及 IP 維度，參數級別；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）流量控制：支持自動和手動，自動是對超大流量自動攔截，通過令牌桶算法實現；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4）智能熔斷：在 Histrix 的基礎上做了改進，支持自動升降級，我們是全部自動的，也支持手動配置立即熔斷，就是發現服務異常比例達到閥值，就自動觸發熔斷；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5）灰度發佈：我對新啓動的機器的流量支持類似 TCP 的慢啓動機制，給機器一個預熱的時間窗口；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6）統一降級：我們對所有轉發失敗的請求都會找統一降級的邏輯，只要業務方配了降級規則，都會降級，我們對降級規則是支持到參數級別的，包含請求頭裏的值，是非常細粒度的，另外我們還會和 varnish 打通，支持 varnish 的優雅降級；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7）流量調度：支持業務根據篩選規則，對流量篩選到對應的機器，也支持只讓篩選的流量訪問這臺機器，這在查問題/新功能發佈驗證時非常用，可以先通過小部分流量驗證再大面積發佈上線；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"8）流量 copy：我們支持對線上的原始請求根據規則 copy 一份，寫入到 MQ 或者其他的 upstream，來做線上跨機房驗證和壓力測試；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"9）請求日誌採樣：我們對所有的失敗的請求都會採樣落盤，提供業務方排查問題支持，也支持業務方根據規則進行個性化採樣，我們採樣了整個生命週期的數據，包含請求和響應相關的所有數據。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面提到的這麼多都是對流量的治理，我們每個功能都是一個 filter，處理失敗都不影響轉發流程，而且所有的這些規則的元數據在網關啓動時就會全部初始化好。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在執行的過程中，不會有 IO 操作，目前有些設計會對多個 filter 做併發執行，由於我們的都是內存操作，開銷並不大，所以我們目前並沒有支持併發執行。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"還有個就是規則會修改，我們修改規則時，會通知網關服務，做實時刷新，我們對內部自己的這種元數據更新的請求，通過獨立的線程處理，防止 IO 在操作時影響業務線程。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.3 服務調用層","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務調用對於代理網關服務是關鍵的地方，一定需要異步，我們通過 Netty 實現，同時也很好的利用了 Netty 提供的連接池，做到了獲取和釋放都是無鎖操作。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"4.3.1）異步 Push：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網關在發起服務調用後，讓工作線程繼續處理其他的請求，而不需要等待服務端返回。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏的設計是我們爲每個請求都會創建一個上下文，我們在發完請求後，把該請求的 context 綁定到對應的連接上，等 Netty 收到服務端響應時，就會在給連接上執行 read 操作。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解碼完後，再從給連接上獲取對應的 context，通過 context 可以獲取到接入端的 session。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣 push 就通過 session 把響應寫回客戶端了，這樣設計也是基於 HTTP 的連接是獨佔的，即連接和請求上下文綁定。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"4.3.2）連接池：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"連接池的原理如下圖：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c6/c640eebff06a6f3f6d92f3d7388077c3.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服務調用層除了異步發起遠程調用外，還需要對後端服務的連接進行管理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"HTTP 不同於 RPC，HTTP 的連接是獨佔的，所以在釋放的時候要特別小心，一定要等服務端響應完了才能釋放，還有就是連接關閉的處理也要小心。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"總結如下幾點：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）Connection:close；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）空閒超時，關閉連接；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）讀超時關閉連接；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4）寫超時，關閉連接；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5）Fin、Reset。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面幾種需要關閉連接的場景，下面主要說下 Connection:close 和空閒寫超時兩種，其他的應該是比較常見的比如讀超時，連接空閒超時，收到 fin，reset 碼這幾個。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"4.3.3）Connection:close：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"後端服務是 Tomcat，Tomcat 對連接重用的次數是有限制的，默認是 100 次。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當達到 100 次後，Tomcat 會通過在響應頭裏添加 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"Connection:close","attrs":{}},{"type":"text","text":"，讓客戶端關閉該連接，否則如果再用該連接發送的話，會出現 400。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"還有就是如果端上的請求帶了 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"connection:close","attrs":{}},{"type":"text","text":"，那 Tomcat 就不等這個連接重用到 100 次，即一次就關閉。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過在響應頭裏添加 ","attrs":{}},{"type":"text","marks":[{"type":"italic","attrs":{}}],"text":"Connection:close","attrs":{}},{"type":"text","text":"，即成了短連接，這個在和 Tomcat 保持長連接時，需要注意的，如果要利用，就要主動 remove 掉這個 close 頭。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic","attrs":{}},{"type":"strong","attrs":{}}],"text":"4.3.4）寫超時：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先網關什麼時候開始計算服務的超時時間，如果從調用 writeAndFlush 開始就計算，這其實是包含了 Netty 對 HTTP 的 encode 時間和從隊列裏把請求發出去即 flush 的時間，這樣是對後端服務不公平的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以需要在真正 flush 成功後開始計時，這樣是和服務端最接近的，當然還包含了網絡往返時間和內核協議棧處理的時間，這個不可避免，但基本不變。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以我們是 flush 成功回調後開始啓動超時任務。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"這裏就有個注意的地方：","attrs":{}},{"type":"text","text":"如果 flush 不能快速回調，比如來了一個大的 post 請求，body 部分比較大，而 Netty 發送的時候第一次默認是發 1k 的大小。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果還沒有發完，則增大發送的大小繼續發，如果在 Netty 在 16 次後還沒有發送完成，則不會再繼續發送，而是提交一個 flushTask 到任務隊列，待下次執行到後再發送。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這時 flush 回調的時間就比較大，導致這樣的請求不能及時關閉，而且後端服務 Tomcat 會一直阻塞在讀 body 的地方，基於上面的分析，所以我們需要一個寫超時，對大的 body 請求，通過寫超時來及時關閉。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"5、全鏈路超時機制","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/51/51cd45a78224d4d7ee90cd0c08d48860.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"上圖是我們在整個鏈路超時處理的機制：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）協議解析超時；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）等待隊列超時；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）建連超時；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4）等待連接超時；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5）寫前檢查是否超時；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6）寫超時；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7）響應超時。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"6、監控報警","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網關業務方能看到的是監控和報警，我們是實現秒級別報警和秒級別的監控，監控數據定時上報給我們的管理系統，由管理系統負責聚合統計，落盤到 influxdb。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們對 HTTP 協議做了全面的監控和報警，無論是協議層的還是服務層的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"協議層：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）攻擊性請求，只發頭，不發/發部分 body，採樣落盤，還原現場，並報警；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）Line or Head or Body 過大的請求，採樣落盤，還原現場，並報警。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"應用層：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）耗時監控：有慢請求，超時請求，以及 tp99，tp999 等；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）OPS 監控和報警；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）帶寬監控和報警：支持對請求和響應的行，頭，body 單獨監控；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4）響應碼監控：特別是 400，和 404；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5）連接監控：我們對接入端的連接，以及和後端服務的連接，後端服務連接上待發送字節大小也都做了監控；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6）失敗請求監控；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7）流量抖動報警：這是非常有必要的，流量抖動要麼是出了問題，要麼就是出問題的前兆。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"總體架構：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/33/3319cf9acf4ec1969b64668ff2aff72d.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"7、性能優化實踐","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"7.1 對象池技術","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於高併發系統，頻繁的創建對象不僅有分配內存的開銷外，還有對gc會造成壓力，我們在實現時會對頻繁使用的比如線程池的任務task，StringBuffer等會做寫重用，減少頻繁的申請內存的開銷。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"7.2 上下文切換","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"高併發系統，通常都採用異步設計，異步化後，不得不考慮線程上下文切換的問題。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"我們的線程模型如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4b/4b5ad7f107cdbfb8b44d4a617398c747.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們整個網關沒有涉及到io操作，但我們在業務邏輯這塊還是和netty的io編解碼線程異步。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"是有兩個原因：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）是防止開發寫的代碼有阻塞；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）是業務邏輯打日誌可能會比較多，在突發的情況下，但是我們在push線程時，支持用netty的io線程替代，這裏做的工作比較少，這裏有異步修改爲同步後(通過修改配置調整)，cpu的上下文切換減少20%，進而提高了整體的吞吐量，就是不能爲了異步而異步，zull2的設計和我們的類似。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"7.3 GC優化","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在高併發系統，gc的優化不可避免，我們在用了對象池技術和堆外內存時，對象很少進入老年代，另外我們年輕代會設置的比較大，而且SurvivorRatio=2，晉升年齡設置最大15，儘量對象在年輕代就回收掉， 但監控發現老年代的內存還是會緩慢增長，通過dump分析，我們每個後端服務創建一個鏈接，都時有一個socket，socket的AbstractPlainSocketImpl，而AbstractPlainSocketImpl就重寫了Object類的finalize方法。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"實現如下：","attrs":{}}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"/**","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" * Cleans up if the user forgets to close it.","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" */","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"protected void finalize() throws IOException {","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    close();","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"}","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"是爲了我們沒有主動關閉鏈接，做的一個兜底，在gc回收的時候，先把對應的鏈接資源給釋放了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於finalize的機制是通過jvm的Finalizer線程來處理的，而且Finalizer線程的優先級不高，默認是8，需要等到Finalizer線程把ReferenceQueue的對象對於的finalize方法執行完，還要等到下次gc時，才能把該對象回收，導致創建鏈接的這些對象在年輕代不能立即回收，從而進入了老年代，這也是爲啥老年代會一直緩慢增長的問題。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"7.4 日誌","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"高併發下，特別是 Netty 的 IO 線程除了要執行該線程上的 IO 讀寫操作，還有執行異步任務和定時任務，如果 IO 線程處理不過來隊列裏的任務，很有可能導致新進來異步任務出現被拒絕的情況。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那什麼情況下可能呢？IO 是異步讀寫的問題不大，就是多耗點 CPU，最有可能 block 住 IO 線程的是我們打的日誌。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前 Log4j 的 ConsoleAppender 日誌 immediateFlush 屬性默認爲 true，即每次打 log 都是同步寫 flush 到磁盤的，這個對於內存操作來說，慢了很多。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時 AsyncAppender 的日誌隊列滿了也會 block 住線程，log4j 默認的 buffer 大小是 128，而且是 block 的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"即如果 buffer 的大小達到 128，就阻塞了寫日誌的線程，在併發寫日誌量大的的情況下，特別是堆棧很多時，log4j 的 Dispatcher 線程會出現變慢要刷盤。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣 buffer 就不能快速消費，很容易寫滿日誌事件，導致 Netty IO 線程 block 住，所以我們在打日誌時，也要注意精簡。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"8、未來規劃","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在我們都是基於 HTTP/1，現在 HTTP/2 相對於 HTTP/1 關鍵實現了在連接層面的服務，即一個連接上可以發送多個 HTTP 請求。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"即 HTTP 連接也能和 RPC 連接一樣，建幾個連接就可以了，徹底解決了 HTTP/1 連接不能複用導致每次都建連和慢啓動的開銷。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們也在基於 Netty 升級到 HTTP/2，除了技術升級外，我們對監控報警也一直在持續優化，怎麼提供給業務方準確無誤的報警，也是一直在努力。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"還有一個就是降級，作爲統一接入網關，和業務方做好全方位的降級措施，也是一直在完善的點，保證全站任何故障都能通過網關第一時間降級，也是我們的重點。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"9、寫在最後","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"網關已經是一個互聯網公司的標配，這裏總結實踐過程中的一些心得和體會，希望給大家一些參考以及一些問題的解決思路，我們也還在不斷完善中，同時我們也在做多活的項目，歡迎交流。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"附錄：更多Netty相關資料","attrs":{}}]},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-306-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"Java新一代網絡編程模型AIO原理及Linux系統AIO介紹","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-163-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"有關“爲何選擇Netty”的11個疑問及解答","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-113-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"MINA、Netty的源代碼（在線閱讀版）已整理髮布","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-426-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"詳解Netty的安全性：原理介紹、代碼演示（上篇）","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-427-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"詳解Netty的安全性：原理介紹、代碼演示（下篇）","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-348-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"詳解Netty的優雅退出機制和原理","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-400-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"NIO框架詳解：Netty的高性能之道","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-97-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"Twitter：如何使用Netty 4來減少JVM的GC開銷（譯文）","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-166-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"絕對乾貨：基於Netty實現海量接入的推送服務技術要點","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-2043-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"新手入門：目前爲止最透徹的的Netty高性能原理和框架架構解析","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-2190-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"寫給初學者：Java高性能NIO框架Netty的學習方法和進階策略","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-2635-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"少囉嗦！一分鐘帶你讀懂Java的NIO和經典IO的區別","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-2640-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"史上最強Java NIO入門：擔心從入門到放棄的，請讀這篇！","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-2663-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"手把手教你用Netty實現網絡通信程序的心跳機制、斷線重連機制","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-2846-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"Java的BIO和NIO很難懂？用代碼實踐給你看，再不懂我轉行！","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-3207-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"史上最通俗Netty框架入門長文：基本介紹、環境搭建、動手實戰","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-1243-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"長連接網關技術專題(一)：京東京麥的生產級TCP網關技術實踐總結","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"《","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-3564-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"長連接網關技術專題(五)：喜馬拉雅自研億級API網關技術實踐","attrs":{}}]},{"type":"text","text":"》","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"本文已同步發佈於“即時通訊技術圈”公衆號。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同步發佈鏈接是：","attrs":{}},{"type":"link","attrs":{"href":"http://www.52im.net/thread-3564-1-1.html","title":null,"type":null},"content":[{"type":"text","text":"http://www.52im.net/thread-3564-1-1.html","attrs":{}}]}]}],"attrs":{}}]}