愛奇藝智能前端異常監控平臺的設計與實踐

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前端監控一般包括三方面："},{"type":"text","marks":[{"type":"strong"}],"text":"異常監控、性能監控"},{"type":"text","text":"（First Meaningful Paint、First Contentful Paint等性能指標監控）及"},{"type":"text","marks":[{"type":"strong"}],"text":"行爲數據監控"},{"type":"text","text":"（PV、UV、頁面停留時長等監控）。其中前端異常監控主要監控因前端代碼執行異常、接口請求異常等導致頁面出現異常，得不到預期結果的情況。從異常導致的問題表現方面來看，前端異常監控平臺應能夠幫助開發者輕鬆應對包括但不限於以下問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務日益擴張，很多重要但較低頻的報錯事件不會使接口成功率降低和預警，導致問題不能及時發現。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於有些報障反饋的問題，利用測試賬號構造類似的數據無法復現。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接口報錯事件反映前端傳參不對，但當前請求相關的前端代碼沒有邏輯問題，需要花費較多時間推測可重現場景及尋找根源。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當前業界涵蓋前端異常監控的成熟方案有很多，但這些平臺也有使用定製困難等不利方面："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這些平臺幾乎都是收費的，有的跟雲服務打包出售，有的單獨出售。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這些前端監控平臺的架構都是不開源的，大多數使用者沒有辦法對這類前端監控平臺通過二次開發得到適合當前項目的前端監控平臺。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這些前端監控平臺大部分都不支持私有化部署，使用者無法得到監控數據做更深入的分析。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這些平臺的功能定製化能力比較差，即使個別定製化能力比較好的平臺，若想達到期望監控的效果，需要做比較複雜的定製或者較多地侵入業務項目邏輯。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此我們自研了"},{"type":"text","marks":[{"type":"strong"}],"text":"愛奇藝智能前端監控平臺"},{"type":"text","text":"——"},{"type":"text","marks":[{"type":"strong"}],"text":"鷹眼"},{"type":"text","text":"(Hawkeye)，目前該平臺已接入了愛奇藝內容創作、分發和變現平臺的大部分業務，經過幾個月的使用，幫助業務及時發現了不少問題，也幫助業務負責人對各自系統的整體的運行狀態有了更深入的瞭解，對線上項目起到了很好的保障作用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文將從設計和實踐方面分別對該平臺進行詳細闡述。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"前端異常監控平臺介紹"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"鷹眼是以幫助及時發現問題、加速業務項目排障爲目標的前端異常監控平臺。尤其善於應對業務繁多的場景，在對相對低頻而又重要的業務進行監控這一方面表現非常理想。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"鷹眼主要有以下三點優勢："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提供事件聚合問題列表，支持業務類型隔離監控報警，支持按照異常業務碼配置去監聽接口請求錯誤，能幫助前後端開發人員更容易、更及時的發現系統問題。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"會記錄發生錯誤事件之前用戶的操作和接口請求，並支持通過前端生成或記錄接口返回的Trace Id來串聯前後端鏈路，爲異常排查提供更多的線索。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接入簡單快捷，對代碼的侵入性低。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整體架構如圖1-1，包括了採集JSSDK, 後端採集服務，監控後臺三部分。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/2c\/2c186dfe325d3608dd54bea088679e6e.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖1-1 鷹眼整體架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"採集JSSDK負責採集異常並上報，其中可採集的信息包括：JS運行時異常（包括通過window.addEventListener監聽error事件收集到的TypeError、ReferenceError等異常以及通過window.addEventListener監聽unhandlerejection事件收集到的promise異常）、接口請求異常、靜態資源加載異常、前端框架錯誤、發生異常之前的用戶操作和請求。採集服務先對數據做校驗清洗，之後按照一定的報警策略進行報警，同時會投遞消息到監控後臺。監控後臺基於公司大數據平臺與流式計算平臺構建了實時計算引擎,對接收的異常消息進行處理，爲監控管理頁面、報表統計、報警平臺等提供數據。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來會從以下四個方面去介紹鷹眼前端異常監控的設計要點，這些是達成及時發現問題、加速業務項目排障這兩個主要目標的關鍵。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"一.  事件智能聚合"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果一個平臺類項目頁面訪問量較高、業務較爲複雜，那麼某一類問題收集到的事件會非常多，龐大的事件量是不利於開發人員查看和發現問題的。因此我們對事件按照錯誤類型、錯誤信息及訪問頁面地址等按照規則等進行分類，爲同一類的事件生成一個錯誤ID，並存入Redis中。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a6\/a621d120bb776fc5edc791f33a03107a.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖1-2 問題管理"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"二.  業務類型隔離監控報警，支持按照異常業務碼配置去監聽接口請求錯誤"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於接入了10+業務類型的單頁面應用，若只按照項目劃分，不利於各業務負責人發現各自關注的問題。因此，我們支持建立業務類型、接口返回錯誤碼以及報警Topic Id等的映射配置，之後在問題收集、監控報警、統計分析等各個過程中可根據配置進行定製化處理，如圖1-3示意。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/bc\/bc1faf867401ec2b24cc0ff82c204a88.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖1-3 按照業務配置監控"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具體說明如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前端統一攔截Ajax\/Fetch接口請求，根據配置傳入的業務錯誤碼去採集接口的報錯信息。這樣的方式對於業務項目邏輯是無侵入的。通過window.addEventListener的方式可以監聽到網絡請求的異常只能監控到Ajax失敗的請求，但無法判斷HTTP狀態碼，並且有的業務接口返回的HTTP狀態碼CODE是200，業務真正的錯誤碼在Response Body中返回。因此對於Ajax請求監控，採用重寫XMLHTTPRequest的方式採集錯誤。同時爲了做到低侵入式的業務隔離監控，我們採用業務錯誤碼配置的方式來監控接口的報錯情況。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"待辦問題列表按照業務配置區分展示，幫助各個業務負責人及時對當天的項目問題進行整體把握。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同一平臺項目下，報警按照業務區分，不同業務的報警不會相互影響。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"三.  收集錯誤上下文，利用Trace Id串聯前後端鏈路"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有些異常事件的發生並不只涉及一次用戶操作或接口請求，而可能是異常發生之前的接口超時引起的或者某個特定的操作系列導致的報錯。因此對於當前報錯之前的用戶操作或請求以及請求耗時也需要進行收集，幫助快速定位問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3f\/3f8ed9960f346472eb81b3de0e4cf106.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖1-4 錯誤上下文"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於接口請求異常，鷹眼支持記錄後端返回的Trace Id串聯前後端的鏈路監控，這樣能夠在前端監控發現問題時，直接根據Trace Id查看前後端的整個鏈路，關聯業務日誌，分析異常環節，快速定位問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時，鷹眼也支持前端生成Trace Id，通過設置HTTP請求頭實現(如圖1-5)，請求頭遵從公司Rover全鏈路追蹤系統的數據規範。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b8\/b83d8c8199a330c5ee0a5b85d251940e.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖1-5 前端生成Trace Id"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"四.  基於公司大數據平臺與流式計算平臺建設監控後臺"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"鷹眼異常監控系統構建在公司大數據平臺與流式計算平臺之上，後臺技術架構如圖1-6所示。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/df\/dfbaebaf8b9c6cc24105c9a496ede8ff.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖1-6 監控後臺技術架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據源：前端上報的異常事件會首先進入消息隊列當中，作爲統一的數據源供後續存儲和計算使用；同時，使用消息隊列也是一種削峯的手段，避免業務高峯時期大量上報的異常事件拖垮服務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"引擎：引擎層分爲存儲引擎和計算引擎。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"存儲引擎：根據數據類型與特點，選擇合適的數據存儲。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用Redis生成異常事件的錯誤ID，根據項目、異常類型、異常值等維度生成唯一的錯誤ID，同一類異常事件將聚合在相同錯誤下。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用ES存儲異常事件中的部分字段，主要用於鷹眼監控平臺中檢索使用。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用HBbase存儲異常事件的全部字段，因爲上報的異常事件中包含了異常堆棧、上下文等信息，數據量較大且並不用於檢索，因此選擇了適合存儲海量數據的HBbase。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL主要存儲一些配置信息，例如項目設置、報警配置等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"計算引擎：主要使用流式計算引擎Flink對上報的異常事件進行實時的計算和聚合。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"展望"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前鷹眼監控已接入了"},{"type":"text","marks":[{"type":"strong"}],"text":"愛奇藝內容創作、分發和變現平臺"},{"type":"text","text":"的大部分業務，在日常工作中大幅度輔助提升了運維效率，具體表現如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過查看當前的問題列表（如圖1-2）及時發現問題，先於用戶報障發現問題。業務開發同學可以選擇各自業務的問題查看，後端同學可以設置默認展示Service API Error（接口報錯）查看。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過Stack Trace、用戶設備環境信息、Trace Id等快速定位問題, 通過錯誤上下文快速排查疑難雜症，如圖1-4，圖1-5。其中利用Trace Id的全鏈路監控可以顯著提升接口報錯的排查效率。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過各種維度的統計，去分析項目的運行狀況。鷹眼的數據方便接入各種可視化平臺統計分析，可分析項目問題發生的趨勢，以及各個項目問題解決的情況。比如利用Grafana出色的多數據源支持和報表功能，進行數據報表的開發和展示，如圖1-7。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/69\/699c99364a30dd20ded8725376055317.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖1-7 統計分析"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上所述，鷹眼已經可以很好的滿足我們日常業務中的監控需求，但還有幾個方向值得我們去思考，例如，在SDK代碼體積，需求程度、開發成本等之間做權衡時，可以考慮是否需要實現以下功能："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"頁面崩潰：頁面崩潰後，正常的上報流程就無法走通了，如果需要投遞頁面崩潰前相關信息，可以通過beforeunload結合sessionStorage，在下次打開頁面時上報，如果項目使用PWA(Progress Web Application)，也可以在SserviceWworker實現上報。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"合併日誌：如果用戶訪問量大，上報日誌多，或用戶反覆觸發同一個錯誤時，我們考慮是否需要將幾次日誌合併到一起再上報。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"HTTP2.0：HTTP2.0上報頭部壓縮，多路複用的優點，會使監控投遞性能更高。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在未來我們將繼續優化擴展鷹眼的能力，比如：儘快提供小程序SDK，提供更多WEB框架的支持，幫助更多技術棧的開發者們及時發現問題、快速排障。同時也會盡快將開源提上日程，以便更多有需要的開發團隊使用和借鑑。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：愛奇藝技術產品團隊（ID：iQIYI-TP）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/hbavJwRjpXxGjGLObENrnA","title":"xxx","type":null},"content":[{"type":"text","text":"愛奇藝智能前端異常監控平臺的設計與實踐"}]}]}]}