爱奇艺智能前端异常监控平台的设计与实践

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前端监控一般包括三方面："},{"type":"text","marks":[{"type":"strong"}],"text":"异常监控、性能监控"},{"type":"text","text":"（First Meaningful Paint、First Contentful Paint等性能指标监控）及"},{"type":"text","marks":[{"type":"strong"}],"text":"行为数据监控"},{"type":"text","text":"（PV、UV、页面停留时长等监控）。其中前端异常监控主要监控因前端代码执行异常、接口请求异常等导致页面出现异常，得不到预期结果的情况。从异常导致的问题表现方面来看，前端异常监控平台应能够帮助开发者轻松应对包括但不限于以下问题："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"业务日益扩张，很多重要但较低频的报错事件不会使接口成功率降低和预警，导致问题不能及时发现。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于有些报障反馈的问题，利用测试账号构造类似的数据无法复现。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接口报错事件反映前端传参不对，但当前请求相关的前端代码没有逻辑问题，需要花费较多时间推测可重现场景及寻找根源。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当前业界涵盖前端异常监控的成熟方案有很多，但这些平台也有使用定制困难等不利方面："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这些平台几乎都是收费的，有的跟云服务打包出售，有的单独出售。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这些前端监控平台的架构都是不开源的，大多数使用者没有办法对这类前端监控平台通过二次开发得到适合当前项目的前端监控平台。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这些前端监控平台大部分都不支持私有化部署，使用者无法得到监控数据做更深入的分析。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这些平台的功能定制化能力比较差，即使个别定制化能力比较好的平台，若想达到期望监控的效果，需要做比较复杂的定制或者较多地侵入业务项目逻辑。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此我们自研了"},{"type":"text","marks":[{"type":"strong"}],"text":"爱奇艺智能前端监控平台"},{"type":"text","text":"——"},{"type":"text","marks":[{"type":"strong"}],"text":"鹰眼"},{"type":"text","text":"(Hawkeye)，目前该平台已接入了爱奇艺内容创作、分发和变现平台的大部分业务，经过几个月的使用，帮助业务及时发现了不少问题，也帮助业务负责人对各自系统的整体的运行状态有了更深入的了解，对线上项目起到了很好的保障作用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文将从设计和实践方面分别对该平台进行详细阐述。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"前端异常监控平台介绍"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"鹰眼是以帮助及时发现问题、加速业务项目排障为目标的前端异常监控平台。尤其善于应对业务繁多的场景，在对相对低频而又重要的业务进行监控这一方面表现非常理想。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"鹰眼主要有以下三点优势："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提供事件聚合问题列表，支持业务类型隔离监控报警，支持按照异常业务码配置去监听接口请求错误，能帮助前后端开发人员更容易、更及时的发现系统问题。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"会记录发生错误事件之前用户的操作和接口请求，并支持通过前端生成或记录接口返回的Trace Id来串联前后端链路，为异常排查提供更多的线索。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接入简单快捷，对代码的侵入性低。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整体架构如图1-1，包括了采集JSSDK, 后端采集服务，监控后台三部分。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/2c\/2c186dfe325d3608dd54bea088679e6e.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图1-1 鹰眼整体架构"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"采集JSSDK负责采集异常并上报，其中可采集的信息包括：JS运行时异常（包括通过window.addEventListener监听error事件收集到的TypeError、ReferenceError等异常以及通过window.addEventListener监听unhandlerejection事件收集到的promise异常）、接口请求异常、静态资源加载异常、前端框架错误、发生异常之前的用户操作和请求。采集服务先对数据做校验清洗，之后按照一定的报警策略进行报警，同时会投递消息到监控后台。监控后台基于公司大数据平台与流式计算平台构建了实时计算引擎,对接收的异常消息进行处理，为监控管理页面、报表统计、报警平台等提供数据。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下来会从以下四个方面去介绍鹰眼前端异常监控的设计要点，这些是达成及时发现问题、加速业务项目排障这两个主要目标的关键。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"一.  事件智能聚合"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果一个平台类项目页面访问量较高、业务较为复杂，那么某一类问题收集到的事件会非常多，庞大的事件量是不利于开发人员查看和发现问题的。因此我们对事件按照错误类型、错误信息及访问页面地址等按照规则等进行分类，为同一类的事件生成一个错误ID，并存入Redis中。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a6\/a621d120bb776fc5edc791f33a03107a.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图1-2 问题管理"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"二.  业务类型隔离监控报警，支持按照异常业务码配置去监听接口请求错误"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于接入了10+业务类型的单页面应用，若只按照项目划分，不利于各业务负责人发现各自关注的问题。因此，我们支持建立业务类型、接口返回错误码以及报警Topic Id等的映射配置，之后在问题收集、监控报警、统计分析等各个过程中可根据配置进行定制化处理，如图1-3示意。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/bc\/bc1faf867401ec2b24cc0ff82c204a88.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图1-3 按照业务配置监控"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"具体说明如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前端统一拦截Ajax\/Fetch接口请求，根据配置传入的业务错误码去采集接口的报错信息。这样的方式对于业务项目逻辑是无侵入的。通过window.addEventListener的方式可以监听到网络请求的异常只能监控到Ajax失败的请求，但无法判断HTTP状态码，并且有的业务接口返回的HTTP状态码CODE是200，业务真正的错误码在Response Body中返回。因此对于Ajax请求监控，采用重写XMLHTTPRequest的方式采集错误。同时为了做到低侵入式的业务隔离监控，我们采用业务错误码配置的方式来监控接口的报错情况。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"待办问题列表按照业务配置区分展示，帮助各个业务负责人及时对当天的项目问题进行整体把握。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同一平台项目下，报警按照业务区分，不同业务的报警不会相互影响。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"三.  收集错误上下文，利用Trace Id串联前后端链路"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有些异常事件的发生并不只涉及一次用户操作或接口请求，而可能是异常发生之前的接口超时引起的或者某个特定的操作系列导致的报错。因此对于当前报错之前的用户操作或请求以及请求耗时也需要进行收集，帮助快速定位问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3f\/3f8ed9960f346472eb81b3de0e4cf106.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图1-4 错误上下文"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对于接口请求异常，鹰眼支持记录后端返回的Trace Id串联前后端的链路监控，这样能够在前端监控发现问题时，直接根据Trace Id查看前后端的整个链路，关联业务日志，分析异常环节，快速定位问题。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同时，鹰眼也支持前端生成Trace Id，通过设置HTTP请求头实现(如图1-5)，请求头遵从公司Rover全链路追踪系统的数据规范。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b8\/b83d8c8199a330c5ee0a5b85d251940e.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图1-5 前端生成Trace Id"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"四.  基于公司大数据平台与流式计算平台建设监控后台"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"鹰眼异常监控系统构建在公司大数据平台与流式计算平台之上，后台技术架构如图1-6所示。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/df\/dfbaebaf8b9c6cc24105c9a496ede8ff.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图1-6 监控后台技术架构"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"数据源：前端上报的异常事件会首先进入消息队列当中，作为统一的数据源供后续存储和计算使用；同时，使用消息队列也是一种削峰的手段，避免业务高峰时期大量上报的异常事件拖垮服务。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"引擎：引擎层分为存储引擎和计算引擎。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"存储引擎：根据数据类型与特点，选择合适的数据存储。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用Redis生成异常事件的错误ID，根据项目、异常类型、异常值等维度生成唯一的错误ID，同一类异常事件将聚合在相同错误下。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用ES存储异常事件中的部分字段，主要用于鹰眼监控平台中检索使用。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用HBbase存储异常事件的全部字段，因为上报的异常事件中包含了异常堆栈、上下文等信息，数据量较大且并不用于检索，因此选择了适合存储海量数据的HBbase。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MySQL主要存储一些配置信息，例如项目设置、报警配置等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"计算引擎：主要使用流式计算引擎Flink对上报的异常事件进行实时的计算和聚合。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"展望"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前鹰眼监控已接入了"},{"type":"text","marks":[{"type":"strong"}],"text":"爱奇艺内容创作、分发和变现平台"},{"type":"text","text":"的大部分业务，在日常工作中大幅度辅助提升了运维效率，具体表现如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过查看当前的问题列表（如图1-2）及时发现问题，先于用户报障发现问题。业务开发同学可以选择各自业务的问题查看，后端同学可以设置默认展示Service API Error（接口报错）查看。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过Stack Trace、用户设备环境信息、Trace Id等快速定位问题, 通过错误上下文快速排查疑难杂症，如图1-4，图1-5。其中利用Trace Id的全链路监控可以显著提升接口报错的排查效率。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通过各种维度的统计，去分析项目的运行状况。鹰眼的数据方便接入各种可视化平台统计分析，可分析项目问题发生的趋势，以及各个项目问题解决的情况。比如利用Grafana出色的多数据源支持和报表功能，进行数据报表的开发和展示，如图1-7。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/69\/699c99364a30dd20ded8725376055317.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图1-7 统计分析"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上所述，鹰眼已经可以很好的满足我们日常业务中的监控需求，但还有几个方向值得我们去思考，例如，在SDK代码体积，需求程度、开发成本等之间做权衡时，可以考虑是否需要实现以下功能："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"页面崩溃：页面崩溃后，正常的上报流程就无法走通了，如果需要投递页面崩溃前相关信息，可以通过beforeunload结合sessionStorage，在下次打开页面时上报，如果项目使用PWA(Progress Web Application)，也可以在SserviceWworker实现上报。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"合并日志：如果用户访问量大，上报日志多，或用户反复触发同一个错误时，我们考虑是否需要将几次日志合并到一起再上报。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"HTTP2.0：HTTP2.0上报头部压缩，多路复用的优点，会使监控投递性能更高。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在未来我们将继续优化扩展鹰眼的能力，比如：尽快提供小程序SDK，提供更多WEB框架的支持，帮助更多技术栈的开发者们及时发现问题、快速排障。同时也会尽快将开源提上日程，以便更多有需要的开发团队使用和借鉴。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文转载自：爱奇艺技术产品团队（ID：iQIYI-TP）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文链接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/hbavJwRjpXxGjGLObENrnA","title":"xxx","type":null},"content":[{"type":"text","text":"爱奇艺智能前端异常监控平台的设计与实践"}]}]}]}