走完线上 Bug 定位最后一公里

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一个小故事"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"周末 12 点的闹钟在回龙观均价 3000 的出租屋急促的响起，程序员小A慵懒的拿过手机，滑开手机通知栏，没有未接电话，点开手机的拦截信箱，没有报警短信，昨晚的发布一定很顺利。小A幸福的伸了个懒腰。戴上 3000 块的 BeatsSolo Pro，音乐逐渐响起来，小A缓缓的闭上了眼睛，正午的阳光从窗户漫进来，撒在小A稀疏的头发上。此时的小A正在脑海中勾勒着自己美好的未来。房东说：十年前住在这间屋的小B，现在已经是某度的 T10 大佬，五年前住在这儿的小T，现在已经在某条带领 200 人的团队，想到这儿，小A的嘴角微微上扬，那我也一定不会太差吧～"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"嘀嘀..耳机里传来两声消息提示音，小A心里咯噔一声，刺骨的寒意弥漫开来，北京三月的阳光突然就不暖了。小A用力的微微睁开双眼，通知栏测试同学小C的头像一闪而过。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"xx线上 BUG 紧急修复群："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"小C: “@小A 昨晚上线的代码好像有点有问题，来公司看下？我在公司等你。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"点开群设置，老板的头像赫然在列。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"怀着愧疚、徘徊、悔恨、无奈、愤怒的心情，小A翻身穿上他在路边买的价值 20 元的人字拖，坐上了前往西二旗的地铁十号线。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不久，西二旗某办公室传来了亘古不变的对话，“这段代码测试过，在我电脑上没问题啊”、\"你重启下试试\"、“是不是代码没上线”、“是不是谁把我代码冲掉了”、“你们测试数据是不是有问题呀”……于是一个下午过去了、一个晚上过去了、一个周末过去了、一个程序员的青春过去了、一个程序员本就不长的职业生涯过去了。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"一个小总结"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面这个虚构的小故事只是想说明一个简单的现象，程序员的很多时间被线上 bug fix 占据。因为线上线下环境不一致、输入输出不一等等原因，很多 bug 定位起来效率低下，耗时巨长，导致很多时候程序员遇到线上 bug 总是头疼不已，不由自主的想要甩锅给外在因素，在确定是自己的问题的时候再排查问题。那么线上问题排查到底难在哪儿？首先来看看我们排查线上问题的一个基本步骤，这个步骤一般是排查大多数线上问题的步骤。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"步骤1：找到能复现问题的输入；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"步骤2：判断该输入能否在日常环境构造, 如果能，调到步骤 5。如果不能，继续步骤 3；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"步骤3：查看线上环境日志，看能否找到异常输入相关的异常日志，辅助排查问题；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"步骤4：初步推断问题原因，尝试修复并加上更多日志输出。然后打包、发布。重复步骤 3 直到定位根因；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"步骤5：日常构造相同输入，单点调试，定位问题；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"实际的场景中，因为线上线下环境隔离的问题，线上的输入很多时候难以在日常环境中构造，大多数时候我们都在步骤 2、3、4 中循环，于是时间就在循环中慢慢的流逝了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面做这么多步骤其实对于查问题而言就是希望可以知道当某段代码执行不符合预期的时候，这段代码的输入是什么，输出是什么，抛出了什么异常，以及代码中每一行的具体执行情况。那么是否有一款产品可以让用户方便快捷的实现这个目标呢？答案是有的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"聊一聊 ARMS"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"阿里云的应用实时监控服务 ARMS 是一款应用性能管理（APM）产品，包含应用监控、Prometheus 监控和前端监控三大子产品，涵盖分布式应用、容器环境、浏览器、小程序、APP 等领域的性能管理，能帮助用户实现全栈式性能监控和端到端全链路追踪诊断。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ARMS 最新推出了 Arthas 诊断功能，其第一个版本主要包含四个能力，分别是 JVM 概览、线程耗时分析、方法执行分析以及性能分析。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"JVM 概览：查看实时的 JVM 内存、GC 信息以及操作系统信息、环境变量、系统变量等信息。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"线程耗时分析：查看实时的线程耗时情况，并可查看每个线程实时的方法堆栈。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"方法执行分析：实时的抓取满足指定条件的方法执行明细、出入参数以及异常。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"性能分析：快捷的通过火焰图的的形式，展示系统性能瓶颈。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ARMS 的 Arthas 功能使用起来也比较简单，详情可参照文档（https:\/\/help.aliyun.com\/document_detail\/204809.html）。下面来简单聊一聊如何利用 ARMS 的 Arthas 诊断能力来进行线上问题的定位。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"聊一聊 ARMS Arthas 诊断"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上一节简单介绍了 ARMS 的 Arthas 诊断具备的能力，那么用这些能力能解决哪些线上问题呢？在这里，我们对线上问题进行了一个归纳总结，将其分为下面四类问题："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"方法执行不符合预期："},{"type":"text","text":"包括方法执行耗时、方法返回值、方法抛出了异常等情况，表现在应用上可能是一些接口或者服务的 RT 增高，错误率增高，返回值异常等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"进程 CPU 耗时突增："},{"type":"text","text":"一般有代码死循环问题、FullGC 导致 GC 线程耗时高、并发使用 HashMap 等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"性能优化问题："},{"type":"text","text":"主要用于分析性能瓶颈，辅助性能优化，包括 CPU 耗时、内存分配、锁竞争、itimer 等情况的性能分析。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"其他问题："},{"type":"text","text":"比如初始化环境变量读取错误、内核版本不符合要求、类冲突等问题。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面就以一个实际的 demo 来演示如何利用 ARMS 的 Arthas 执行不符合预期这种问题的诊断，后续的文章会继续介绍如何利用 Arthas 进行其他类型问题的诊断。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"利用 ARMS Arthas 诊断方法执行不符合预期类问题"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"问题背景：product 应用的 com.alibabacloud.hipstershop.productserviceapi.service.ProductService@confirmInventory   接口某次发布后平均 RT 到达 400，发布以前的平均 RT 在 1ms 以下，如下图所示。现在想定位耗时具体耗在哪儿。 "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/78\/78e1991eb1295a7480529bf77e2a3338.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图 1"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，进入 ARMS Arthas 诊断的页面。当我们进行 Bug 定位的时候，首先需要知道出问题的类名和方法名，按照图示截图中的红色注释输入相应的类名和方法名。如果你是 EDAS（https:\/\/help.aliyun.com\/document_detail\/42934.html）用户，可直接选择一个服务或者接口，后台会自动推断相应的实现类和方法。对应到本案例，对应的类是 com.alibabacloud.xxx.xxx.xxx.ProductService，方法是 confirmInventory。填写完毕后点击确定。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/cc\/cc602625915ce93436044c3e9481312e.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图 2"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下图所示，点击确定后可以得到 confirmInventory 方法执行的纪录，包含执行的入参，返回值异常以及方法执行明细。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a2\/a244ab655d948c1cab87b96aea549a58.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图 3"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是这次执行的耗时 2.89 ms，不是我们预期中的一次耗时高调用。此时，可点击右上角修改诊断参数，设定抓取耗时大于 300ms 的方法调用(除此以外还可以设置更多的过滤条件，包括方法参数满足的条件等等，具体可查看文档https:\/\/help.aliyun.com\/document_detail\/204809.html)。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/4a\/4a1fa8a3816caa0218b7369a2204e891.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图 4"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"点击确定后，点击右上角刷新图标再次诊断，这次抓取到一次耗时 1501ms 的方法调用，发现原来是在该方法的执行过程中，执行了 Thread.sleep() 方法。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/61\/61247e1472506fcdfebd69b4b5beb2ad.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图5"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"到这里，你可能还会好奇，为什么会执行 sleep 方法呢？这块代码的逻辑是怎样的呢？点击右上角查看方法源码，一目了然的将方法源码与方法执行明细相结合。如下图所示，confirmInventory 方法中执行的每一次方法调用最后会以“\/\/-”为前缀展示该方法执行的耗时情况。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/fc\/fcc080894c9b7dbf0f3e11eb5a7c1eb3.png","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"图 6"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，你还可以点击图5 ，列表最右侧的操作列的下钻，快捷的进一步分析 confirmInventory 调用的子方法的执行情况。这在根因比较深的场景下十分方便好用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至此，完成了我们这个问题的一个定位演示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相信 ARMS 的 Arthas 诊断功能一定给你留下了深刻的印象，也一定会成为您线上问题诊断的利器，帮助您更快更方便的诊断线上故障。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"写在最后"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"快速免费体验 ARMS 功能：https:\/\/arms.console.aliyun.com\/。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，企业级分布式应用服务EDAS K8s（https:\/\/help.aliyun.com\/document_detail\/199295.html) 作为一款一体化的产品，既具备了应用的托管能力，也集成了 ARMS 的监控诊断能力，同样可以体验 ARMS 的 Arthas 诊断功能，可根据您目前的实际情况选择一款产品来体验 ARMS 的 Arthas 诊断能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"备注：上述功能目前仅对部署在 K8s 为集群中的 Java 应用有效，后续会支持部署的 ECS 上的 Java 应用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文转载自：阿里巴巴中间件（ID：Aliware_2018）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文链接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/UBSizddouMxc27A5_PfkmQ","title":"xxx","type":null},"content":[{"type":"text","text":"走完线上 Bug 定位最后一公里"}]}]}]}