今日頭條 ANR 優化實踐系列 - 監控工具與分析思路

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在前文，我們對ANR 設計原理及影響因素進行了介紹，並對影響 ANR 的不同場景進行歸類。但是依靠現有的系統日誌，不足以完成複雜場景的問題歸因，而且有些信息從應用側無法獲取，這就導致很多線上問題更加棘手；因此我們在應用側探索了新的監控能力，以彌補信息獲取不足的短板。同時對日常分析過程中用到日誌信息和分析思路進行總結，以幫忙大家更好的掌握分析技巧，下面我們就來看看相關實現。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Raster 監控工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"俗話說：“工欲善其事，必先利其器”，日常分析 ANR 問題也是如此，一個好的監控工具不僅可以幫助我們在解決常規問題時達到錦上添花的效果，在面對線上複雜隱蔽的問題時，也能爲我們打開視野，提供更多線索和思路。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"工具介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該工具主要是在主線程消息調度過程進行監控，並按照一定策略聚合，以保證監控工具本身對應用性能和內存抖動影響降至最低。同時對應用四大組件消息執行過程進行監控，便於對這類消息的調度及耗時情況進行跟蹤和記錄。另外對當前正在調度的消息及消息隊列中待調度消息進行統計，從而在發生問題時，可以回放主線程的整體調度情況。此外，我們將系統服務的 CheckTime 機制遷移到應用側，應用爲線程 CheckTime 機制，以便於系統信息不足時，從線程調度及時性推測過去一段時間系統負載和調度情況。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此該工具用一句話來概括就是：由點到面，回放過去，現在和將來。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因其實現原理和消息聚合後的效果，直觀展示主線程調度過程長短不一的耗時片段，猶如一道道光柵，故將該工具命名爲 "},{"type":"text","marks":[{"type":"strong"}],"text":"Raster[ˈræstər]"},{"type":"text","text":"。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"監控工具由來"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"舉個例子："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"例如下圖，是線下遇到的 ANR 問題，從手機端獲取的 Trace 日誌，可以看到從主線程堆棧上基本得不到太多有效信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/47\/47c7cd5dbc7d738a700a0be250979fd1.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"繼續從 Trace 中分析其它信息，包含了各個進程的虛擬機和線程狀態信息，以及 ANR 之前或之後一段時間，CPU 使用率比較高的進程乃至系統負載(CPU，IO)的相關信息等等，如下圖："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/20\/20e03aa894635cbe75d1801e0f00a363.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是從這些信息中，相信很多同學都很難再進一步分析，因爲這些信息只是列舉了當前各個進程或線程的狀態，並沒有很好的監控和記錄影響這些指標的過程。而現實中這類場景的問題，每天都在線上大量發生。那麼針對這種情況該如何更好的解決呢？下面就來介紹一下我們是如何應對的。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"消息調度監控"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Android 系統的 ANR 設計原理及影響因素一文中，我們講到，ANR 問題很多場景都是歷史消息耗時較長並不斷累加後導致的，但是在 ANR 發生時我們並不知道之前都調度了哪些消息，如果可以監控每次消息調度的耗時並記錄，當發生 ANR 時，獲取這些記錄信息，並能計算出當前正在執行消息的耗時，是不是就可以清晰的知道 ANR 發生前主線程都發生了什麼？按照這個思路，整理出如下示意圖："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/eb\/ebf739fe5b8aaa509da174425a49a8a6.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是通過上面示意圖並結合實際業務場景，我們發現，對於大多數業務消息，單次耗時都很少，如果每個消息都單獨記錄，要想跟蹤記錄 ANR 前 10S 甚至更長時間範圍內的所有消息，可能需要成千上萬條記錄，這個顯然是不合理的，而且這麼多的消息也不方便我們後續查看。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"消息聚合分類"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"聯想到實際業務場景很多情況都是耗時較少的消息，而在排查這類問題過程耗時較少的消息基本是可以直接忽略的，因此我們是不是可以對這類消息按照一定條件進行聚合，將一段時間以內的消息進行累加計算，如果累計耗時超過我們規定的閾值，那麼就將這些消息的數量和累計耗時合併成一條記錄，如 16 個消息累計耗時剛好超過 300ms，則生成一條記錄。以此類推存儲 100 條的話，就能監控到 ANR 發生前 30S 主線程的調度歷史了(實際可能是大於 15S，至於爲何是這個範圍，我們會在後面說明)，如此一來就可以較好的解決大量記錄和頻繁更新帶來的內存抖動問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根據上面的思路，我們對消息監控及記錄又進一步的進行聚合優化和關鍵消息過濾；總結來看，分爲以下幾種類型："}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"多消息聚合："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"該場景主要用於主線程連續調度多個消息，並且每個消息耗時都很少的情況下，將這些消息耗時累加，直到這些消息累計耗時超過設置的閾值，則彙總並生成一條記錄"},{"type":"text","text":"，並在這條記錄中標明本次聚合總共調度了多少個消息。按照消息聚合的思路，發生問題時主線程消息調度示意圖如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/93\/93ca09b4028cd4f697c5a2607f6bff39.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Count 表示本條記錄包含了多少個消息；Wall 表示本輪消息執行的累計耗時"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"消息聚合拆分："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對上面多消息聚合策略，會存在一些特殊情況，例如在將多個消息進行累計統計過程中，如果前 N 次消息調度結束後，累計耗時都沒有超過閾值，但是調度完下一個消息之後，發現累計耗時超過閾值，並且還明顯超出，如設置閾值是 300ms，但是前 N 個消息累計 200ms，加上本次消息累計耗時達到了 900ms，那麼這種情況，我們明顯知道是最後一次消息耗時嚴重，因此"},{"type":"text","marks":[{"type":"strong"}],"text":"需要將本次消息單獨記錄，並標記其耗時和相關信息，同時將之前 N 次消息調度耗時和消息數聚合在一起並單獨記錄，這種場景相當於一次生成 2 條記錄。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了考慮監控工具對性能的影響，我們只在每輪統計需要保存時，更新線程 cpu 執行時間，如果發生消息聚合拆分的場景，會默認前一條記錄的 cpu 時間爲-1，因爲本條記錄並不是我們重點關注的，所以會把本輪統計的 cpu 時間全部歸到後一條消息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在一些極端場景下，如本輪監控第一個消息執行耗時爲 1ms，但是加上本次消息耗時，累計超過 600ms，因此兩次消息累計耗時遠大於設定的閾值 300ms，那麼就需要對本次耗時嚴重的消息單獨記錄，而前面那個 1ms 的消息也需要被單獨進行記錄；類似情形如此反覆，就會出現上面說的保存 100 條記錄，整體監控可回溯的時長區間存在波動的情況；該場景的示意圖如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/72\/72c3d455775678fd533c33de6cbe300b.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Count 表示本條記錄包含了多少個消息；Wall 表示本輪消息執行的累計耗時"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"關鍵消息聚合："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了上面單次耗時嚴重的消息需要拆分並單獨記錄之外，還有一類消息也需要我們單獨標記，以達到更好的識別，那就是可能會引起 ANR 的應用組件，如 Activity，Service，Receiver，Provider 等等。爲了監控這幾種組件的執行過程，我們需要對 ActivityThread.H 的消息調度進行監控，當發現上面這些組件有關的消息在執行時，不管其耗時多少，都對其進行單獨記錄，並將之前監控的一個或多個消息也保存爲一條記錄。該場景的示意圖如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b6\/b6cfb1594af611a6cdb69d23ddaad39c.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Count 表示本條記錄包含了多少個消息；Wall 表示本輪消息執行的累計耗時"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"IDLE 場景聚合："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"熟悉消息隊列的同學都知道，主線程是基於消息隊列的方式進行調度，在每次消息調度完成之後都會從消息隊列讀取下一個待調度的消息，如果消息隊列沒有消息，或者下一個消息沒有到設定的調度時間，並且也沒有 IDLE 消息等待調度，那麼主線程將會進入 IDLE 狀態，如下示意圖："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/c2\/c2796e9d26836030bf150ace2faf0721.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正是因爲上面的調度邏輯，使得主線程在消息調度過程中會多次進入 IDLE 狀態，而這個過程也涉及到線程上下文切換(如：Java 環境切換到 Native 環境)，會去檢測是否有掛起請求，所以對調用頻繁的接口來說，會在 ANR 發生時被命中，理論上調用越頻繁的接口被命中的概率越大，如下圖堆棧："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0b\/0bd5c571823ee931bca7264da769ffc0.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是上面這種場景的 IDLE 停留時長可長可短，如果按照完全上面那幾類消息聚合策略，多個消息連續聚合的話，可能會把這類場景也給聚合進來，一定程度造成干擾，這顯然不是我們想要的。爲此需要進一步優化，在每次消息調度結束後，獲取當前時間，在下次消息調度開始前，再次獲取當前時間，並統計距離上次消息調度結束的間隔時長。如果間隔較長，那麼也需要單獨記錄，如果間隔時間較短，我們認爲可以忽略，並將其合併到之前統計的消息一起跟蹤，到這裏就完成了各類場景的監控和歸類；該場景的示意圖如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f5\/f5c40690e76090c508afbfa3ed62a77f.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"Count 表示本條記錄包含了多少個消息；Wall 表示本輪消息執行的累計耗時"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"耗時消息堆棧採樣："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在上面重點講述了主線程消息調度過程的監控和聚合策略，便於發生 ANR，在線下進行回放。但是那些耗時較長的消息，僅僅知道其耗時和消息 tag 是遠遠不夠的，因爲每個消息內部的業務邏輯對於我們來說都是黑盒，各個接口耗時也存在很多不確定性，如鎖等待、Binder 通信、IO 等系統調用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此需要知道這些耗時消息內部接口的耗時情況，我們選取了 2 種方案進行對比："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第一種方案：是對每個函數進行插樁，在進入和退出過程統計其耗時，並進行聚合和彙總。該方案的優點是可以精確的知道每個函數的真實耗時，缺點是很影響包體積和性能，而且不利於其他產品高效複用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二種方案，在每個消息開始執行時，觸發子線程的超時監控，如果在超時之後本次消息還沒執行結束，則抓取主線程堆棧，並繼續對該消息設置下一次超時監控，直到該消息執行結束並取消本輪監控。如果在觸發超時之前已經執行完畢，則取消本次監控並在下次消息開始執行時，再次設置超時監控，但是因爲大部分消息耗時都很少，如果每次都頻繁設置和取消，將會帶來性能影響，因此我們對此進行優化，採用系統 ANR 超時監控相同的時間對齊方案，具體來說就是： "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以消息開始時間加上超時時長爲目標超時時間，每次超時時間到了之後，檢查當前時間是否大於或等於目標時間，如果滿足，則說明目標時間沒有更新，也就是說本次消息沒結束，則抓取堆棧。如果每次超時之後，檢查當前時間小於目標時間，則說明上次消息執行結束，新的消息開始執行並更新了目標超時時間，這時異步監控需對齊目標超時，再次設置超時監控，如此往復。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根據上面的思路，整理流程圖如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a7\/a7382b4082f72e9e164b6f3a5f1fd4c5.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要注意的是，消息採樣堆棧的超時時長不可設置太短，否則頻繁抓取堆棧對主線程性能影響較大，同時也不能設置太長，否則會因爲採樣過低導致數據置信度降低；具體時長根據每個產品複雜度靈活調整即可。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"監控正在調度消息及耗時："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了監控 ANR 發生之前主線程歷史消息調度及耗時之外，也需要知道 ANR 發生時正在調度的消息及其耗時，以便於在看到 ANR 的 Trace 堆棧時，可以清晰的知道當前 Trace 邏輯到底執行了多長時間，幫忙我們排除干擾，快速定位；藉助這個監控可以很好的回答大家，ANR 發生時當前 Trace 堆棧是否耗時以及耗時多久的問題，避免陷入“Trace 堆棧”誤區。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"獲取 Pending 消息："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時除了監控主線程歷史消息調度及耗時之外，也需要在 ANR 發生時，獲取消息隊列中待調度的消息，爲我們分析問題時提供更多線索，如："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"消息隊列中待調度消息是否被 Block 以及被 Block 時長，根據 Block 時長可以推測主線程的繁忙程度；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以觀察消息隊列中是否存在發生 ANR 的應用組件消息，如 Service 消息，以及其在待調度消息隊列中的位置和 Block 時長；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以觀察消息隊列中都有哪些消息，這些消息是否有一定規律，如大量重複消息，如果有大量重複消息，則說明很有可能與該消息相關的業務邏輯發生異常，頻繁和主線程交互(後面的案例分析中我們也會有介紹)。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d0\/d0c0ee86cdeb07b2751c4cc36baa570b.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之前我們講到，對於一次消息調度，它的耗時可以從兩個維度進行統計，即 Wall Duration 和 Cpu Duration，通過這兩個維度的統計，可以幫助我們更好的推測一次嚴重耗時，是執行大量業務邏輯還是處於等待或被搶佔的情況。如果是後者，那麼可以看到這類消息的 Wall Duration 和 Cpu Duration 比例會比較大，當然如何更好更全面的區分一次消息耗時是等待較多還是線程調度被搶佔，我們將會在後面結合其他參考信息進行介紹。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"完整示意圖"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這裏我們再把 Cpu Duration 耗時也給統計之後，那麼整個有關主線程完整的消息調度監控功能就基本完成了。示意圖如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/94\/941320937cb4e721e53d89291d89274e.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過這個消息調度監控工具，我們就可以很清晰的看到發生 ANR 時，主線程歷史消息調度情況；當前正在調度消息耗時，以及消息隊列待調度消息及相關信息；而且利用這個監控工具，一眼便知 ANR 發生時主線程 Trace 實際耗時情況，因此很好解決了部分同學對當前堆棧是否耗時以及耗時多久的疑問。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上面介紹可以看出，爲了重點標記單次耗時消息和關鍵消息，我們使用了多種聚合策略，因此監控過程記錄的信息可能會代表不同類型的消息，爲了便於區分，我們在可視化展示時加上 Type 標識，便於區別。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"應用示例："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"例如下圖，從 Trace 日誌可以看到，ANR 發生時主線程被 Block 在 Binder 通信過程，可能很多同學第一反應是 WMS 服務沒有及時響應 Binder 請求導致的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f0\/f08a5ff00ac7cf3739f249e655302761.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是再結合下面的消息調度監控來覈實一下，我們發現當前調度的消息 Wall duration 只有 44ms，而在該消息之前有兩次歷史消息耗時比較嚴重，分別爲 2166ms，3277ms，也就是說本次 Binder 調用耗時並不嚴重，真正的問題是前面 2 次消息耗時較長，影響了後續消息調度，只有同時解決這 2 個消息耗時嚴重問題，該 ANR 問題纔可能解決。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/31\/31782b52abf4e7181adbf9a9a6f919eb.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果沒有消息調度監控工具，上去就盲目的分析當前邏輯調用的 IPC 問題，可能就犯了方向性的錯誤，掉入“Trace 堆棧”陷阱中。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來再來看一個發生在線上的另外一個實例，從下圖可以看到主線程正在調度的消息耗時超過 1S，但是在此之前的另一個歷史消息耗時長達 9828ms。繼續觀察下圖消息隊列待調度的消息狀態(灰色示意)，可以看到第一個待調度的消息被 Block 了 14S 之久。由此我們可以知道 ANR 消息之前的這個歷史消息，纔是導致 ANR 的罪魁禍首，當然這個正在執行的消息也需要優化一下性能，因爲我們在前面說過：“發生 ANR 時，沒有一個消息是無辜的。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/e7\/e717aaf79e7a8270c8d938c29b793fbf.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正是因爲有了上面這些監控能力，讓我們在日常面對 Trace 日誌中的業務邏輯是否耗時以及耗時多久的困惑，瞬間就會變得清晰起來。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Checktime"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Checktime 背景介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Checktime 是 Android 系統針對一些系統服務(AMS，InputService 等)中高頻訪問的接口，執行時間的監控，當這類接口真實耗時超過預期值將會給出提示信息，此類設計爲了在真實環境監測進程被調度和響應能力的一種結果反饋。具體實現是，在每個函數執行前和執行後分析獲取當前系統時間，並計算差值，如果該接口耗時超過其設定的閾值，則會觸發“slow operation”的信息提醒，部分代碼實現如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3c\/3c32025d40caca9dd3d9989c57205d15.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Checktime 邏輯很簡單，用當前系統時間減去對比時間，如果超過 50ms，則給出 Waring 日誌提示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0f\/0fba5a9dbc501e0a0e989681dc0877de.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們在分析線下問題，或者在系統層面分析這類問題時，經常會在 logcat 中看到這類消息，但是對於線上的三方應用來說，因爲權限問題無法獲取系統日誌，只能自己實現了。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"線程 Checktime："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"瞭解完系統 Checktime 設計思路及實現之後，我們就可以在應用層實現類似的功能了。通過藉助其它子線程的週期檢測機制，"},{"type":"text","marks":[{"type":"strong"}],"text":"在每次調度前獲取當前系統時間，然後減去我們設置延遲的時間，即可得到本次線程調度前的真實間隔時間，如設置線程每隔 300ms 調度一次，結果發現實際響應時間間隔有時會超過 300ms，如果偏差越大，則說明線程沒有被及時調度，進一步反映系統響應能力變差。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過這樣的方式，即使在線上環境獲取不到系統日誌，也可以從側面反映不同時段系統負載對線程調度影響，如下圖示意，當連續發生多次嚴重 Delay 時，說明線程調度受到了影響。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d8\/d854ada21409e7255a83dcd0f226301d.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"小結："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過上述監控能力，我們可以清晰的知道 ANR 發生時主線程歷史消息調度以及耗時嚴重消息的採樣堆棧，同時可以知道正在執行消息的耗時，以及消息隊列待中調度消息的狀態。同時通過線程 CheckTime 機制從側面反映線程調度響應能力，由此完成了應用側監控信息從點到面的覆蓋。但是在面對 ANR 問題時，只有這個監控，是遠遠不夠的，需要結合其他信息整體分析，以應對更爲複雜的系統環境。下面就結合監控工具來介紹一下 ANR 問題的分析思路。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"ANR 分析思路："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在介紹分析思路之前，我們先來說一下分析這類問題需要用到哪些日誌，當然在不同的環境下，獲取信息能力會有很大差別，如線下環境和線上環境，應用側和系統角度都有差異；這裏我們會將我們日常排查問題常用的信息都介紹一下，便於大家更好的理解，主要包括以下幾種："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Trace 日誌"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"AnrInfo"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Kernel 日誌"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Logcat 日誌"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Meminfo 日誌"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Raster 監控工具"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於應用側來說，在線上環境可能只能拿到當前進程內部的線程堆棧(取決於實現原理，參見：Android 系統的 ANR 設計原理及影響因素)以及 ANR Info 信息。在系統側，幾乎能獲取到上面的所有信息，對於這類問題獲取的信息越多，分析定位成功率就越大，例如可以利用完整的 Trace 日誌，分析跨進程 Block 或死鎖問題，系統內存或 IO 緊張程度等等，甚至可以知道硬件狀態，如低電狀態，硬件頻率(CPU，IO，GPU)等等。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"關鍵信息解讀："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這裏我們把上面列舉的日誌進行提取並解讀，以便於大家在日常開發和麪對線上問題，根據當前獲取的信息進行參考。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Trace 信息"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在前文Android 系統的 ANR 設計原理及影響因素中，我們講到了在發生 ANR 之後，系統會 Dump 當前進程以及關鍵進程的線程堆棧，狀態(紅框所示關鍵信息，稍後詳細說明)，示例如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/67\/67b3f6ac84a76799f8f10eec7ecf26e3.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面的日誌包含很多信息，這裏將常用的關鍵信息進行說明，如下："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"線程堆棧："}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個比較好理解，也就是發生 ANR 時，線程正在執行的邏輯，但是很多場景下，獲取的堆棧耗時並不長，原因詳見Android 系統的 ANR 設計原理及影響因素。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"線程狀態："}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"見上圖“state=xxx”，表示當前線程工作狀態，Running 表示當前線程正在被 CPU 調度，Runnable 表示線程已經 Ready 等待 CPU 調度，如上圖 SignalCatcher 線程狀態。Native 態則表示線程由 Java 環境進入到 Native 環境，可能在執行 Native 邏輯，也可能是進入等待狀態；Waiting 表示處於空閒等待狀態。除此之外還有 Sleep，Blocked 狀態等等；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"線程耗時："}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"見上圖“utmXXX，stmXXX”，表示該線程從創建到現在，被 CPU 調度的真實運行時長，不包括線程等待或者 Sleep 耗時，其中線程 CPU 耗時又可以進一步分爲用戶空間耗時(utm)和系統空間耗時(stm)，這裏的單位是 jiffies，當 HZ=100 時，1utm 等於 10ms。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"utm："},{"type":"text","text":" Java 層和 Native 層非 Kernel 層系統調用的邏輯，執行時間都會被統計爲用戶空間耗時；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"stm："},{"type":"text","text":" 即系統空間耗時，一般調用 Kernel 層 API 過程中會進行空間切換，由用戶空間切換到 Kernel 空間，在 Kernel 層執行的邏輯耗時會被統計爲 stm，如文件操作，open,write,read 等等；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"core"},{"type":"text","text":"：最後執行這個線程的 cpu 核的序號。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"線程優先級："}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"nice："},{"type":"text","text":" 該值越低，代表當前線程優先級越高，理論上享受的 CPU 調度能力也越強。對於應用進程(線程)來說，nice 範圍基本在 100~139。隨着應用所在前後臺不同場景，系統會對進程優先級進行調整，廠商可能也會開啓 cpu quota 等功能去限制調度能力；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"調度態："}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"schedstat："},{"type":"text","text":" 參見“schedstat=( 1813617862 14167238 546 )”，分別表示線程 CPU 執行時長(單位 ns)，等待時長，Switch 次數。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"AnrInfo 信息"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了 Trace 之外，系統會在發生 ANR 時獲取一些系統狀態，如 ANR 問題發生之前和之後的系統負載以及 Top 進程和關鍵進程 CPU 使用率。這些信息如果在本地環境可以從 Logcat 日誌中拿到，也可以在應用側通過系統提供的 API 獲取(參見："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s?__biz=MzI1MzYzMjE0MQ==&mid=2247488116&idx=1&sn=fdf80fa52c57a3360ad1999da2a9656b&chksm=e9d0d996dea750807aadc62d7ed442948ad197607afb9409dd5a296b16fb3d5243f9224b5763&token=569762407&lang=zh_CN&scene=21#wechat_redirect","title":null,"type":null},"content":[{"type":"text","text":"Android 系統的 ANR 設計原理及影響因素"}],"marks":[{"type":"strong"}]},{"type":"text","text":")，Anr Info 節選部分信息如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7b\/7b005e643da709bbe22b6229ba25114c.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於上圖信息，主要對以下幾部分關鍵信息進行介紹："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"ANR 類型(longMsg)："}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"表示當前是哪種類型的消息或應用組件導致的 ANR，如 Input，Receiver，Service 等等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"系統負載(Load)："}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"表示不同時間段的系統整體負載，如：\"Load：45.53 \/ 27.94 \/ 19.57\"，分佈代表 ANR 發生前 1 分鐘，前 5 分鐘，前 15 分鐘各個時間段系統 CPU 負載，具體數值代表單位時間等待系統調度的任務數(可以理解爲線程)。如果這個數值過高，則表示當前系統中面臨 CPU 或 IO 競爭，此時，普通進程或線程調度將會受到影響。如果手機處於溫度過高或低電等場景，系統會進行限頻，甚至限核，此時系統調度能力也會受到影響。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，可以將這些時間段的負載和應用進程啓動時長進行關聯。如果進程剛啓動 1 分鐘，但是從 Load 數據看到前 5 分鐘，甚至前 15 分鐘系統負載已經很高，那很大程度說明本次 ANR 在很大程度會受到系統環境影響。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"進程 CPU 使用率："}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖，表示當前 ANR 問題發生之前(CPU usage from XXX to XXX ago)或發生之後(CPU usage from XXX to XXX later)一段時間內，都有哪些進程佔用 CPU 較高，並列出這些進程的 user，kernel 的 CPU 佔比。當然很多場景會出現 system_server 進程 CPU 佔比較高的現象，針對這個進程需要視情況而定，至於 system_server 進程 CPU 佔比爲何普遍較高。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"minor 表示次要頁錯誤，文件或其它內存被加載到內存後，但是沒有被映射到當前進程，通過內核訪問時，會觸發一次 Page Fault。如果訪問的內容還沒有加載到內存，那麼會觸發 major，所以對比可以看到，major 的系統開銷會比 minor 大很多。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"關鍵進程："}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"kswapd："},{"type":"text","text":" 是 linux 中用於頁面回收的內核線程，主要用來維護可用內存與文件緩存的平衡，以追求性能最大化，當該線程 CPU 佔用過高，說明系統可用內存緊張，或者內存碎片化嚴重，需要進行 file cache 回寫或者內存交換(交換到磁盤)，線程 CPU 過高則系統整體性能將會明顯下降，進而影響所有應用調度。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"mmcqd："},{"type":"text","text":" 內核線程，主要作用是把上層的 IO 請求進行統一管理和轉發到 Driver 層，當該線程 CPU 佔用過高，說明系統存在大量文件讀寫，當然如果內存緊張也會觸發文件回寫和內存交換到磁盤，所以 kswapd 和 mmcqd 經常是同步出現的。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"系統 CPU 分佈："}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如下圖，反映一段時間內，系統整體 CPU 使用率，以及 user，kernel，iowait 方向的 CPU 佔比，如果發生大量文件讀寫或內存緊張的場景，則 iowait 佔比較高，這個時候則要進一步觀察上述進程的 kernel 空間 CPU 使用情況，並通過進程 CPU 使用，再進一步對比各個線程的 CPU 使用，找出佔比最大的一個或一類線程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ad\/ad7d5f15f315e2b8f9f7e6d3067f0824.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Logcat 日誌："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 log 日誌中，我們除了可以觀察業務信息之外，還有一些關鍵字也可以幫我們去推測當前系統性能是否遇到問題，如下圖， "},{"type":"text","marks":[{"type":"strong"}],"text":"“Slow operation”，“Slow delivery”"},{"type":"text","text":" 等等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/fe\/fe1f4623da367297537d14cb79eea2da.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/07\/0727bdcc7005d9296468fef0be363a2c.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","text":"Slow operation"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Android 系統在一些頻繁調用的接口中，分別在方法前後利用 checktime 檢測，以判斷本次函數執行耗時是否超過設定閾值，通常這些值都會設置的較爲寬鬆，如果實際耗時超過設置閾值，則會給出“Slow XXX”提示，表示系統進程調度受到了影響，一般來說系統進程優先級比較高，如果系統進程調度都受到了影響，那麼則反映了這段時間內系統性能很有可能發生了問題。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Kernel 日誌："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於應用側來說，這類日誌基本是拿不到的，但是如下是在線下測試或者從事系統開發的同學，可以通過 dmesg 命令進行查看。對於 kernel 日誌，我們主要分析的是 lowmemkiller 相關信息，如下圖："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/fc\/fc9a2f9399c69a481d14c96426cd82c8.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","text":"Lowmemkiller："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從事性能(內存)優化的同學對該模塊都比較熟悉，主要是用來監控和管理系統可用內存，當可用內存緊張時，從 kernel 層強制 Kill 一些低優先級的應用，以達到調節系統內存的目的。而選擇哪些應用，則主要參考進程優先級(oom_score_adj)，這個優先級是 AMS 服務根據應用當前的狀態，如前臺還是後臺，以及進程存活的應用組件類型而計算出來的。例如：對於用戶感知比較明顯的前臺應用，優先級肯定是最高的，此外還有一些系統服務，和後臺服務(播放器場景)優先級也會比較高。當然廠商也對此進行了大量的定製(優化)，以防止三方應用利用系統設計漏洞，將自身進程設置太高優先級進而達到保活目的。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"消息調度時序圖："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/e7\/e717aaf79e7a8270c8d938c29b793fbf.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上圖，在我們分析完系統日誌之後，會進一步的鎖定或縮小範圍，但是最終我們還是要回歸到主線程進一步的分析 Trace 堆棧的業務邏輯以及耗時情況，以便於我們更加清晰的知道正在調度的消息持續了多長時間。但是很多情況當前 Trace 堆棧並不是我們期待的答案，因此需要進一步的確認 ANR 之前主線程的調度信息，評估歷史消息對後續消息調度的影響，便於我們尋找“真兇”。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當然，有時也需要進一步的參考消息隊列中待調度消息，在這些消息裏面，除了可以看到 ANR 時對應的應用組件被 Block 的時長之外，還可以瞭解一下都有哪些消息，這些消息的特徵有時對於我們分析問題也會提供有力的證據和方向。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"分析思路"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在上面我們對各類日誌的關鍵信息進行了基本釋義，下面就來介紹一下，當我們日常遇到 ANR 問題時，是如何分析的，總結思路如下："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"分析堆棧"},{"type":"text","text":"，看看是否存在明顯業務問題(如死鎖，業務嚴重耗時等等)，如果無上述明顯問題，則進一步通過 ANR Info 觀察系統負載是否過高，進而導致整體性能較差，如 CPU，Mem，IO。然後再進一步分析是本進程還是其它進程導致，最後再分析進程內部分析對比各個線程 CPU 佔比，找出可疑線程。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"綜合上述信息，利用監控工具收集的信息，觀察和找出 ANR 發生前一段時間內，主線程耗時較長的消息都有哪些，並查看這些耗時較長的消息執行過程中採樣堆棧，根據堆棧聚合展示，進一步的對比當前耗時嚴重的接口或業務邏輯。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上分析思路，進一步細分的話，可以分爲以下幾個步驟："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"一看 Trace："}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"死鎖堆棧："},{"type":"text","text":" 觀察 Trace 堆棧，確認是否有明顯問題，如主線程是否與其他線程發生死鎖，如果是進程內部發生了死鎖，那麼恭喜，這類問題就清晰多了，只需找到與當前線程死鎖的線程，問題即可解決；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"業務堆棧："},{"type":"text","text":" 觀察通過 Trace 堆棧，發現當前主線程堆棧正在執行業務邏輯，你找到對應的業務同學，他承認該業務邏輯確實存在性能問題，那麼恭喜，你很有可能解決了該問題，爲什麼只是有可能解決該問題呢？因爲有些問題取決於技術棧或框架設計，無法在短時間內解決。如果業務同學反饋當前業務很簡單，基本不怎麼耗時，而這種場景也是日常經常遇到的一類問題，那麼就可能需要藉助我們的監控工具，追溯歷史消息耗時情況了；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"IPC Block 堆棧："},{"type":"text","text":" 觀察通過 Trace 堆棧，發現主線程堆棧是在跨進程(Binder)通信，那麼這個情況並不能當即下定論就是 IPC block 導致，實際情況也有可能是剛發送 Binder 請求不久，以及想要進一步的分析定位，這時也需要藉助我們的自研監控工具了；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"系統堆棧："},{"type":"text","text":" 通過觀察 Trace，發現當前堆棧只是簡單的系統堆棧，想要搞清楚是否發生嚴重耗時，以及進一步的分析定位，如我們常見的 NativePollOnce 場景，那麼也需要藉助我們的自研監控工具進一步確認了。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"二看"},{"type":"text","text":"關鍵字："},{"type":"text","marks":[{"type":"strong"}],"text":"Load，CPU，Slow Operation，Kswapd，Mmcqd，Kwork，Lowmemkiller 等等"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"剛纔我們介紹到，上面這些關鍵字是反應系統 CPU，Mem，IO 負載的關鍵信息，在分析完主線程堆棧信息之後，還需要進一步在 ANRInfo，logcat 或 Kernel 日誌中搜索這些關鍵字，並根據這些關鍵字當前數值，判斷當前系統是否存在資源(CPU，Mem，IO)緊張的情況；"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"三看系統負載分佈：觀察系統整體負載：User,Sys,IOWait"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過觀察系統負載，則可以進一步明確是 CPU 資源緊張，還是 IO 資源緊張；如果系統負載過高，一定是有某個進程或多個進程引起的。反之系統負載過高又會影響到所有進程調度性能。通過觀察 User，Sys 的 CPU 佔比，可以進一步發分析當前負載過高是發生在應用空間，還是系統空間，如大量調用邏輯(如文件讀寫，內存緊張導致系統不斷回收內存等等)，知道這些之後，排查方向又會進一步縮小範圍。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"四看進程 CPU：觀察 Top 進程的 CPU 佔比"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上面分析，在我們知道當前系統負載過高，是發生在用戶空間還是內核空間之後，那麼我們就要通過 Anrinfo 的提供的進程 CPU 列表，進一步鎖定是哪個(些)進程導致的，這時則要進一步的觀察每個進程的 CPU 佔比，以及進程內部 user，sys 佔比。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在分析進程 CPU 佔比過程，有一個關鍵的信息，"},{"type":"text","marks":[{"type":"strong"}],"text":"要看統計這些進程 CPU 過高的場景是發生在 ANR 之前的一段時間還是之後一段時間"},{"type":"text","text":"，如下圖表示 ANR 之前 4339ms 到 22895ms 時間內進程的 CPU 使用率。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/52\/523ce839a7167f5930b89e1755039a4f.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"五看 CPU 佔比定線程 ：對比各線程 CPU 佔比，以及線程內部 user 和 kernel 佔比"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在通過系統負載(user,sys,iowait)鎖定方向之後，又通過進程列表鎖定目標進程，那麼接下來我們就可以從目標進程內部分析各個線程的(utm,stm)，進一步分析是哪個線程有問題了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Trace 日誌的線程信息裏可以清晰的看到每個線程的 utm，stm 耗時。至此我們就完成了從系統到進程，再到進程內部線程方向的負載分析和排查。當然，有時候可能導致系統高負載的不是當前進程，而是其他進程導致，這時同樣會影響其他進程，進而導致 ANR。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"六看消息調度鎖定細節 ："}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在分析和明確系統負載是否正常，以及負載過高是哪個進程引起的結論之後，接下來便要通過我們的監控工具，進一步排查是當前消息調度耗時導致，歷史消息調度耗時導致，還是消息過於頻繁導致。同時通過我們的線程 CheckTime 調度情況分析當前進程的 CPU 調度是否及時以及影響程度，在鎖定上述場景之後，再進一步分析耗時消息的採樣堆棧，纔算找到解決問題的終極之鑰。當然耗時消息內部可能存在一個或多個耗時較長的函數接口，或者會有多個消息存在耗時較長的函數接口，這就是我們前文中提到的：“發生 ANR 時，沒有一個消息是無辜的”"}]}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"更多信息："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除了上面的一些信息，我們還可以結合 logcat 日誌分析 ANR 之前的一些信息，查看是否存在業務側或系統側的異常輸出，如搜索“Slow operation”，\"Slow delivery\"等關鍵字。也可以觀察當前進程和系統進程是否存在頻繁 GC 等等，以幫忙我們更全面的分析系統狀態。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"總結："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面我們重點介紹了基於主線程消息調度的監控工具，實現了“由點到面”的監控能力，以便於發生 ANR 問題時，可以更加清晰直觀的整體評估主線程的“過去，現在和將來”。同時結合日常實踐，介紹了在應用側分析 ANR 問題經常用到的日誌信息和分析思路。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，Raster 監控工具因爲其很好的提升了問題定位效率和成功率，成爲 ANR 問題分析利器，並融合到公司性能穩定性監控平臺，爲公司衆多產品廣泛使用。接下來我們將利用該工具並結合上面的分析思路，講一講實際工作中遇到不同類型的 ANR 問題時，是如何快速分析和定位問題的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：字節跳動技術團隊（ID：toutiaotechblog）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/_Z6GdGRVWq-_JXf5Fs6fsw","title":"xxx","type":null},"content":[{"type":"text","text":"今日頭條 ANR 優化實踐系列 - 監控工具與分析思路"}]}]}]}