抖音 Android 性能優化系列：新一代全能型性能分析工具 Rhea

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"「抖音 Android 性能優化」系列文章是由抖音 Android 基礎技術部門技術專家傾力打造的技術乾貨內容，和大家分享基礎技術團隊在打造極致用戶體驗的抖音的過程中，收穫的性能優化方法論、工具和實踐，與各位技術同學一起交流成長。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用戶交互響應的耗時，作爲 Android 用戶日常感知最深的一項性能指標，在日常開發中有着非常重要的意義。而抖音 Android 基礎技術團隊爲打造極致的交互響應體驗，一直在致力於極致性能的探索，其中就包括如何打造極致的耗時檢測工具。"}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"概述"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"俗話說，工欲善其事，必先利其器，我們要做好性能優化，首要是要能夠發現性能的問題，這就需要有靠譜的工具來幫助我們做性能分析。市面上主流的性能分析工具有：Systrace、TraceView、Android Studio 的 CPU Profiler。相信做性能優化的同學對這些工具應該都是非常熟悉了，抖音最早也是用 Systrace 作爲主要的分析工具，在優化前期也發揮了比較大的作用。隨着抖音的性能優化來到了深水區，我們需要發現並解決更細粒度、更多維度的性能問題，我們會關注幾毫秒的耗時，關注線上一些低端機用戶遇到的鎖阻塞和 IO 等待問題。而市面上這些主流的性能分析工具因其使用的侷限性和較大的性能損耗，已經無法滿足抖音性能優化的需求。爲了能夠百尺竿頭更進一步，我們需要開發更加靈活、精細化以及多元的信息和工具來輔助我們進行高效的優化工作。在這樣的背景之下，抖音 Android 基礎技術團隊開發了 Rhea( [ˈriːə] 瑞亞，寓意時光女神）跟蹤器（Tracer），其是一種通過靜態代碼插樁技術自動添加 Trace，用來分析 APP 運行時耗時的性能分析工具，意思是要做一個功能全面、追求效率、大家都喜歡的女神，也符合我們工具的核心設計原則。Rhea 跟蹤器獲取 Trace 不僅要性能損耗低，還要能脫離 PC 端工具在 App 側直接抓取，跟蹤更多常規函數耗時的同時還要可以跟蹤系統調用，如：鎖信息、I\/O 耗時、Binder IPC 以及更多其他信息。最後，還提供轉換腳本工具，用於將原始跟蹤文件生成可視化報告，便於用戶分析性能問題。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"優勢對比"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Rhea 當前因其無侵入、高性能、信息全等優勢已在字節多個 APP 上落地使用，效果明顯，已多次幫助大家快速發現性能問題，其包含的信息包括不限層級的應用層函數、IO、鎖、Binder、CPU 調度等耗時信息等，其部分效果如下所示：圖片相對於其他 Android 性能排查工具，其具體優勢表現爲：圖片當前，Systrace 只能監控特定系統信息，監控應用層的耗時則需要手動打點；TraceView 性能跟採樣率關係密切，採樣過於頻繁性能開銷巨大，採樣過低又難以精準發現問題函數；Nanoscope 雖然幾乎沒有性能損耗，但每次都需定製 ROM 刷機，使用成本非常高，並且這些工具都只支持 debugable 的應用程序線下分析，這些工具在針對 APP 性能優化都有不甚完美之處，而 Rhea 是一個集大成者，融合了各工具優勢並彌補了相關缺陷。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"架構演進之路"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"第一階段：基於 Systrace 補充函數耗時 Trace"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Systrace 是 Android 性能調試優化的常用工具，它可以收集進程的活動信息，如函數調用耗時、鎖等；也可以收集內核信息，如 CPU 調度、IO 活動、Binder 調用信息等；這些信息會統一時間軸，在 Chrome 瀏覽器中顯示出來，方便工程師性能調試、優化卡頓等工作。因此，抖音早期性能優化首選 Systrace 作爲主要工具，其大致流程如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/e2\/e2ed70240dec27221b34ff644a5511c7.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"功能改造"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Systrace 工具只能監控特定系統調用的耗時情況，它不支持應用程序代碼的耗時分析，所以在使用時有一些侷限性。原生 Systrace 需要開發者在方法的起止位置手動加入 Trace.beginSection 與 Trace.endSection 方法對，這個過程就變成了開發者預判耗時位置，然後在手動加入監控函數對，通過不斷重複添加監控點、打包、運行、採集數據，從而一步步完成耗時方法定位，這也使得 Systrace 的使用成本變得極高。爲了提高 Systrace 的易用性，我們開發了 Rhea 1.0 對 Systrace 功能進行了改造，加入了自動插樁機制：通過字節碼插樁自動完成 Trace.beginSection 和 Trace.endSection 方法對的插入，並且通過運行時限制方法層級的方式，來有效控制因引入監控帶來的性能損耗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"插樁類及樁方法僞代碼："}]}]}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nclass Tracer{\n method_stack = list()\n max_size = 6\n\n methodIn(method_id, method_name){\n if(method_stack.size()<=max_size){\n method_stack.push(method_id)\n Trace.beginSection(method_name)\n }\n }\n\n methodOut(){\n if(method_stack.size>0){\n method_stack.pop()\n Trace.end()\n }\n }\n}"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"被插樁方法"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nmethod1(){\n Tracer.methodIn(1,method1)\n ...\n Tracer.methodOut()\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"輸出數據如下所示，指定層級內所有方法即可按照預期展示在輸出 html 中："}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/fa\/fad96eb4b3ef571895385f4f5400d4ce.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2. 方法 Did not finished 問題"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在使用改造後的 systrace 時，我們時常會遇到如下問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/32\/32db1e2377687e63ee044d47c402ee64.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分析發現，主要原因在於方法在運行期執行中被中斷，例如：方法執行過程中發生異常後，被其調用者方法捕獲，發生異常方法的 Systracer.o 方法未被調用。如圖：test 方法中的 error 方法執行時出現 arr[2]的數組越界，導致 test 方法中的插樁方法 SysTracer.o(13L)未調用，異常被 onCreate 中的 catch 塊捕獲，從而導致 test 的插樁方法沒有被成對調用，最終導致了 test 外層所有的方法調用都無法正確閉合。"},{"type":"text","marks":[{"type":"strong"}],"text":"（注意：本小結提到的樁方法，即 SysTracer 相關方法，均是通過字節碼插樁自動插入）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/0f\/0f07286b295d7b3a64783766cb5b3434.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解決辦法，在外層所有異常捕獲的位置，額外插入樁方法，重新這種異常調用鏈下的樁方法不成對問題。如下圖："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/8c\/8c2cd33b596bd67ddcd3bfbec9882793.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":3,"normalizeStart":3},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"依然存在的問題"}]}]}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"性能問題"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着 Rhea 1.0 功能的深入使用，在帶來極大便利的同時，功能本身的不足也逐漸暴露出來。在採集數據過程中，其本身的性能損耗會導致在一些實際性能優化過程中會帶偏方向。經我們嚴格測試，其性能損耗有 11.5%左右，如下所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7a\/7a27674b384792efd9e1e51e581df719.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在實際使用過程中發現，在開啓 Systrace 之後，對應 Sleep 耗時佔比在極端情況下會超過 40%以上。一方面是 APP 鎖帶來的 Sleep 耗時。例如，在抖音啓動路徑上 SharedPreference 優化過程中，在開啓 Rhea 1.0 的 Systrace 功能後，發現 SP 調用存在明顯的鎖耗時，當時針對 SP 進行了一番鎖的優化後，上線發現效果並不明顯，後續經過一系列排查，發現鎖的耗時是由於開啓 Systrace 功能後導致。另一方面是 IO 帶來的 Uninterrupt Sleep 耗時。例如，我們在一次性能優化過程中看到了很多__fdget_pos 操作，對_fdget_pos 操作相對 Uninterruptible Sleep 的佔比統計了下，至少佔了 Uninterruptible Sleep 總耗時的 60%左右。我們花了比較長時間，額外加了很多 IO 的信息，最終定位原因是在開啓 Systrace 後，由於所有線程的 trace 都會寫入同一個文件，所有線程會同步競爭內核態的文件 pos 鎖導致。工具本身的性能問題誤導了我們的排查方向，同時也暴露了在排查這種 IO Wait 問題的時候由於 IO 信息不全導致排查效率不高的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"限制層級導致的調用缺失"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Systrace 的原理決定了當我們在應用層插入更多函數插樁以定位應用層耗時問題的時候，會導致非常嚴重的性能問題，所以我們在線下使用該工具會通過限制插樁層級的方式以減少運行時性能損耗。但是層級的限制使得超過既定層級的函數調用數據缺失，在分析調用層級較深的函數耗時的時候，無法定位到準確的耗時點。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用場景限制"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於 Systrace 在採集數據的過程中，需要依賴 PC，對於一些需要脫離 PC 採集數據的場景，Systrace 就無法滿足需求了。比如我們產品運營同學經常會在線下場景實地測試抖音的使用性能，例如地鐵、餐館、咖啡廳等，這些實際使用場景下的性能數據，systrace 就無法支持到了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"低端機無法正常使用"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們在針對低端機進行耗時優化時，發現諸如三星、oppo 等一些早期的低端機型，systrace 也不能支持其數據抓取。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對以上問題，我們對工具進行了深入的探索和優化。於是，工具的開發進入了第二階段。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"第二階段：高性能全場景的 Trace 抓取工具"}]},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"功能升級"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了彌補第一階段功能短板，進一步提高性能，同時能滿足更多使用場景，我們找到了新的解決方案：在 Java 層，通過記錄方法首末位置時間戳、所在線程等信息，過濾出大於指定耗時閾值的函數後，將數據異步記錄到文件。數據採集結束後，將輸出文件轉換成指定格式後，便可通過 SDK 提供的 Systrace 工具轉化成方便查看的 Html 格式，從而實現和 Systrace 相同的可視化效果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d2\/d2f0c84476237a4aa1a42da425399f25.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2. 實現原理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Rhea 2.0 如何採集數據並生成和 Systrace 相同可視化效果的 html 呢？SDK 中 Systrace 工具的--from-file 命令可將原始的.trace 格式數據轉成 html 格式，分析.trace 數據內部格式："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f2\/f2fe551dc9e3f519425ccecad16471b2.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多次嘗試後得出結論，可被 SDK Systrace 工具解析的 .trace 文件需滿足如下格式："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f4\/f470219a30ca6ab22e155f0cd73e2b6e.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"格式說明："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"：線程名，若爲主線程，可指定爲包名。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"：線程 ID。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"：方法開始或者結束的時間，單位 s。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"：標記該條記錄爲方法開始(B)還是結束(E)。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"：所在進程 ID。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"：方法標記，字符長度不可超過 127。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由此可知，Mtrace 採集的數據至少需要包含以上內容。以下則是對應 Trace 格式："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\ndepth,methodID,inTime,outTime,threadName,threadID\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"depth,methodID,inTime,outTime,threadName,threadID相較於 Rhea 1.0 的 Systrace，Rhea 2.0 的 Method Trace 性能損耗有了顯著的降低，性能損耗也由 11.5%下降至 3%，效果如下所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/c5\/c56a3fdd3f0d3079bdcdb5f8ba62132f.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3. 最佳實踐"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"功能使用"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MTrace 相較於 Systrace，提供了更豐富的線下功能，其中包括：解決針對真實用戶點對點的卡頓耗時問題反饋功能，解決產品、運營、QA 同學外出走查場景的問題反饋功能。總之，不管你在哪，性能反饋都一觸即達！如圖爲完整操作流程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b4\/b4ff612b215c0811d657ca9ef51eebc4.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"線上案例"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一個真實的案例：抖音灰度版本線上用戶反饋卡頓，通過 MTrace 功能包實現遠程卡頓問題分析排查！以下則是通過用戶配合回傳的真實卡頓數據，經過解析即可發現耗時調用點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d1\/d1f33f9f438a5fc4e964a0c531f74a22.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/df\/df30c195a9296379fc1b19ef63694124.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"4. 存在的問題"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於這個階段採集的只有 Java 方法層的數據，在抖音啓動 IO 耗時優化工作中，Method Trace 無法提供哪些函數進行了 IO 操作，以及 IO 操作讀取\/寫入了哪些文件，給優化工作帶來了較大的難度。另外在一些複雜場景中，Method Trace 只記錄函數執行時長，但是不能準確定位是由於多線程同步等鎖或者系統 IO 導致的執行時間變長。針對上面的問題，我們意識到一套優秀的 Trace 工具還需要融合更多的系統事件，於是工具進入了第三階段的打磨。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"第三階段：動態一體化 Trace 工具規劃"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Rhea 1.0 和 2.0 在抖音早期的性能優化工作中成績顯著，但隨着優化工作的深入同時也暴漏諸多侷限與不便。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一方面，使用常規的 Systrace 工具做性能優化，本身有諸多侷限性。一是 Trace 信息少，在默認情況下，只包含系統預置的耗時打點信息，並不足以支持常規的耗時分析需要在 App 側手動調用 Trace.beginSection 和 Trace.endSection 方法才能獲取更多函數耗時信息，爲避免影響線上包大小，使用完以後又需手動移除，一上一下事倍而功半。二是 Systrace 本身性能損耗大，特別是應用通過插樁的方式對業務代碼進行大量的打點時，極端情況性能損耗會超過 50%。三是 Systrace 完全依賴 PC 端工具抓取，不夠靈活。尤其是需要能夠穩定復現性能問題的場景，對於一些特定區域或者特定用戶羣體才能復現的問題無法獲直接高效的取到有效信息，依賴研發或者測試走查，甚至用戶反饋的部分概率問題即使走查也無法通過 Systrace 獲取到對應的信息，從而導致優化效率低。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另一方面，通過簡單定製 Trace 的獲取函數耗時相較於 Systrace，雖然有顯著的性能提升，和更高的靈活性，但數據只包含基本的耗時信息，在部分複雜場景（如持有鎖引起的耗時），數據仍存在侷限。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如上工具都均已無法完全滿足抖音的啓動、首刷以及低端機等核心場景的性能優化工作，我們需要重新設計和規劃功能更加強大的動態一體化 Trace 工具來輔助分析性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"該工具要非常靈活，可以不依賴 PC 端的抓取腳本，同時支持線上線下，能夠在應用任何想要抓取數據的時候運行，作爲一個平臺性工具，Rhea 還需要支持動態擴展，支持多種場景的配置和動態開關，可以將任意需要的信息進行採集。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":2,"normalizeStart":2},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"該工具抓取的 Trace 信息要全面，能夠採集和追蹤包括 ATrace 插樁、等鎖信息、I\/O 信息以及 Binder 耗時等在內的多種信息。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":3,"normalizeStart":3},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"要支持可視化，統一的格式進行輸出和格式化，最終以兼容 Systrace 的結果進行前端展示和使用，儘量不要改變使用者習慣。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":4,"normalizeStart":4},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"性能損耗要低，以免帶偏性能優化方向。因此，我們重新設計了新一代 Trace 分析工具：圖片整體上，App 通過集成 Rhea SDK 在打包時不限層級插入函數耗時樁方法，在運行時插入 IO、Binder、Lock 等相關 Trace 信息，支持動態配置，統一 Trace 格式爲 atrace，同時支持獲取系統級別的 Linux ftrace、Android Framework atrace 和 App 插入的 atrace 信息，能夠不依賴 PC 抓取，最終提供可視化顯示。具體實現如下："}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"一、不依賴 PC 抓取 Trace"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了實現不依賴 PC 抓取 Trace，我們有必要先了解下 Android atrace 的實現機制。首先，是 atrace 包括的數據源包括："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a9\/a9dc30bf933b131be86157e805e5bbe0.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中，用戶空間的數據包括了應用層的自定義 Trace、系統層的 gfx 渲染相關 Trace、系統層打的鎖相關的 Trace 信息等，其最終都是通過調用 Android SDK 提供的Trace.beginSection或者 ATRACE_BEGIN 記錄到同一個文件點\/sys\/kernel\/debug\/tracing\/trace_marker 中的。此節點允許用戶層寫入字符串，ftrace 會記錄該寫入操作時的時間戳，當用戶在上層調用不同函數時，寫入不同的調用信息，比如函數進入和退出分別寫入，那麼 ftrace 就可以記錄跟蹤函數的運行時間。atrace 在處理用戶層的多種 trace 類別時，只是激活不同的 TAG，如選擇了 Graphics，則激活 ATRACE_TAG_GRAPHICS，將渲染事件記錄到 trace_marker。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而內核空間的數據主要是一些補充的分析數據 freq、sched、binder 等，常用的比如 CPU 調度的相關信息包括："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CPU 頻率變化"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"情況任務執行情況"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大小核的調度情況"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CPU Boost 調度情況"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9c\/9c39e530897c6496c485073ce42c48ad.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這些信息是 App 可以通過直接讀取\/sys\/devices\/system\/cpu 節點下相關信息獲得，而另外一部分標識線程狀態的信息則只能通過系統或者 adb 才能獲取，且這些信息不是統一的一個節點控制，其需要激活各自對應的事件節點，讓 ftrace 記錄下不同事件的 tracepoint。內核在運行時，根據節點的使能狀態，會往 ftrace 緩衝中打點記錄事件。例如，激活線程調度狀態信息記錄，需要激活類似如下相關節點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nevents\/sched\/sched_switch\/enable\nevents\/sched\/sched_wakeup\/enable"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"激活後，則可以獲取到線程調度狀態相關的信息，比如："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Running: 線程在正常執行代碼邏輯"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Runnable: 可執行狀態，等待調度，如果長時間調度不到，說明 CPU 繁忙"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Sleeping: 休眠，一般是在等待事件驅動"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uninterruptible Sleep: 不可中斷的休眠，需要看 Args 的描述來確定當時的狀態"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Uninterruptible Sleep - Block I\/O: IO 阻塞"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/35\/350ff4473b852c22c0497917a5da0e03.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最終，上述兩大類事件記錄都彙集到內核態的同一緩衝中，PC 端上 Systrace 工具腳本是通過指定抓取 trace 的類別等參數，然後觸發手機端的\/system\/bin\/atrace 開啓對應文件節點的信息，接着 atrace 會讀取 ftrace 的緩存，生成只包含 ftrace 信息的 atrace_raw 信息，最終通過腳本轉換成可視化 HTML 文件。大致流程如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/c1\/c1285dfe5b464ce3d2778266084fc9d5.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，我們基於 Android atrace 的實現原理，我們同步參考了 Facebook 的 profilo 用於在 APP 側直接獲取 atrace 的方案，實現了不依賴 PC 抓取 Trace 的方法。我們通過 dlopen 獲取 libcutils.so 對應句柄，通過對應 symbol 從中找到 atrace_enabled_tags 和 atrace_marker_fd 對應指針，從而設置 atrace_enabled_tags 用以打開 atrace 開關，具體實現如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\n\n std::string lib_name(\"libcutils.so\");\n std::string enabled_tags_sym(\"atrace_enabled_tags\");\n std::string marker_fd_sym(\"atrace_marker_fd\");\n\n if (sdk < 18) {\n lib_name = \"libutils.so\";\n \/\/ android::Tracer::sEnabledTags\n enabled_tags_sym = \"_ZN7android6Tracer12sEnabledTagsE\";\n \/\/ android::Tracer::sTraceFD\n marker_fd_sym = \"_ZN7android6Tracer8sTraceFDE\";\n }\n\n if (sdk < 21) {\n handle = dlopen(lib_name.c_str(), RTLD_LOCAL);\n } else {\n handle = dlopen(nullptr, RTLD_GLOBAL);\n }\n \/\/ safe check the handle\n if (handle == nullptr) {\n ALOGE(\"atrace_handle is null\");\n return false;\n }\n\n atrace_enabled_tags_ = reinterpret_cast<:atomic> *>(\n dlsym(handle, enabled_tags_sym.c_str()));\n if (atrace_enabled_tags_ == nullptr) {\n ALOGE(\"atrace_enabled_tags not defined\");\n goto fail;\n }\n\n atrace_marker_fd_ = reinterpret_cast(\n dlsym(handle, marker_fd_sym.c_str()));"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來，我們通過 hook libcutils 動態庫中的 write、write_chk 方法通過判定 atrace_marker_fd 來將對應 atrace 信息攔截下來轉儲到到本地或上傳到雲端分析。實現如下所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nssize_t proxy_write_chk(int fd, const void* buf, size_t count, size_t buf_size) {\n BYTEHOOK_STACK_SCOPE();\n if (Atrace::Get().IsAtrace(fd, count)) {\n Atrace::Get().LogTrace(buf, count);\n return count;\n }\n\n ATRACE_BEGIN_VALUE(\"__write_chk:\", FileInfo(fd, count).c_str());\n\n size_t ret = BYTEHOOK_CALL_PREV(proxy_write_chk, fd, buf, count, buf_size);\n\n ATRACE_END();\n\n return ret;\n}"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"二、提供更加全面 Trace 信息"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"鎖耗時"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Java 層的鎖，無論是同步方法還是同步塊，最終都會走到虛擬機的 MonitorEnter 和 MonitorExit，在 MonitorEnter 中實現了多種鎖狀態的切換，包括從無鎖到輕鎖，輕鎖中的偏向和重入，出現競爭並超過自旋的次數之後升級成重鎖分配 monitor 對象，其中 art 現在的自旋不是真的自旋，而是用 sched_yield 主動讓出 CPU 等待下次調度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而我們需要首先關注的就是出現鎖競爭升級成重鎖後的等待耗時信息，這個信息從 Android 6.x 開始會通過 ATrace 的方式輸出到 trace_marker 中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但是想要輕鎖的信息還需要做一些額外的工作，因爲是否輸出輕鎖的 ATrace 信息除了 ATRACE_ENABLE 條件之外，還有另外一個 systrace_lock_logging 的開關變量控制，這個變量是虛擬機中一個全局變量的成員，這個成員變量的值正常情況下是由虛擬機啓動的時候確定，默認是 false，可以通過啓動虛擬機的時候傳遞-verbose:sys-locks 參數來打開，但是作爲普通應用我們沒有辦法通過這種方式來打開，所以需要用非常規手段在運行時動態打開："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"首先確認從 Android7.x 開始，這個結構的大小、成員順序是否有發生變化；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"如果沒有變化，則可以自己定義一個相同的結構，因爲裏面都是原始的 bool 類型變量，不會引入其他依賴；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"如果有變化，但是向前兼容，我們想要訪問的成員位置沒有變化，只是往後追加了成員，也同樣可以自己定義相同的結構；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"通過 dlsym 找到虛擬機的全局符號 gLogVerbosity；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"將其類型轉換爲預先定義的結構體類型；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null},"content":[{"type":"text","text":"訪問 systrace_lock_logging 成員並賦值爲 true；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":7,"align":null,"origin":null},"content":[{"type":"text","text":"輕鎖的 ATrace 信息即可正常輸出；"}]}]}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nstd::string lib_name(\"libart.so\");\n\/\/ art::gLogVerbosity\nstd::string log_verbosity_sym(\"_ZN3art13gLogVerbosityE\");\n\nvoid *handle = nullptr;\nhandle = npth_dlopen_full(lib_name.c_str());\nif (handle == nullptr) {\n ALOGE(\"libart handle is null\");\n return false;\n}\n\nlog_verbosity_ = reinterpret_cast(\n npth_dlsym(handle, log_verbosity_sym.c_str()));\nif (log_verbosity_ == nullptr) {\n ALOGE(\"gLogVerbosity not defined\");\n npth_dlclose(handle);\n return false;\n}\n\nnpth_dlclose(handle);"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2. IO 耗時"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在做抖音在啓動路徑上性能優化時，我們統計了冷啓動的耗時，其中佔比最長的是進程處於 D 狀態（不可中斷睡眠態，Uninterruptible Sleep ，通常我們用 PS 查看進程狀態顯示 D，因此俗稱 D 狀態）的時間，這部分耗時佔比佔總啓動耗時的 40%左右，進程爲什麼會被置於 D 狀態呢？處於 uninterruptible sleep 狀態的進程通常是在等待 IO，比如磁盤 IO，其他外設 IO，正是因爲得不到 IO 的響應，進程才進入了 uninterruptible sleep 狀態，所以要想使進程從 uninterruptible sleep 狀態恢復，就得使進程等待的 IO 恢復。類似如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/67\/6782cf7edb8f8f0e5fb030602d3fbd4c.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但我們在使用 Systrace 進行優化時僅能得到如上內核態的調用狀態，卻無法得知具體的 IO 操作是什麼。因此，我們專門設計了一套獲取 IO 耗時信息的方案，其包括用戶空間和內核空間兩部分。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一是在用戶空間，爲了採集到需要的 I\/O 耗時信息，我們通過 Hook I\/O 操作時標準的關鍵函數族，包括 open，write，read，fsync，fdatasync 等，插入對應的 trace 埋點用於統計對應的 IO 耗時。以 fsync 爲例："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nint proxy_fsync(int fd) {\n BYTEHOOK_STACK_SCOPE();\n ATRACE_BEGIN_VALUE(\"fsync:\", FileInfo(fd).c_str());\n\n int ret = BYTEHOOK_CALL_PREV(proxy_fsync, fd);\n\n ATRACE_END();\n return ret;\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/bc\/bcfb7d4073b28bb7f6f7dc5f3937102e.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"二是在內核空間，除了可由 systrace 或 atrace 直接支持啓用的功能之外，ftrace 還提供了其他功能，並且包含一些對調試性能問題至關重要的高級功能（這些功能需要 root 訪問權限，通常可能也需要新的內核）。因此，我們基於此添加了顯示定製 IO 信息等功能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在線下模式，我們開啓了\/sys\/kernel\/debug\/tracing\/events\/android_fs 節點下 ftrace 信息，用於收集 IO 相關的信息，這時候，我們追本溯源，先找到 Systrace 之母，Google Android 和 Chrome 團隊的所有開源項目 Catapult 。正是 Catapult 生成了 Systrace 及其解析器的工具，在 Catapult 中，採用 javascript 實現了一個跨平臺的 trace 解析工具，我們在此基礎上開發了 Rhea 工具腳本將轉換成 systrace 可顯示化的格式，用於快速診斷髮現 IO 性能瓶頸。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"例如，我們線上監控發現我們某個 View 方法調用 setText 方法會導致 ANR，線下通過 Systrace 抓取 Trace 如下：圖片此時，看到主線程處於 D 狀態，卻束手無策，而通過我們的 Rhea 工具，獲取 Trace 如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/be\/bea5b88a85af159656fa10184c94b909.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們很容易就定位到此時是由於讀取對應字體帶來的 IO 耗時導致的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3. Binder 耗時"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在抖音啓動性能性能優化過程中，我們通常還會遇到 Sleep 帶來的耗時問題，這部分耗時通常佔據總耗時的 30%左右，處在這種睡眠狀態，進程通常是在等鎖或是 Binder 調用耗時導致，通常在線下，我們可以通過開啓 tracing\/events\/binder 節點獲取到，但是在線上由於權限問題我們很難獲取到這部分信息。因此，我們通過 Hook libbinder.so 對應的 android_os_BinderProxy_transact 方法來統計對應 binder 調用耗時。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nif (TraceProvider::Get().isEnableBinder()) {\n \/\/ static jboolean android_os_BinderProxy_transact(JNIEnv* env, jobject obj,jint code, jobject dataObj, jobject replyObj, jint flags)\n bytehook_stub_t stub = bytehook_hook_single(\n \"libbinder.so\",\n NULL,\n \"_ZN7android14IPCThreadState8transactEijRKNS_6ParcelEPS1_j\",\n reinterpret_cast(proxy_transact),\n NULL,\n NULL);\n stubs.push_back(stub);\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"後，統計對應 binder 耗時，如果耗時超過指定閾值，則將對應堆棧打印出來用於輔助分析 Sleep 耗時問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nstatic void log_binder(int64_t start, int64_t end, int64_t flags) {\n JNIEnv *env = context.env;\n env->CallStaticVoidMethod(context.javaRef, context.logBinder, start, end, flags);\n}\n\nstatus_t proxy_transact(void *pIPCThreadState, int32_t handle, uint32_t code,\n const void *data, void *reply, uint32_t flags) {\n \/\/ todo: add more informations\n nsecs_t start = systemTime();\n status_t status = BYTEHOOK_CALL_PREV(proxy_transact, pIPCThreadState, handle, code, data, reply,\n flags);\n nsecs_t end = systemTime();\n nsecs_t cost_us = ns2us(end - start);\n if (is_main_thread() && cost_us > 10000) {\n log_binder(ns2us(start), ns2us(end), flags);\n nsecs_t end_ = systemTime();\n }\n\n return status;\n}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"效果如圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f9\/f9227cd6ba6f664d72cd0425a4bcfa8d.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"4. 支持後續增加更多數據源"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當然，僅僅支持上述這些信息不可能完全覆蓋我們性能優化過程中未來還可能遇到的其他問題，因此，我們支持了動態配置的功能，後續僅需要在現有框架下，簡單添加對應配置項及其功能即可快速方便收集到我們所需要的信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\nenum TraceConfigKey {\n kIO = 0,\n kBinder,\n kThinLock,\n kStopTraceUnhook,\n kLockStack,\n\n kKeyEnd,\n};\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"5. 不限層級插樁獲取函數耗時"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"限制插樁的層級固然可以提升運行時性能，但是限制層級後面臨兩個問題:函數調用數據採集不全面;難以定位深層的耗時調用;因此在用戶態，爲了獲取 App 更多的 Trace 信息，便於性能優化。我們採用不限制層級的插樁方案。開發了在編譯階段不限制層級插樁的插件，通過靜態代碼插樁方式，在 App 調用方法的起始和結束位置分別插入 Trace.beginSection 和 Trace.endSection 。效果如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3a\/3aaa1f6c15a73656bb40cbf0f97f28ce.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"三、優化降低性能損耗"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"插樁性能優化"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在插樁階段, 我們做了如下優化:"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持自定義插樁作用域, 減少 Trace 對於其他無關模塊的運行損耗;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對 Trace 數據出現不閉合的問題, 對 catch 代碼塊進行全插樁;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對高頻調用函數, 可以選擇性的添加到黑名單中, 提升運行時性能;"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲支持生產環境使用，我們採用在 proguard 後進行插樁，由於函數內聯等優化, 相較於混淆前插樁插樁數量可以減少 2.6%。對於線上模式，直接插入方法 ID，收集 Trace 後需在主機端或服務端對方法 id 重新映射成方法名，但又考慮到線下用戶的易用性，在線下模式打包階段直接插入方法名；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在編譯階段通過分析字節碼信息，過濾掉不耗時函數的插樁。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"2. 優化 App 側啓停 Trace 性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於 App 側抓取 Trace 的實現要依賴於 hook，我們參考了 Facebook Profilo 的實現，但其實現存在動態庫過大、啓停 Trace 耗時問題，因此我們進一步優化了 App 本地獲取 atrace 依賴的動態庫大小和性能。如下所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/82\/824c0b6ebda07c837569a60598564368.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"3. 優化 Trace 寫入性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於在 App 方法中插入大量 Trace 信息，在開啓 atrace 後，所有線程會將所有的 trace 都寫入到 trace_marker 文件，會帶來 IO 損耗劇增，會掩蓋真實性能問題，原因是所有線程都在短時間向 trace_marker 文件進行寫入操作，同時競爭內核態 pos 鎖，導致獲取到的 trace 文件無法真實反映性能問題，如下圖所示："}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/cc\/cc9fb9bd7e1f2084555dbae7d4e735df.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，我們將原本直接寫入內核態文件的 Trace 在用戶態進行攔截，緩存起來，再以異步 IO 的方式轉儲。既避免了大量用戶態與內核態切換帶來的上下文損耗，又避免了直接 IO 帶來的 IO 損耗。效果如下所示："}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/e1\/e129c6c44b3a511fb8cc29eeaeafd132.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"四、可視化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於我們將用戶態 atrace 和內核態 ftrace 分別存儲在對應空間下的 ringbuffer 中，原生的 systrace 只能分別進行可視化，因此我們開發了統一整合 trace 的腳本工具，將多個 trace 信息將成爲單個的 html 文件，當瀏覽 trace 信息時，可在 Chrome（chrome:\/\/tracing 訪問）中可視化顯示。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"未來規劃"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，Rhea 對 Native 的支持還不夠全；性能優化還不夠極致，特別在用於分析卡頓問題時需要定位幾毫秒甚至更細粒度耗時的情況下，性能損耗仍然會有些偏大，在一定程度上會帶偏優化方向；目前 Trace 工具更多的還是在線下使用，由於插樁過多影響了包大小，使得我們線上部分只能對小規模的用戶羣體定向打開，沒法全量上線定位線上大規模用戶的性能問題。未來我們會重點解決如上問題，將 Trace 工具打造到極致。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"小結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前新一代 Trace 分析工具 Rhea 其主要優勢如下:"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、使用靈活，不依賴 PC 抓取腳本，同時支持線上線下多種模式和配置開關；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、支持採集和追蹤包括不限層級 ATrace 函數耗時插樁、等鎖信息、I\/O 信息以及 Binder 耗時等在內的多種信息；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3、兼容性高，支持 API 16~30 全機型的 trace 抓取；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4、零侵入代碼，通過 gradle 完成插件全部配置，無任何代碼直接調用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：字節跳動技術團隊（ID：outiaotechblog）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/vkBeZ6hmVn_RaXS5Xv_L2g","title":"xxx","type":null},"content":[{"type":"text","text":"抖音 Android 性能優化系列：新一代全能型性能分析工具 Rhea"}]}]}]}