閒魚性能測試那些事兒

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“小明，小明，性能報告麻煩給一下” "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“小明，小明，你這數據有問題啊，跟我測的不一致，重新再測試”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"“小明，小明，我們搞了個體驗優化，幫忙跑一下數據吧” "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上述故事純屬虛構，如有雷同，純屬巧合。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"引言"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如文章開頭的故事所言，在閒魚APP發版期間，也會遇到性能報告生成不及時、性能數據不準確的問題。先一起來看個圖回顧一下閒魚最初的性能測試流程。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/57\/570d9254007baff5039d0bd38d04923e.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"從上圖中我們可以發現閒魚之前的性能測試流程自動化覆蓋率特別低，基本上所有環節都需要人力介入，十分耗時耗力，主要問題如下： "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"人工成本高"},{"type":"text","text":" 閒魚發版時需要回歸多個主幹場景的性能信息，投入人力成本至少1人日，適配同學偶爾苦不堪言。 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"手動操作誤差大，頻繁返工"},{"type":"text","text":" 參與跟版的同學是輪崗制，他們的操作習慣不同、打開頁面不同會導致數據存在誤差，偶爾需要返工，最多一次需要反覆測試四五次。 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"提測無性能卡點"},{"type":"text","text":" 人工測試時性能卡點都是放到了發版的階段，往往會耗費過多精力去排查什麼引起的，我們把性能卡點前置就可以避免"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"數據沒有沉澱"},{"type":"text","text":" 在自動化之前，性能數據都是用一個Excel來統計，然後生成報告，整個數據都沒有沉澱，我們也沒有抓手去判斷到底是哪裏產生了問題。 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"研發自測難"},{"type":"text","text":" 開發同學經常會做個體驗優化，這個時候想驗證一下，每次都在等資源，然後適配同學做這種枯燥的事情又很枯燥、很無奈。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於以上原因，我們開始着手設計開發閒魚端性能測試平臺，通過流程化、規範化、自動化來提升測試效率，釋放人力去做更多有意義的事情，本文主要介紹了閒魚性能自動化測試平臺的解決方案以及落地過程中踩過的坑。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"解決方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其實我們主要想解決的問題有3個：人力成本、誤差問題、數據沉澱，對於底層基礎能力，SLM和TMQ已經足夠完善，我們不需要重複造輪子，所以主要集中測試流程、腳本以及數據沉澱，底層實現直接使用集團現有能力。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"系統設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在整個系統設計中，我們秉承着極簡主義，絕不重複造輪的理由，整個系統底層上直接使用SLM、TMQ現有能力進行性能數據收集，同時報表數據使用deepinsight生成。在腳本管理方面，通過Git倉庫來進行管理，每個分支代表某個版本的性能測試腳本，這樣我們可以同時回溯多個版本的性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在任務觸發方法，我們在閒魚質量平臺中開發了性能測試模塊，用於性能測試腳本、任務的管理，用戶不需要關注底層的平臺實現，只需要在閒魚質量平臺上提供包地址以及可選參數，即可觸發任務，而這些任務可能是在TMQ運行，也可能是在SLM運行。另外我們和整個打包構建流程打通，當有release包構建成功時，能夠自動觸發性能任務進行數據收集，減少人力介入。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"任務完成後，通過數據迴流，可直接在平臺上進行數據查看，同時對於版本數據，能夠直接進行歸檔，歸檔後，數據通過odps同步到deepinsight實現報表的自動化和生成。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/83\/8354282581585f62669b96d723c0e7cd.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們的測試目標不應該是爲了出數據而出數據，而是能夠給出開發排查建議，具體定位到性能是受哪塊業務、哪個合併的影響，這也是我們規劃中的內容，後續會在該方面持續發力，利用歷史數據沉澱進行問題自檢、排查建議。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"具體測試流程"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如圖3所示，整個測試流程分爲三個階段：任務觸發、自動化執行、PTM\/PM數據確認以及問題排查。對於研發同學而已，他們做完性能優化之後，可以直接在平臺觸發任務進行性能測試，不需要等待質量同學和適配同學介入就能夠自己快速驗證效果。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/5d\/5dd7921a15eeeba3bcca36b240f06602.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在發版流程中，我們將性能測試左移到每個release包構建完成後，只要release渠道包構建完成，系統會自動觸發性能測試任務，一旦迴歸完成，我們就能夠立刻獲得該版本的性能數據。這個過程減少了人力介入，能夠避免人爲因素帶來的版本節奏推遲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在手工測試時，經常會出現數據不準確導致返工反覆測試，中間的溝通其實浪費了很多的時間和人力成本，所以我們在平臺中加入了自動重試機制，當某個版本的數據計算完成後，會自動與歷史版本數據進行對比，如果對比結果數據差異過大，系統會自動在不同的機器上重跑最近兩個版本的數據並進行比較，將數據驗證前移。由於平臺做了一輪數據比對和校驗，所以在PTM\/PM數據確認以及問題排查階段一般情況下不需要再投入額外的精力，除非APP性能確實出現了問題。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"踩過的坑與雷"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"讓整個流程跑起來很簡單，但是在實際使用中有不少坑，所以整個系統上線後依然做了不少調整。以下是我們遇到的問題以及解決方案，希望對大家的性能測試工作有所幫助。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"機器溫度對性能的影響"},{"type":"text","text":" 由於機房的機器整天都在執行任務，再加上一直充電，就會導致設備溫度升高，升高之後CPU會進行降頻，這時候FPS、CPU都不準確。爲了解決這個問題，我們在每次任務執行前，先把設備佔用，停止執行任務10分鐘，讓設備的溫度降低，這樣能夠減小溫度帶來的誤差。 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"FPS受到網絡加載的影響"},{"type":"text","text":" 在Feeds測試時，不同頁面網絡的加載會導致滑動的幅度有差異，這個時候每次在測量FPS時都會有波動，雖然求取平均值能夠減少誤差，但是並不準確。於是我們在測量時，將整個性能測試分爲兩個階段：CPU\\Memory\\流量收集階段，即上滑時收集內存、內存、和流量信息，下滑時收集FPS，因爲在下滑時，數據都已經加載完畢，我們就可以規避網絡帶來的誤差。 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"設備對性能的影響"},{"type":"text","text":" 設備的影響，不同的設備條件不同，在測量時CPU、FPS均有誤差，爲了解決這個問題，我們已經開始自建閒魚自建的設備庫，對於版本性能測試，使用專門的設備進行測試。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結與展望"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經過一階段的實踐，目前該能力已經應用在閒魚發版流程中（6890-6940），總體效果如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"人力成本降低"},{"type":"text","text":" 版本性能測試從1人日->1h以內，同時數據有爭議時，能夠同步自動化快速進行驗證，不需要考慮適配成本，這樣可以投入更多的人力做更需要人力的地方。 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"高效穩定"},{"type":"text","text":" 相比於之前，目前的測試數據更加穩定，除了極個別場景需要特別回顧以外，大多數場景都能夠準確測試，基本上在跟進的幾個版本中沒有出現重複跑數據的情況。在兩次發版過程中出現2次性能數據差異過大，經排查都是代碼變更問題導致。 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"數據沉澱"},{"type":"text","text":" 目前我們的數據沉澱能夠幫助我們在後續進行更深層次的性能分析。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"展望"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"測試左移"},{"type":"text","text":" 我們希望把腳本通用化，開發只需要點點點、選選選就能夠指定需要測試的場景性能，而不需要維護腳本。 "}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"接入持續集成，提測卡點"},{"type":"text","text":" 在release包打出之後，我們的性能自動化平臺就能夠自動觸發，對其性能進行測試，發版時報告自動發送。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"深入化"},{"type":"text","text":" 當前的測試流程提供了問題發現能力，但在問題定位能力上還有所欠缺。後續我們會持續深入探索如何通過數據沉澱、分析來主動拋出指標異常可能是由於哪個模塊、甚至是哪個代碼塊導致，這樣不但能夠減少開發排查的成本，也能增加數據的可信性。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：閒魚技術（ID：XYtech_Alibaba）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/481fCIJmyncOYY_U1pWkIA","title":"xxx","type":null},"content":[{"type":"text","text":"閒魚性能測試那些事兒"}]}]}]}