乾貨 | 攜程機票前端安卓虛擬機測試集羣建設實踐

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在攜程內部業務高頻率敏捷迭代發佈節奏下，線上生產服務質量需要同步快速提升。這就依賴自動化測試的覆蓋率提升，測試任務執行頻次提升，測試任務執行速率提升。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過調研機票前臺內部，以及攜程其他事業部前臺研發和測試團隊，發現存在普遍共性痛點。需要清晰化解耦自動化測試體系的三層架構，解決當前核心瓶頸。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SaaS上層服務，例如UI測試，API測試，兼容性測試，穩定性測試。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"PaaS中層服務，例如測試用例管理，測試任務調度，測試框架驅動。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"IaaS底層服務，測試用例代碼最終執行和測試報告輸出功能。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"laas底層服務，公司各事業部的前端團隊普遍通過採購手機設備，組建自動化測試設備小集羣方案，或者基於開發測試人員的個人測試手機進行本地測試執行工作。這兩種方式都存在一些問題。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.1 IaaS 底層服務現狀問題"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）測試手機採購成本較高。需要覆蓋衆多系統版本，衆多廠商型號，以及持續更新換代。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）測試手機的廠商型號差異化，導致自建小型測試設備集羣的技術方案和運維管理困難。並且長期運行導致手機電池老化加速，電池膨脹損壞主板芯片，甚至存在潛在火災隱患。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）測試手機的用例執行穩定性和規模化受限，導致PaaS及SaaS上層的測試任務集整體執行效率降低。測試用例執行需要排隊等待，測試任務容易運行失敗，測試任務執行總耗時長。在集中發佈日或大版本發佈期間，大批量集成和迴歸測試用例集的任務堵塞擁擠。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.2 IaaS 底層基建不穩對業務研發的影響"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）自動化測試工作的推廣普及率和覆蓋率受限。開發和測試人員在自動化測試工作中，很多耗時用於分析 Failed Job 日誌，重啓重試測試任務，修復運行兼容性問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）海量測試用例集的總運行耗時較長，導致DevOps閉環反饋流程遲緩。於是大部分團隊將自動化測試任務的頻率延長到每天一次，每週一次，甚至每個版本一次。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）自動化測試落地受阻，導致研發團隊的生產交付質量被迫繼續依賴手工測試團隊的人工密集重複執行。手工測試容易人爲疏忽遺漏，以及帶來研發成本的固定支出。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、項目目標"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"重構IaaS底層基建系統，降低採購和運維成本，提升測試任務可靠性和性能速度。提供通用測試設備服務，無縫支持多種上層測試框架，方便全公司各種前端團隊的測試框架系統低成本接入。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、系統選型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"行業方案主要有三種：公有云真機集羣、私有云真機集羣和私有云虛擬機集羣。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1 公有云真機集羣"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"公有云真機集羣是指使用行業內一些公司提供的雲真機服務，例如 Testin，WeTest，以及華爲，三星等廠商提供的真機雲測實驗室等，通常按照使用時間和使用設備數量收費。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"優點：無需自建，公有云真機集羣的設備型號較爲完備。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"缺點：費用成本中等，但是通常僅支持少量幾種主流成熟測試框架，公司內各團隊歷史積累的測試用例集遷移成本較高。並且無法支持測試任務運行時依賴的衆多內網系統，例如Mock服務、SOA服務等等。導致真正可測試覆蓋的場景受限，對線上生產交付的質量保障有限。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.2 私有云真機集羣"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自建方式，需要採購真機設備和專用機櫃，如需滿足大規模測試用例集的高頻率高併發執行，就要相應採購大量手機設備。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"優點：真機設備的性能較好，並且可以針對性覆蓋一些特殊測試場景，例如挖孔屏、摺疊屏，特定廠商API等等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"缺點：設備數量決定了測試用例集合的併發執行速度，因此前期成本投入較高，只有當大量研發團隊和測試用例任務接入後，才能逐漸平攤降低成本，並且始終存在設備運維更新換代的成本支出。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現狀：攜程機票前端自動化測試開發團隊自2017年至今使用該方案實現小型測試設備集羣，通過若干臺Mac mini驅動管理大約20臺手機真機設備。目前該方案仍然持續運維，作爲私有云虛擬機集羣的補充。真機設備來源於常規採購的日常開發調試工作的淘汰換置。隨着使用時間增加，安卓系統更新換代，部分設備性能逐漸下降，給研發人員日常開發調試使用帶來不便。於是我們就將其換置託管到自動化測試集羣，發揮餘熱。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.3 私有云虛擬機集羣"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自建方式，使用安卓虛擬機鏡像（Android Virtual Device，以下簡稱AVD）執行測試，以此組成測試設備集羣，搭配一套管理系統對其進行統一調度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"優點：投入成本低，無需採購真機，便於根據使用量進行快速擴縮容，統一標準化管理，7x24小時可用，並且可以無縫銜接各種內部測試框架和內網依賴服務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"缺點：性能較真機略低，無法滿足少量特殊測試場景，無法覆蓋特定廠商機型定製功能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着Google Android官方團隊對AVD相關組件的逐步優化，其在 X86 桌面環境模擬運行ARM指令的速度大大提升，並逐步剝離其與Android Studio \/ SDK的耦合，更加易於獨立部署。隨着K8S，Docker技術的成熟與普及，Google開源了 android-emulator-container-scripts 實驗性項目，使得AVD + Docker + K8S 技術方案具備高可行性。因此我們啓動了該項目的研發工作。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、系統架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主要由三部分組成："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"容器實例層：基於K8S進行AVD容器的調度、編排，AVD 容器內包含Android 模擬器及完整運行環境、驅動程序和預安裝的系統依賴項，採用OVS、Neuron、Quota、Harbor組件實現網絡通信、虛擬化管理、鏡像存儲和分發。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"調度管理層：Android 模擬器的創建、銷燬、縮擴容、佔用、釋放、重啓、日誌排查等，以API方式提供服務。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"操作使用層：Android 模擬器的Web GUI可視化和CLI命令行操作使用。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/cb\/cbd2897d4763178585a38edf15e18c48.jpeg","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.1 容器實例層"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"AVD Container的調度編排，使用了內部現有的K8S管理服務，爲適應AVD容器化的使用要求，K8S管理服務做了以下適配處理："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Node配置修改：開啓KVM嵌套虛擬化支持"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"K8S容器化參數修改：設置 containers securityContext privileged = true"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"K8S管理服務爲每臺AVD容器設備分配固定IP，保證在部署和啓動AVD Container後，調用者可通過固定IP訪問。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Node資源配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"攜程App大部分是信息數據檢索頁面型應用場景，對於圖像和音視頻處理的要求不高，Node採用Intel(R) Xeon(R) CPU E5-2680 V3 @2.5GHZ 24Core 48Thread 192G Memory配置，未配置GPU顯卡。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當前Node配置的特點是內存足夠，CPU不足，所以考慮對CPU進行超分。雖然CPU超配可能會導致應用高併發下運行變慢，考慮到服務於測試環境，一定的性能損耗可以容忍。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"超配策略的原則是在保證服務質量的同時，儘量提高資源的利用率。通過進行節點壓測，分析歷史真實使用數據，確定了20%超分，高併發下AVD container仍然可以正常工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Pod資源配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"CPU：4C"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"內存：8G（初始啓動消耗2.9G，運行時消耗4G）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"顯卡：使用Google SwiftShader 軟加速庫，基於CPU進行圖形渲染"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"SwiftShader介紹"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"SwiftShader是Vulkan圖形API12的基於CPU的高性能實現。其目標是爲高級3D圖形提供硬件獨立性，可支持Android和Chrome（OS）構建環境。爲了向用戶提供最佳性能，SwiftShader 使用多種方法高效地在 CPU 上執行圖形計算。動態代碼生成使在運行時針對現有任務自定義代碼成爲可能，與更常見的編譯時優化完全不同。通過使用 Reactor 簡化這種複雜的方法，Reactor 是一種自定義 C++ 嵌入式語言，具有直觀的命令式語法。SwiftShader 還可以單指令多線程 (SIMT) 方式使用向量運算，並結合使用多線程技術來提高 CPU 可用內核和矢量單元之間的並行性。這樣可實現實時渲染，其用途包括在 Android 上進行應用串流等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用Airtest測試框架，選取機票流程中的常用頁面，從頁面渲染、元素獲取，模擬點擊元素這幾個維度，對AVD設備和普通Android設備進行性能對比測試。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

性能數據	AVD設備	普通Android真機
頁面渲染（等待關鍵元素完成加載）	5-8秒	3-5秒
模擬點擊元素	2秒	2.1秒
元素獲取	600ms	500ms"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"綜上所述，AVD設備除了頁面渲染性能比真機略慢，其他維度和普通Android真機性能基本持平，能夠滿足日常集成測試需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"AVD Container內運行AVD Docker鏡像，鏡像構建採用了Google開源的 android-emulator-container-scripts 技術方案，基於公司內部統一的Linux系統基礎鏡像，自定義Dockfile生成AVD  Image，並上傳至內部Docker Hub系統，鏡像文件主要包含："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Linux操作系統"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Android模擬器引擎"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"驅動程序和一些預安裝的系統工具、網絡服務代理"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持的Android系統版本"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Android 11 (API 級別 30)"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Android 10（API 級別 29）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Android 9（API 級別 28）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Android 8.1（API 級別 27）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Android 8.0（API 級別 26）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Android 7.1（API 級別 25）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Android 7.0（API 級別 24）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Android 6.0（API 級別 23）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Android 5.1（API 級別 22）"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Android 5.0（API 級別 21）"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/29\/296658fc257f0ea978925f106a692053.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"安卓模擬器架構圖（源自Google Android官網）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7b\/7b8462acf7940e25292f52d7dbf1b290.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於Google官方提供的 android-emulator-container-scripts只在Debian和Ubuntu下進行過測試，我們在Centos系統下發現該腳本有諸多問題，因此沿用Ubuntu作爲基礎鏡像。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按該腳本幫助文檔中的方案激活虛擬環境，通過運行emu-docker interactive --start命令，以交互方式選擇要使用的android和模擬器版本，之後將創建一個docker文件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"編輯Dockfile文件，並做以下修改："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"修改from基礎鏡像"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"安裝並配置ssh服務，用於遠程管理容器"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置容器代理服務器，使APP可訪問外網地址(可選)"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"修復容器啓動過程中發現的Dockfile腳本問題等"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將以上內容組合在一起生成Docker鏡像，這樣可以創建一個完整的運行環境，在其中運行Android模擬器，使得查找系統映像，管理系統依賴以及運行安卓模擬器變得非常容易。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.2 調度管理層"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實現AVD設備的創建、銷燬、擴縮容與使用管理、設備狀態監控等。restAPI採用java spring boot技術實現，主要包括以下幾個接口："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"獲取設備列表接口：獲取設備狀態、IP"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"佔用\/釋放設備接口：佔用時分配可用時長，時間到達後自動釋放"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"擴容設備接口：耗時小於2分鐘，單批最大50臺"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"縮容設備接口：耗時小於10秒，單批最大50臺"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"重啓設備接口：耗時小於10秒"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.3 操作使用層"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲方便用戶使用，系統提供了UI交互界面和CLI命令行模式，以下是命令行操作示例："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\n獲取設備列表、IP\n$ avd devices\n設備訪問連接\n$ adb connect device_ip_address：5555\n注：device_ip_address如上文所述由paas平臺在創建AVD Docker主機時分配，該IP在測試環境和辦公環境均可訪問。\n確認主機已連接到目標設備：\n$ adb devices\nList of devices attached\ndevice_ip_address:5555 device"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"五、AVD Iaas 高可用保障"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"AVD測試設備作爲laas基礎設施，穩定性是非常重要的指標，要能夠爲用戶提供7x24的穩定保障，我們基於以下幾個維度系統設計了保障策略。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）API服務架構層面"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"避免單點服務，多機部署"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"防止服務之間相互干擾，重要服務單獨部署"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"防止數據庫異常導致服務不可用，增加緩存處理"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）運維層面"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通用指標監控：CPU、內存、API請求量、響應時間、錯誤數"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"業務指標監控：自動化使用設備量、可用設備池庫存、設備申請失敗率"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接入攜程內部的報警系統，故障分鐘級別響應"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）代碼層面"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"保證代碼異常不會導致服務掛掉"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"保證服務是無狀態的，可以支持水平擴展"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4）設備彈性調度"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"防止設備不足導致任務積壓排隊，系統會監控任務隊列情況自動擴容"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"防止任務低谷下的設備大量閒置，系統會監控任務隊列情況自動縮容"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5）設備自動維護"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"防止設備被長期空閒佔用，系統針對設備的使用情況進行定期檢測，一段時間內未使用的設備，會自動收回到可用設備池，並通過IM消息通知到用戶"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根據設備監控報警，自動拉出異常設備重啓維護，確認正常後加入設備池"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"綜合以上幾點，穩定性較原先真機設備大幅提升，滿足7x24小時使用的需求，SLA指標達到99%，我們也在持續努力，繼續提升。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.1 遇到的問題"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於ARM編譯APP在X86架構Node運行時，會將ARM 指令都轉換成 x86 指令，造成較高的性能負荷，因此與基於 x86 編譯的APP相比，ARM編譯APP在 x86 宿主機上的運行速度會慢很多，而且它還無法使用 x86 處理器提供的硬件加速和 CPU 虛擬化技術。爲了保障應用的執行性能，我們的最初方案是將測試應用APP編譯爲X86模式，這樣可以減少Android系統指令轉換的性能開銷。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着規模的逐漸發展和更多用戶場景的提出，這套方案也逐漸暴露出了一些問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一些APP不支持x86編譯"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"編譯爲x86後，少量場景運行時，底層so文件會出現異常，而同樣的場景下，使用ARM編譯的APP卻沒有問題"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"綜合以上兩個問題，我們開始尋找優化方案，可以更好的支持ARM架構編譯的APP應用。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.2 問題解決方案"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Google官方在2020年開始推出Android11，新版本帶來了新特性。引入Android11，可以解決ARM編譯APP的性能問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"全新的Android 11 系統映像與 ARM 兼容，它不僅允許整個系統在本機運行 x86 指令，而且還可以照常使用虛擬化技術。當應用的某個進程需要使用 ARM 二進制代碼時，代碼僅會在該進程內被轉換成 x86 指令，其餘進程將繼續在 x86 環境內執行，包括 Android 運行時 (ART) 以及其它性能關鍵庫，例如libGLES和libvulkan。除此以外，指令轉換器也不會執行底層的硬件特定庫，從而避免高成本的內存訪問檢測和相應的性能影響。經過測試，在X86服務器上基於Android11運行ARM架構APP，性能確實比之前版本提升很多，因此我們引入Android11，用戶可根據APP編譯類型選擇合適的AVD容器。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"六、AVD Iaas服務中臺化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2020年攜程無線公共團隊提出建設無線CTest測試中臺的目標，AVD Iaas作爲底層基礎設施方案，也加入CTest中臺提供給攜程各事業部使用。目前已經有15個事業部接入，總使用次數超過10000+，滿足公司內多版本、多技術棧的測試任務執行和兼容性驗證需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7d\/7def9eed9f7f1b8f4552c5390a289e25.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6.1 大規模無線UI自動化集成測試應用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"無線UI自動化集成測試，是APP應用持續交付過程中繞不開的關鍵環節。過去在每個月交互1個版本的模式下，還能夠通過增加手工測試人力勉強滿足版本的驗證需求，隨着交付節奏的不斷加快，攜程內部的交付頻率不斷提速，如機票前臺研發團隊，期望提速爲每週交付2次。基於此大背景下，自動化集成測試的效率是必須要解決的問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖爲攜程機票前臺研發團隊的CICD流程圖。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過在特性分支和發佈分支高頻觸發自動化測試，並設置對應的質量卡口，大部分的測試工作交由自動化完成，幫助快速發現問題，減少開發修復成本，這對自動化集成測試提出了很高的性能要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過應用AVD Iaas，基於AVD容器設備的快速擴縮容能力，在項目測試開始時，系統會根據項目的case數量，動態創建、分配測試設備，保證單個項目的UI自動化集成可以在幾分鐘內完成。使用結束後，自動縮容回收。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最終較好的支撐了機票前臺研發團隊每週發佈2個版本，業務快速上線的要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b6\/b602404e185cdd235fcae9a5bd9ac36f.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下圖爲使用AVD laas設備後的測試執行耗時對比。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過按需動態擴縮容，緩解任務排隊瓶頸，提高併發能力，測試用例執行耗時平均降低74%。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/57\/5720254ebdb92913443ad758372c62f0.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6.2 接入AVD laas時遇到的典型問題"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在接入AVD laas的過程中，部分團隊出現了UI自動化case執行通過率降低的現象。通過分析，發現主要是case代碼不規範導致，比如：操作等待時長、滑動距離等操作存在硬編碼，導致case代碼只能在固定環境中執行通過。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲此團隊總結了自動化代碼規範，幫助接入方優化case代碼，更好的支持不同設備環境下的穩定執行，這裏就不再詳細展開了。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"七、總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前AVD laas系統已經支撐了攜程絕大部分業務線在不同場景下的移動端自動化測試設備需求。我們一直在努力豐富AVD容器設備的功能場景，不斷提升系統穩定性和性能，此外我們也在積極構建BDD測試執行框架、用戶流量回放等自研的研發工具，通過和AVD laas形成組合拳，解鎖研發活動中更多的適用場景，幫助業務團隊更好的提升能效。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者簡介"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Liang，攜程研發總監，關注DevOps，前端&服務端質量保障、能效提升；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Tony，攜程資深測試經理，關注自動化測試框架及平臺類工具開發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：攜程技術中心（ID：ctriptech）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/YD-bhGQo9LI77D5rlVaUJg","title":"xxx","type":null},"content":[{"type":"text","text":"乾貨 | 攜程機票前端安卓虛擬機測試集羣建設實踐"}]}]}]}