字節跳動精準測試實踐，SmartEye背後的設計邏輯

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"簡介"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在移動互聯網高度發展下，移動APP趨近於大型化，功能也趨近於複雜化，諸如各種國民級應用，功能\/玩法等迭代不停，其背後對應的研發協作團隊也非常龐大。在此種場景下，依靠人力堆疊去保障產品質量，並非最優解，在軟件規模達到某個臨界點之後，問題逃逸將無法避免。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"字節跳動在代碼覆蓋率監測平臺基礎上，建立了SmartEye-智能化精準中臺，幫助研發團隊快速定位代碼變更影響域，精準推薦測試用例\/自動化巡檢任務，爲業務減負，爲研發團隊提效。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在日常的研發活動中，我們經常會遇到下列場景："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這次需要研發自測保障了， 我的用例集是不是全都有效覆蓋了？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這次技術重構改動挺大的，會影響哪些已有功能？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基礎工具SDK有重大升級，我是涉及到的業務方，哪些功能需要測試驗證？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"版本要上線了，大家都走一下全量回歸Case，測試重點在哪裏？迴歸測試用例集全量執行是不是必要的？"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在項目研發團隊中的每個同學質量標準是不是都統一了？"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上述問題在產品快速迭代下會越來越突出，每一個軟件工程師應該都經歷過上述問題的“拷問”。在此背景下，精準測試也應運而生。精準測試是國內軟件工業界首創的技術概念。近年來，隨着國內工業界深入探索，精準測試已經有較爲廣泛的影響力和統一的認知。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"精準測試是質量工程智能化建設的重要趨勢，爲項目研發驗收全流程建立公信力度量指標，解決用例與代碼的雙向關聯，精準匹配和投放測試資源（人力\/機架），削弱\/打破項目規模增長與質量保障資源增長的強正相關關係。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"工作流程"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/b6\/b66be60329b9aaa0ea9549e01f9687d6.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上述是字節跳動研發團隊比較常見的研發流程，在這裏我們把研發需求劃分爲5個階段：開發、自測、測試、集成迴歸、上線。精準測試發生的區間主要在自測、功能驗收測試和迴歸測試階段。在這三個階段的節點1、2、3，業務通過請求SmartEye服務，可以分別獲取到自測推薦用例，功能測試推薦用例，迴歸測試推薦用例。其中，自測用例和功能測試推薦用例用於驗證代碼變動是否引起存量功能異常，將功能缺陷發現時機前置，爲研發團隊爭取更多的修復時間。迴歸測試用例推薦用來精簡迴歸測試人力，幫助業務節省迴歸測試人效。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/19\/19397b1629c19bc5912d6f56db57e2fc.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"用例知識庫"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們前期先建設了基礎代碼覆蓋率監測平臺，收集程序運行時的動態代碼覆蓋率數據，以此爲基礎來構建用例知識庫。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/45\/45adab7c9065c792a7b5ea0817db1fc4.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"用例錄製"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"客戶端手工用例"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/e8\/e801a07ed4e86c8c1ba31170c5099262.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"人工用例錄製，利用內嵌到客戶端的SDK，提供UI界面供用戶進行錄製操作，同時由SDK內部保障錄製過程中數據清理，採集，上報。用例錄製結束後由SDK對當次錄製數據進行上報，服務端實時解析，落盤用例知識庫，供後續數據分析推薦，並及時給予用戶錄製反饋。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"客戶端自動化用例"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f7\/f7007438283ad6dc5a95b427aa911007.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"自動化用例錄製全流程無人工干預，錄製時需要以單個用例爲分割，記錄單條用例的代碼覆蓋率數據和用例的關聯關係。SmartEye結合客戶端高級調試模式SDK，使用公司基礎長鏈接服務控制自動機架的指定設備進行用例數據的錄製\/停止，全流程自動化，可以以很小的成本，批量錄製客戶端自動化用例。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"用例結構化信息"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/55\/5543b32fd3ec4ebed78699dd68a4207b.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"用例索引"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"利用當前變動信息、調用鏈信息、覆蓋率信息等基礎能力，建設代碼-用例的映射數據集，以索引形式存儲海量用例，爲測試用例推薦等上層服務產品提供數據基礎。基於用例的關聯方法、關聯代碼塊、上下游調用鏈路以及對應覆蓋率等信息，建設不同粒度不同版本的"},{"type":"text","marks":[{"type":"strong"}],"text":"調用信息-CaseIDs倒排索引服務"},{"type":"text","text":"，直接支撐推薦引擎的用例召回功能；此外，服務適配用例錄製接口，實現了用例錄製->用例信息解析->用例信息入庫->索引生成的全流程自動化處理，並支持雙端、人工&&自動化用例的多維度用例獨立存儲。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/10\/108213f03d765d08c61f5cff306e2c64.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"推薦引擎"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"代碼影響域分析"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"準確定位MR的代碼變動信息及其代碼調用上下游影響域，能夠幫助後續精準測試準確分析與之關聯的測試用例，提升測試充分性與問題攔截率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/1d\/1d190b3035fac32539d983c4ad2ac75f.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們利用公司CodeGraph平臺的雙端靜態代碼調用鏈分析能力與整體調用鏈路拓撲數據，完成基於MR的變動DIFF分析以及上下游調用鏈分析，並針對變動DIFF部分，完成代碼變動率的統計。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/fd\/fd0e39140511e208421587edcd34e029.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"關聯用例召回與多樣性平衡"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於改動影響的邊界範圍，召回對應方法、代碼塊、調用鏈路關聯的用例集合，聚合去重後生成用例召回集合；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用例召回集合中，存在大量冗餘用例。爲實現向線上高頻調用模塊的測試傾斜，增加多樣性控制邏輯。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於對應方法&&代碼塊的關聯用例數、用例覆蓋率以及線上熱度歸一化指標，生成對應方法&&代碼塊的多樣性控制因子，並最終將該因子作用在用例排序與推薦集生成階段；"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/d1\/d17219a1f77984bbaa9ccf17fa99e931.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"推薦集生成"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"模型層"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在用例排序階段基於用例與變動影響域內各方法&&代碼塊的關聯信息、用例後驗指標、用例多樣性控制因子等多維度信息，通過機器學習用例排序模型，計算用例與本次改動的相關性得分；基於相關性得分進行用例排序，並結合業務方指定的推薦用例數量，完成用例的篩選，生成用例篩選集。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"策略層"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對底層代碼關聯大量測試用例、用例路徑多樣性不足等導致的測試冗餘問題，通過向業務側賦能，結合業務方的先驗知識體系，利用用例定向投放、底層代碼規避等策略，以更加貼近業務的策略形式，在保證測試質量的同時，進一步降低測試冗餘，提升測試效率。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"整體方案"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/1f\/1f5fcbad30b6016dfbd9f1ab8915525a.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"使用效果"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，我們與火山客戶端等研發團隊一起，搭建了字節跳動精準測試平臺Cover，致力於爲公司內所有產品線研發團隊提供標準的精準測試解決方案，期望在保證質量的前提下爲業務提效。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該平臺已在火山、抖音、西瓜等團隊客戶端迭代流程中落地，並取得了不錯的效果。其中一個典型業務線利用精準測試，在保證100%攔截全量測試用例可發現問題的基礎上，提升人效約40%，具體數據如下:"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/60\/605b2fe0171b2fb4d8f098ba61b63801.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前，SmartEye精準引擎每日推薦用例數1K+，效率提升約40%。藉助SmartEye的能力，業務可以精確評估變更影響，清晰代碼變動之後需要測試\/迴歸的用例集。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於SmartEye的精準測試，在字節跳動的落地得到了很多團隊的支持，包括火山、抖音、頭條、基礎架構等多個研發團隊，業務落地當前正在持續推進中。後續重點，我們將延伸SmartEye的精準能力，覆蓋各種類型的測試用例與測試需求，在更大範圍內成爲DevOps基礎服務，爲更多的研發團隊提效。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們相信，越來越多的智能化測試工具落地，將會加速質量工程領域的變革，推動國內質量工程技術水平走到全球質量工程工業界前沿。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：字節跳動技術團隊（ID：BytedanceTechBlog）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/wUrm8gycgLHhCSHbpG8Hkw","title":"xxx","type":null},"content":[{"type":"text","text":"字節跳動精準測試實踐，SmartEye背後的設計邏輯"}]}]}]}