精準測試二三談

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們都在使用敏捷開發，敏捷測試，維護着我們的項目，我們寫着少量的test case，甚至不寫一條case，敏捷宣言其中一條原則是工作的軟件 “高於” 詳盡的文檔，詳解文檔包括各種計劃書，總結報告，詳盡測試用例等，我們將大量時間用在自動化測試以及手工探索性測試上面，而我們的用例則以BDD形式存在於代碼之中，這樣來幫助儘可能早的發現問題。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但最近我發現幾個客戶在質量問題，存在一些共性，這些基於黑盒測試的項目在測試過程中存在以下幾個共同的問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"（1）大量的黑盒測試用例"},{"type":"text","text":"，有的項目甚至用例數超過5w，測試工作大都是手工爲主，受主觀人爲因素影響太大：每次版本發佈，QA全憑個人經驗來確定改動對系統影響範圍，通常情況，要麼測試範圍定小了，造成漏測，要麼測試範圍過大，付出的代價過高，造成項目不能如期按時交付。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"（2）代碼與測試沒有數據可衡量"},{"type":"text","text":"：沒有單元測試，其他類型測試對代碼覆蓋程度，質量高低，沒有數據能夠衡量，例如我們說api測試覆蓋率是100%，這個數據大多都是根據用例業務場景估算出的。QA只能增加更多的黑盒測試，而實際功能測試覆蓋率隨着時間和用例增多，便會觸達覆蓋率的天花板，更多的是重複的無效測試。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"（3）自動化測試無法發揮作用"},{"type":"text","text":"：對於web\/api或app 後端服務系統，測試人員對除手工測試外，我們將大量的時間與精力投放在api接口測試的實現上，隨着項目的迭代，自動化用例積累越來越多，從幾百到上千，這時候我們需要考慮測試穩定性，運行時長，大量重複測試場景與代碼，整個測試ROI並不是隨着用例數增多而上升，反而維護和排查問題成爲QA日常工作的重擔，疲於應付，沒有精力將時間投入到更有用的探索性測試和分析工作中，進而造成bug頻出，整個團隊便對自動化失去信任，直至廢棄，這也是很多傳統行業無法規模化實施敏捷測試原因之一。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼如何幫助團隊樹立信心，準確定位到變更影響的範圍？精準測試。這個概念是最近幾年逐漸興起的話題。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"先來談談什麼是精準測試？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f5\/f5df075daf7ffbc05730a4cee054f0e1.png","alt":"圖片","title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"定義：利用技術手段對測試過程產生的數據進行採集存儲，計算，彙總，可視化最終幫助團隊提升軟件測試的效率、並對項目整體質量進行改進和優化的這一系列操作。通俗點講：核心基於源代碼變更分析，結合分析算法，確定影響範圍，提升測試效率。精準測試並沒有改變傳統的軟件測試方法論，只不過是幫助我們將測試用例與程序代碼之間的邏輯映射關係建立起來， 而這個過程則是通過算法和工具去採集測試過程執行的代碼邏輯及測試數據，在測試過程加入採集過程，形成正向和逆向的追溯。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正向追溯，開發人員可以看到QA執行用例的代碼細節，例如用例執行過程中，調用具體方法與實現類，方便進行缺陷的修復與定位。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"逆向追溯，測試人員通過release前的增量代碼快速確定測試用例的範圍，極大減少迴歸測試的盲目性和工作量，提升ROI，達到測試覆蓋率最大化。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"精準測試原理"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/95\/95044c45c1bb7e21185f29c7defcceb4.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/a6\/a6e468f72c0dda0afd587c8ef8c7f7de.png","alt":"圖片","title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖是我基於業界通用精準測試架構畫出的架構圖與流程圖。這套精準測試架構既可以用作手工測試，也可用在任意自動化測試上。整個架構分爲以下幾部分："}]},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"建立用例與代碼覆蓋率之間的映射關係"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"影響面評估，分析識別增量與變更代碼"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"測試範圍評估，用例篩選，鏈路分析"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"建立用例與代碼覆蓋率之間的映射關係"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前建立用例與代碼之間的關係通過統計調用產生的覆蓋率與路徑進行關聯。"}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在線模式"}]}]}]},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"功能測試關聯方案"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前通用的解決方案是利用Jacocoon-the-fly模式基於 On-the-fly 方式無須入侵應用服務代碼，啓動腳本時，只需在 JVM 中通過 -javaagent 參數指定 jar 文件以啓動 Instrumentation 的代理程序，該代理程序通過 Class Loader 裝載一個 class 前判斷是否需要注入 class 文件，再將統計代碼插入 class ，測試覆蓋率分析就可以在 JVM 執行測試的過程中完成。Jacoco提供了自己的Agent，完成插樁的同時，還提供了豐富的dump輸出機制，如File,Tcp Server,Tcp Client。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"覆蓋率信息可以通過文件或是Tcp的形式輸出。通過外部服務在任意機器上通過api請求獲取被測程序的覆蓋率與執行路徑。對功能測試用例，我們可以通過執行單個用例，經過上述步驟後拿到該條用例影響代碼的覆蓋率與執行路徑。在通過關聯服務，將具體用例的ID與生成的覆蓋率信息（類，方法，行等）建立映射關係，最後將關聯數據存到數據庫中保存。基於Jacocoon-the-fly 模式獲取單個用例覆蓋率建立映射關係的流程如下："}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/1c\/1c5056411b30801074ea330247ff3af8.png","alt":"圖片","title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上是基於java語言來關聯用例與代碼直接的關係，前端也有類似原理的工具，利用istanbul-middleware也可以實現同樣的功能，具體請查閱一下這兩篇文章，這裏不再複述。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":"2","normalizeStart":"2"},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"自動化測試關聯方案"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"利用AOP原理，在自動化框架的執行器加一個攔截器，在覆蓋率收集開關打開時，請求執行前，這個過程類似於測試框架中的before hook：reset 被測服務樁數據，也就是上一次測試產生的dump 文件，請求執行後，這個過程類似於測試框架中的after hook：用api導出內存中的覆蓋率數據，我們利用反射拿到用例方法名，利用關聯服務將之與對應的 dump結果關聯起來，實現自動插樁功能，快速幫助我們建立自動化用例與覆蓋率之間的關係。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/77\/77dd489aa668c870844d68b6d05fe14b.png","alt":"圖片","title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Offline模式（unit test採用模式，不是本文）："}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在測試前先對文件進行插樁，然後生成插過樁的class或jar包，測試插過樁 的class和jar包後，會生成動態覆蓋信息到文件，最後統一對覆蓋信息進行處理，並生成報告。Offline模式適用於以下場景："}]},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"運行環境不支持java agent"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"部署環境不允許設置JVM參數"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"字節碼需要被轉換成其他虛擬機字節碼，如Android Dalvik VM"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"動態修改字節碼過程中和其他agent衝突"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"無法自定義用戶加載類"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"影響面評估，分析識別增量與變更代碼"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們有了代碼與用例直接的關係的映射，我們需要將之用在開發流程中，首先我們需要得知我們的改動是什麼，最直接的是通過 git diff 得知具體改動代碼，但過於繁重，且太多幹擾例如註釋，空行等，最好的方法是實現比對算法，經過降噪處理，消除干擾，進而拿到處理後變更數據。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3f\/3f48bead0f5247dd0de30040ba81b583.png","alt":"圖片","title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"測試範圍評估，用例篩選，鏈路分析"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們有了用例與代碼之間的關係映射，有了提交增量代碼差異記錄，就可以實現逆向回溯。利用代碼的差異，通過查詢服務就可以在上面提到關聯關係數據庫中反推影響的用例，以及上層的業務。這樣可以幫助QA快速劃分測試範圍，減少過度測試。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/9f\/9f32e80c7d19315a4370ab93869668cd.png","alt":"圖片","title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結精準測試的優點"}]},{"type":"numberedlist","attrs":{"start":"1","normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"這種代碼與用例，業務之間的關聯關係，能夠加深我們對被測系統及架構的瞭解，在不斷的版本迭代過程中，能夠實時瞭解任何類型測試對於當前版本的覆蓋率，是否有遺漏的場景等，幫助團隊更好的建立信心，使質量真正走上可持續化道路。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"精準測試在項目的中後期不在不依賴個人能力以及業務熟悉度等特點，大幅降低了團隊測試的成本，使得團隊QA能夠有大量的時間做探索性測試以及質量度量上，提高QA對於團隊的ROI，帶給團隊更清晰的質量數據。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但事物總是存在對立面的，獲得巨大的收益同時，必然相應的存在缺點。否則也應當像UI自動化測試一樣流行於各個公司以及團隊中。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"精準測試存在的問題"}]},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"基於手工測試的精準測試建立映射關係繁雜，如果需求改變頻繁，用例維護以及之間的關係維護需要耗費大量時間精力。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"精準測試需要一定的自動化測試的覆蓋，這樣做起來更有意義，例如api自動化測試，如果本身用例過少，與代碼之間關聯關係不多時，變更代碼後可能不會得出什麼結果。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"最好有對應的用例管理系統，能夠方便的幫助我們建立與代碼之間的關係。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"需要投入開發能力強的QA或者測試開發建立整套系統環境，但長遠考慮，將精準測試嵌入整個公司的質量平臺中，不管對於新項目還說維護項目來說都是一種提升。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"項目生命週期需要較長，短期項目花費巨大精力開發和維護整套精準測試系統得不償失。短期項目可以利用精準測試以api測試覆蓋率作爲衡量標準。不去建立繁雜的關係，只監控UI API測試覆蓋率迭代時的變更來達到目的。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"精準測試不是銀彈，需要巨大的投入，用的好，能夠成倍的提升質量，生產效率，用不好的話，就成了領導的KPI項目，棄之可惜，食之無味，雞肋也。 "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"齊磊，前Thoughtworks高級質量分析師，現任HSBC測試諮詢專家，擅長敏捷測試，測試開發，devops等領域。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：ThoughtWorks洞見（ID：TW-Insights）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/WLqDVQdI1bSv8VnYCx6eIg","title":"xxx","type":null},"content":[{"type":"text","text":"精準測試二三談"}]}]}]}