Facebook 怎樣做自動化測試
最近Quora上有個討論，原意是：“facebook是如何做自動化測試的，他們是怎樣測試才能保證每週的升級都可以不出差錯的呢？” 參見Link
來自Facebook的Steven Grimm很好地回答了這個問題，覺得還不錯，這裏以第一人稱翻譯了一下。
- 對於PHP的代碼，我們寫了非常多的基於PHPUnit測試框架的測試類，這些測試類覆蓋範圍比較大，從簡單的判讀真假的單元測試到大規模的後端服務的集成測試。開發人員把運行這些基於PHPUnit的測試用例作爲他們工作中的一部分，同時這些用例也在一些專用的設備上不停地被運行（注：持續集成模式）。當開發人員對一些代碼做了比較大的修改時，在開發機器上的自動化工具會運行這些測試用例的同時也會生成相應的代碼覆蓋率數據，對於需要提交到代碼庫的diff，在做代碼review的時候回自動地產生一份帶有覆蓋率的測試報告。
- 對於前端的代碼，我們使用Waitir（注：Waitir是前端UI的自動化測試框架）做了基於瀏覽器的界面自動化測試。這些測試用例涵蓋了網站頁面的功能，特別是針對隱私方面，比如：“用戶X發佈了Y，而Y應該/不應該被用戶Z看到”，有着大量的基於瀏覽器級別的這種用例。（這些隱私規則當然也會使用一些更低級別的方法被測試到，但是這些規則的實現是必須要嚴格執行的，並有着非常高的優先級，因此這部分必須要有足夠的測試用例來覆蓋）
- 除了一些使用watir的全自動化用例以外，我們也有一些半自動化的測試。這些測試也使用了waitir技術，這樣可以使一些表格填充或者點擊button來完成整改界面上的流程的測試不太單調乏味，而且我們可以很清楚地檢查和驗證當前的步驟或流程是否正確合理。
- 我們也在嘗試開始使用JSSpec （注：JavaScript單元測試框架）去做一些JavaScript代碼的單元測試，但當前也是剛剛開始做。
- 對於後端服務的測試，根據不同的服務特性我們採用了許多不同的測試框架與方法。對於一些需要開源發佈的項目，我們會使用開源的測試框架，像Boost和JUnit測試框架（注：Boost是針對C++/JUnit是針對Java的測試框架）；對於另外一些項目，可能永遠都不會發布到外界，我們就是使用內部開發的可以很緊密地與我們build系統集成在一起的C++測試框架。還有少數項目會使用項目級別的測試工具。多數後端服務的測試都會緊緊地和持續集成/Build系統結合在一起，這些持續集成的build系統會不停地針對源代碼自動地運行測試用例並生成測試結果，測試結果在存儲在數據庫的同時會發送到通知系統中去。
- HipHop(注：HipHop for PHP是Facebook的PHP項目)有一套類似的持續集成系統，HipHop的單元測試和所有基於PHPUnit的測試都會被運行。所有的這些測試結果會和基於普通的PHP解釋器的結果做對比，從而可以看到不同PHP上的行爲的不同；
Facebook的測試工具將測試結果存儲在數據庫的同時會發送一份通知郵件，這個郵件會包含執行失敗的信息並且郵件的接收範圍是開發同學可以自己調整的。（例如，你可以選擇只有在測試連續失敗一段時候的時候才接收到通知郵件，或者當一個用力失敗的時候立刻收到通知）。在瀏覽器UI上，測試結果和 缺陷/開發任務跟蹤系統會結合在一起，可以很容易的將測試失敗與開發任務關聯起來。
測試中一個非常重要的現象是“導致阻塞”，也就是一個測試用例失敗有可能會阻止發佈（在Facebook，有發佈工程師會來評估是否可以將帶有問題的代碼發佈到生產環境，發佈工程師在必要的情況下會得到授權去阻止產品的發佈）。阻止產品發佈上線的事情是被認爲是非常嚴重的問題，因爲在Facebook大家對於這種快速發佈的模式是深深引以爲豪的。
我所在的團隊是測試工程部門，主要職責是打造通用基礎工具，這些工具會被上述的所有人用到，同時我們也在維護測試框架，像PHPUnit和Watir。Facebook沒有專職的測試團隊，所有的工程師都需要爲他們的代碼寫自動化測試用例，並維護這些測試用例，保證產品代碼改變的同時這些測試代碼可以正確地運行。
Facebook的測試還處於一個初期起步嘗試階段，上面的介紹都只是我們在當前運行的方法而已。
公直 2012/2/27
附錄原文，
What kind of automated testing does Facebook do? How do they make sure they aren’t breaking things in their weekly pushes?
Steven Grimm, 2005-2012
We do several kinds of testing. Some specifics:
- For our PHP code, we have a suite of a few thousand test classes using the PHPUnit framework. They range in complexity from simple true unit tests to large-scale integration tests that hit our production backend services. The PHPUnit tests are run both by developers as part of their workflow and continuously by an automated test runner on dedicated hardware. Our developer tools automatically use code coverage data to run tests that cover the outstanding edits in a developer sandbox, and a report of test results is automatically included in our code review tool when a patch is submitted for review.
- For browser-based testing of our Web code, we use the Watir framework. We have Watir tests covering a range of the site’s functionality, particularly focused on privacy—there are tons of “user X posts item Y and it should/shouldn’t be visible to user Z” tests at the browser level. (Those privacy rules are, of course, also tested at a lower level, but the privacy implementation being rock-solid is a critical priority and warrants redundant test coverage.)
- In addition to the fully automated Watir tests, we have semi-automated tests that use Watir so humans can avoid the drudgery of filling out form fields and pressing buttons to get through UI flows, but can still examine what’s going on and validate that things look reasonable.
- We’re starting to use JSSpec for unit-testing JavaScript code, though that’s still in its early stages at this point.
- For backend services, we use a variety of test frameworks depending on the specifics of the services. Projects that we release as open source use open-source frameworks like Boost’s test classes or JUnit. Projects that will never be released to the outside world can use those, or can use an internally-developed C++ test framework that integrates tightly with our build system. A few projects use project-specific test harnesses. Most of the backend services are tied into a continuous integration / build system that constantly runs the test suites against the latest source code and reports the results into the results database and the notification system.
- HipHop has a similar continuous-integration system with the added twist that it not only runs its own unit tests, but also runs all the PHPUnit tests. These results are compared with the results from the same PHP code base run under the plain PHP interpreter to detect any differences in behavior.
Our test infrastructure records results in a database and sends out email notifications on failure with developer-tunable sensitivity (e.g., you can choose to not get a notification unless a test fails continuously for some amount of time, or to be notified
the instant a single failure happens.) The user interface for our test result browser is integrated with our bug/task tracking system, making it really easy to associate test failures with open tasks.
A significant fraction of tests are “push-blocking”—that is, a test failure is potential grounds for holding up a release (this is at the discretion of the release engineer who is pushing the code in question out to production, but that person is fully empowered to stop the presses if need be.) Blocking a push is taken very seriously since we pride ourselves on our fast release turnaround time.
My team, Test Engineering, is responsible for building the common infrastructure used by all the above stuff, as well as for maintaining PHPUnit and Watir. Facebook has no dedicated QA team; all Facebook engineers are responsible for writing automated tests for their code and keeping the tests maintained as the underlying code changes.
Facebook’s test setup is still very much a work in progress, but the above is at least a taste of what we do in that area.
From http://www.quora.com/What-kind-of-automated-testing-does-Facebook-do
