Here's a familiar scenario: Testers spend months or longer, testing a product. Once the product is released, users report bugs that were not found by the testing team. The obvious question that gets asked is - how did the testing team miss these issues ?
Listed below are some of the common reasons why users catch issues that the software testing team may have missed.
- The testing team has not tested in an environment that is similar to what the user uses. This could happen for a variety of factors. It could be due to a lack of awareness of the user environment or usage scenario. Where there is awareness of the user environment, it may be that the testing team did not have the time or bandwidth available to test with this scenario. Where there is awareness and time, it may be that the testing team could not replicate the scenario due to physical or logistical constraints such as availability of required hardware, software, peripherals, etc. While it is not possible to replicate every possible usage scenario, testing must consider the most possible or widely used cases.
- The steps that users followed differed from what the testing team followed. This can happen either when users follow a different set of steps than what the testing team may have followed or when the order of the steps followed differs. Even when the same set of steps are followed, if the ordering of the steps differ, it can have different consequences.
- The user entered a set of input data that was not covered during testing. This can occur for the simple reason that it is physically not possible to test every possible set of input data. When a product is deployed widely, the chances that some users somewhere will enter a set of values that was untested is likely. While designing tests, testers choose sets of input values to test with. Errors in making these choices can also contribute to user reported defects.
- The defect that users reported could come from code that was not tested. It could either be due to having released untested code or the existing set of tests did not exercise the piece of code where users found defects. The challenges encountered by the software testing team increases as our products become more complex.