Testing vs field observed defects

Myers put forth the counter-intuitive principle in software testing which states that the more defects found during formal testing the more that remained to be found later.

There seems to be a positive co-relation between the rate of defects found during formal testing with the rate of defects reported from the field. Higher rate of defects reported during a formal testing exercise usually means that there has either been a higher rate of error injection during the development process or that a new and more effective approach to testing has been followed. It could also be the case that a lot of additional and extra-ordinary test effort was expended resulting in the higher rate of defects being found.

A popular analogy to describe the relationship between defect rates during formal testing and field trials is to consider the overall defect rate as an iceberg. The tip that is visible is likened to the defects found during testing and the submerged portion as the latent field defect rate. The overall size of this iceberg is determined by the level of error injection during development. Formal testing normally happens once the code is developed and integrated by which time the “iceberg” is already formed. The larger the tip that is visible, the larger would be the entire iceberg.

This does not mean that we just get into a mode of acceptance about the latent defects that would be revealed during field usage. We can take steps to reduce the extent of the latent defects and bring up more of the iceberg above water. It must be stressed that managing of quality of the development process is important and can contribute towards reducing the rate of error injection. Prevention is definitely better than trying to determine and fix defects (in the process probably introducing other defects). Even with robust processes, some amount of error injection cannot be ruled out and this is where practices such as good design & code reviews and inspections are needed. Additionally, unit and integration tests by developers prior to checking in code into the repository should help reduce the number of defects that are left lurking around. The testing team must also continually enhance their tests, improve coverage and analyse defect rates & trends across releases to make sure that testing is doing its best to find as many issues as it can.