Number of Interactions Involved in Software Failures - empirical data

One of the most important questions in software testing is "how much is enough"?  For combinatorial testing, this question takes the form of determining the appropriate level of interaction that should be tested.  That is, if some failure is triggered only by an unusual combination of more than two  values, how many testing combinations are enough to detect all errors? What degree of interaction occurs in real system failures?  This page summarizes what we know from empirical studies from a variety of application domains.  Although certainly not conclusive, the available data suggest that the number of interactions invovled in system failures is relatively low, with a maximum from 4 to 6 in the six studies cited below.  (Note:  TCAS study used seeded errors, all others are "naturally occuring".)  This suggests that combinatorial testing of all 4-way to 6-way combinations should have a high probability of detecting nearly all faults.

If you have have questions, or would like to contribute data to this collection, please email me at kuhn@nist.gov.  


Interaction faults graph

FTFI Medical
Devices		 Browser Server NASA
GSFC		 Network
Security		 TCAS
1 66 29 42 68 17 *
2 97 76 70 93 62 53
3 99 95 89 98 87 74
4 100 97 96 100 98 89
5 99 96 100 100
6 100 100
* = not reported
** Note:  TCAS study used seeded errors, all others are "naturally occuring"
Medical Devices:  D.R. Wallace, D.R. Kuhn, Failure Modes in Medical Device Software: an Analysis of 15 Years of Recall Data, International Journal of Reliability, Quality, and Safety Engineering, Vol. 8, No. 4, 2001.    

Browser, Server:  D.R. Kuhn, M.J. Reilly, An Investigation of the Applicability of Design of Experiments to Software Testing, 27th NASA/IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, 4-6 December, 2002 .  

NASA database:  D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr., Software Fault Interactions and Implications for Software Testing, IEEE Trans. on Software Engineering, vol. 30, no. 6, June, 2004.  

Network Security:  K.Z. Bell, Optimizing Effectiveness and Efficiency of Software Testing: a Hybrid Approach,  PhD Dissertation, North Carolina State University, 2006.    

TCAS module:  D. R. Kuhn, V. Okun, Pseudo-exhaustive Testing For Software, 30th NASA/IEEE Software Engineering Workshop, April 25-27, 2006.    (seeded errors)