|
As part of NIST's mission to advance the science of testing and measurement, we have conducted extensive analyses of software failure in real-world systems. Combinatorial testing takes advantage of what we refer to as the interaction rule, which is based on our analysis of thousands of software failures. The interaction rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. Our focus is on empirical results and real-world problems. For a quick summary of combinatorial and pairwise testing, see this article or this one. Tools provided freely on this site are used by hundreds of organizations around the world for software and hardware testing of reliability, safety, and security. |
NEW ITEMS Talks for 2013: Carnegie Mellon U, Feb 12 (Kuhn), Loyola U.of Md., Feb 26 (Kuhn) ; IEEE ECS workshop seminar, Singapore, July 16 Tool releases: Version 2.7 of ACTS testing tool and Version 0.7 of CCM coverage measurement tool now available. Combinatorial Coverage Measurement - NIST IR 7878 (released Sept. 2012) 2nd International Workshop on Combinatorial Testing association with 5th IEEE International Conference on Software Testing, Verification and Validation (ICST 2013, March 18-22) Luxembourg. Please consider submitting a paper! Full details at the link. |
Our research
program
currently includes:
Some of our accomplishments to date include:
|
2009 Excellence in Technology Transfer Federal Laboratory Consortium, Mid-Atlantic Region |
FREELY
AVAILABLE SOFTWARE:
Software on this site is free
of charge and will remain free in the future. No
license is required and there are no restrictions on use.
NIST is
an agency of the United
States Government. We have over 1,200 users as of January 2013, in nearly all major industries. Here is a breakdown of our user base. 1) Combinatorial testing tool. The tool, named Advanced Combinatorial Testing System (ACTS) can compute tests for 2-way through 6-way interactions. An easy-to-use GUI is included. More here. A comparison of ACTS with similar tools shows that ACTS produces smaller test sets (with the same degree of coverage) and is faster than others. To request a copy, send email to Rick Kuhn - kuhn@nist.gov. Please include your first and last name, and company or university name (this helps us with management support for the project!) 2) Combinatorial coverage measurement tool. Interested in combinatorial methods but concerned about disrupting existing test practices? The CCM measurement tool can analyze existing tests for 2-way through 6-way interactions they already have. An easy-to-use GUI is included. To request a copy, send email to Rick Kuhn - kuhn@nist.gov. Please include your first and last name, and company or university name (this helps us with management support for the project!) 3) Comprehensive tutorial, Practical Combinatorial Testing, NIST SP800-142 (pdf 81 pages, 841Kb). This publication provides a self-contained tutorial on using combinatorial testing for real-world software. It introduces the key concepts and methods, explains use of software tools for generating combinatorial tests, and discusses advanced topics. The material is accessible to an undergraduate student of computer science or engineering, and includes an extensive set of references to papers that provide more depth on each topic. Oct. 2010. Public domain, distribution unlimited; 81 pages. Over 21,000 downloads! 4) Security policy testing tool. The Access Control Policy Test (ACPT) tool allows policy authors to conveniently specify access control models (such as RBAC and Multi-Level models) and rules as well as access control properties. From the specified models and rules, the ACPT tool automatically synthesizes deployable policies in XACML and generates combinatorial tests to verify security policy implementations. Complete test cases are generated, consisting of test inputs and expected output for each set of inputs. ACPT uses ACTS to provide 2-way to 4-way combinatorial testing of policies. To request a copy, send email to Vincent Hu - vhu@nist.gov. 5) Web app testing tool. CPUT (Combinatorial-based Prioritization of User-session-based Testsuites) applies combinatorial methods to testing web applications. Test prioritization is used to make web app testing much more manageable. The tool allows testers to easily collect, prioritize, and reduce user-session-based test cases. CPUT provides (1) guidance to users on how to configure their web server to log important usage information, (2) automated parsing of web logs into XML formatted test cases that can be used by test replay tools, (3) automated prioritization of test cases by length-based and combinatorial-based criteria, and (4) automated reduction of testcases by combinatorial coverage. |
S.
Manchester, N. Samant, R. Bryce, S. Sampath D. R. Kuhn, and
R. Kacker, “Applying Higher Strength
Combinatorial Criteria to Test Prioritization: a Case Study”, Journal
of Combinatorial Mathematics and
Combinatorial Computing - to appear TBD 2012.