Test Accelerator
(www.nist.gov/ta)
Mary Brady (mary.brady@nist.gov)
Overview:
Many enterprises have
become driven by digital data, including healthcare, e-commerce,
and homeland security. Each of these application areas are
developing standards that allow for the exchange and manipulation of
this data. As the plethora of standards is promulgated throughout
the industry, the demand for metrics to measure conformance to these
standards has increased. Effective use of these metrics dictates
that they be developed in a short amount of time, and be done so in
a way that allows changes to the specification to be easily
propagated through a given set of tests.
Industry Needs Addressed: It has
become well-known that software is not adequately tested and is
rushed to market full of bugs. A recent NIST study says that the
annual cost to the country for inadequate testing is as high as 59
billion dollars. One of the main reasons for inadequate testing is
the time and expense of developing comprehensive test suites. A
test accelerator that provides core reusable components, yet allows
input and output formats to be defined by the user, will facilitate
building conformance tests. Making the testing process faster, more
flexible, and extensible provides the IT community with tools to
quickly develop tests that will be used to identify and correct
errors in their implementations and standards.
NIST/ITL Approach: NIST developed
a prototype test accelerator that is flexible and extensible.
Building upon our prior experience in XML-based testing, we
developed a data-driven approach that consists of encapsulating the
structure and semantics of a specification in a NIST developed XML
test language. The test accelerator takes the XML input and creates
an input control graph, which is intelligently traversed, resulting
in a set of test instances. The test instances are transformed
using NIST developed templates to generate specific tests for a
given test suite. Development of additional tests can then be
accomplished via incremental changes to the pre- and post
processors. Work has begun on developing additional methods to trim
the control graph and generate appropriate data values for specific
data types.
This approach was
applied to the Linux Standard Base stdlib and to XML Query expressions. These
standards were selected to scope the work and provide significant
complexity to truly investigate this approach. Each of these
communities are developing standards that are large and complex,
thus requiring an automated testing approach.
Impact: NIST developed conformance
tests provide the information technology industry with the necessary
tools to ensure that their standards are correctly implemented,
which will lead to interoperability in many application areas. The
NIST test development system will allow us to focus on producing
high quality tests while reusing capabilities from prior testing
projects. It will also allow us to develop methods for defining
semantic capabilities for complex systems.
|