Software
bugs, or errors, are so prevalent and so detrimental that
they cost the U.S. economy an estimated $59.5 billion annually,
or about 0.6 percent of the gross domestic product, according
to a newly released study commissioned by the Department
of Commerce's National
Institute of Standards and Technology (NIST). At the
national level, over half of the costs are borne by software
users and the remainder by software developers/vendors.
The study also found that, although all errors cannot be
removed, more than a third of these costs, or an estimated
$22.2 billion, could be eliminated by an improved testing
infrastructure that enables earlier and more effective identification
and removal of software defects. These are the savings associated
with finding an increased percentage (but not 100 percent)
of errors closer to the development stages in which they
are introduced. Currently, over half of all errors are not
found until "downstream" in the development process
or during post-sale software use.
NIST
funded the study, which was conducted by the Research Triangle
Institute (RTI) in North Carolina, as part of a joint planning
process with industry to help identify and assess technical
needs that would improve software-testing capabilities.
Findings of the 309-page report are intended to identify
the infrastructure needs that NIST can meet through its
research programs.
"The
impact of software errors is enormous because virtually
every business in the United States now depends on software
for the development, production, distribution, and after-sales
support of products and services," said NIST Director
Arden Bement. "Innovations in fields ranging from robotic
manufacturing to nanotechnology and human genetics research
have been enabled by low-cost computational and control
capabilities supplied by computers and software."
In
2000, total sales of software reached approximately $180
billion, supported by a large workforce encompassing 697,000
software engineers and 585,000 computer programmers.
Software
is error-ridden in part because of its growing complexity.
The size of software products is no longer measured in thousands
of lines of code, but in millions. Software developers already
spend approximately 80 percent of development costs on identifying
and correcting defects, and yet few products of any type
other than software are shipped with such high levels of
errors. Other factors contributing to quality problems include
marketing strategies, limited liability by software vendors,
and decreasing returns on testing and debugging, according
to the study. At the core of these issues is difficulty
in defining and measuring software quality.
The
increasing complexity of software, along with a decreasing
average product life expectancy, has increased the economic
costs of errors. The catastrophic impacts of some failures
are well-known. For example, a software failure interrupted
the New York Mercantile Exchange and telephone service to
several East Coast cities in February 1998. But high-profile
incidents are only the tip of a pervasive pattern that software
developers and users agree is causing substantial economic
losses.
Study
Design and Background Facts
In the
study, RTI identified a set of quality attributes and used
them to construct metrics for estimating the cost of an
inadequate testing infrastructure. Two in-depth case studies
were conducted, one in the manufacturing sector (transportation
equipment) and one in the service sector (financial services).
For
the analysis of transportation equipment industries, data
were collected from 10 vendors of computer-aided design/manufacturing/engineering
(CAD/CAM/CAE) and product data management (PDM) software,
and from 179 users, primarily automotive and aerospace companies.
Approximately 60 percent of the automotive and aerospace
manufacturers surveyed reported significant software errors
in the previous year. Respondents who experienced errors
reported an average of 40 major and 70 minor software bugs
per year in their CAD/CAM/CAE or PDM software systems.
The
total cost impact on these manufacturing sectors from an
inadequate software-testing infrastructure is estimated
to be $1.8 billion, and the potential cost reduction from
feasible infrastructure improvements is $0.6 billion. Users
of CAD/CAM/CAE and PDM software absorb approximately three-fourths
of the total impact,
with the automotive industry representing about 65 percent
and the aerospace industry representing 10 percent. Software
developers experience the remaining one-fourth of the costs.
For
the analysis of financial services, data were collected
from four developers of financial electronic data interchange
(FEDI) and clearinghouse software as well as the software
embedded in routers and switches that support electronic
data exchange, and from 98 software users, primarily banks
and credit unions. Approximately two-thirds of the software
users surveyed reported experiencing major software errors
in the previous year. Respondents that did have major errors
reported an average of 40 major and 49 minor software bugs
per year in their FEDI or clearinghouse software systems.
Approximately 16 percent of those bugs were attributed to
router and switch problems, and 48 percent were attributed
to transaction software problems. The source of the remaining
36 percent of errors was unknown. Typical problems encountered
due to bugs were increased person-hours used to correct
posting errors, temporary shut down leading to lost transactions,
and delay of transaction processing.
The
total cost impact on the financial services sector from
an inadequate software-testing infrastructure is estimated
to be $3.3 billion. Potential cost reduction from feasible
infrastructure improvements is $1.5 billion. Software developers
absorb about 75 percent of the economic impacts. Users experience
the remaining 25 percent of costs, with banks accounting
for the majority of user costs.
The
annual cost to these two major industry groups from inadequate
software infrastructure is estimated to be $5.18 billion.
Based on similarities across industries with respect to
software development and use and, in particular, software-testing
labor costs, RTI projected the cost to the entire U.S. economy.
Using the per-employee impacts for the two case studies,
an extrapolation to other manufacturing and service industries
yields an approximate estimate of $59.5 billion as the annual
cost to the nation of inadequate software testing infrastructure.
Thus,
if all software bugs could be identified and removed instantly
(in real time), the combined economic benefits to the two
industry groups and to the economy would be $5.85 billion
and $59.5 billion, respectively. Realizing that such a "perfect
infrastructure" is not attainable, industry experts
were asked for estimates of a plausible reduction in delayed
identification and removal of software errors. Based on
this information, a "feasible improved infrastructure"
scenario was constructed. For this scenario, software developers
were asked to estimate the potential cost savings associated
with enhanced testing tools, and users were asked to estimate
cost savings if the software they purchase had 50 percent
fewer bugs and errors. This improved infrastructure scenario
is estimated to result in a combined annual benefit of $2.10
billion to the two industry groups studied, and $22.2 billion
to the U.S. economy.
Next
Steps
The
path to higher software quality is significantly improved
software testing. Standardized testing tools, suites, scripts,
reference data, reference implementations and metrics that
have undergone a rigorous certification process would have
a large impact on the inadequacies currently plaguing software
markets. For example, the availability of standardized test
data, metrics and automated test suites for performance
testing would make benchmarking tests less costly to perform.
Standardized automated testing scripts, along with standard
metrics, also would provide a more consistent method for
determining when to stop testing.
Electronic
copies of NIST Planning Report 02-3, The Economic Impacts
of Inadequate Infrastructure for Software Testing, can be
obtained from http://www.nist.gov/director/prog-ofc/report02-3.pdf.
(To read these files, you can download Adobe
Acrobat Reader free.) Paper copies can be requested
by e-mail from dherbert@nist.gov
(refer to the title or Planning Report 02-3).
A non-regulatory
agency of the U.S. Department of Commerce's Technology Administration,
NIST develops and promotes measurement, standards, and technology
to enhance productivity, facilitate trade and improve the
quality of life.