Summary

n 2008, the cumulative cost growth in the Department of Defense's portfolio of 96 major defense acquisition programs was $296 billion and the average delay in delivering promised capabilities to the warfighter was 22 months. These poor outcomes mean that other critical defense and national priorities may go unfunded and that warfighters may go without the equipment they need to counter the changing threats that they face. GAO has examined extensively the issues in DOD's weapon system programs from the perspective of best practices in product development, and believes that significant improvements are possible. Because the ability to measure knowledge, processes, and outcomes is critical to achieving these improvements, GAO has used metrics to review the management and health of these programs from within the framework of best practices. This testimony discusses: 1) "knowledge metrics," used to determine how well programs manage technology, design, and manufacturing risks; 2) outcome metrics--concerning cost, schedule, and capability--that serve as "health indicators" of how well programs are being executed in terms of predicted outcomes; and 3) the prerequisites that GAO believes must be met in order for a program's plans and goals to be realistic.

GAO employs a set of knowledge metrics to determine whether programs have attained the right knowledge at critical points over the course of a weapon system acquisition, and facilitate the identification of potential problems that could lead to cost, schedule, or performance shortfalls. In essence, knowledge supplants risk over time. Key knowledge points and metrics include 1) achieving a high level of technology maturity at the start of program development, 2) reaching design stability at the system-level critical design review, and 3) demonstrating that critical manufacturing processes are in control before starting production. By applying these metrics to selected programs in DOD's 2008 portfolio of major defense acquisitions, GAO found that most programs have started system development without mature technologies and moved into system demonstration with low levels of design stability. GAO has determined that programs with immature technologies and unstable designs have experienced significant cost and schedule growth. Program outcome metrics--quantitative measures of cost, schedule, and performance over time--provide useful indicators of the health of acquisition programs and whether they are meeting their intended goals. When assessed regularly for changes and the reasons that cause changes, these indicators can be valuable tools for improving insight into and oversight of individual programs as well as DOD's total portfolio of major defense acquisitions. The collective performance of the programs in DOD's portfolio is a key indicator of how well the acquisition system generates the return on investment that it promises to the warfighter, Congress and taxpayers. GAO recently reported that outcome metrics for DOD's 2008 major defense acquisition portfolio show worsening performance when compared to the department's 2003 portfolio. For example, total acquisition costs for programs in the 2008 portfolio increased 25 percent from first estimates compared to a 19-percent increase for programs in the 2003 portfolio. DOD is working with GAO and the Office of Management and Budget to develop a comprehensive set of outcome metrics to better assess its portfolio of programs. While knowledge and outcome metrics provide valuable information about the potential problems and health of programs, they are of limited value if DOD does not do a better job ensuring acquisitions begin with realistic plans and baselines prior to development start. GAO believes there is a clear set of prerequisites that must be met by each program's acquisition strategy before a measurement of the program's health will be of real value. These prerequisites include: 1) establishing an evolutionary, knowledge-based business case for each acquisition; 2) separating technology development from product development; 3) limiting time and requirements for product development to manageable levels; 4) employing systems engineering early on in the process to arrive at realistic cost and schedule estimates; 5) committing to fully funding a program once it is approved; and 6) setting priorities from the top to ensure that candidate programs are truly needed and have a solid plan for delivery.