Detailed Information on the
Air Combat Program Assessment

The DoD evaluate the specific requirements for air combat capabilities in order to support the 2006 Quadrennial Defense Report goals and Secretary of Defense Priorities.

Proposes that DoD refine methods for assessing the efficiency and effectiveness (or otherwise) of the overall air combat program in light of the needs of the 2001 QDR defense strategy and the global war on terrorism.

Measure: Number of performance objectives for individual weapons systems unmet

Explanation:

Measure: Percentage growth in program costs

Explanation:For those programs reviewed through Selected Acquisition Reports, cost growth ranged from +4.3% (F/A-18 E/F) down to -.9% (EA-18G).

Measure: When normalized to constant year dollars, there is no change from the previous year's Average Procurement Unit Cost (APUC) for the F-22, F/A-18E/F and F-35 fighter aircraft programs.

Explanation:The APUC is the average cost for procuring a specific major acquisition item and is reported quarterly for all major acquisition programs in the Department of Defense Selected Acquisition Report. This measure's goal is for no annual increase in the cost of producing the following fighter aircraft programs: F-22, F/A-18E/F and F-35. This goal is achieved if the APUC for the end of the fiscal year does not change from year to year when adjusted for inflation.

Is the program purpose clear?

Explanation: Air combat capability is seen as a vital component of the US' ability to project force and respond to the full spectrum of international crises .

Evidence: The annual performance 2002 Government Performance and Results Act (GPRA) goal for DoD is to maintain trained and ready forces with the ability to respond to the full spectrum of crises. This capability is also one of the Department's core competencies.

Does the program address a specific interest, problem or need?

Explanation: Recent crises have shown the vital need for air combat forces, both to protect American forces and to defeat enemy forces.

Evidence: The role of air forces in recent crises has been documented in numerous Congressional testimonies and in the 2002 Air Force Posture Statement. The need for this capability has also been stated in the Secretary of Defense's Annual Report to the President and the Congress, 2002.

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: The individual acquisition programs within the Air Combat program all contribute to US air combat capabilities.

Evidence: Over $15.7 billion is devoted to investment in tactical combat aircraft in FY2003. Specific program detail can be obtained from budget documents.

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: The program is not redundant or duplicative of other Federal or non-federal efforts.

Evidence: There is no evidence of duplicative efforts.

Is the program optimally designed to address the interest, problem or need?

Explanation: There are many different elements in this Air Combat program and many possible combinations of these elements could provide air combat capabilities to the United States. Although there has not been a recent, careful overall analysis of capabilities trade-offs that could inform on the optimal mix of these elements, there is no evidence that the planned future mix will not be optimal.

Evidence: In response to the Secretary of Defense's FY2004 Planning Guidance, the department conducted a combat air forces study to evaluate overall air combat capabilities. This process determined the appropriate aircraft mix and quantities of DoD combat aircraft for the planned 2004 program. However, due to other factors, such as industrial base capacity and budgetary pressures, DoD is not always able to procure aircraft in the quantities required. The Department, in developing its Future Years Defense Program, assesses requirements in all mission areas and allocates resources in order to best ensure accomplishment of its National Security missions.

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The overall goal of the air combat program is to be able to achieve air dominance over any expected threat aircraft. The program strives to achieve this goal by acquiring new and more capable aircraft that meet strict performance criteria. The DoD acquisition process is designed to ensure that all new weapon systems adhere to performance goals established in the Operational Requirements Document, Acquisition strategy, and other relevant documents.

Evidence: Selected Acquisition Reports for weapon systems identify specific, ambitious performance goals that must be accomplished in order for the system to address the threat.

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: DoD establishes performance goals for each aircraft program. These include cost, schedule and capability goals that DoD tracks on an annual basis.

Evidence: The President's Budget documentation provides details on program estimated costs, contract award dates, delivery schedules, construction dates. Comparisons of one fiscal year budget to the next reveals variances. Selected Acquisition Reports, which are provided by DoD to Congress at least annually, show deviations from the program baselines.

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation:

Evidence:

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation:

Evidence:

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: By statute and regulation, DoD conducts evaluations of programs on a regular basis to monitor program progress against cost, schedule, and performance criteria.

Evidence: Prior to establishing a new program, or approving the continuation of an aircraft program, DoD develops an acquisition program baseline (APB) that sets out the cost, schedule and performance goals for the program. This process requires an extensive analysis of each program and its potential contribution to the overall mission. Two examples of reports that include this information are the Annual Selected Acquisition Reports, which are sent to Congress, and the quarterly Defense Acquisition Executive Summary reports.

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: The DoD program planning and budgeting system requires budgeting based on a determination of the resources needed to achieve the acquisition goals of the program elements within the overall air combat program.

Evidence: DoD budget documents lay out the required funding and programmatic objectives to be achieved for each year.

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: The overall goal of the air combat program is to be able to achieve air dominance over any expected threat aircraft. The program strives to achieve this goal through the construction of new and more capable air combat aircraft.

Evidence: DoD's FY2003 budget documents for the F-22, Joint Strike Fighter and F/A-18E/F aircraft programs describe how each will meet its goals and thereby meet the overall air combat capability program objective.

Are acquisition program plans adjusted in response to performance data and changing conditions?

Explanation: DoD conducts regular reviews of acquisition programs when changes to programs can be,and sometimes are, made. For example, based on a review of problems in the Army's Comanche program, DoD restructured the program, added more test aircraft and changed the total number of aircraft that would be procured in the program.

Evidence: DoD Defense Acquisition Directives describe milestones when reviews should be undertaken, and goals that should be achieved.

Has the agency/program conducted a recent, meaningful, credible analysis of alternatives that includes trade-offs between cost, schedule and performance goals?

Explanation: DoD conducts an analysis of alternatives at the start of an acquisition program or when there is a fundamental change to a program. In addition, in response to the Secretary of Defense's Planning Guidance for FY 2004, the department conducted a combat air forces study to evaluate overall air combat capabilities. This process was used by both requirements and acquisition to determine the appropriate aircraft mix and quantities of DoD combat aircraft for the FY2004 program.

Evidence: An Analysis of Alternatives was conducted for the Joint Strike Fighter October 2001, and for the Comanche April 2000, recently updated in October 2002. In addition, the combat air forces study was completed in August 2002.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: DoD regularly collects performance information on the elements of the air combat program and uses the data to inform senior leadership and to make program decisions. Program progress reflecting performance in cost, schedule, and attainment of system performance parameters by both the program office and the contractor(s) is regularly collected and reported.

Evidence: Several documents are produced by DoD giving performance information on combat aircraft investment programs including the annual Selected Acquisition Reports, Analyses of Alternatives and Defense Acquisition Executive Summaries contain performance information on air combat programs.

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: Acquisition directives assign accountability to program managers for cost, schedule, and performance. Contractual requirements are stated in performance terms. An example of the seriousness of accountability is the recent change of program managers for a major aircraft program because of cost overruns.

Evidence: An Oct 30, 2002 memorandum from the Deputy Secretary of Defense entitled "The Defense Acquisition System" provides guidance on program manager (PM) accountability. PMs receive charters upon assignment, giving authority, responsibility, and accountability.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: DoD budget reviews include a review of obligation and expediture data to ensure that funds are obligated in accordance with DoD standards.

Evidence: Expenditure data are reported monthly on standard form 1002 by the Defense Finance and Accounting Service.

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: While there are no efficiency targets, before programs are initiated, analyses of alternatives are conducted to ensure selection of the best solution for the requirement. Program performance plans contain performance metrics that focus on cost schedule, and performance. Application of Earned Value Management also helps to ensure efficiency and effectiveness in program execution. Selected Acquisition Reports monitor total and unit costs.

Evidence: Periodic management reports, such as the Selected Acquisition Reports, Defense Acquisition Executive Summary, and program performance plans are developed and evaluated periodically to ensure adherence to cost, schedule, and performance. When deviations occur, DoD management makes adjustments to ensure the most effective use of resources.

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: The DoD budget documents include estimates for all programatic costs.

Evidence: DoD budget documents lay out the costs for each program on an annual basis. Moreover, the Selected Acquisition Reports for each major acquisition program show total cost estimates for the program.

Does the program use strong financial management practices?

Explanation: DoD's financial management system is being improved, but slowly. DoD has no audit reports showing that the air combat program is free of internal control weaknesses have been provided. Nor are there any reports showing showing a lack of internal controls.

Evidence:

Has the program taken meaningful steps to address its management deficiencies?

Explanation: DoD uses an earned value management system to track program performance. For example, for the F-22 program the summary report shows developmental testing as an area of concern. Changes have now been made in management of the testing program, specifically giving the contractor more responsibility for conducting aerodynamic testing.

Evidence: DoD management receives quarterly reports showing how air combat programs are progressing.

Does the program define the required quality, capability, and performance objectives for deliverables?

Explanation: All elements of the air combat program have well defined key performance parameters.

Evidence: Selected acquisition reports and program requirements documents lay out the performance objectives of each program in detail.

Has the program established appropriate, credible, cost and schedule goals?

Explanation: A major challenage for DoD in managing all of its major acquisition programs is ensuring that programs meet cost and schedule goals. DoD experiences cost growth and schedule delays on many programs, including combat aircraft, due to a variety of factors including unanticipated technical problems. GAO has placed management of the acquisition system on its high risk list.

Evidence: For example, the Army recently had to add $3.5 billion over the life of the Comanche program in order to meet revised program goals and objectives. Also, the F-22 program has experienced cost increases and is now "building to budget", that is, building as many, or as few, aircraft as can be built within its budget instead of setting a target cost for each aircraft and requesting the appropriate budget for the number of aircraft that it wishes to build.

Has the program conducted a recent, credible, cost-benefit analysis that shows a net benefit?

Explanation: A cost benefit analysis is not normally done for DoD capital programs.

Evidence:

Does the program have a comprehensive strategy for risk management that appropriately shares risk between the government and contractor?

Explanation: DoD uses an earned value management system to track program performance and, in addition, cost plus type contracts are used by DoD for highly risky programs. Each aircraft acquisition element within the air combat program normally also includes a risk mitigation plan. For example, the Joint Strike Fighter program has a risk management plan specifically identifying areas of concern and the mitigation steps required to reduce risk.

Evidence: Quarterly management reports and specific contract language describe the extent of risk and how it is shared between the government and contractor.

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: The overall, primary goal is to maintain theater air dominance. Current fielded aircraft are superior to current threats. To maintain this edge, newer aircraft must be developed and fielded. The F/A-18E/F is in the early stages of fielding; other aircraft programs are still in various research and development stages, with fielding projected in the next few years. Progress towards the long term goals is adequate.

Evidence: Selected Acquisition Reports, various Service fact sheets on current and future combat aircraft capabilities, describe program goals.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Annual performance goals relate to cost, schedule, and performance parameters. These goals must be met in order for the programs to efficiently attain the long-term outcome goals.

Evidence: Selected Acquisition Reports, President's Budget Procurement and research and development justification materials lay out program goals.

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: Various elements of the F/A-18E/F Airframe contract are performing under cost.

Evidence: These data are included in the Selected Acquisition Reports.

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation:

Evidence:

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation:

Evidence:

Were program goals achieved within budgeted costs and established schedules?

Explanation: Several of the elements within the air combat program have experienced schedule delays and cost overruns. Others have had changes made to their targets, but some are progressing well.

Evidence: The FY2003 budget documents, and selected acquisition reports, lay out the cost and schedule goals and aircraft delivery numbers for the air combat programs.