Summary
The Air Force has turned to service contracts for the F-15C, F-16, Airborne Warning and Control System, and F-15E, and the Army has done the same for helicopter simulator training at its Flight School XXI. The contractors own, operate, and maintain the simulator hardware and software. The military services rely on industry to capitalize the required up-front investment, with the understanding that the contractors will amortize this investment by selling training services by the hour. GAO was asked to address (1) the factors that led the Air Force and Army to acquire simulator training as a service and whether the decision to use this approach was adequately supported; (2) whether implementation of the approach has resulted in the planned number of simulator training sites being activated; and (3) whether the Air Force and Army are effectively tracking the return on their expenditure of taxpayer dollars. GAO makes recommendations to the Secretary of Defense intended to improve management and oversight of these service contracts to help ensure that the best approach is used to provide the war-fighter with needed training. In written comments on a draft of this report, DOD concurred with all but one of the recommendations, only partially concurring with one pertaining to the Army's simulator utilization rates. GAO continues to believe that the Army needs to track the extent to which it is using simulator availability.
The Air Force and Army turned to service contracts for simulator training primarily because efforts to modernize existing simulator hardware and software had lost out in the competition for procurement funds. As a result, the simulators were becoming increasingly obsolete. Buying training as a service meant that operation and maintenance (O&M) funds could be used instead of procurement funds. Shifting the responsibility for simulator ownership, operation, and maintenance from the government to the contractor was thought to more quickly enable simulator upgrades to match the changing configurations of aircraft. However, the decision to take a service contract approach was not supported by a thorough analysis of the costs and benefits as compared to other alternatives, despite a Department of Defense directive that provided for such an analysis. While Air Force and Army officials told GAO the new simulators are significant improvements over the previous ones, the expected number of Air Force training sites have not been activated. For the Air Force, O&M funds have not been allocated at the anticipated levels, leading to schedule slippages. The F-16 simulator contractor cited the funding problems and subsequent schedule slippages as the basis for notifying the Air Force that its situation under the contract was no longer financially viable. The Air Force is in the process of re-competing the F-16 training contract, which will likely result in a training gap for pilots--possibly over 2 years--and additional costs to the Air Force. The start date of the Army's flight simulator training was rebaselined twice, but Army officials told us that adequate training was in place for the flight school participants. The return on expenditure of taxpayer dollars is not being effectively tracked in three key ways: Air Force utilization of simulator training frequently falls well below the hours for which the government is paying. The Army is not collecting data on utilization rates at all. The government has little insight into what it is paying for during the development period before training is activated, which can take more than a year. While invoices for preparatory efforts reflect only discrete tasks such as training capabilities assessments, the wide range of invoice amounts and GAO's discussions with contractor representatives suggest that the government is actually making milestone payments to the contractors for a portion of their up-front costs to acquire and develop the simulators. Most of the contracts contain award-term provisions, where the contractors can earn an extension of the contract period for good performance. GAO found that the award-term evaluation factors do not always measure key acquisition outcomes such as simulator availability and concurrency with aircraft upgrades.
Recommendations
Our recommendations from this work are listed below with a Contact for more information. Status will change
from "In process" to "Implemented" or "Not implemented" based on our follow up work.
Director:
Team:
Phone:
William T. Woods
Government Accountability Office: Acquisition and Sourcing Management
(202) 512-2649
Recommendations for Executive Action
Recommendation: To help ensure that the best approach is used to provide the war-fighter with needed training, the Secretary of Defense should direct the Secretaries of the Air Force and Army to conduct a thorough analysis of the costs and benefits of using service contracts for simulator training to determine if it is indeed the best approach. The analysis should proactively address potential risks associated with the service contract approach and identify the level of simulator training needed to meet requirements.
Agency Affected: Department of Defense
Status: Implemented
Comments: Because the National Defense Authorization Act for FY07 restricted use of service contracts to acquire military flight simulators, this recommendation can be considered closed. The Air Force will begin purchasing the simulators and have them in place before the current contracts expire (except for the block 50 F-16, which will have a bridge contract).
Recommendation: To help ensure that the required training is provided to pilots, the Secretary of the Air Force should reconcile the funding level needed for simulator training with the requirements identified in the evaluation of costs and benefits of the service contract approach and take steps to allocate funds accordingly.
Agency Affected: Department of Defense: Department of the Air Force
Status: Implemented
Comments: DOD states that Air Force has fully programmed funds to buy simulators.
Recommendation: To help ensure that the incentives motivate contractor performance toward achieving desired training outcomes, the Secretary of Defense should direct the Secretaries of the Air Force and Army to determine whether it is in the government's best interest to retain the award-term incentive under these service contracts.
Agency Affected: Department of Defense
Status: Implemented
Comments: Air Force has determined that award term is not an effective incentive and does not anticipate using them in the future; the F-16 block 50 bridge service contract does not have award term provisions. The recommendation is moot for other contracts because in accordance with the FY07 National Defense Authorization Act, DOD will no longer be using service contracts for flight simulators. The Army has decided that the award term is still in the government's best interest and will reexamine the use of award term incentives for future contracts for Flight Simulator Service contracts.
Recommendation: To help ensure that the incentives motivate contractor performance toward achieving desired training outcomes, the Secretary of Defense should direct the Secretaries of the Air Force and Army to, if the award-term incentive is retained, take appropriate steps to improve the approach by reassessing the areas to be rated and the definitions of performance levels for the various grade categories. For the Air Force, improvements to the approach should include a determination as to whether to continue allowing rollover of award-term points.
Agency Affected: Department of Defense
Status: Implemented
Comments: Air Force has determined that award term is not an effective incentive and does not anticipate using them in the future. Awaiting documentation as to lack of award term incentive in the F-16 block 50 bridge service contract. The recommendation is moot for other contracts because in accordance with the FY07 National Defense Authorization Act, DOD will no longer be using service contracts for flight simulators. The Army states that the award term process allows them to reassess the evaluation criteria in each phase of the contract or when a significant modification takes place. The Army was modifying their award term plan based on the transition to a more steady state mode of operation.
Recommendation: To help ensure greater transparency into what the government is paying for preparatory tasks during the development phase, the Secretary of Defense should direct the Secretaries of the Air Force and Army to reassess the pricing of any up-front payments made to the contractors during the development period on future replacement or restructured contracts.
Agency Affected: Department of Defense
Status: Implemented
Comments: Because of the provisions in FY07 National Defense Authorization Act limiting DOD's use of service contracts for flight simulators, the recommendation is applicable only to the Air Force's block 50 F-16 bridge contract. DOD's response says it will evaluate the proposals to obtain best value and anticipates a minimal percentage of the contract value will be for preparatory costs. Army states they will structure future contracts to insure that all milestone payments are made based on work performed and that any preparatory services are based upon actual work effort.
Recommendation: To help ensure greater transparency into what the government is paying for preparatory tasks during the development phase, the Secretary of Defense should direct the Secretaries of the Air Force and Army to, if retained, take appropriate measures to (1) create an appropriate and transparent contract payment mechanism, separate from the preparatory tasks, if development costs are to be reimbursed; and (2) increase visibility into the percentage of upfront development costs contractors are recouping from these preparatory tasks and development payments.
Agency Affected: Department of Defense
Status: Implemented
Comments: Because the FY07 National Defense Authorization Act restricts use of service contracts for flight simulators, the Air Force is transitioning to a procurement acquisition strategy for future military flight simulators. Therefore, implementation of this recommendation will no longer be required for the long-term simulator replacement acquisitions. With regard to the interim training service contract (block 50 of F-16), the RFP was released 6 months prior to our report, so according to DOD it was impracticable to incorporate the recommendation for restructuring of potential preparatory service rates. Due to the required start dates for the interim F-16 block 50 contract, the Air Force anticipates preparatory tasks will be a minimal percentage of the contract value. The Air Force states it is receiving the maximum visibility into the contractors' pricing structure, as permitted by the Truth in Negotiations Act, for a competitive environment. The Army has stated it will not pay reimburse for development costs and that any preparatory costs will be priced upon estimates of the cost to perform such services.
Recommendation: To help ensure that available simulator training for the warfighter is used in the most effective and efficient manner, the Secretary of Defense should direct the Secretaries of the Air Force and Army to determine whether and how simulator utilization can be increased in order to maximize use of taxpayer dollars.
Agency Affected: Department of Defense
Status: Implemented
Comments: The Air Force stated that it is implementing this recommendation on existing service contracts and will continue until those contracts expire. The Army states they have increased the Army's average ratio of simulator training to total pilot training hours from 19% to 45% with the new Flight School.
Recommendation: To help ensure that available simulator training for the warfighter is used in the most effective and efficient manner, the Secretary of Defense should direct the Secretary of the Army to track and record monthly utilization rates on Flight School XXI contracted simulator training in order to have the data necessary to adjust training requirements and contract provisions, as necessary.
Agency Affected: Department of Defense
Status: In process
Comments: According to the Army response, prior to Flight School XXI training devices, simulator use was only at 19%. The Army established a goal of 40% use of simulator hours since simulator flight hours are less expensive than live flight hours. The number and availability of Flight School simulators has increased the simulator ratio to 45%
Recommendation: To help ensure that available simulator training for the warfighter is used in the most effective and efficient manner, the Secretary of Defense should direct the Secretary of the Air Force to revise Air Force Instruction 36-2248, Operation and Management of Aircrew Training Devices, to ensure that, for the purposes of reporting utilization rates, the usage of individual training simulators is calculated.
Agency Affected: Department of Defense
Status: In process
Comments: The Air Force's initial response to the recommendation stated it would revise AFI 36-2248, Operation and Management of Aircrew Training Devices, to prescribe utilization rates using a single standard and in greater detail to accurately determine the training utilization of individual simulator systems. The updated response states that the Air Force is complying with the recommendation, but has altered its original approach. It is drafting a new instruction that will "clarify utilization tracking and reporting requirements" to comply with this recommendation. As of June 2008, the Air Force is still working on a revision to the instruction.
Recommendation: To help ensure that available simulator training for the warfighter is used in the most effective and efficient manner, the Secretary of Defense should direct the Secretary of the Air Force to ensure that all sites consistently track and report simulator utilization.
Agency Affected: Department of Defense
Status: In process
Comments: The Air Force's initial response to the recommendation stated it would revise AFI 36-2248, Operation and Management of Aircrew Training Devices, to prescribe utilization rates using a single standard and in greater detail to accurately determine the training utilization of individual simulator systems. The updated response states that the AF is complying with the recommendation, but has altered its original approach. It is drafting a new instruction that will "clarify utilization tracking and reporting requirements" to comply with this recommendation.
