Summary

We are reviewing the processes the Department of Defense (DOD) is using to conduct its Mobility Capabilities Study (MCS). The MCS is to address changes in DOD's transportation force structure and mobility requirements due to changes in threats and certain national security and military strategies. The study results may underpin decisions on future strategic airlift, aerial refueling aircraft, and sealift procurements. The study relies on the use of various models and data inputs to develop and evaluate transportation alternatives, including variations in alternative transportation modes (air, land, sea) and sources (military, civilian, foreign), as well as factors that affect transportation mode and source decisions. The Senate Armed Services Committee directed us to monitor the conduct of the MCS and report on the adequacy and completeness of the report no later than 30 days after DOD completes the study. DOD plans to issue the MCS report during 2005. This letter is intended to bring to the Secretary of Defense's attention preliminary observations on certain aspects of the MCS methodology to permit you to ensure the credibility of this and future studies. In our letter, we address the adequacy of the department's verification, validation, and accreditation (VV&A) of the models and simulations being used to conduct the MCS--that is, the process the MCS team is using to identify the models' capabilities, limitations, and performance relative to the real world events they simulate. We will continue to monitor the MCS and will report on the adequacy and completeness of the methodology after DOD issues its report.

We are unable to assess the adequacy of the process DOD used to verify, validate, and accredit the models used to conduct the MCS. Although officials in the Office of Program Analysis and Evaluation stated that they have performed an equivalent VV&A process for the models used in the MCS, there is little documentation available to describe the equivalent process that was used. An adequate evaluation of this self-described equivalent process cannot be conducted due to this absence of documentation, which is compounded because DOD currently does not plan to disclose how it conducted its equivalent VV&A process in its MCS report. This could negatively impact the credibility of the MCS report. DOD guidance requires that models and data go through a VV&A process, but officials in the Office of Program Analysis and Evaluation believe that this guidance is not relevant for models that have been used for many years, called legacy models, because, in their view, the models and data have already undergone an equivalent VV&A process consisting of actual use, although the guidance does not identify actual use as an appropriate equivalent process. Moreover, DOD was conducting VV&A on one legacy model being used in the MCS, raising questions about the need for such actions given the department's statements that it is unnecessary.

Recommendations

Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Implemented" or "Not implemented" based on our follow up work.

Recommendations for Executive Action