This is the accessible text file for GAO report number GAO-08-704R entitled 'Defense Transportation: DOD Should Ensure that the Final Size and Mix of Airlift Force Study Plan Includes Sufficient Detail to Meet the Terms of the Law and Inform Decision Makers' which was released on April 28, 2008. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. April 28, 2008: The Honorable Robert M. Gates: The Secretary of Defense: General Larry D. Welch, USAF (Ret.): President and CEO: Institute for Defense Analyses: Subject: Defense Transportation: DOD Should Ensure that the Final Size and Mix of Airlift Force Study Plan Includes Sufficient Detail to Meet the Terms of the Law and Inform Decision Makers: Global mobility[Footnote 1] is a key component of U.S. national security. Since the end of the Cold War, senior decision makers have relied upon Department of Defense (DOD) mobility studies to provide insights they need to build and maintain the right mix of mobility capabilities. The most recent study, the Mobility Capabilities Study,[Footnote 2] identified the mobility support needed for the full range of strategic operations in the context of the September 11, 2001 attacks, the global war on terror, and DOD's evolving global defense posture, all in support of the National Military Strategy. According to DOD officials, the department plans to issue the next mobility study-- the Mobility Capabilities Requirements Study--in the spring of 2009. The 2005 mobility study also assessed requirements for two overlapping war fights, DOD support to homeland defense, civil support, lesser contingency operations, sustainment of forward-deployed forces, and national strategic missions. In accomplishing these missions, DOD depends on its airlift force. The National Defense Authorization Act for Fiscal Year 2008[Footnote 3] (hereafter referred to as the Act) mandated a requirements-based study on alternatives for the proper size and mix of the airlift force to meet the needs of the National Military Strategy to be done by a Federally Funded Research and Development Center (FFRDC).[Footnote 4] The Act specifically defined what the study plan should include and set time frames for the completion of various events. The FFRDC was to submit a study plan to the appropriate congressional committees, the Secretary of Defense, and the Comptroller General 60 days after the enactment of the Act. The Act required us to review the study plan to determine if it is complete and objective and whether it has any flaws or weaknesses in scope or methodology and report to the Secretary of Defense and the FFRDC within 30 days. It also required us to include in the report any recommendations that the Comptroller General considers appropriate for improvements to the study plan. DOD selected the Institute for Defense Analyses (IDA) to accomplish the study and signed a task order with IDA outlining the study framework. On March 28, 2008, IDA delivered the draft Size and Mix of Airlift Force Study Plan to DOD, congressional committees, and us. The draft study plan comprises 34 pages of bulleted information, graphs, and diagrams. The seven major sections are introduction, background, scope, objective, study management, staged approach, and schedule. The single objective of the study is to address the numerous airlift issues identified in the Act and to report to the Secretary of Defense and to the Congress by January 10, 2009. We assessed the draft Size and Mix of Airlift Force Study Plan that IDA delivered on March 28, 2008, for completeness, but we were unable to evaluate objectivity or identify flaws or weaknesses in scope or methodology.[Footnote 5] We also commented on another ongoing airlift-related study, the Mobility Capabilities Requirements Study, because it is related to the scope of the draft Size and Mix of Airlift Force Study Plan. DOD uses studies to inform decision making, and study plans are important because they define what will be accomplished, what methodologies and assumptions will be used in a study, and how it will be done. DOD has used detailed study plans to address large and complex issues, including analyses of alternatives supporting acquisitions and execution plans for force management initiatives. Although DOD has not published departmentwide guidance, the Army, Air Force, and Navy have publications that describe study plans and/or note the use and importance of studies in a variety of efforts.[Footnote 6] Service publications describe a number of detailed elements that could be considered for use in study plans and that characterize successful studies. Our prior work shows that a detailed study plan is a critical part of a well-executed study, but may not guarantee a fully successful study. To conduct our evaluation, we reviewed IDA's submission to determine if it fulfilled the study plan elements in the Act. We also assessed the completeness of the study plan by reviewing the plan and comparing it with study plan descriptions in Army, Air Force, and Navy publications that identified the key elements that could be included in a study plan. We reviewed prior GAO work that discussed study plans. To obtain information on the IDA draft study plan, we interviewed officials from IDA; officials from the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics; and U.S. Transportation Command. We conducted this performance audit from March to April 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Results In Brief: The draft Size and Mix of Airlift Force Study Plan does not meet the terms of the Act and lacks sufficient detail for assessment. We are unable to fully assess the draft Size and Mix of Airlift Force Study Plan as required in section 1046 of the Act because the plan does not contain sufficient detail for us to evaluate its objectivity and its scope and methodology. Because the draft study plan did not address all of the specified elements in the Act, it is not complete. The draft plan did not include specific and explicit references that can be traced directly to the Act, such as the assumptions to be included in the study plan and assessments to be accomplished. This absence of detail also precludes us from evaluating the scope and methodology. Moreover, the plan lacked key details expected in such plans, such as assumptions and measures of effectiveness. The lack of details discussed above precludes us from making any recommendations concerning improvements to the study plan. DOD officials stated that because DOD selected IDA and issued a task order for the study only shortly before the mandated deadline, sufficient time was not available to produce a more detailed study plan. Nevertheless, DOD is responsible for ensuring the statutorily required elements of the study plan are fulfilled. IDA officials told us that IDA plans to submit to the Secretary of Defense in June 2008 a final Size and Mix of Airlift Force Study Plan that will be more robust. In addition to the independent draft Size and Mix of Airlift Force Study Plan, we note that DOD is conducting another study that may also inform decision makers on airlift issues. Accordingly, we are recommending that the Secretary of Defense direct the Under Secretary of Defense for Acquisition, Technology and Logistics to ensure that the final Size and Mix of Airlift Force Study Plan include (1) sufficient detail to address, at a minimum, elements mandated in section 1046 of the National Defense Authorization Act for Fiscal Year 2008; and (2) sufficient detail to inform decision makers on airlift issues. In oral comments on a draft of this report, DOD concurred with our recommendation. DOD's comments are discussed in more detail at the end of this report. DOD also provided technical comments and we have incorporated them where appropriate. Background: In previous GAO work, we noted the importance of study plans and usage of study plans as a best practice.[Footnote 7] Specifically, we found that governmental agencies and the private sector rely on detailed study plans, or data collection and analysis plans, to guide the development of studies and the collection and analysis of data. Additionally, we stated that a study plan provides a feedback loop that links the outcomes of the study and subsequent analysis to the original goals and objectives of the study. We also found that particularly large and complex issues may benefit from a study plan. GAO also identified a best practice for a study plan process that mirrors information found in service publications concerning study plans. Service publications describe the role of studies in complex decision making in areas such as acquisitions, comparing alternatives, and evaluations of force capabilities. Service publications also describe elements, such as assumptions and measures of effectiveness (MOEs), that may be considered for inclusion in a study plan. The Navy publication that discusses studies describes determining assumptions, statements related to the study that are taken as true in the absence of facts, as a key step in study plan development.[Footnote 8] The Army publication that discusses preparation of a study also lists assumptions as a part of a study plan.[Footnote 9] The Air Force analysis handbook contains a study plan outline that includes assumptions as well.[Footnote 10] According to these service publications, MOEs are the level of success to be achieved or how well tasks are performed. All of the service publications refer to MOEs as part of a study plan. In the Army and Navy publications, selection of the MOEs is described as perhaps the most crucial part of any analysis. The Draft Size and Mix of Airlift Force Study Plan Does Not Meet the Terms of the Act and Cannot Be Fully Assessed: We are unable to fully assess the draft Size and Mix of Airlift Force Study Plan as required in section 1046 of the Act because it does not contain sufficient detail for us to evaluate the objectivity or to evaluate the scope and methodology. Since the draft study plan did not address all of the specified elements in the Act, it is not complete. This absence of detail also precludes us from evaluating the scope and methodology. In the absence of a complete study plan, the final study may not sufficiently inform decision makers concerning the alternatives for the size and mix of the airlift force to meet requirements of the National Military Strategy. The draft study plan is not complete because it did not address all of the required elements in the Act. While IDA officials described the draft as generally addressing the three main elements cited in the Act, we found that it lacks specific and explicit references that can be traced directly to the Act. Specifically, the Act identified 11 assumptions, at a minimum, to be included in the study plan.[Footnote 11] For example, one required assumption involved the new capability in airlift to be provided by the KC(X) tanker aircraft. The draft study plan included a 5-bullet slide concerning the assessment of the KC(X) as an airlifter, but did not include the required assumptions. Additionally, the Act required the study plan to include assumptions concerning airlift mobility requirements in support of homeland defense and national emergencies. However, the plan only included a number of bulleted statements about the homeland defense and national emergencies missions and contained no assumptions. Without the required assumptions, the draft Size and Mix of Airlift Force Study Plan does not satisfy the Act and is not complete. The draft study plan also does not contain key details sufficient for us to evaluate the scope and methodology. Service publications reflect the importance of key steps such as identification of MOEs. According to the service publications, perhaps the most crucial part of any analysis is the selection of appropriate measures of effectiveness, which are central to evaluating alternatives. They enable decision makers to compare the results in a study.[Footnote 12] For example, MOEs could allow decision makers to compare the results of two analyses that measure different airlift force mixes. In the draft study plan, mixes of two aircraft (C27 and C-130) are to be compared for their fleet effectiveness. In such an example, the time required for an aircraft fleet to accomplish a mission, such as moving cargo or people, can be measured in terms of time required, tons moved, or miles flown. The task order for the draft Size and Mix of Airlift Force Study Plan specified the use of MOEs. The draft Size and Mix of Airlift Force Study Plan includes a slide concerning effectiveness analyses but, for the purposes of our analysis, it does not detail the MOEs or how they are to be used. Without critical details such as MOEs, the final study may not sufficiently inform decision makers concerning airlift issues. In our discussion with IDA officials concerning the draft study plan, they acknowledged the importance of the use of assumptions and MOEs in study plans. DOD officials stated that DOD selected IDA only shortly before the mandated deadline for a draft study plan and added that sufficient time was not available to produce a more detailed draft study plan. The plan states that its final stage (phase) will ensure all issues raised in the Act are addressed. In our discussion with IDA officials, they explained that the March 2008 version of the Size and Mix of Airlift Force Study Plan is a draft and that the final study plan, scheduled for completion in June 2008, will be more robust and detailed. We note that while DOD is using IDA to perform the study plan pursuant to a task order, DOD is responsible for ensuring that the statutorily required elements of the study plan are fulfilled. We also note that, pursuant to the Act, we were to provide any recommendations that the Comptroller General considered appropriate for improvement to the study plan. The lack of details discussed above precludes us from making any recommendations concerning improvements to the study plan. In the Navy study guide publication, one of the first steps in initiating a study is identifying potential uses for anticipated study results. While our scope only included the draft Size and Mix of Airlift Force Study Plan, we note that DOD plans to publish results from both the Size and Mix of Airlift Force Study and the Mobility Capabilities Requirements Study in 2009 as decision makers consider airlift requirements. We believe that it is possible that decision makers may compare the findings and assumptions of the Size and Mix of Airlift Force Study with the airlift portion of the Mobility Capabilities Requirements Study. If the Mobility Capabilities Requirements Study does not consider the objectives and assumptions of the Size and Mix of Airlift Force Study, it is unclear as to whether the results of the two studies can be compared and therefore the study results may not be fully useful to decision makers. Conclusion: Well-executed airlift and mobility studies that inform decision making are important as DOD continues its efforts to sustain, modernize, and recapitalize its airlift programs. Detailed study plans are a critical part of well-executed studies and should include key details such as assumptions and MOEs. Although the draft Size and Mix of Airlift Force Study Plan does not fulfill the requirements of the Act, the final version may include corrections and refinements. Recommendations for Executive Action: We recommend that the Secretary of Defense direct the Under Secretary of Defense for Acquisition, Technology and Logistics to ensure that the final Size and Mix of Airlift Force Study Plan includes sufficient detail to address, at a minimum, elements mandated in section 1046 of the National Defense Authorization Act for Fiscal Year 2008 and ensure that the study plan includes sufficient detail to inform decision makers on airlift issues. Agency Comments and Our Evaluation: In oral comments on a draft of this report, DOD concurred with our recommendation that DOD ensure that the final study plan include sufficient detail to address, at a minimum, elements mandated in law and sufficient detail to inform decision makers on airlift issues. As part of the comments, an official from the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics reiterated that IDA was not given adequate time to create a detailed study plan in compliance with established standards. DOD stated the intention to continue monitoring IDA's work on the final study plan, adding that DOD fully expects the final study plan to comply with established DOD standards regarding studies. DOD also provided technical comments and we have incorporated them where appropriate. We are sending copies of this report to the Senate and House Appropriations and Armed Services Committees and other interested congressional committees; the Under Secretary of Defense for Acquisition, Technology and Logistics; the Commander of U.S. Transportation Command; and the Director, Office of Management and Budget. In addition, this report will be available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you have any questions regarding this report, please contact me at (202) 512-8365 or solisw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made contributions to this report include Ann Borseth, Ron La Due Lake, Charles Perdue, Karen Thornton, Karen Werner, and Steve Woods. Signed by: William M. Solis, Director: Defense Capabilities and Management: [End of section] Related GAO Products: Defense Acquisition: KC-135 Recapitalization Analysis of Alternatives Does Not Inform Decision Makers Regarding Cost, Effectiveness, and Suitability. GAO-08-69CR. Washington, D.C.: January 8, 2008. Defense Acquisitions: Air Force Decision to Include a Passenger and Cargo Capability in Its Replacement Refueling Aircraft Was Made without Required Analyses. GAO-07-367R. Washington, D.C.: March 6, 2007. Defense Transportation: Study Limitations Raise Questions about the Adequacy and Completeness of the Mobility Capabilities Study and Report. GAO-06-938. Washington, D.C.: September 20, 2006. Military Readiness: Navy's Fleet Response Plan Would Benefit from a Comprehensive Management Approach and Rigorous Testing. GAO-06-84. Washington, D.C.: November 22, 2005. [End of section] Footnotes: [1] The Air Force defines global mobility as the ability to rapidly establish an air-bridge and move military capability in support of operations anywhere in the world under any conditions. [2] The intent of the December 2005 Mobility Capabilities Study (MCS) was to identify and quantify the mobility capabilities needed to support U.S. strategic objectives into the next decade. The MCS determined that the projected mobility capabilities are adequate to achieve U.S. objectives with an acceptable level of risk during the period from fiscal years 2007 through 2013; that is, the existing U.S. inventory of aircraft, ships, prepositioned assets, and other capabilities were concluded to be sufficient, in conjunction with host nation support. The MCS emphasized that continued investment in the mobility system, in line with current departmental priorities and planned spending, is required to maintain these capabilities in the future. This included, for example, fully funding Army prepositioned assets as planned and completing a planned reengineering of the C-5 aircraft. The MCS report also made recommendations to conduct further studies, develop plans and strategies, and improve data collection and mobility models. [3] Pub. L. No. 110-181, �46 (2008). [4] The Federal Acquisition Regulation sets forth federal policy regarding the establishment and use of Federally Funded Research and Development Centers. An FFRDC meets some special long-term research or development need which cannot be met as effectively by existing in- house or contractor resources. FFRDCs enable agencies to use private sector resources to accomplish tasks that are integral to the mission and operation of the sponsoring agency. The FFRDC is required to conduct its business in a manner befitting its special relationship with the government, to operate in the public interest with objectivity and independence, to be free from organizational conflicts of interest, and to provide full disclosure of its affairs to the sponsoring agency. FFRDCs are operated, managed, and/or administered by either a university or consortium of universities, another not-for-profit or nonprofit organization, or an industrial firm, as an autonomous organization or as an identifiable separate operating unit of a parent organization. [5] See GAO, Government Auditing Standards: July 2007 Revision, GAO-07- 731G (Washington, D.C.: July 2007). Government auditing standards define scope as the boundaries of a study that are directly tied to the study objectives. The scope defines the subject matter that the executors will assess. Government auditing standards define methodology as describing the nature and extent of procedures for gathering and analyzing evidence to address study objectives. Methodology is to obtain sufficient, appropriate evidence to address objectives and provide reasonable assurance that the evidence is sufficient and appropriate to support study findings and conclusions. Methodology includes both the nature and extent of procedures used to address objectives. [6] Studies that might be guided by a study plan include but are not limited to cost, benefit, or effectiveness analysis of concepts, plans, training, tactics, forces, systems, policies, personnel management methods, and policies or programs; cost and operational effectiveness analyses (COEA); evaluations of force capabilities, organizational structure, administrative policies, procedures, methods, systems, and distribution of functions; research and development of data bases, models, and methodologies for accomplishing specific studies and analyses; analyses of materiel, personnel, logistics, and management systems; and studies to establish materiel requirements. [7] See GAO, Military Readiness: Navy's Fleet Response Plan Would Benefit from a Comprehensive Management Approach and Rigorous Testing, GAO-06-84 (Washington, D.C.: Nov. 22, 2005). [8] Navy Warfare Development Command, Study Planning and Conduct Guide, Mr. Richard C. Rigazio, Operations Research Analyst (August 2007). [9] Department of the Army Pamphlet 5-5, Guidance for Army Study Sponsors, Sponsor's Study Directors, Study Advisory Groups, and Contracting Officer Representatives, Headquarters (Washington, D.C. Nov 1, 1996). [10] Office of Aerospace Studies, Analysis Handbook, A Guide for Performing Analysis Studies: For Analysis of Alternatives or Functional Solution Analyses (July 2004). [11] Pub. L. No. 110-181, �46 (d)(1) (2008). [12] According to the Air Force analysis handbook, MOEs are important to a warfighter because they express both system worth (ability to contribute to a warfighter's immediate goal) and military worth (ability to contribute to high-level goals of winning the war). GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: