This is the accessible text file for GAO report number GAO-08-822 
entitled 'DOD Business Systems Modernization: Key Marine Corps System 
Acquisition Needs to Be Better Justified, Defined, and Managed' which 
was released on July 28, 2008.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Subcommittee on Readiness and Management Support, 
Committee on Armed Services, U.S. Senate: 

United States Government Accountability Office: 
GAO: 

July 2008: 

DOD Business Systems Modernization: 

Key Marine Corps System Acquisition Needs to Be Better Justified, 
Defined, and Managed: 

GAO-08-822: 

GAO Highlights: 

Highlights of GAO-08-822, a report to the Subcommittee on Readiness and 
Management Support, Committee on Armed Services, U.S. Senate. 

Why GAO Did This Study: 

GAO has designated the Department of Defense’s (DOD) business systems 
modernization as a high-risk program because, among other things, it 
has been challenged in implementing key information technology (IT) 
management controls on its thousands of business systems. The Global 
Combat Support System-Marine Corps program is one such system. 
Initiated in 2003, the program is to modernize the Marine Corps 
logistics systems. The first increment is to cost about $442 million 
and be deployed in fiscal year 2010. GAO was asked to determine whether 
the Department of the Navy is effectively implementing IT management 
controls on this program. To accomplish this, GAO analyzed the 
program’s implementation of several key IT management disciplines, 
including economic justification, earned value management, risk 
management, and system quality measurement. 

What GAO Found: 

DOD has not effectively implemented key IT management controls provided 
for in DOD and related acquisition guidance on this program. If 
implemented effectively, these and other IT management disciplines 
increase the likelihood that a given system investment will produce the 
right solution to fill a mission need and that this system solution 
will be acquired and deployed in a manner that maximizes the chances of 
delivering promised system capabilities and benefits on time and within 
budget. Neither of these outcomes is being fully realized on this 
program, as evidenced by the fact that its first increment has already 
slipped more than 3 years and is expected to cost about $193 million 
more than envisioned. These slippages and cost overruns can be 
attributed in part to the management control weaknesses discussed in 
this report and summarized below. Moreover, additional slippages and 
overruns are likely if these and other IT management weaknesses are not 
addressed. 

* Investment in the system has not been economically justified on the 
basis of reliable estimates of both benefits and costs. Specifically, 
while projected benefits were risk-adjusted to compensate for limited 
data and questionable assumptions, the cost side of the benefit/cost 
equation is not sufficiently reliable because it was not derived in 
accordance with key cost estimating practices. In particular, it was 
not based on historical data from similar programs and it did not 
account for schedule risks, both of which are needed for the estimate 
to be considered accurate and credible. 

* Earned value management that the program uses to measure progress has 
not been adequately implemented. Specifically, the schedule baseline 
against which the program gauges progress is not based on key 
estimating practices provided for in federal guidance, such as 
assessing schedule risks and allocating schedule reserves to address 
these risks. As a result, program progress cannot be adequately 
measured, and likely program completion dates cannot be projected based 
on actual work performed. 

* Some significant program risks have not been adequately managed. 
While a well-defined risk management plan and supporting process have 
been put in place, the process has not always been followed. 
Specifically, mitigation steps for significant risks either have not 
been implemented or proved ineffective, allowing the risks to become 
actual problems. 

* The data needed to produce key indicators of system quality, such as 
trends in the volume of significant and unresolved problems and change 
requests, are not being collected. Without such data, it is unclear 
whether the system is becoming more or less mature and stable. 

The reasons for these weaknesses range from limitations of DOD guidance 
and tools, to not collecting relevant data. Until they are addressed, 
DOD is at risk of delivering a solution that does not cost-effectively 
support mission operations and falls short of cost, schedule, and 
capability expectations. 

What GAO Recommends: 

GAO is making recommendations to the Secretary of Defense aimed at 
limiting investment in the program and addressing its cost and schedule 
estimating, risk management, and system quality measurement weaknesses. 
DOD agreed in full or in part with GAO’s recommendations and described 
ongoing and planned actions intended to address the recommendations. 

To view the full product, including the scope and methodology, click on 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-822]. For more 
information, contact Randolph C. Hite at (202) 512-3439 or 
hiter@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

Key DOD and Related IT Acquisition Management Controls Have Not Been 
Fully Implemented on GCSS-MC: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objective, Scope, and Methodology: 

Appendix II: Comments from the Department of Defense: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Description of Legacy Systems Scheduled for Retirement in 
2010: 

Table 2: Organizations Responsible for GCSS-MC Oversight and 
Management: 

Table 3: Summary of Selected System Acquisition Best Practices: 

Table 4: Summary of Cost Estimating Characteristics That the Cost 
Estimate Satisfies: 

Table 5: IMS Satisfaction of Nine Schedule Estimating Key Practices: 

Table 6: MCITS Priority Levels: 

Table 7: Change Request Priorities: 

Figures: 

Figure 1: GCSS-MC Program Schedule Status: 

Figure 2: GCSS-MC Program Cost Status: 

Abbreviations: 

BEA: business enterprise architecture: 

BTA: Business Transformation Agency: 

CIO: Chief Information Officer: 

DAS: defense acquisition system: 

DBSMC: Defense Business Systems Management Committee: 

DOD: Department of Defense: 

DON: Department of the Navy: 

ERP: enterprise resource planning: 

EVM: earned value management: 

FOC: full operational capability: 

GCSS-MC: Global Combat Support System-Marine Corps: 

I-RMIS: Issues-Risk Management Information System: 

IEEE: Institute of Electrical and Electronics Engineers: 

IMS: integrated master schedule: 

IRB: investment review board: 

IT: information technology: 

MCITS: Marine Corps Issue Tracking System: 

MDA: milestone decision authority: 

NTCSS: Naval Tactical Command Support System: 

OMB: Office of Management and Budget: 

TC-AIMS II: Transportation Coordinators' Automated Information for 
Movements System II: 

USD AT&L: Under Secretary of Defense for Acquisition, Technology and 
Logistics: 

[End of section] 

United States Government Accountability Office:
Washington, DC 20548: 

July 28, 2008: 

The Honorable Daniel K. Akaka: 
Chairman: 
The Honorable John Thune: 
Ranking Member: 
Subcommittee on Readiness and Management Support: 
Committee on Armed: 
Services United States Senate: 

The Honorable John Ensign: 
United States Senate: 

For decades, the Department of Defense (DOD) has been challenged in 
modernizing its timeworn business systems.[Footnote 1] In 1995, we 
designated DOD's business systems modernization program as high risk 
and continue to do so today.[Footnote 2] Our reasons include the 
modernization's large size, complexity, and critical role in addressing 
other long-standing transformation and financial management challenges. 
Other reasons are that DOD has yet to institutionalize key system 
modernization management controls, and it has not demonstrated the 
ability to consistently deliver promised system capabilities and 
benefits on time and within budget. 

Nevertheless, DOD continues to invest billions of dollars in thousands 
of business systems, including about a hundred that the department has 
labeled as business transformational programs, 12 of which account for 
about 50 percent of these programs' costs. The Global Combat Support 
System-Marine Corps (GCSS-MC) is one such program. Initiated in 2003, 
GCSS-MC is to modernize the Marine Corps logistics systems and thereby 
provide decision makers with timely and complete logistics information 
to support the warfighter. As envisioned, the program consists of a 
series of major increments, the first of which is expected to cost 
approximately $442 million and be fully deployed in fiscal year 2010. 

As agreed, our objective was to determine whether the Department of the 
Navy is effectively implementing information technology (IT) management 
controls on GCSS-MC. To accomplish this, we focused on the first 
increment of GCSS-MC by analyzing a range of program documentation and 
interviewing cognizant officials relative to the following management 
areas: architectural alignment, economic justification, earned value 
management, requirements management, risk management, and system 
quality measurement. We conducted our performance audit from June 2007 
to July 2008, in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objective. We 
believe that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objective. Additional 
details on our objective, scope, and methodology are in appendix I. 

Results in Brief: 

DOD has not effectively implemented key IT management controls on its 
GCSS-MC program. Collectively, these management controls are intended 
to reasonably ensure that investment in a given system represents the 
right solution to fill a mission need--and if it is, that the system is 
acquired and deployed the right way, meaning that it is done in a 
manner that maximizes the chances of delivering defined system 
capabilities and benefits on time and within budget. Given that 
deployment of GCSS-MC is more than 3 years behind schedule and expected 
to cost about $193 million more than envisioned, these goals are 
already not being met, in part because DOD program management and 
oversight entities have not adequately implemented several key IT 
management controls. As a result, the department does not have a 
sufficient basis for knowing that GCSS-MC, as defined, is the best 
system solution to meeting its mission needs, and the program is likely 
to experience further schedule slips and cost overruns, along with 
reduced system capabilities. Weaknesses associated with DOD's 
implementation of five key IT management controls, as well as recent 
actions to correct weaknesses with another management control, are as 
follows: 

* GCSS-MC compliance with DOD's federated business enterprise 
architecture (BEA) has not been sufficiently demonstrated. To its 
credit, the program office has followed DOD's BEA compliance guidance. 
However, the program's compliance assessment (1) did not include all 
relevant architecture products, such as those that describe technical 
and system elements; (2) was not used to identify potential areas of 
duplication across programs; and (3) did not address compliance with 
the Department of the Navy's enterprise architecture. These important 
steps were not performed because of policy, guidance, and tool 
limitations, and because aspects of the corporate BEA and the 
Department of the Navy's enterprise architecture, which are both major 
components of DOD's federated BEA, have yet to be sufficiently defined 
to permit thorough compliance determinations in these areas. In 
addition, program oversight and approval authorities did not validate 
the program office's compliance assessments. As a result, the 
department does not have a sufficient basis for knowing if GCSS-MC has 
been defined to optimize the DOD and Department of the Navy business 
operations. 

* Investment in GCSS-MC has not been economically justified. According 
to the program's economic analysis, the first increment will have an 
estimated life cycle cost of about $442 million and deliver about $1.04 
billion in risk-adjusted, estimated benefits. While the most recent 
cost estimate was derived using some effective estimating practices, it 
was not based on other practices that are essential to having an 
accurate and credible estimate. For example, the estimate was not 
grounded in a historical record of comparable data from similar 
programs and did not account for significant risks associated with the 
program's aggressive schedule, both of which limit the estimate's 
accuracy and credibility. These important practices were not employed 
for various reasons, including a lack of historical data from similar 
programs and a schedule risk analysis to assess the cost estimate's 
variability. As a result, an adequate basis for informed investment 
decision making does not exist, and actual program costs will likely 
not be consistent with estimates. 

* Earned value management (EVM), which is a means for determining and 
disclosing actual program cost and schedule performance in comparison 
with estimates, is not being effectively performed because the schedule 
baseline is not reliable. Specifically, while the program's current 
schedule baseline was derived using some key estimating practices, it 
is not reflective of other important practices, such as conducting a 
schedule risk assessment and allocating schedule reserve. These 
important practices were not followed, according to program officials, 
because doing so would have pushed back the estimated completion date 
for the first increment, which they said would not have been consistent 
with DOD oversight and approval authorities' direction to complete the 
increment as soon as possible. In other words, the program office 
adopted what amounted to an imposing completion date but did not adjust 
the scope and schedule of work to be completed to make this date 
attainable. The result is a schedule that is not reliable and does not 
provide a sufficient baseline for performing EVM. 

* Despite limitations in earlier efforts to manage GCSS-MC's 
requirements, improvements have since been made. During the initial 
phase of our review, the program office could not trace all of its 
1,375 system-level requirements to design specifications and test 
documentation. For example, about 30 percent of the program's design 
documents had yet to be validated, approved, and linked to the 
requirements. This was significant because it had already contributed 
to lengthy delays to the program's schedule and, without adequate 
traceability, the risk of the system not performing as intended and 
requiring expensive rework is increased. Following our inquiries into 
the traceability process being used, program officials changed the 
process. Since then, our analysis of 61 randomly selected, system-level 
requirements confirmed that 60 are now traceable backward to 
operational requirements and forward to design specifications and test 
plans. If implemented effectively, the new process should address 
previous requirements' traceability weaknesses and thereby avoid a 
repeat of past problems. 

* Despite having a well-defined process for managing program risks, not 
all program risks have been adequately managed. For example, of 25 
medium risks that the program office reported as closed as of February 
2008, 4 were closed because they became actual issues (problems). In 
each of these cases, the risk mitigation strategy was not fully 
implemented. In addition, 4 were closed because, even though the risk 
strategies were implemented, the strategies did not mitigate the risks, 
resulting in each becoming an actual problem. Program officials 
attributed the lack of success in mitigating these risks to, among 
other things, resource constraints and an aggressive program schedule. 
Unless program risks are effectively mitigated, GCSS-MC will experience 
further cost, schedule, and performance shortfalls. 

* Sufficient data for measuring trends in the number of high priority, 
unresolved system issues (problems) and system change requests, both of 
which are recognized indicators of a system's quality or maturity, are 
not available. On the positive side, the program office has established 
processes for (1) collecting and tracking data on the status of program 
issues, including problems discovered during early test events, and (2) 
capturing data on the status of requests for changes to the system. 
Moreover, program documentation emphasizes the importance of monitoring 
trends in program problems and change requests that have yet to be 
addressed or resolved. However, an effective means for producing the 
full complement of data that are needed for a reliable picture of such 
trends does not exist. In particular, data on problems and change 
request priority levels and closure dates are not consistently 
maintained, and program oversight of contractor-identified issues or 
defects is limited. Program officials stated that they intend to 
address these data limitations, but stated that oversight of contractor-
identified issues is not their responsibility. Without sufficient data 
available to understand trends in the volume and severity of program 
problems and changes, it is unclear whether GCSS-MC's quality and 
stability are moving in the right direction. 

Until the above discussed IT management controls are effectively 
implemented, DOD is at risk of investing in incremental GCSS-MC system 
solutions that do not optimally support corporate mission needs and 
mission performance, and do not satisfy defined requirements and meet 
schedule and cost commitments. These risks have already been realized, 
as evidenced by the more than 3-year delay and $193 million increase in 
the expected costs to deploy the first system increment. Accordingly, 
we are making recommendations to the Secretary of Defense aimed at 
ensuring that any decision to invest in the next acquisition phase of 
GCSS-MC's first increment is made in light of the status of efforts to 
address the risks discussed in this report and that investment in all 
future increments be conditional upon the program having fully 
addressed the control weaknesses discussed in this report. We are also 
making recommendations intended to correct the cost estimating, 
schedule estimating, risk management, and system quality measurement 
control weaknesses discussed in this report. However, we are not making 
recommendations in this report relative to addressing the architecture 
compliance weaknesses because we have work under way that is more 
broadly focused on this area across multiple programs, and we will be 
making recommendations in that report. We are also not making 
recommendations regarding requirements traceability because the 
weaknesses that we found have recently been corrected. 

In written comments on a draft of this report, signed by the Deputy 
Under Secretary of Defense (Business Transformation) and reprinted in 
appendix II, the department stated that it concurred with two of our 
recommendations and partially concurred with the remaining five. In 
general, it partially concurred with the five recommendations because 
it said that efforts were either under way or planned that will address 
some of the weaknesses that these recommendations are aimed at 
correcting. We support these efforts because they are generally 
consistent with the intent of the recommendations and believe that, if 
fully and properly implemented, they will go a long way in addressing 
the management control weaknesses that our recommendations are intended 
to correct. DOD's comments are reprinted in their entirety in appendix 
II of this report. 

Background: 

The Department of the Navy (DON) is a major component of DOD, 
consisting of two uniformed services: the Navy and the Marine Corps. 
The Marine Corps' primary mission is to serve as a "total force in 
readiness" by responding quickly in a wide spectrum of 
responsibilities, such as attacks from sea to land in support of naval 
operations, air combat, and security of naval bases. As the only 
service that operates in three dimensions--in the air, on land, and at 
sea, the Marine Corps must be equipped to provide rapid and precise 
logistics[Footnote 3] support to operating forces in any environment. 

The Marine Corps' many and dispersed organization components rely 
heavily on IT to perform their respective mission-critical operations 
and related business functions, such as logistics and financial 
management. For fiscal year 2008, the Marine Corps budget for IT 
business systems is about $1.3 billion, of which $746 million (57 
percent) is for operations and maintenance of existing systems and $553 
million (43 percent) is for systems development and modernization. Of 
the approximately 904 systems in DON's current inventory, the Marine 
Corps accounts for 81, or about 9 percent, of the total. The GCSS-MC is 
one such system investment. According to DOD, it is intended to address 
the Marine Corps' long-standing problem of stove-piped logistics 
systems that collectively provide limited data visibility and access, 
are unable to present a common, integrated logistics picture in support 
of the warfighter, and do not provide important decision support tools. 

GCSS-MC: A Brief Description: 

In September 2003, the Marine Corps initiated GCSS-MC[Footnote 4] to 
(1) deliver integrated functionality across the logistics areas (e.g., 
supply and maintenance), (2) provide timely and complete logistics 
information to authorized users for decision making, and (3) provide 
access to logistics information and applications regardless of 
location. The system is intended to function in three operational 
environments--deployed operations (i.e., in theater of war or exercise 
environment on land or at sea), in-transit, and in garrison.[Footnote 
5] When GCSS-MC is fully implemented, it is to support about 33,000 
users located around the world. 

GCSS-MC is being developed in a series of large and complex increments 
using commercially available enterprise resource planning 
(ERP)[Footnote 6] software and hardware components. The first increment 
is currently the only funded portion of the program and is to provide a 
range of asset management capabilities, including: 

* planning inventory requirements to support current and future 
demands; 

* requesting and tracking the status of products (e.g., supplies and 
personnel) and services (e.g., maintenance and engineering); 

* allocating resources (e.g., inventory, warehouse capacity, and 
personnel) to support unit demands for specific products; and: 

* scheduling maintenance resources (e.g., manpower, equipment, and 
supplies) for specific assets, such as vehicles. 

Additionally, the first increment is to replace four legacy systems 
scheduled for retirement in 2010. Table 1 describes these four systems. 

Table 1: Description of Legacy Systems Scheduled for Retirement in 
2010: 

System: Supported Activities and Supply System; 
Description: Thirty-five year old mainframe ground supply system that 
is used for procuring, distributing, and managing Marine Corps' 
supplies. 

System: Marine Corps Integrated Maintenance Management System; 
Description: Twenty-nine year old mainframe system that is used for 
maintaining ground equipment, including planning and managing work 
orders and parts, and for performing equipment readiness reporting and 
status tracking. 

System: Asset Tracking and Logistics System; 
Description: Fifteen-year-old data entry system dedicated to supporting 
the Supported Activities and Supply System and the Marine Corps 
Integrated Maintenance Management System for controlling, distributing, 
and replenishing equipment and supplies in assigned areas of operation 
and receiving supply support from and providing it to other military 
departments. 

System: Personal Computer Marine Corps Integrated Maintenance 
Management System; 
Description: Fourteen-year-old stand-alone system that is used for 
performing a limited number of ground maintenance management logistics 
functions. 

Source: DOD. 

[End of table] 

Future increments are to provide additional functionality (e.g., 
transportation and wholesale inventory management), enhance existing 
functionality, and potentially replace up to 44 additional legacy 
systems. 

The program office estimates the total life cycle cost for the first 
increment to be about $442 million, including $169 million for 
acquisition and $273 million for operations and maintenance.[Footnote 
7] The total life cycle cost of the entire program has not yet been 
determined because future increments are currently in the planning 
stages and have not been defined. As of April 2008, the program office 
reported that approximately $125 million has been spent on the first 
increment. 

Program Oversight and Management Roles and Responsibilities: 

To manage the acquisition and deployment of GCSS-MC, the Marine Corps 
established a program management office within the Program Executive 
Office for Executive Information Systems. The program office is led by 
the Program Manager who is responsible for managing the program's scope 
and funding and ensuring that the program meets its objectives. To 
accomplish this, the program office is responsible for key acquisition 
management controls, such as architectural alignment, economic 
justification, EVM, requirements management, risk management, and 
system quality measurement. In addition, various DOD and DON 
organizations share program oversight and review activities relative to 
these and other acquisition management controls. A listing of key 
entities and their roles and responsibilities is in table 2. 

Table 2: Organizations Responsible for GCSS-MC Oversight and 
Management: 

Entity: Under Secretary of Defense for Acquisition, Technology, and 
Logistics (USD AT&L); 
Roles and responsibilities: Serves as the milestone decision authority 
(MDA), which according to DOD, has overall responsibility for the 
program, to include approving the program to proceed through its 
acquisition cycle on the basis of, for example, the acquisition plan, 
an independently evaluated economic analysis, and the Acquisition 
Program Baseline. 

Entity: Assistant Secretary of the Navy, Research, Development, and 
Acquisition; 
Roles and responsibilities: Serves as DON's oversight organization for 
the program, to include enforcement of USD AT&L policies and 
procedures. 

Entity: Department of the Navy, Program Executive Office for Executive 
Information Systems; 
Roles and responsibilities: Oversees a portfolio of large-scale 
projects and programs designed to enable common business processes and 
provide standard capabilities, to include reviewing the acquisition 
plan, economic analysis, and the Acquisition Program Baseline prior to 
approval by the MDA. 

Entity: Department of the Navy Chief Information Officer (CIO); 
Roles and responsibilities: Supports the department's planning, 
programming, budgeting, and execution processes by ensuring that the 
program has achievable and executable goals and conforms to financial 
management regulations, and DON, DOD, and federal IT policies in 
several areas (e.g., security, architecture, and investment 
management); it also works closely with the program office during 
milestone review assessments. 

Entity: Office of the Secretary of Defense, Office of the Director for 
Program Analysis and Evaluation; 
Roles and responsibilities: Verifies and validates the reliability of 
cost and benefit estimates in economic analyses and provides its 
results to the MDA. 

Entity: Naval Center for Cost Analysis; 
Roles and responsibilities: Performs independent program cost 
estimates. 

Entity: Defense Cost and Resource Center; 
Roles and responsibilities: Collects current and historical major 
defense acquisition program cost and software resource data in a joint 
service environment and makes those data available for use by 
authorized government analysts when estimating the cost of ongoing and 
future government programs. 

Entity: Deputy Commandant, Installations and Logistics, Headquarters, 
Marine Corps; 
Roles and responsibilities: Provides guidance and direction for overall 
logistics modernization effort and develops/approves formal capability 
requirements for the program. 

Entity: Defense Business Systems Management Committee (DBSMC); 
Roles and responsibilities: Serves as the highest ranking governance 
body for business systems modernization activities and approves 
investments costing more than $1 million, as, for example, being 
compliant with the BEA. 

Entity: Investment Review Board (IRB); 
Roles and responsibilities: Reviews business system investments and has 
responsibility for recommending certification for all business system 
investments costing more than $1 million that are asserted as compliant 
with the BEA. 

Entity: Business Transformation Agency (BTA); 
Roles and responsibilities: Coordinates business transformation efforts 
across DOD and supports the IRBs and DBSMC. 

Entity: GCSS-MC Program Management Office; 
Roles and responsibilities: Performs day-to-day program management and, 
as such, is the single point of accountability for managing the 
program's objectives through development, production, and sustainment, 
such as risk management and coordinating all testing activities with 
requirements, including measuring system quality and managing change 
requests. 

Source: DOD. 

[End of table] 

Overview of GCSS-MC's Status: 

The program reports that the first increment of GCSS-MC is currently in 
the system development and demonstration phase of the defense 
acquisition system (DAS).[Footnote 8] The DAS consists of five key 
program life cycle phases and three related milestone decision points. 
These five phases and related milestones are described along with a 
summary of key program activities completed during, or planned, for 
each phase as follows: 

1. Concept refinement: The purpose of this phase is to refine the 
initial system solution (concept) and create a strategy for acquiring 
the investment solution. During this phase, the program office defined 
the acquisition strategy and analyzed alternative solutions. The first 
increment completed this phase on July 23, 2004, which was 1 month 
later than planned, and the MDA approved a Milestone A decision to move 
to the next phase. 

2. Technology development: The purpose of this phase is to determine 
the appropriate set of technologies to be integrated into the 
investment solution by iteratively assessing the viability of various 
technologies while simultaneously refining user requirements. During 
this phase, the program office selected Oracle's E-Business Suite 
[Footnote 9] as the commercial off-the-shelf ERP software. In addition, 
the program office awarded Accenture the system integration contract 
to, among other things, configure the software, establish system 
interfaces, and implement the new system. This system integration 
contract was divided into two phases--Part 1 for the planning, 
analysis, and conceptual design of the solution and Part 2 for detailed 
design, build, test, and deployment of the solution. The program office 
did not exercise the option for Part 2 of the contract to Accenture and 
shortly thereafter established a new program baseline in June 2006. In 
November 2006, it awarded a time-and-materials system integration 
contract[Footnote 10] valued at $28.4 million for solution design to 
Oracle. The first increment completed this phase on June 8, 2007, which 
was 25 months later than planned due in part to contractual performance 
shortfalls, and the MDA approved a Milestone B decision to move to the 
next phase. 

3. System development and demonstration: The purpose of this phase is 
to develop the system and demonstrate through developer testing that 
the system can function in its target environment. During this phase, 
the program office extended the solution design contract and increased 
funding to $67.5 million due, in part, to delays in completing the 
detailed design activities. As a result, the program office has not yet 
awarded the next contract (which includes both firm-fixed-price and 
time-and-materials task orders) for build and testing activities, 
originally planned for July 2007. Instead, it entered what it termed a 
"transition period" to complete detailed design activities. According 
to the program's baseline, the MDA is expected to approve a Milestone C 
decision to move to the next phase in October 2008. However, program 
officials stated that Milestone C is now scheduled for April 2009, 
which is 35 months later than originally planned. 

4. Production and deployment: The purpose of this phase is to achieve 
an operational capability that satisfies the mission needs, as verified 
through independent operational test and evaluation, and implement the 
system at all applicable locations. The program office plans to award a 
separate firm-fixed-price plus award fee contract[Footnote 11] for 
these activities with estimated costs yet to be determined. 

5. Operations and support: The purpose of this phase is to 
operationally sustain the system in the most cost-effective manner over 
its life cycle. The details of this phase have not yet been defined. 

Overall, GCSS-MC was originally planned to reach full operational 
capability (FOC)[Footnote 12] in fiscal year 2007 at an estimated cost 
of about $126 million over a 7-year life cycle.[Footnote 13] This cost 
estimate was later revised in 2005 to about $249 million over a 13-year 
life cycle.[Footnote 14] However, the program now expects to reach FOC 
in fiscal year 2010 at a cost of about $442 million over a 12-year life 
cycle. Figures 1 and 2 show the program's current status against 
original milestones and original, revised, and current cost estimates. 

Figure 1: GCSS-MC Program Schedule Status: 

This figure is a timeline of the GCSS-MC Program schedule status, as 
follows: 

DAS Phases: Concept refinement; 
May 2004 schedule: Late 2003 to Late 2004; 
March 2008 schedule: Late 2003 to Late 2004. 

DAS Phases: Technology development; 
May 2004 schedule: Late 2004 to mid-2005; 
March 2008 schedule: Late 2004 to late 2007 (rebaselined mid-2006). 

DAS Phases: System development and demonstration (solution design and 
build/test activities); 
May 2004 schedule: Mid-2005 to mid-2006; 
March 2008 schedule: Mid 2007 to mid-2009; 

DAS Phases: Production and deployment; 
May 2004 schedule: Mid-2006 to mid-2007 (FOC); 
March 2008 schedule: Mid-2009 to mid-2010 (FOC). 

July 28, 2008: Issue date of GAO-08-822. 

Source: GAO analysis of Marine Corps data. 

[End of figure] 

Figure 2: GCSS-MC Program Cost Status: 

[See PDF for image] 

This figure is a vertical bar graph depicting the following data: 

Fiscal year: 2004 (May 2004: Fiscal years 2004-2011); 
Dollars in millions: $126 million. 

Fiscal year: 2005 (July 2005: Fiscal years 2005-2018); 
Dollars in millions: $249 million. 

Fiscal year: 2007 (January 2007: Fiscal years 2007-2019); 
Dollars in millions: $442 million. 

Source: GAO analysis of Marine Corps data. 

[End of figure] 

Use of IT Acquisition Management Best Practices Maximizes Chances for 
Success: 

Acquisition best practices are tried and proven methods, processes, 
techniques, and activities that organizations define and use to 
minimize program risks and maximize the chances of a program's success. 
Using best practices can result in better outcomes, including cost 
savings, improved service and product quality, and a better return on 
investment. For example, two software engineering analyses of nearly 
200 systems acquisition projects indicate that teams using systems 
acquisition best practices produced cost savings of at least 11 percent 
over similar projects conducted by teams that did not employ the kind 
of rigor and discipline embedded in these practices.[Footnote 15] In 
addition, our research shows that best practices are a significant 
factor in successful acquisition outcomes and increase the likelihood 
that programs and projects will be executed within cost and schedule 
estimates.[Footnote 16] 

We and others have identified and promoted the use of a number of best 
practices associated with acquiring IT systems.[Footnote 17] See table 
3 for a description of several of these activities. 

Table 3: Summary of Selected System Acquisition Best Practices: 

Business practice: Architectural alignment; To ensure that the 
acquisition is consistent with the organization's enterprise 
architecture; 
Description: Architectural alignment is the process for analyzing and 
verifying that the proposed architecture of the system being acquired 
is consistent with the enterprise architecture for the organization 
acquiring the system. Such alignment is needed to ensure that acquired 
systems can interoperate and are not unnecessarily duplicative of one 
another. 

Business practice: Economic justification; To ensure that system 
investments have an adequate economic justification; 
Description: Economic justification is the process for ensuring that 
acquisition decisions are based on reliable analyses of the proposed 
investment's likely costs versus benefits over its useful life, as well 
as an analysis of the risks associated with actually realizing the 
acquisition's forecasted benefits for its estimated costs. Economic 
justification is not a one-time event but rather is performed 
throughout an acquisition's life cycle in order to permit informed 
investment decision making. 

Business practice: Earned value management; To ensure that actual 
progress is being monitored against expected progress; 
Description: Earned value management is a tool that integrates the 
technical, cost, and schedule parameters of a contract and measures 
progress against them. During the planning phase, an integrated program 
baseline is developed by time phasing budget resources for defined 
work. As work is performed and measured against the baseline, the 
corresponding budget value is "earned." Using this earned value metric, 
cost and schedule variances, as well as cost and time to complete 
estimates, can be determined and analyzed. 

Business practice: Requirements management; To ensure that requirements 
are traceable, verifiable, and controlled; 
Description: Requirements management is the process for ensuring that 
the requirements are traceable, verifiable, and controlled. 
Traceability refers to the ability to follow a requirement from origin 
to implementation and is critical to understanding the interconnections 
and dependencies among the individual requirements, as well as the 
impact when a requirement is changed. Requirements management begins 
when the contractual requirements are documented and ends when system 
responsibility is transferred to the support organization. 

Business practice: Risk management; To ensure that risks are identified 
and systematically mitigated; 
Description: Risk management is the process for identifying potential 
acquisition problems and taking appropriate steps to avoid their 
becoming actual problems. Risk management occurs early and continuously 
in the acquisition life cycle. 

Business practice: System quality measurement; To ensure the maturity 
and stability of system products; 
Description: System quality measurement is the process for 
understanding the maturity and stability of the system products being 
developed, operated, and maintained so that problems can be identified 
and addressed early, therefore limiting their overall impact on program 
cost and schedule. One indicator of system quality is the volume and 
significance of system defect reports and change proposals. 

Source: GAO. 

[End of table] 

Prior GAO Reviews Have Identified IT Acquisition Management Weaknesses 
in DOD's Business System Investments: 

We have previously reported[Footnote 18] that DOD has not effectively 
managed a number of business system investments. Among other things, 
our reviews of individual system investments have identified weaknesses 
in such areas as architectural alignment and informed investment 
decision making, which are also the focus areas of the Fiscal Year 2005 
National Defense Authorization Act[Footnote 19] business system 
provisions. Our reviews have also identified weaknesses in other system 
acquisition and investment management areas--such as EVM, economic 
justification, requirements management, risk management, and test 
management. 

Most recently, for example, we reported that the Army's approach to 
investing about $5 billion over the next several years in its General 
Fund Enterprise Business System, Global Combat Support System-Army 
Field/Tactical,[Footnote 20] and Logistics Modernization Program did 
not include alignment with Army enterprise architecture or use a 
portfolio-based business system investment review process.[Footnote 21] 
Moreover, we reported that the Army did not have reliable analyses, 
such as economic analyses, to support its management of these programs. 
We concluded that until the Army adopts a business system investment 
management approach that provides for reviewing groups of systems and 
making enterprise decisions on how these groups will collectively 
interoperate to provide a desired capability, it runs the risk of 
investing significant resources in business systems that do not provide 
the desired functionality and efficiency. Accordingly, we made 
recommendations aimed at improving the department's efforts to achieve 
total asset visibility and enhancing its efforts to improve its control 
and accountability over business system investments. The department 
agreed with our recommendations. 

We also reported that DON had not, among other things, economically 
justified its ongoing and planned investment in the Naval Tactical 
Command Support System (NTCSS)[Footnote 22] and had not invested in 
NTCSS within the context of a well-defined DOD or DON enterprise 
architecture. In addition, we reported that DON had not effectively 
performed key measurement, reporting, budgeting, and oversight 
activities and had not adequately conducted requirements management and 
testing activities. We concluded that, without this information, DON 
could not determine whether NTCSS, as defined, and as being developed, 
is the right solution to meet its strategic business and technological 
needs. Accordingly, we recommended that the department develop the 
analytical basis to determine if continued investment in the NTCSS 
represents prudent use of limited resources and to strengthen 
management of the program, conditional upon a decision to proceed with 
further investment in the program. The department largely agreed with 
these recommendations. 

In addition, we reported that the Army had not defined and developed 
its Transportation Coordinators' Automated Information for Movements 
System II (TC-AIMS II)--a joint services system with the goal of 
helping to manage the movement of forces and equipment within the 
United States and abroad--in the context of a DOD enterprise 
architecture.[Footnote 23] We also reported that the Army had not 
economically justified the program on the basis of reliable estimates 
of life cycle costs and benefits and had not effectively implemented 
risk management. As a result, we concluded that the Army did not know 
if its investment in TC-AIMS II, as planned, is warranted or represents 
a prudent use of limited DOD resources. Accordingly, we recommended 
that DOD, among other things, develop the analytical basis needed to 
determine if continued investment in TC-AIMS II, as planned, represents 
prudent use of limited defense resources. In response, the department 
largely agreed with our recommendations and has since reduced the 
program's scope by canceling planned investments. 

Key DOD and Related IT Acquisition Management Controls Have Not Been 
Fully Implemented on GCSS-MC: 

DOD IT-related acquisition policies and guidance, along with other 
relevant guidance, provide an acquisition management control framework 
within which to manage business system programs like GCSS-MC. Effective 
implementation of this framework can minimize program risks and better 
ensure that system investments are defined in a way to optimally 
support mission operations and performance, as well as deliver promised 
system capabilities and benefits on time and within budget. Thus far, 
GCSS-MC has not been managed in accordance with key aspects of this 
framework, which has already contributed to more than 3 years in 
program schedule delays and about $193 million in cost increases. These 
IT acquisition management control weaknesses include: 

* compliance with DOD's federated BEA not being sufficiently 
demonstrated; 

* expected costs not being reliably estimated; 

* earned value management not being adequately implemented; 

* system requirements not always being effectively managed, although 
this has recently improved; 

* key program risks not being effectively managed; and: 

* key system quality measures not being used. 

The reasons that these key practices have not been sufficiently 
executed include limitations in the applicable DOD guidance and tools, 
and not collecting relevant data, each of which is described in the 
applicable sections of this report. By not effectively implementing 
these key IT acquisition management controls, the program has already 
experienced sizeable schedule and cost increases, and it is at 
increased risk of (1) not being defined in a way that best meets 
corporate mission needs and enhances performance and (2) costing more 
and taking longer than necessary to complete. 

GCSS-MC Compliance with DOD's Federated BEA Has Not Been Sufficiently 
Demonstrated: 

DOD and federal guidance[Footnote 24] recognize the importance of 
investing in business systems within the context of an enterprise 
architecture.[Footnote 25] Moreover, the 2005 National Defense 
Authorization Act requires that defense business systems be compliant 
with DOD's federated BEA.[Footnote 26] Our research and experience in 
reviewing federal agencies show that not making investments within the 
context of a well-defined enterprise architecture often results in 
systems that are duplicative, are not well integrated, are 
unnecessarily costly to interface and maintain, and do not optimally 
support mission outcomes.[Footnote 27] 

To its credit, the program office has followed DOD's BEA compliance 
guidance.[Footnote 28] However, this guidance does not adequately 
provide for addressing all relevant aspects of BEA compliance. 
Moreover, DON's enterprise architecture, which is a major component of 
DOD's federated BEA, as well as key aspects of DOD's corporate BEA, 
have yet to be sufficiently defined to permit thorough compliance 
determinations. In addition, current policies and guidance do not 
require DON investments to comply with its enterprise architecture. 
This means that the department does not have a sufficient basis for 
knowing if GCSS-MC has been defined to optimize DON and DOD business 
operations. Each of these architecture alignment limitations is 
discussed as follows: 

* The program's compliance assessments did not include all relevant 
architecture products. In particular, the program did not assess 
compliance with the BEA's technical standards profile, which outlines, 
for example, the standards governing how systems physically communicate 
with other systems and how they secure data from unauthorized access. 
This is particularly important because systems, like GCSS-MC, need to 
employ common standards in order to effectively and efficiently share 
information with other systems. A case in point is GCSS-MC and the Navy 
Enterprise Resource Planning program.[Footnote 29] Specifically, GCSS- 
MC has identified 13 technical standards that are not in the BEA 
technical standards profile, and Navy Enterprise Resource Planning has 
identified 25 technical standards that are not in the profile. Of 
these, some relate to networking protocols, which could limit 
information sharing between these and other systems. 

In addition, the program office did not assess compliance with the BEA 
products that describe system characteristics. This is important 
because doing so would create a body of information about programs that 
could be used to identify common system components and services that 
could potentially be shared by the programs, thus avoiding wasteful 
duplication. For example, our analysis of GCSS-MC program documentation 
shows that they contain such system functions as receiving goods, 
taking physical inventories, and returning goods, which are also system 
functions cited by the Navy Enterprise Resource Planning program. 
However, because compliance with the BEA system products was not 
assessed, the extent to which these functions are potentially 
duplicative was not considered. 

Furthermore, the program office did not assess compliance with BEA 
system products that describe data exchanges among systems. As we 
previously reported, establishing and using standard system interfaces 
is a critical enabler to sharing data.[Footnote 30] For example, GCSS- 
MC program documentation indicates that it is to exchange order and 
status data with other systems. However, the program office has not 
fully developed its architecture product describing these exchanges and 
thus does not have the basis for understanding how its approach to 
exchanging information differs from that of other systems that it is to 
interface with. Compliance against each of these BEA products was not 
assessed because DOD's compliance guidance does not provide for doing 
so and, according to BTA and program officials, some BEA and program- 
level architecture products are not sufficiently defined. According to 
these officials, BTA plans to continue to define these products as the 
BEA evolves. 

* The compliance assessment was not used to identify potential areas of 
duplication across programs, which DOD has stated is an explicit goal 
of its federated BEA and associated investment review and decision- 
making processes. More specifically, even though the compliance 
guidance provides for assessing programs' compliance with the BEA 
product that defines DOD operational activities, and GCSS-MC was 
assessed for compliance with this product, the results were not used to 
identify programs that support the same operational activities and 
related business processes. Given that the federated BEA is intended to 
identify and avoid not only duplications within DOD components, but 
also between DOD components, it is important that such commonality be 
addressed. For example, program-level architecture products for GCSS-MC 
and Navy Enterprise Resource Planning, as well as two Air Force 
programs (Defense Enterprise Accounting and Management System-Air Force 
and the Air Force Expeditionary Combat Support System) show that each 
supports at least six of the same BEA operational activities (e.g., 
conducting physical inventory, delivering property, and services) and 
three of these four programs support at least 18 additional operational 
activities (e.g., performing budgeting, managing receipt, and 
acceptance). As a result, these programs may be investing in 
duplicative functionality. Reasons for not doing so were that 
compliance guidance does not provide for such analyses to be conducted, 
and programs have not been granted access rights to use this 
functionality. 

* The program's compliance assessment did not address compliance 
against the DON's enterprise architecture, which is one of the biggest 
members of the federated BEA. This is particularly important given that 
DOD's approach to fully satisfying the architecture requirements of the 
2005 National Defense Authorization Act is to develop and use a 
federated architecture in which component architectures are to provide 
the additional details needed to supplement the thin layer of corporate 
policies, rules, and standards included in the corporate BEA.[Footnote 
31] As we recently reported, the DON's enterprise architecture is not 
mature because, among other things, it is missing a sufficient 
description of its current and future environments in terms of business 
and information/data. However, certain aspects of an architecture 
nevertheless exist and, according to DON, these aspects will be 
leveraged in its efforts to develop a complete enterprise architecture. 
For example, the FORCEnet architecture documents DON's technical 
infrastructure. Therefore, opportunities exist for DON to assess its 
programs in relation to these architecture products and to understand 
where its programs are exposed to risks because products do not exist, 
are not mature, or are at odds with other DON programs. According to 
DOD officials, compliance with the DON architecture was not assessed 
because DOD compliance policy is limited to compliance with the 
corporate BEA, and the DON enterprise architecture has yet to be 
sufficiently developed. 

The program's compliance assessment was not validated by DOD or DON 
investment oversight and decision-making authorities. More 
specifically, neither the DOD IRBs nor the DBSMC, nor the BTA in 
supporting both of these investment oversight and decision-making 
authorities, reviewed the program's assessments. According to BTA 
officials, under DOD's tiered approach to investment accountability, 
these entities are not responsible for validating programs' compliance 
assessments. Rather, this is a component responsibility, and thus they 
rely on the military departments and defense agencies to validate the 
assessments. 

However, the DON Office of the CIO, which is responsible for 
precertifying investments as compliant before they are reviewed by the 
IRB, did not evaluate any of the programs' compliance assessments. 
According to CIO officials, they rely on Functional Area Managers 
[Footnote 32] to validate a program's compliance assessments. However, 
no DON policy or guidance exists that describes how the Functional Area 
Managers should conduct such validations. 

Validation of program assessments is further complicated by the absence 
of information captured in the assessment tool about what program 
documentation or other source materials were used by the program office 
in making its compliance determinations. Specifically, the tool is only 
configured, and thus was only used, to capture the results of a 
program's comparison of program architecture products to BEA products. 
Thus, it was not used to capture the system products used in making 
these determinations. 

In addition, the program office did not develop certain program-level 
architecture products that are needed to support and validate the 
program's compliance assessment and assertions. According to the 
compliance guidance, program-level architecture products, such as those 
defining information exchanges and system data requirements are not 
required to be used until after the system has been deployed. This is 
important because waiting until the system is deployed is too late to 
avoid the costly rework required to address areas of noncompliance. 
Moreover, it is not consistent with other DOD guidance,[Footnote 33] 
which states that program-level architecture products that describe, 
for example, information exchanges, should be developed before a 
program begins system development. 

The limitations in existing BEA compliance-related policy and guidance, 
the supporting compliance assessment tool, and the federated BEA, puts 
programs like GCSS-MC at increased risk of being defined and 
implemented in a way that does not sufficiently ensure interoperability 
and avoid duplication and overlap. We currently have a review under way 
for the Senate Armed Services Committee, Subcommittee on Readiness and 
Management Support, which is examining multiple programs' compliance 
with the federated BEA. 

Investment in GCSS-MC Was Not Economically Justified Using Reliable 
Estimates of Costs: 

The investment in the first increment of GCSS-MC has not been 
economically justified on the basis of reliable analyses of estimated 
system costs over the life of the program. According to the program's 
economic analysis, the first increment will have an estimated life 
cycle cost of about $442 million and deliver an estimated $1.04 billion 
in risk-adjusted estimated benefits during this same life cycle. This 
equates to a net present value of about $688 million. While the most 
recent cost estimate was derived using some effective estimating 
practices, it did not make use of other practices that are essential to 
having an accurate and credible estimate. As a result, the Marine Corps 
does not have a sufficient basis for deciding whether GCSS-MC, as 
defined, is the most cost-effective solution to meeting its mission 
needs, and it does not have a reliable basis against which to measure 
cost performance. 

The Cost Estimate Was Not Derived Using Practices Necessary for an 
Accurate and Credible Estimate: 

A reliable cost estimate is critical to the success of any IT program, 
as it provides the basis for informed investment decision making, 
realistic budget formulation and program resourcing, meaningful 
progress measurement, proactive course correction, and accountability 
for results. According to the Office of Management and Budget (OMB), 
[Footnote 34] programs must maintain current and well-documented cost 
estimates, and these estimates must encompass the full life cycle of 
the program. OMB states that generating reliable cost estimates is a 
critical function necessary to support OMB's capital programming 
process. Without reliable estimates, programs are at increased risk of 
experiencing cost overruns, missed deadlines, and performance 
shortfalls. 

Our research has identified a number of practices for effective program 
cost estimating. We have issued guidance that associates these 
practices with four characteristics of a reliable cost estimate. 
[Footnote 35] These four characteristic are specifically defined as 
follows: 

* Comprehensive: The cost estimates should include both government and 
contractor costs over the program's full life cycle, from the inception 
of the program through design, development, deployment, and operation 
and maintenance, to retirement. They should also provide a level of 
detail appropriate to ensure that cost elements are neither omitted nor 
double counted and include documentation of all cost-influencing ground 
rules and assumptions. 

* Well-documented: The cost estimates should have clearly defined 
purposes and be supported by documented descriptions of key program or 
system characteristics (e.g., relationships with other systems, 
performance parameters). Additionally, they should capture in writing 
such things as the source data used and their significance, the 
calculations performed and their results, and the rationale for 
choosing a particular estimating method or reference. Moreover, this 
information should be captured in such a way that the data used to 
derive the estimate can be traced back to, and verified against, their 
sources. The final cost estimate should be reviewed and accepted by 
management on the basis of confidence in the estimating process and the 
estimate produced by the process. 

* Accurate: The cost estimates should provide for results that are 
unbiased and should not be overly conservative or optimistic (i.e., 
should represent the most likely costs). In addition, the estimates 
should be updated regularly to reflect material changes in the program, 
and steps should be taken to minimize mathematical mistakes and their 
significance. The estimates should also be grounded in a historical 
record of cost estimating and actual experiences on comparable 
programs. 

* Credible: The cost estimates should discuss any limitations in the 
analysis performed that are due to uncertainty or biases surrounding 
data or assumptions. Further, the estimates' derivation should provide 
for varying any major assumptions and recalculating outcomes based on 
sensitivity analyses, and the estimates' associated risks and inherent 
uncertainty should be disclosed. Also, the estimates should be verified 
based on cross-checks using other estimating methods and by comparing 
the results with independent cost estimates. 

The $442 million life cycle cost estimate for the first increment 
reflects many of the practices associated with a reliable cost 
estimate, including all practices associated with being comprehensive 
and well-documented, and several related to being accurate and 
credible. (See table 4.) However, several important accuracy and 
credibility practices were not satisfied. 

Table 4: Summary of Cost Estimating Characteristics That the Cost 
Estimate Satisfies: 

Characteristic of reliable estimates: Comprehensive; 
Satisfied?[A]: Yes. 

Characteristic of reliable estimates: Well-documented; 
Satisfied?[A]: Yes. 

Characteristic of reliable estimates: Accurate; 
Satisfied?[A]: Partially. 

Characteristic of reliable estimates: Credible; 
Satisfied?[A]: Partially. 

Source: GAO analysis of Marine Corps data. 

[A] "Yes" means that the program office provided documentation 
demonstrating satisfaction of the criterion. "Partially" means that the 
program office provided documentation demonstrating satisfaction of 
part of the criterion. "No" means that the program office has yet to 
provide documentation demonstrating satisfaction of the criterion. 

[End of table] 

The cost estimate is comprehensive because it includes both the 
government and contractor costs specific to development, acquisition 
(nondevelopment), implementation, and operations and support over the 
program's 12-year life cycle. Moreover, the estimate clearly describes 
how the various subelements are summed to produce the amounts for each 
cost category, thereby ensuring that all pertinent costs are included, 
and no costs are double counted. Lastly, cost-influencing ground rules 
and assumptions, such as the program's schedule, labor rates, and 
inflation rates are documented. 

The cost estimate is also well-documented in that the purpose of the 
cost estimate was clearly defined, and a technical baseline has been 
documented that includes, among others things, the relationships with 
other systems and planned performance parameters. Furthermore, the 
calculations and results used to derive the estimate are documented, 
including descriptions of the methodologies used and traceability back 
to source data (e.g., vendor quotes, salary tables). Also, the cost 
estimate was reviewed both by the Naval Center for Cost Analysis and 
the Office of the Secretary of Defense, Director for Program Analysis 
and Evaluation, which ensures a level of confidence in the estimating 
process and the estimate produced. 

However, the estimate lacks accuracy because not all important 
practices related to this characteristic were satisfied. Specifically, 
while the estimate is grounded in documented assumptions (e.g., 
hardware refreshment every 5 years), and periodically updated to 
reflect changes to the program, it is not grounded in historical 
experience with comparable programs. As stated in our guide, estimates 
should be based on historical records of cost and schedule estimates 
from comparable programs, and such historical data should be maintained 
and used for evaluation purposes and future estimates on other 
comparable programs. The importance of doing so is evident by the fact 
that GCSS-MC's cost estimate has increased by about $193 million since 
July 2005, which program officials attributed to, among other things, 
schedule delays, software development complexity, and the lack of 
historical data from similar ERP programs. While the program office did 
leverage historical cost data from other ERP programs, including the 
Navy's Enterprise Resource Planning Pilot programs and programs at the 
Bureau of Prisons and the Department of Agriculture, program officials 
told us that these programs' scopes were not comparable. For example, 
none of the programs had to utilize a communication architecture as 
complex as the Marine Corps, which officials cited as a significant 
factor in the cost increases and a challenge in estimating costs. 

The absence of analogous cost data for large-scale ERP programs is due 
in part to the fact that DOD has not established a standardized cost 
element structure for ERP programs that can be used to capture actual 
cost data. According to officials with the Defense Cost and Resource 
Center, such cost element structures are needed, along with a 
requirement for programs to report on their costs, but approval and 
resources have yet to be gained for either these structures or the 
reporting of their costs. Until a standardized data structure exists, 
programs like GCSS-MC will continue to lack a historical database 
containing cost estimates and actual cost experiences of comparable ERP 
programs. This means that the current and future GCSS-MC cost estimates 
will lack sufficient accuracy for effective investment decision making 
and performance measurement. 

Compounding the estimate's limited accuracy are limitations in its 
credibility. Specifically, while the estimate satisfies some of the key 
practices for a credible cost estimate (e.g., confirming key cost 
drivers, performing sensitivity analyses,[Footnote 36] having an 
independent cost estimate prepared by the Naval Center for Cost 
Analysis that was within 4 percent of the program's estimate, and 
conducting a risk analysis that showed a range of estimated costs of 
$411 million to $523 million), no risk analysis was performed to 
determine the program schedule's risks and associated impact on the 
cost estimate. As described earlier in this report, the program has 
experienced about 3 years in schedule delays and recently experienced 
delays in completing the solution design phase. Therefore, the 
importance of conducting a schedule risk analysis and using the results 
to assess the variability in the cost estimate is critical for ensuring 
a credible cost estimate. Program officials agreed that the program's 
schedule is aggressive and risky and that this risk was not assessed in 
determining the cost estimate's variability. Without doing so, the 
program's cost estimate is not credible, and thus the program is at 
risk of cost overruns as a result of schedule delays. 

Benefits Estimate Has Been Adjusted to Reflect Questionable Assumptions 
and Limited Data: 

Forecasting expected benefits over the life of a program is also a key 
aspect of economically justifying an investment. OMB guidance[Footnote 
37] advocates economically justifying investments on the basis of net 
present value. If net present value is positive, then the corresponding 
benefit-to-cost ratio will be greater than 1 (and vice versa).[Footnote 
38] This guidance also advocates updating the analyses over the life of 
the program to reflect material changes in expected benefits, costs, 
and risks. Since estimates of benefits can be uncertain because of the 
imprecision in both the underlying data and modeling assumptions used, 
effects of this uncertainty should be analyzed and reported. By doing 
this, informed investment decision making can occur through the life of 
the program, and a baseline can be established against which to compare 
the accrual of actual benefits from deployed system capabilities. 

The original benefit estimate for the first increment was based on 
questionable assumptions and insufficient data from comparable 
programs. The most recent economic analysis, dated January 2007, 
includes monetized, yearly benefit estimates for fiscal years 2010-2019 
in three key areas--inventory reductions, reductions in inventory 
carrying costs, and improvements in maintenance processes. 
Collectively, these benefits totaled about $2.89 billion (not risk- 
adjusted). However, these calculations were made using questionable 
assumptions and limited data. For example, 

* The total value of the Marine Corps inventory needed to calculate 
inventory reductions and reductions in carrying costs could not be 
determined because of limitations with existing logistic systems. 

* The cost savings resulting from improvements in maintenance processes 
were calculated based on assumptions from an ERP implementation in the 
commercial sector that, according to program officials, is not 
comparable in scope to GCSS-MC. 

To account for the uncertainty inherent in the benefits estimate, the 
program office performed a Monte Carlo simulation.[Footnote 39] 
According to the program office, this risk analysis generated a 
discounted and risk-adjusted benefits estimate of $1.04 billion. As a 
result of the $1.85 billion adjustment to estimated benefits, the 
program office has a more realistic benefit baseline against which to 
compare the accrual of actual benefits from deployed system 
capabilities. 

EVM Has Not Been Adequately Implemented: 

The program office has elected to implement EVM, which is a proven 
means for measuring program progress and thereby identifying potential 
cost overruns and schedule delays early, when they can be minimized. In 
doing so, it has adopted a tailored EVM approach that focuses on 
schedule. However, this schedule-focused approach has not been 
effectively implemented because it is based on a baseline schedule that 
was not derived using key schedule estimating practices. According to 
program officials, the schedule was driven by an aggressive program 
completion date established in response to direction from oversight 
entities to complete the program as soon as possible. As a result, they 
said that following these practices would have delayed this completion 
date. Regardless, this means that the schedule baseline is not 
reliable, and progress will likely not track to the schedule. 

A Tailored EVM Approach Is Being Used to Measure Program Progress: 

The program office has adopted a tailored approach to performing EVM 
because of the contract type being used. As noted earlier, the contract 
types associated with GCSS-MC integration and implementation vary, and 
include, for example, firm-fixed-price contracts[Footnote 40] and time- 
and-materials contracts. Under a firm-fixed-price contract, the price 
is not subject to any adjustment on the basis of the contractor's cost 
experience in performing the contract. For a time-and-materials 
contract, supplies or services are acquired on the basis of (1) an 
undefined number of direct labor hours that are paid at specified fixed 
hourly rates and (2) actual cost for materials. 

According to DOD guidance,[Footnote 41] EVM is generally not encouraged 
for firm-fixed-price, level of effort, and time-and-material contracts. 
[Footnote 42] In these situations, the guidance states that programs 
can use a tailored EVM approach in which an integrated master schedule 
(IMS) is exclusively used to provide visibility into program 
performance. 

DON has chosen to implement this tailored EVM approach on GCSS-MC. In 
doing so, it is measuring progress against schedule commitments, and 
not cost commitments, using an IMS for each program phase. According to 
program officials, the IMS describes and guides the execution of 
program activities. Regardless of the approach used, effective 
implementation depends on having a reliable IMS. 

Schedule Baseline Was Not Derived in Accordance with Key Estimating 
Practices: 

The success of any program depends in part on having a reliable 
schedule specifying when the program's set of work activities will 
occur, how long they will take, and how they are related to one 
another. As such, the schedule not only provides a road map for the 
systematic execution of a program, but it also provides the means by 
which to gauge progress, identify and address potential problems, and 
promote accountability. Our research has identified nine practices 
associated with effective schedule estimating.[Footnote 43] These 
practices are (1) capturing key activities, (2) sequencing key 
activities, (3) assigning resources to key activities, (4) integrating 
key activities horizontally and vertically, (5) establishing the 
duration of key activities, (6) establishing the critical path for key 
activities, (7) identifying "float time"[Footnote 44] between key 
activities, (8) distributing reserves to high-risk activities, and (9) 
performing a schedule risk analysis. 

The current IMS for the solution design and transition-to-build phase 
of the first increment was developed using some of these practices. 
However, it does not reflect several practices that are fundamental to 
having a schedule baseline that provides a sufficiently reliable basis 
for measuring progress and forecasting slippages. To the program 
office's credit, its IMS captures and sequences key activities required 
to complete the project, integrates the tasks horizontally, and 
identifies the program's critical path. However, the program office is 
not monitoring the actual durations of scheduled activities so that it 
can address the impact of any deviations on later scheduled activities. 
Moreover, the schedule does not adequately identify the resources 
needed to complete the tasks and is not integrated vertically, meaning 
that multiple teams executing different aspects of the program cannot 
effectively work to the same master schedule. Further, the IMS does not 
adequately mitigate schedule risk by identifying float time between key 
activities, introducing schedule reserve for high-risk activities, or 
including the results of a schedule risk analysis. See table 5 for the 
results of our analyses relative to each of the nine practices. 

Table 5: IMS Satisfaction of Nine Schedule Estimating Key Practices: 

Practice: Capturing key activities; 
Explanation: The schedule should reflect all key activities (e.g., 
steps, events, outcomes) as defined in the program's work breakdown 
structure, to include activities to be performed by both the government 
and its contractors; 
Satisfied?[A]: Yes; 
GAO analysis: The program's schedule reflects both government and 
contractor activities, such as building and testing of the software 
components, as well as key milestones for measuring progress. 

Practice: Sequencing key activities; 
Explanation: The schedule should be planned so that it can meet 
critical program dates. To meet this objective, key activities need to 
be logically sequenced in the order that they are to be carried out. In 
particular, activities that must finish prior to the start of other 
activities (i.e., predecessor activities), as well as activities that 
cannot begin until other activities are completed (i.e., successor 
activities) should be identified. By doing so, interdependencies among 
activities that collectively lead to the accomplishment of events or 
milestones can be established and used as a basis for guiding work and 
measuring progress;
Satisfied?[A]: Yes; 
GAO analysis: The schedule includes the sequencing of key activities, 
meaning that it includes both the predecessor and successor activities 
and thus establishes interdependencies among the activities that form 
the basis for guiding work and measuring progress. 

Practice: Establishing the duration of key activities; 
Explanation: The schedule should realistically reflect how long each 
activity will take to execute. In determining the duration of each 
activity, the same rationale, historical data, and assumptions used for 
cost estimating should be used for schedule estimating. Further, these 
durations should be as short as possible, and they should have specific 
start and end dates. Excessively long periods needed to execute an 
activity should prompt further decomposition of the activity so that 
shorter execution durations will result. The schedule should be 
continually monitored to determine when forecasted completion dates 
differ from the planned dates, which can be used to determine whether 
schedule variances will affect downstream work; 
Satisfied?[A]: Partially; 
GAO analysis: The schedule establishes the durations of key activities 
based on government and contractor opinions, as well as historical data 
from the contractor's experience. However, the program office does not 
monitor the actual start and completion dates of work activities so 
that the impact of deviations on downstream scheduled work can be 
proactively addressed. Program officials agreed that they have not been 
tracking actual start and end dates, but stated that they intend to do 
so for future phases. 

Practice: Assigning resources to key activities; 
Explanation: The schedule should reflect what resources (e.g., labor, 
material, and overhead) are needed to do the work, whether all required 
resources will be available when they are needed, and whether any 
funding or time constraints exist; 
Satisfied?[A]: Partially; 
GAO analysis: The schedule allocated some, but not all, resources to 
all key activities. For example, it did not allocate labor hours and 
materials. Program officials stated that these resources are assigned 
to more detailed activities at the team level. However, they have yet 
to provide adequate documentation to show that this is occurring. 

Practice: Integrating key activities horizontally and vertically; 
Explanation: The schedule should be horizontally integrated, meaning 
that it should link the products and outcomes associated with already 
sequenced activities. These links are commonly referred to as 
"handoffs" and serve to verify that activities are arranged in the 
right order to achieve aggregated products or outcomes. The schedule 
should also be vertically integrated, meaning that traceability exists 
among varying levels of activities and supporting tasks and subtasks. 
Such mapping or alignment among levels enables different groups to work 
to the same master schedule; 
Satisfied?[A]: Partially; 
GAO analysis: The schedule is horizontally integrated, meaning that the 
activities across the multiple teams are arranged in the right order to 
achieve aggregated products or outcomes. However, the schedule is not 
vertically integrated, meaning that traceability does not exist among 
varying levels of activities. Program officials stated that team-level 
schedules can be traced vertically to the IMS, but they have not 
provided adequate documentation to show that this is occurring. 

Practice: Establishing the critical path for key activities; 
Explanation: Using scheduling software, the critical path--the longest 
duration path through the sequenced list of key activities--should be 
identified. The establishment of a program's critical path is necessary 
for examining the effects of any activity slipping along this path. 
Potential problems that might occur along or near the critical path 
should also be identified and reflected in the scheduling of the time 
for high-risk activities (see next); 
Satisfied?[A]: Yes; 
GAO analysis: The program's critical path has been defined using 
scheduling software and includes, among other things, the preparation 
for testing activities and the execution of six test scenarios. 

Practice: Identifying float time between key activities; 
Explanation: The schedule should identify float time--the time that a 
predecessor activity can slip before the delay affects successor 
activities--so that schedule flexibility can be determined. As a 
general rule, activities along the critical path typically have the 
least amount of float time; 
Satisfied?[A]: No; 
GAO analysis: The program office could not identify the amount of float 
time allocated to key activities to account for potential problems that 
might occur along or near the critical path. Therefore, if the schedule 
for an activity near the critical path were to slip, it is likely that 
the delay would impact the critical path. 

Practice: Distributing reserves to high-risk activities; 
Explanation: The baseline schedule should include a buffer or a reserve 
of extra time. Schedule reserve for contingencies should be calculated 
by performing a schedule risk analysis (see next). As a general rule, 
the reserve should be applied to high-risk activities, which are 
typically found along the critical path; 
Satisfied?[A]: No; 
GAO analysis: The program office did not allocate schedule reserve for 
high-risk activities on the critical path. Schedule reserve should be 
calculated by performing a schedule risk analysis, and allocating 
additional reserve for those activities identified as high risk. 
Without this reserve, any delays on activities on the critical path 
increase the risk of delays to the scheduled completion date. 

Practice: Performing a schedule risk analysis; 
Explanation: A schedule risk analysis should be performed using 
statistical techniques to predict the level of confidence in meeting a 
program's completion date. This analysis focuses not only on critical 
path activities but also on activities near the critical path, since 
they can potentially affect program status; 
Satisfied?[A]: No; 
GAO analysis: The program office did not perform a schedule risk 
analysis to determine the level of confidence in meeting the program's 
activities and its completion date. 

Sources: GAO analysis of Marine Corps data. 

[A] "Yes" means that the program office provided documentation 
demonstrating satisfaction of the criterion. "Partially" means that the 
program office provided documentation demonstrating satisfaction of 
part of the criterion. "No" means that the program office has yet to 
provide documentation demonstrating satisfaction of the criterion. 

[End of table] 

According to program officials, they intend to begin monitoring actual 
activity start and completion dates so that they can proactively adjust 
later scheduled activities that are affected by deviations. However, 
they do not plan to perform the three practices related to 
understanding and managing schedule risk because doing so would likely 
extend the program's completion date, and they set this date to be 
responsive to direction from DOD and DON oversight entities to complete 
the program as soon as possible. In our view, not performing these 
practices does not allow the inherent risks in meeting this imposed 
completion date to be proactively understood and addressed. The 
consequence of omitting these practices is a schedule that does not 
provide a reliable basis for performing EVM. 

Requirements Management Weakness Was Recently Corrected: 

Well-defined and managed requirements are recognized by DOD guidance as 
essential and can be viewed as a cornerstone of effective system 
acquisition. One aspect of effective requirements management is 
requirements traceability.[Footnote 45] By tracing requirements both 
backward from system requirements to higher level business or 
operational requirements and forward to system design specifications 
and test plans, the chances of the deployed product satisfying 
requirements are increased, and the ability to understand the impact of 
any requirement changes, and thus make informed decision about such 
changes, is enhanced. 

The program office recently strengthened its requirements traceability. 
In November 2007, and again in February 2008, the program office was 
unable to demonstrate for us that it could adequately trace its 1,375 
system requirements to both design specifications and test 
documentation. Specifically, the program office was at that time using 
a tool called DOORS®, which if implemented properly, allows each 
requirement to be linked from its most conceptual definition to its 
most detailed definition, as well as to design specifications and test 
cases. In effect, the tool maintains the linkages among requirement 
documents, design documents, and test cases even if requirements 
change. However, the system integration contractor was not using the 
tool. Instead the contractor was submitting its 244 work products, 
[Footnote 46] accompanied by spreadsheets that linked each work product 
to one or more system requirements and test cases. The program office 
then had to verify and validate the spreadsheets and import and link 
each work product to the corresponding requirement and test case in 
DOORS. Because of the sheer number of requirements and work products 
and its potential to impact cost, schedule, and performance, the 
program designated this approach as a medium risk. It later closed the 
risk because the proposed mitigation strategy failed to mitigate it, 
and it was realized as a high-priority program issue (i.e., problem). 

According to program officials, this requirements traceability approach 
resulted in time-consuming delays in approving the design work products 
and importing and establishing links between these products and the 
requirements in DOORS, in part because the work products were not 
accompanied by complete spreadsheets that established the traceability. 
As a result, about 30 percent of the contractor's work products had yet 
to be validated, approved, and linked to requirements when the design 
phase was originally scheduled to be complete. Officials stated that 
the contractor was not required to use DOORS because it was not 
experienced with this tool and becoming proficient with it would have 
required time and resources, thereby increasing both the program's cost 
and schedule. Ironically, however, not investing the time and resources 
to address the limitations in the program's traceability approach 
contributed to recent delays in completing the solution design 
activities, and additional resources had to be invested to address its 
requirements traceability problems. 

The program office now reports that it can trace requirements backward 
and forward. In April 2008, we verified this by tracing 60 out of 61 
randomly sampled requirements backward to system requirements and 
forward to approved design specifications and test plans. Program 
officials explained that the reason that we could not trace the one 
requirement was that the related work products had not yet been 
approved. In addition, they stated that there were additional work 
products that had yet to be finalized and traced. 

Without adequate traceability, the risk of a system not performing as 
intended and requiring expensive rework is increased. To address its 
requirements traceability weakness, program officials told us that they 
now intend to require the contractor to use DOORS during the next phase 
of the program (build and test). If implemented effectively, the new 
process should address previous requirements traceability weaknesses 
and thereby avoid a repeat of past problems. 

Not All Program Risks Have Been Adequately Managed: 

Proactively managing program risks is a key acquisition management 
control and, if done properly, can greatly increase the chances of 
programs delivering promised capabilities and benefits on time and 
within budget. To the program office's credit, it has defined a risk 
management process that meets relevant guidance. However, it has not 
effectively implemented the process for all identified risks. As a 
result, these risks have become actual program problems that have 
impacted the program's cost, schedule, and performance commitments. 

DOD acquisition management guidance,[Footnote 47] as well as other 
relevant guidance,[Footnote 48] advocates identifying facts and 
circumstances that can increase the probability of an acquisition's 
failing to meet cost, schedule, and performance commitments and then 
taking steps to reduce the probability of their occurrence and impact. 
In brief, effective risk management consists of: (1) establishing a 
written plan for managing risks; (2) designating responsibility for 
risk management activities; (3) encouraging project-wide participation 
in the identification and mitigation of risks; (4) defining and 
implementing a process that provides for the identification, analysis, 
and mitigation of risks; and (5) examining the status of identified 
risks in program milestone reviews. 

The program office has developed a written plan for managing risks, and 
established a process that together provide for the above cited risk 
management practices, and it has followed many key aspects of its plan 
and process. For example, 

* The Program Manager has been assigned overall responsibility for 
managing risks. Also, individuals have been assigned ownership of each 
risk, to include conducting risk analyses, implementing mitigation 
strategies, and working with the risk support team.[Footnote 49] 

* The plan and process encourage project-wide participation in the 
identification and mitigation of risks by allowing program staff to 
submit a risk for inclusion in a risk database[Footnote 50] and take 
ownership of the risk and the strategy for mitigating it. In addition, 
stakeholders can bring potential risks to the Program Manager's 
attention through interviews, where potential risks are considered and 
evaluated. 

* The program office has thus far identified and categorized individual 
risks. As of December 2007, the risk database contained 27 active 
risks--2 high, 15 medium, and 10 low.[Footnote 51] 

* Program risks are considered during program milestone reviews. 
Specifically, our review of documentation for the Design Readiness 
Review,[Footnote 52] a key decision point during the system development 
and demonstration phase leading up to a Milestone C decision, showed 
that key risks were discussed. Furthermore, the Program Manager reviews 
program risks' status through a risk watch list[Footnote 53] and 
bimonthly risk briefings. 

However, the program office has not consistently followed other aspects 
of its process. For example, it did not perform key practices for 
identifying and managing schedule risks, such as conducting a schedule 
risk assessment and building in reserve time to its schedule. In 
addition, mitigation steps for several key risks were either not 
performed in accordance with the risk management strategy, or risks 
that were closed as having been mitigated were later found to be actual 
program issues (i.e., problems). 

For 25 medium risks in the closed risk database, as of February 2008, 4 
were closed because mitigation steps were not performed in accordance 
with the strategy and the risks ultimately became actual issues. 
Examples from these medium risks are as follows: 

* In one case, the mitigation strategy was for the contractor to 
deliver certain design documents that were traced to system 
requirements and to do so before beginning the solution build phase. 
The design documents, however, were not received in accordance with the 
mitigation strategy. Specifically, program officials told us that the 
design documents contained inaccuracies or misinterpretations of the 
requirements and were not completed on time because of the lack of 
resources to correct these problems. As a result, the program 
experienced delays in completing its solution design activities. 

* In another case, the mitigation strategy included creating the 
documentation needed to execute the contract for monitoring the build 
phase activities. However, the mitigation steps were not performed due 
to, among other things, delays in approving the contractual approach. 
As a result, the risk became a high-priority issue in February 2008. 
According to a program issue report, the lack of a contract to monitor 
system development progress may result in unnecessary rework and thus 
additional program cost overruns, schedule delays, and performance 
shortfalls. 

Four of the same 25 medium risks were retired because key mitigation 
steps for each one were implemented, but the strategies proved 
ineffective, and the risks became actual program issues. Included in 
these 4 risks were the following: 

* In one case, the program office closed a risk regarding data exchange 
with another DON system because key mitigation steps to establish 
exchange requirements were fully implemented. However, in February 
2008, a high-priority issue was identified regarding the exchange of 
data with this system. According to program officials, the risk was 
mitigated to the fullest extent possible and closed based on the 
understanding that continued evaluation of data exchange requirements 
would be needed. However, because the risk was retired, this evaluation 
did not occur. 

* In another case, a requirements management risk was closed on the 
basis of having implemented mitigation steps, which involved 
establishing a requirements management process, including having 
complete requirements traceability spreadsheets. However, although 
several of the mitigation steps were not fully implemented, the risk 
was closed on the basis of what program officials described as an 
understanding reached with the contractor regarding the requirements 
management process. Several months later, a high-priority issue 
concerning requirements traceability was identified because the program 
office discovered that the contractor was not adhering to the 
understanding. 

Unless risk mitigation strategies are monitored to ensure that they are 
fully implemented and that they produce the intended outcomes, and 
additional mitigation steps are taken when they are not, the program 
office will continue to be challenged in preventing risks from 
developing into actual cost, schedule, and performance problems. 

Important Aspects of System Quality and Program Maturity Are Not Being 
Measured: 

Effective management of programs like GCSS-MC depends in part on the 
ability to measure the quality of the system being acquired and 
implemented. Two measures of system quality are trends in (1) the 
number of unresolved severe system defects and (2) the number of 
unaddressed high-priority system change requests. 

GCSS-MC documentation recognizes the importance of monitoring such 
trends. Moreover, the program office has established processes for (1) 
collecting and tracking data on the status of program issues, including 
problems discovered during early test events, and (2) capturing data on 
the status of requests for changes to the system. However, its 
processes do not provide the full complement of data that are needed to 
generate a reliable and meaningful picture of trends in these areas. In 
particular, data on problems and change request priority levels and 
closure dates are either not captured or not consistently maintained. 
Further, program office oversight of contractor-identified issues or 
defects is limited. Program officials acknowledged these data 
limitations, but they stated that oversight of contractor-identified 
issues is not their responsibility. Without tracking trends in key 
indicators, the program office cannot adequately understand and report 
to DOD decision makers whether GCSS-MC's quality and stability are 
moving in the right direction. 

Data to Understand Trends in Program Problems Are Limited: 

Program guidance and related best practices[Footnote 54] encourage 
trend analysis and the reporting of system defects and program problems 
as measures or indicators of system quality and program maturity. As we 
have previously reported,[Footnote 55] these indicators include trends 
in the number of unresolved problems according to their significance or 
priority. 

To the program office's credit, it collects and tracks what it calls 
program issues, which are problems identified by program office staff 
or the system integrator that are process, procedure, or management 
related. These issues are contained in the program's Issues-Risk 
Management Information System (I-RMIS). Among other things, each issue 
in I-RMIS is to have an opened and closed date and an assigned priority 
level of high, medium, or low. In addition, the integration contractor 
tracks issues that its staff identifies related to such areas as system 
test defects. These issues are contained in the contractor's Marine 
Corps Issue Tracking System (MCITS). Each issue in MCITS is to have a 
date when it was opened and is to be assigned a priority on a scale of 
1-5. According to program officials, the priority levels are based on 
guidance from the Institute of Electrical and Electronics Engineers 
(IEEE). (See table 6 for a description of each priority level.) 

Table 6: MCITS Priority Levels: 

Priority: 1; 
Description: 
* Prevents the accomplishment of an essential capability; 
* Jeopardizes safety, security, or other requirement designated 
critical. 

Priority: 2; 
Description: 
* Adversely affects the accomplishment of an essential capability, and 
no work-around solution is known; 
* Adversely affects technical, cost, or schedule risks to the project 
or to life cycle support of the system, and no work-around solution is 
known. 

Priority: 3; 
Description: 
* Adversely affects the accomplishment of an essential capability, but 
a work-around solution is known; 
* Adversely affects technical, cost, or schedule risks to the project 
or to life cycle support of the system, but a work-around solution is 
known. 

Priority: 4; 
Description: 
* Results in user/operator inconvenience or annoyance but does not 
affect a required operational or mission-essential capability; 
* Results in inconvenience or annoyance for development or maintenance 
personnel but does not prevent the accomplishment of the 
responsibilities of those personnel. 

Priority: 5; 
Description: 
* Any other effect. 

Sources: GCSS-MC and IEEE. 

[End of table] 

However, neither I-RMIS nor MCITS contain all the data needed to 
reliably produce key measures or indicators of system quality and 
program maturity. Examples of these limitations are as follows: 

* For I-RMIS, the program office has not established a standard 
definition of the priority levels used. Rather, according to program 
officials, each issue owner is allowed to assign a priority based on 
the owner's definition of what high, medium, and low mean. By not using 
standard priority definitions for categorizing issues, the program 
office cannot ensure that it has an accurate and useful understanding 
of the problems it is facing at any given time, and it will not know if 
it is addressing the highest priority issues first. 

* For MCITS, the integration contractor does not track closure dates 
for all issues. For example, as of April 2008, over 30 percent of the 
closed issues did not have closure dates. This is important because it 
limits the contractor's ability to understand trends in the number of 
high-priority issues that are unresolved. Program officials 
acknowledged the need to have closure dates for all closed issues and 
stated that they intend to correct this. If it is not corrected, the 
program office will not be able to create a reliable measure of system 
quality and program maturity. 

Compounding the above limitations in MCITS data is the program office's 
decision not to use contractor-generated reports that are based on 
MCITS data. Specifically, reports summarizing MCITS issues are posted 
to a SharePoint[Footnote 56] site for the program office to review. 
However, program officials stated that they do not review these reports 
because the MCITS issues are not their responsibility, but the 
contractor's. However, without tracking and monitoring contractor- 
identified issues, which include such things as having the right skill- 
sets and having the resources to track and monitor issues captured in 
separate databases, the program office is missing an opportunity to 
understand whether proactive action is needed to address emerging 
quality shortfalls in a timely manner. 

Data to Understand Trends in Changes to the System Are Limited: 

Program guidance and related best practices[Footnote 57] encourage 
trend reporting of change requests as measures or indicators of system 
stability and quality. These indicators include trends in the number 
and priority of approved changes to the system's baseline functional 
and performance capabilities that have yet to be resolved. 

To its credit, the program office collects and tracks changes to the 
system, which can range from minor or administrative changes to more 
significant changes that propose or impact important system 
functionality. These changes can be identified by either the program 
office or the contractor, and they are captured in a master change 
request spreadsheet. Further, the changes are to be prioritized 
according to the level described in table 7, and the dates that change 
requests are opened and closed are to be recorded. 

Table 7: Change Request Priorities: 

Priority: 1; 
Description: 
* Prevents the accomplishment of an essential capability; 
* Jeopardizes safety, security, or other requirement designated 
critical. 

Priority: 2; 
Description: 
* Adversely affects the accomplishment of an essential capability, and 
no work-around solution is known; 
* Adversely affects technical, cost, or schedule risks to the project 
or to life cycle support of the system, and no work-around solution is 
known. 

Priority: 3; 
Description: 
* Adversely affects the accomplishment of an essential capability, but 
a work-around solution is known; 
* Adversely affects technical, cost, or schedule risks to the project 
or to life cycle support of the system, but a work-around solution is 
known. 

Priority: 4; 
Description: 
* Results in user/operator inconvenience or annoyance but does not 
affect a required operational or mission-essential capability; 
* Results in inconvenience or annoyance for development or maintenance 
personnel but does not prevent the accomplishment of the 
responsibilities of those personnel. 

Priority: 5; 
Description: 
* Any other effect. 

Sources: GCSS-MC and IEEE. 

[End of table] 

However, the change request master spreadsheet does not contain the 
data needed to reliably produce key measures or indicators of system 
stability and quality. Examples of these limitations are as follows: 

* The program office has not prioritized proposed changes or managed 
these changes according to their priorities. For example, of the 572 
change requests as of April 2008, 171 were assigned a priority level, 
and 401 were not. Of these 171, 132 were categorized as priority 1. 
Since then, the program office has temporarily recategorized the 401 
change requests to priority 3 until each one's priority can be 
evaluated. The program office has yet to establish a time frame for 
doing so. 

* The dates that change requests are resolved are not captured in the 
master spreadsheet. Rather, program officials said that these dates are 
in the program's IMS and are shown there as target implementation 
dates. While the IMS does include the dates changes will be 
implemented, these dates are not actual dates, and they are not used to 
establish trends in unresolved change requests. 

Without the full complement of data needed to monitor and measure 
change requests, the program office cannot know and disclose to DOD 
decision makers whether the quality and stability of the system are 
moving in the right direction. 

Conclusions: 

DOD's success in delivering large-scale business systems, such as GCSS- 
MC, is in large part determined by the extent to which it employs the 
kind of rigorous and disciplined IT management controls that are 
reflected in DOD policies and related guidance. While implementing 
these controls does not guarantee a successful program, it does 
minimize a program's exposure to risk and thus the likelihood that it 
will fall short of expectations. In the case of GCSS-MC, living up to 
expectations is important because the program is large, complex, and 
critical to supporting the department's warfighting mission. 

The department has not effectively implemented a number of essential IT 
management controls on GCSS-MC, which has already contributed to 
significant cost overruns and schedule delays, and has increased the 
program's risk going forward of not delivering a cost-effective system 
solution and not meeting future cost, schedule, capability, and benefit 
commitments. Moreover, GCSS-MC could be duplicating the functionality 
of related systems and may be challenged in interoperating with these 
systems because compliance with key aspects of DOD's federated BEA has 
not been demonstrated. Also, the program's estimated return on 
investment, and thus the economic basis for pursing the proposed system 
solution, is uncertain because of limitations in how the program's cost 
estimate was derived, raising questions as to whether the nature and 
level of future investment in the program needs to be adjusted. In 
addition, the program's schedule was not derived using several key 
schedule estimating practices, which impacts the integrity of the cost 
estimate and precludes effective implementation of EVM. Without 
effective EVM, the program cannot reliably gauge progress of the work 
being performed so that shortfalls can be known and addressed early, 
when they require less time and fewer resources to overcome. Another 
related indicator of progress, trends in system problems and change 
requests, also cannot be gauged because the data needed to do so are 
not being collected. Collectively, these weaknesses have already helped 
to push back the completion of the program's first increment by more 
than 3 years and added about $193 million in costs, and they are 
introducing a number of risks that, if not effectively managed, could 
further impact the program. However, whether these risks will be 
effectively managed is uncertain because the program has not always 
followed its defined risk management process and, as a result, has 
allowed yesterday's potential problems to become today's actual cost, 
schedule, and performance problems. 

While the program office is primarily responsible for ensuring that 
effective IT management controls are implemented on GCSS-MC, other 
oversight and stakeholder organizations share some responsibility. In 
particular, even though the program office has not demonstrated its 
alignment with the federated BEA, it nevertheless followed established 
DOD architecture compliance guidance and used the related compliance 
assessment tool in assessing and asserting its compliance. The root 
cause for not demonstrating compliance thus is not traceable to the 
program office, but rather is due to, among other things, the 
compliance guidance and tool being limited, and the program's oversight 
entities not validating the compliance assessment and assertion. Also, 
even though the program's cost estimate was not informed by the cost 
experiences of other ERP programs of the same scope, the program office 
is not to blame because the root cause for this is that the Defense 
Cost and Resource Center has not maintained a standardized cost element 
structure for its ERP programs and a historical database of ERP program 
costs for program's like GCSS-MC to use. In contrast, other weaknesses 
are within the program office's control, as evidenced by its positive 
actions to address the requirements traceability shortcomings that we 
brought to its attention during of the course of our work and its well- 
defined risk management process. 

All told, this means that addressing the GCSS-MC IT management control 
weaknesses require the combined efforts of the various DOD 
organizations that share responsibility for defining, justifying, 
managing, and overseeing the program. By doing so, the department can 
better assure itself that GCSS-MC will optimally support its mission 
operations and performance goals and will deliver promised capabilities 
and benefits, on time and within budget. 

Recommendations for Executive Action: 

To ensure that each GCSS-MC system increment is economically justified 
on the basis of a full and reliable understanding of costs, benefits, 
and risks, we recommend that the Secretary of Defense direct the 
Secretary of the Navy to ensure that investment in the next acquisition 
phase of the program's first increment is conditional upon fully 
disclosing to program oversight and approval entities the steps under 
way or planned to address each of the risks discussed in this report, 
including the risk of not being architecturally compliant and being 
duplicative of related programs, not producing expected mission 
benefits commensurate with reliably estimated costs, not effectively 
implementing EVM, not mitigating known program risks, and not knowing 
whether the system is becoming more or less mature and stable. We 
further recommend that investment in all future GCSS-MC increments be 
limited if the management control weaknesses that are the source of 
these risks, and which are discussed in this report, have not been 
fully addressed. 

To address each of the IT management control weaknesses discussed in 
this report, we are also making a number of additional recommendations. 
However, we are not making recommendations for the architecture 
compliance weaknesses discussed in this report because we have a 
broader review of DON program compliance to the BEA and DON enterprise 
architecture that will be issued shortly and will contain appropriate 
recommendations. 

To improve the accuracy of the GCSS-MC cost estimate, as well as other 
cost estimates for the department's ERP programs, we recommend that the 
Secretary of Defense direct the appropriate organization within DOD to 
collaborate with relevant organizations to standardize the cost element 
structure for the department's ERP programs and to use this standard 
structure to maintain cost data for its ERP programs, including GCSS- 
MC, and to use this cost data in developing future cost estimates. 

To improve the credibility of the GCSS-MC cost estimate, we recommend 
that the Secretary of Defense direct the Secretary of the Navy, through 
the appropriate chain of command, to ensure that the program's current 
economic analysis is adjusted to reflect the risks associated with it 
not reflecting cost data for comparable ERP programs, and otherwise not 
having been derived according to other key cost estimating practices, 
and that future updates to the GCSS-MC economic analysis similarly do 
so. 

To enhance GCSS-MC's use of EVM, we recommend that the Secretary of 
Defense direct the Secretary of the Navy, through the appropriate chain 
of command, to ensure that the program office (1) monitors the actual 
start and completion dates of work activities performed so that the 
impact of deviations on downstream scheduled work can be proactively 
addressed; (2) allocates resources, such as labor hours and material, 
to all key activities on the schedule; (3) integrates key activities 
and supporting tasks and subtasks; (4) identifies and allocates the 
amount of float time needed for key activities to account for potential 
problems that might occur along or near the schedule's critical path; 
(5) performs a schedule risk analysis to determine the level of 
confidence in meeting the program's activities and completion date; (6) 
allocates schedule reserve for high-risk activities on the critical 
path; and (7) discloses the inherent risks and limitations associated 
with any future use of the program's EVM reports until the schedule has 
been risk-adjusted. 

To improve GCSS-MC management of program risks, we recommend that the 
Secretary of Defense direct the Secretary of the Navy, through the 
appropriate chain of command, to ensure that the program office (1) 
adds each of the risks discussed in this report to its active inventory 
of risks, (2) tracks and evaluates the implementation of mitigation 
plans for all risks, (3) discloses to appropriate program oversight and 
approval authorities whether mitigation plans have been fully executed 
and have produced the intended outcome(s), and (4) only closes a risk 
if its mitigation plan has been fully executed and produced the 
intended outcome(s). 

To strengthen GCSS-MC system quality measurement, we recommend that the 
Secretary of Defense direct the Secretary of the Navy, through the 
appropriate chain of command, to ensure that the program office (1) 
collects the data needed to develop trends in unresolved system defects 
and change requests according to their priority and severity and (2) 
discloses these trends to appropriate program oversight and approval 
authorities. 

Agency Comments and Our Evaluation: 

In written comments on a draft of this report, signed by the Deputy 
Under Secretary of Defense (Business Transformation) and reprinted in 
appendix II, the department stated that it concurred with two of our 
recommendations and partially concurred with the remaining five. In 
general, the department partially concurred because it said that 
efforts were either under way or planned that will address some of the 
weaknesses that these recommendations are aimed at correcting. For 
example, the department stated that GCSS-MC will begin to use a 
recently developed risk assessment tool that is expected to assist 
programs in identifying and mitigating internal and external risks. 
Further, it said that these risks will be reported to appropriate 
department decision makers. We support the efforts that DOD described 
in its comments because they are generally consistent with the intent 
of our recommendations and believe that if they are fully and properly 
implemented, they will go a long way in addressing the management 
control weaknesses that our recommendations are aimed at correcting. In 
addition, we have made a slight modification to one of these five 
recommendations to provide the department with greater flexibility in 
determining which organizations should provide for the recommendation's 
implementation. 

We are sending copies of this report to interested congressional 
committees; the Director, Office of Management and Budget; the 
Congressional Budget Office; the Secretary of Defense; and the 
Department of Defense Office of the Inspector General. We also will 
make copies available to others upon request. In addition, the report 
will be available at no charge on the GAO Web site at [hyperlink, 
http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-3439 or hiter@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. GAO staff who made major contributions to 
this report are listed in appendix III. 

Signed by: 

Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 

[End of section] 

Appendix I: Objective, Scope, and Methodology: 

Our objective was to determine whether the Department of the Navy is 
effectively implementing information technology management controls on 
the Global Combat Support System-Marine Corps (GCSS-MC). To accomplish 
this, we focused on the first increment of GCSS-MC relative to the 
following management areas: architectural alignment, economic 
justification, earned value management, requirements management, risk 
management, and system quality measurement. In doing so, we analyzed a 
range of program documentation, such as the acquisition strategy, 
program management plan, and Acquisition Program Baseline, and 
interviewed cognizant program officials. 

To determine whether GCSS-MC was aligned with the Department of 
Defense's (DOD) federated business enterprise architecture (BEA), we 
reviewed the program's BEA compliance assessments and system 
architecture products, as well as versions 4.0, 4.1, and 5.0 of the BEA 
and compared them with the BEA compliance requirements described in the 
Fiscal Year 2005 National Defense Authorization Act[Footnote 58] and 
DOD's BEA compliance guidance and evaluated the extent to which the 
compliance assessments addressed all relevant BEA products. We also 
determined the extent to which the program-level architecture 
documentation supported the BEA compliance assessments. We obtained 
documentation, such as the BEA compliance assessments from the GCSS-MC 
and Navy Enterprise Resource Planning programs, as well as the Air 
Force's Defense Enterprise Accounting and Management System and Air 
Force Expeditionary Combat Support System programs. We then compared 
these assessments to identify potential redundancies or opportunities 
for reuse and determined if the compliance assessments examined 
duplication across programs and if the tool that supports these 
assessments is being used to identify such duplication. In doing so, we 
interviewed program officials and officials from the Department of the 
Navy, Office of the Chief Information Officer, and reviewed recent GAO 
reports to determine the extent to which the programs were assessed for 
compliance against the Department of the Navy enterprise architecture. 
We also interviewed program officials and officials from the Business 
Transformation Agency and the Department of the Navy, including the 
logistics Functional Area Manager, and obtained guidance documentation 
from these officials to determine the extent to which the compliance 
assessments were subject to oversight or validation. 

To determine whether the program had economically justified its 
investment in GCSS-MC, we reviewed the latest economic analysis to 
determine the basis for the cost and benefit estimates. This included 
evaluating the analysis against Office of Management and Budget 
guidance and GAO's Cost Assessment Guide.[Footnote 59] In doing so, we 
interviewed cognizant program officials, including the Program Manager 
and cost analysis team, regarding their respective roles, 
responsibilities, and actual efforts in developing and/or reviewing the 
economic analysis. We also interviewed officials at the Office of 
Program Analysis and Evaluation and Naval Center for Cost Analysis as 
to their respective roles, responsibilities, and actual efforts in 
developing and/or reviewing the economic analysis. 

To determine the extent to which the program had effectively 
implemented earned value management, we reviewed relevant 
documentation, such the contractor's monthly status reports, 
Acquisition Program Baselines, and schedule estimates and compared them 
with DOD policies and guidance.[Footnote 60] We also reviewed the 
program's schedule estimates and compared them with relevant best 
practices[Footnote 61] to determine the extent to which they reflect 
key estimating practices that are fundamental to having a reliable 
schedule. In doing so, we interviewed cognizant program officials to 
discuss their use of best practices in creating the program's current 
schedule. 

To determine the extent to which the program implemented requirements 
management, we reviewed relevant program documentation, such as the 
baseline list of requirements and system specifications and evaluated 
them against relevant best practices[Footnote 62] to determine the 
extent to which the program has effectively managed the system's 
requirements and maintained traceability backward to high-level 
business operation requirements and system requirements, and forward to 
system design specifications, and test plans. To determine the extent 
to which the requirements were traceable, we randomly selected 61 
program requirements and traced them both backward and forward. This 
sample was designed with a 5 percent tolerable error rate at the 95 
percent level of confidence, so that, if we found 0 problems in our 
sample, we could conclude statistically that the error rate was less 
than 5 percent. Based upon the weight of all other factors included in 
our evaluation, our verification of 60 out of 61 requirements was 
sufficient to demonstrate traceability. In addition, we interviewed 
program officials involved in the requirements management process to 
discuss their roles and responsibilities for managing requirements. 

To determine the extent to which the program implemented risk 
management, we reviewed relevant risk management documentation, such as 
risk plans and risk database reports demonstrating the status of the 
program's major risks and compared the program office's activities with 
DOD acquisition management guidance[Footnote 63] and related best 
practices.[Footnote 64] We also reviewed the program's mitigation 
process with respect to key risks, including 25 medium risks in the 
retired risk database that were actively addressed by the program 
office, to determine the extent to which these risks were effectively 
managed. In doing so, we interviewed cognizant program officials 
responsible, such as the Program Manager, Risk Manager, and subject 
matter experts to discuss their roles and responsibilities and obtain 
clarification on the program's approach to managing risks associated 
with acquiring and implementing GCSS-MC. 

To determine the extent to which the program is collecting the data and 
monitoring trends in the number of unresolved system defects and the 
number of unaddressed change requests, we reviewed program 
documentation such as the testing strategy, configuration management 
policy, test defect reports, change request logs, and issue data logs. 
We compared the program's data collection and analysis practices 
relative to these areas to program guidance and best practices[Footnote 
65] to determine the extent to which the program is measuring important 
aspects of system quality. We also interviewed program officials such 
as system developers, relevant program management staff, and change 
control managers to discuss their roles and responsibilities for system 
quality measurement. 

We conducted our work at DOD offices and contractor facilities in the 
Washington, D.C., metropolitan area, and Triangle, Va., from June 2007 
to July 2008, in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objective. We 
believe that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objective. 

[End of section] 

Appendix II: Comments from the Department of Defense: 

Office Of The Under Secretary Of Defense: 
Acquisition Technology And Logistics: 
3000 Defense Pentagon: 
Washington, DC 20301-3000: 

July 21, 2008: 

Mr. Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 
U.S. Government Accountability Office: 
441 G Street, N.W. 
Washington, DC 20548: 

Dear Mr. Hite: 

This is the Department of Defense (DoD) response to the GAO draft 
report GAO-08-822, "DOD Business Systems Modernization: Key Marine 
Corps System Acquisition Needs to be Better Justified, Defined and 
Managed," dated June 9, 2008 (GAO Code 310648). Detailed comments on 
the report recommendations are enclosed. 

DoD concurred with two recommendations and partially concurred with the 
remaining five. Overall, the Department has already taken steps to 
address some of GAO's findings. including updating guidance for 
Business Enterprise Architecture (BEA) 5.0 compliance. Further, the 
Department's Business Capability Lifecycle (BCL) is expected to provide 
stakeholders at the program-, Component-, and governance-levels with 
additional visibility into program risks via the Enterprise Risk 
Assessment Methodology (ERAM) as well as further visibility into 
existing management control issues. BCL will also support the 
collection of cost data for Enterprise Resource Planning (ERP) for 
programs' future cost estimation purposes. 

We welcome the GAO's insight on the Department's progress with its 
business transformation efforts and continue to value our partnership. 

Sincerely, 

Signed by: 

Paul A. Brinkley: 
Deputy Under Secretary of Defense (Business Transformation): 

Enclosure: As stated: 

GAO DRAFT REPORT DATED JUNE 9, 2008: 
GAO-08-822 (GAO CODE 310648): 

DOD Business Systems Modernization: Key Marine Corps System Acquisition 
Needs To Be Better Justified, Defined, And Managed: 

Department Of Defense Comments To The GAO Recommendations: 

Recommendation 1: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy to ensure that investment in the next 
acquisition phase of the program's first increment is conditional upon 
fully disclosing to program oversight and approval entities the steps 
underway or planned to address each of the risks discussed in this 
report, including the risk of not being architecturally compliant and 
being duplicative of related programs, not producing expected mission 
benefits commensurate with reliably estimated costs, not effectively 
implementing earned value management (EVM), not mitigating known 
program risks, and not knowing whether the system is becoming more or 
less mature and stable. (Page 54/GAO Draft Report) 

DOD Response: Partially Concur. 

The Department is already taking steps to promote greater visibility 
into program risks through its implementation of the Business 
Capability Lifecycle (BCL). Per a May 18, 2007 memo from the Under 
Secretary of Defense for Acquisition, Technology and Logistics 
(AT&L)/Vice Chair of the Defense Business Systems Management Committee 
(DBSMC), the Global Combat Support Systems - Marine Corps (GCSS-MC) 
program is one of the programs that will begin to use the Enterprise 
Risk Assessment Methodology (ERAM) for the duration of its lifecycle, 
which will assist the program in identifying and mitigating internal 
and external risks. ERAM findings are shared with appropriate team 
members and decision makers within the Component as well as Investment 
Review Board (IRB) and DBSMC leadership, and programs utilize ERAM 
findings to create risk mitigation plans as needed. Given that future 
development of the GCSS-MC capabilities will be achieved via an 
incremental approach as part of the Department's BCE process, it is 
envisioned that any emerging risk issues will be identified and 
addressed. 

Additionally, the GCSS-MC Program Office is promoting greater 
visibility of risks at the program level by developing and using a 
database to track emerging risks for mitigation in future increments, 
and the program manager is kept apprised of the status of these risks 
through a watch list and regular briefings by the risk owners. 

The program is also taking several steps to ensure that the program is 
architecturally compliant in accordance with current DoD guidance and 
policies. As the Business Enterprise Architecture (BEA) evolves, the 
GCSS-MC program has continued to update system maps as required as part 
of the acquisition review process. Additionally, as part of the Marine 
Corps' overall Logistics Modernization effort, a Logistics Operational 
Architecture (LOG OA) was developed to guide future development of 
Logistics IT systems used by the Marine Air Ground Task Forces (MAGTFs) 
and the supporting establishment organizations. This architecture has 
been mapped to the BEA. the USTRANSCOM Deployment and Distribution 
Enterprise Architecture and efforts are ongoing to develop a Joint Army-
Marine Corps Integrated Architecture in support of logistics 
operations. GCSS-MC, as the centerpiece of the Logistics Modernization 
effort, has been aligned to the LOG OA to provide a basis for 
identifying capability gaps and overlaps in architecture that are not 
being used to develop future capability requirements. 

The Business Transformation Agency (BTA) will also continue to revise 
the BEA compliance guidance and further define program-level 
architecture products to provide the Components with clearer 
requirements. For example, the latest issuance of the BEA compliance 
guidance in May 2008 (attached) updated the BEA compliance requirements 
to enable Components to assert to the BEA 5.0 data synonym and 
attributes that were developed in support of Enterprise Data 
Initiatives, and to enable the IRBs to enforce program compliance to a 
more defined set of requirements that promote interoperability and data 
standardization. The new guidance document will better assist the 
Components as they map their programs to BEA 5.0 requirements in future 
increments. 

Recommendation 2: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy to ensure that investment in all 
future Global Combat Support Systems-Marine Corps (GCSS-MC) increments 
be limited if the management control weaknesses that arc the source of 
these risks, and which are discussed in this report, have not been 
fully addressed. (Page 54/GAO Draft Report) 

DOD Response: Partially concur. 

The Department is already taking steps to address some of the 
management control issues GAO noted in the report. As previously 
discussed, the BEA compliance guidance was recently updated to provide 
Components with additional clarity on requirements for asserting 
compliance to BEA 5.0. In addition, the implementation of BCL is 
expected to provide the program offices, Components, and IRB/DBSMC 
leadership with enhanced visibility of risks. This visibility will 
allow the Department to examine the root causes of identified risks and 
take action as needed to make improvements in its management controls 
as business transformation efforts continue. 

Going forward, the GCSS-MC Program Office will address the correction 
of all identified weaknesses prior to seeking investment decisions for 
future increments in accordance with all existing Department of the 
Navy and Department of Defense guidance and policies, including IRB 
policy and Guidance. 

Recommendation 3: The GAO recommends that the Secretary of Defense 
direct the Director of the Office of Program Analysis and Evaluation to 
ensure the Defense Cost and Resource Center, in collaboration with 
other relevant organizations, standardize the cost element structure 
for the department's Enterprise Resource Planning (ERP) programs and to 
use this standard structure to maintain cost data for its ERP programs, 
including GCSS-MC, and to use this cost data in developing future cost 
estimates. (Page 55/GAO Draft Report) 

DOD Response: Partially Concur. 

The Department concurs with GAO's recommendation that actual cost data 
for ERP programs should be maintained for use in developing future cost 
estimates. Assembling this data would be best achieved by taking steps 
to standardize the Work Breakdown Structure (WBS) for ERPs. The BTA 
will support the Office of AT&L/Acquisition Resources and Analysis 
(ARA), the organization responsible for issuing DoD's handbook on work 
breakdown structures for defense materiel items (MIL-HDBK-881 A), in 
developing the standard WBS for ERP systems. 

In addition, the Department intends to track and maintain cost data for 
ERPs through BCL's Integrated Management Information Environment 
(IMIE). Since BCL unifies reporting requirements of the Defense 
Acquisition System (DAS) 5000-series policies, the Chairman of the 
Joint Chiefs of Staff Instruction (CJCSI) 3170, and the IRB Concept of 
Operations (CONOPS) for business systems, the required information is 
presented to the IRB/DBSMC governance structure, including the 
program's economic analysis. As ERPs undergo the BCL process and more 
business cases are submitted to the IRBs, cost data will be captured in 
IMIE for use in developing future cost estimates, and in developing 
economic analysis guidance for both acquisition and Clinger-Cohen Act 
compliance decisions. 

Recommendation 4: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy, through the appropriate chain of 
command, to ensure that the program's current economic analysis is 
adjusted to reflect the risks associated with it not reflecting cost 
data for comparable ERP programs, and otherwise not having been derived 
according to other key costing estimating practices, and that future 
updates to the GCSSMC economic analysis similarly do so. (Page 55/GAO 
Draft Report) 

DOD Response: Partially concur 

To date, economic analyses for GCSS-MC have leveraged historical cost 
data from ERP programs; however this is not analogous historical cost 
data for large-scale ERP programs with similar architectural 
complexities. Per DoD's response to Recommendation 3, the Department 
intends to track and maintain cost data for ERPs through the BCL's 
IMIE. which will assist programs in developing future economic 
analyses, and a standard WBS for ERP systems will be developed by 
AT&L/ARA and BTA. 

Through the Naval Center for Cost Analysis (NCCA), the Department of 
Navy conducted a risk analysis of the GCSS-MC estimate, which included 
a schedule component. This information was provided as part of the data 
collected through NCCA, and cited in the Department of the Navy Cost 
Analysis Improvement Group (DON CAIG) memo.
Moving forward, the program will also include risk analysis, as a point 
comparison to that developed through the DON CAIG. The program is 
currently preparing a cost estimate in support of the GCSS-MC Block 1 
Milestone C and will ensure that these and future costs are adjusted to 
reflect risks associated with the lack of comparable historical data. 

Recommendation 5: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy, through the appropriate chain of 
command, to ensure that the program office: (1) monitors the actual 
start and completion dates of work activities performed so that the 
impact of deviations on downstream scheduled work can he proactively 
addressed; (2) allocates resources, such as labor hours and material, 
to all key activities on the schedule; (3) integrates key activities 
and supporting tasks and sub-tasks; (4) identities and allocates the 
amount of float time needed for key activities to account for potential 
problems that might occur along or near the schedule's critical path; 
(5) performs a schedule risk analysis to determine the level of 
confidence in meeting the program's activities and completion date; (6) 
allocates schedule reserve for high risk activities on the critical 
path; and (7) discloses the inherent risks and limitations associated 
with any future use of the program's EVM reports until the schedule has 
been risk-adjusted. (Page 55/GAO Draft Report) 

DOD Response: Concur. 

The Department's EVM policy recognizes that the preparation of an 
integrated master schedule (IMS) is a best practice, and requires the 
preparation of an IMS whenever EVM compliance is required. GCSS-MC is 
currently rebaselining its own integrated master schedule (IMS) to 
adjust for schedule risk. The rebaselined GCSS-MC IMS will be used to: 

* Provide a baseline for monitoring and reporting planned versus actual 
start and completion dates of work activities performed, so that the 
impact of deviations on downstream scheduled work can be proactively 
addressed; 

* Allocate resources to activities; 

* Integrate key activities and supporting tasks and sub-tasks; 

* Identify and allocate the amount of float time needed for key 
activities; 

* Allocate schedule reserve for high risk activities on the critical 
path. 

As the schedule is rebaselined, GCSS-MC will perform a schedule risk 
analysis to determine the level of confidence in meeting the program's 
activities and completion dates. The program will disclose the inherent 
risk and limitations associated with the program's EVM reports until 
the schedule has been rebaselined and risk adjusted. 

Recommendation 6: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy, through the appropriate chain of 
command, to ensure that the program office: (1) adds each of the risks 
discussed in this report to its active inventory of risks; (2) tracks 
and evaluates the implementation of mitigation plans for all risks; (3) 
discloses to appropriate program oversight and approval authorities 
whether mitigation plans have been fully executed and have produced the 
intended outcome(s); and (4) only closes a risk if its mitigation plan 
has been fully executed and produced the intended outcome(s). (Page 
55/GAO Draft Report) 

DOD Response: Partially concur. 

GCSS-MC has addressed the majority of these risk factors through the 
240-plus risks in the risk database, although not necessarily in the 
general wording of the GAO report, but instead has addressed them under 
a broader range of risk topics in details. The Program Office will, 
however, re-examine the risks in the database to ensure that the risks 
in this report are covered. In addition, program risks are discussed 
during program milestone reviews and regularly reviewed by the program 
manager. 

Recommendation 7: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy, through the appropriate chain of 
command, to ensure that the program office: (1) collects the data 
needed to develop trends in unresolved system defects and change 
requests according to their priority and severity; and (2) discloses 
these trends to appropriate program oversight and approval authorities. 

DOD Response: Concur. 

The GCSS-MC program office is currently working with an independent 
software measurement analysis company to collect the data needed to 
develop trends in unresolved system defects and change requests 
according to their priority and severity levels. Results will be 
reported to appropriate program oversight authorities on a monthly 
basis. 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Randolph C. Hite, (202) 512-3439, or hiter@gao.gov: 

Staff Acknowledgments: 

In addition to the individual named above, key contributors to this 
report were Neelaxi Lakhmani, Assistant Director; Monica Anatalio; 
Harold Brumm; Neil Doherty; Cheryl Dottermusch; Nancy Glover; Mustafa 
Hassan; Michael Holland; Ethan Iczkovitz; Anh Le; Josh Leiling; Emily 
Longcore; Lee McCracken; Madhav Panwar; Karen Richey; Melissa 
Schermerhorn; Karl Seifert; Sushmita Srikanth; Jonathan Ticehurst; 
Christy Tyson; and Adam Vodraska. 

[End of section] 

Footnotes: 

[1] Business systems are information systems, including financial and 
nonfinancial systems, that support DOD business operations, such as 
civilian personnel, finance, health, logistics, military personnel, 
procurement, and transportation. 

[2] GAO, High-Risk Series: An Update, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-07-310] (Washington, D.C.: 
January 2007). 

[3] DOD defines logistics as the science of planning and carrying out 
the movement and maintenance of forces. Logistics includes the aspects 
of military operations that deal with: (1) design and development, 
acquisition, storage, movement, distribution, maintenance, evacuation, 
and disposition of materiel; (2) movement, evacuation, and 
hospitalization of personnel; (3) acquisition or construction, 
maintenance, operation, and disposition of facilities; and (4) 
acquisition or furnishing of services. 

[4] GCSS-MC was formally designated a Major Automated Information 
System Acquisition program in July 2004, which is a program or 
initiative that is so designated by the Assistant Secretary of Defense 
(Networks and Information Integration)/Chief Information Officer or 
that is estimated to require program costs in any single year in excess 
of $32 million (fiscal year 2000 constant dollars), total program costs 
in excess of $126 million (fiscal year 2000 constant dollars), or total 
life cycle costs in excess of $378 million (fiscal year 2000 constant 
dollars). 

[5] A garrison location is a home station, usually in the continental 
United States, for a unit that is not deployed. 

[6] An ERP solution is commercial off-the-shelf software package 
consisting of multiple, integrated functional modules that perform a 
variety of business related tasks, such as payroll, general ledger 
accounting, and supply chain management. 

[7] The current life cycle cost estimate is from fiscal year 2007 
through fiscal year 2019, in base year 2007 dollars, and excludes costs 
associated with supporting and maintaining legacy systems during GCSS- 
MC development, totaling $8.2 million, and does not reflect program 
costs of $61.4 million from fiscal year 2004 through fiscal year 2006 
that are considered sunk costs. According to the Office of Management 
and Budget, sunk costs are those incurred in the past that will not be 
affected by any present or future decision and should be ignored when 
determining whether a new investment is worthwhile. 

[8] DAS is a framework-based approach that is intended to translate 
mission needs and requirements into stable, affordable, and well- 
managed acquisition programs. 

[9] E-business Suite is a Web-based software system consisting of 
various software (packaged software, applications server, Web server, 
database server and administrative software) and hardware (servers, 
switches, storage, and ancillary equipment) to support the computing 
environment. 

[10] Time-and-materials contracts provide for acquiring supplies or 
services on the basis of (1) direct labor hours at specified fixed 
hourly rates that include wages, overhead, general and administrative 
expenses, and profit and (2) actual cost for materials. 

[11] Fixed-price contracts with award fees include a fixed priced 
(including normal profit) and an additional, separate award fee amount. 
The fixed price is paid for satisfactory performance; the award fee, if 
any, is earned based on periodic evaluation by the government. 

[12] FOC means that the system has been deployed in all intended 
locations. 

[13] According to the May 10, 2004, analysis of alternatives, this 
estimate was a "rough order of magnitude" for research and development, 
procurement and operations and support from fiscal years 2004 through 
2011. 

[14] According to the July 15, 2005, economic analysis, program costs 
are estimated from fiscal years 2005 through 2018, in base year 2005 
dollars, and exclude $9.6 million associated with supporting and 
maintaining legacy systems during GCSS-MC development and $11.9 million 
in fiscal year 2004 sunk costs. 

[15] Donald E. Harter, Mayuram S. Krishnan, and Sandra A. Slaughter, 
"Effects of Process Maturity on Quality, Cycle Time, and Effort in 
Software Product Development," Management Science, 46, no. 4, 2000; and 
Bradford K. Clark, "Quantifying the Effects of Process Improvement on 
Effort," IEEE Software (November/December 2000). 

[16] GAO, Information Technology: DOD's Acquisition Policies and 
Guidance Need to Incorporate Additional Best Practices and Controls, 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-722] (Washington, 
D.C.: July 30, 2004). 

[17] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-722]. 

[18] See, for example, GAO, DOD Business Transformation: Lack of an 
Integrated Strategy Puts the Army's Asset Visibility System Investments 
at Risk, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-860] 
(Washington, D.C.: July 27, 2007); GAO, Information Technology: DOD 
Needs to Ensure That Navy Marine Corps Intranet Program Is Meeting 
Goals and Satisfying Customers, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-07-51] (Washington, D.C.: Dec. 8, 2006); GAO, Defense 
Travel System: Reported Savings Questionable and Implementation 
Challenges Remain, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-
980] (Washington, D.C.: Sept. 26, 2006); GAO, DOD Systems 
Modernization: Uncertain Joint Use and Marginal Expected Value of 
Military Asset Deployment System Warrant Reassessment of Planned 
Investment, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-171] 
(Washington, D.C.: Dec. 15, 2005); and GAO, DOD Systems Modernization: 
Planned Investment in the Navy Tactical Command Support System Needs to 
Be Reassessed, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-
215] (Washington, D.C.: Dec. 5, 2005). 

[19] Ronald W. Reagan National Defense Authorization Act for Fiscal 
Year 2005, Pub. L. No. 108-375, § 332 (2004) (codified at 10 U.S.C. §§ 
186 and 2,222). 

[20] Field/tactical refers to Army units that are deployable to 
locations around the world, such as Iraq or Afghanistan. 

[21] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-860]. 

[22] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-215]. 

[23] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-171]. 

[24] Department of Defense Architecture Framework, Version 1.0, Volume 
1 (February 2004); GAO, Information Technology: A Framework for 
Assessing and Improving Enterprise Architecture Management (Version 
1.1), [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-584G] 
(Washington, D.C.: April 2003); Chief Information Officer Council, A 
Practical Guide to Federal Enterprise Architecture, Version 1.0 
(February 2001); and Institute of Electrical and Electronics Engineers 
(IEEE), Standard for Recommended Practice for Architectural Description 
of Software-Intensive Systems 1471-2000 (Sept. 21, 2000). 

[25] A well-defined enterprise architecture provides a clear and 
comprehensive picture of an entity, whether it is an organization 
(e.g., a federal department) or a functional or mission area that cuts 
across more than one organization (e.g., personnel management). This 
picture consists of snapshots of both the enterprise's current or "As 
Is" environment and its target or "To Be" environment, as well as a 
capital investment road map for transitioning from the current to the 
target environment. These snapshots consist of integrated "views," 
which are one or more architecture products that describe, for example, 
the enterprise's business processes and rules; information needs and 
flows among functions, supporting systems, services, and applications; 
and data and technical standards and structures. 

[26] DOD has adopted a federated approach for developing its business 
mission area enterprise architecture, which includes the corporate BEA 
representing the thin layer of DOD-wide corporate architectural 
policies, capabilities, rules, and standards; component architectures 
(e.g., DON enterprise architecture); and program architectures (e.g., 
GCSS-MC architecture). 

[27] See, for example, GAO, Information Technology: FBI Is Taking Steps 
to Develop an Enterprise Architecture, but Much Remains to Be 
Accomplished, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-363] 
(Washington, D.C.: Sept. 9, 2005); GAO, Homeland Security: Efforts 
Under Way to Develop Enterprise Architecture, but Much Work Remains, 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-777] (Washington, 
D.C.: Aug. 6, 2004); GAO, Information Technology: Architecture Needed 
to Guide NASA's Financial Management Modernization, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-04-43] (Washington, D.C.: Nov. 
21, 2003); GAO, DOD Business Systems Modernization: Important Progress 
Made to Develop Business Enterprise Architecture, but Much Work 
Remains, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-1018] 
(Washington, D.C.: Sept. 19, 2003); GAO, Information Technology: DLA 
Should Strengthen Business Systems Modernization Architecture and 
Investment Activities, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-01-631] (Washington, D.C.: June 29, 2001); and GAO, 
Information Technology: INS Needs to Better Manage the Development of 
Its Enterprise Architecture, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO/AIMD-00-212] (Washington, D.C.: Aug. 1, 2000). 

[28] DOD, Business Enterprise Architecture Compliance Guidance (April 
10, 2006). 

[29] Navy Enterprise Resource Planning was initiated in 2003 to 
modernize and standardize certain Navy business operations, such as 
financial, acquisition, workforce management, supply, and maintenance. 
Moreover, according to program officials, both Navy Enterprise Resource 
Planning and GCSS-MC are under the leadership of Navy's Program 
Executive Office for Enterprise Information Systems, which is 
responsible for developing, acquiring and deploying seamless enterprise-
wide IT systems. 

[30] GAO, DOD Business Systems Modernization: Progress in Establishing 
Corporate Management Controls Needs to Be Replicated Within Military 
Departments, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-705]. 
(Washington, D.C.: May 15, 2008). 

[31] As we recently reported, while the corporate BEA includes 
corporate policies, capabilities, rules, and standards, it is still 
evolving and will continue to add additional details. See GAO-08-705. 

[32] Functional Area Managers are responsible for reducing the number 
of DON applications within specific portfolios, such as logistics and 
financial management, and reviewing program-level compliance assertions 
before they are submitted to the DON CIO. 

[33] Department of Defense, Business Transformation Guidance. Version 
1.1. (July 2007). 

[34] Office of Management and Budget, Circular No. A-11, Preparation, 
Submission, and Execution of the Budget (Washington, D.C.: Executive 
Office of the President, June 2006); Circular No. A-130 Revised, 
Management of Federal Information Resources (Washington, D.C.: 
Executive Office of the President, Nov. 28, 2000); and Capital 
Programming Guide: Supplement to Circular A-11, Part 7, Preparation, 
Submission, and Execution of the Budget (Washington, D.C.: Executive 
Office of the President, June 2006). 

[35] GAO, Cost Assessment Guide: Best Practices for Estimating and 
Managing Program Costs, Exposure Draft, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP] (Washington, D.C.: 
July 2007). 

[36] Sensitivity analysis reveals how the cost estimate is affected by 
a change in a single assumption or cost driver at a time while holding 
all other variables constant. 

[37] Office of Management and Budget, Circular No. A-94, Guidelines and 
Discount Rates for Benefit-Cost Analysis of Federal Programs (Oct. 29, 
1992). 

[38] A benefit-to-cost ratio is calculated by dividing the present 
value of benefits by the present value of costs, while net present 
value is the present value of benefits minus the present value of 
costs. Present value is the worth of a future stream of costs or 
benefits in terms of money paid or received immediately. Prevailing 
interest rates provide the basis for converting future amounts into 
equivalent present amounts. These interest rates serve as discount 
rates in the discounting process--in the calculation of present values. 

[39] A Monte Carlo simulation allows the model's parameters to vary 
simultaneously according to their associated probability distribution. 
The result is a set of estimated probabilities of achieving alternative 
outcomes, given the uncertainty in the underlying parameters. 

[40] A firm-fixed-price contract provides for a price that is not 
subject to any adjustment on the basis of the contractor's cost 
experience in performing the contract. This contract type places upon 
the contractor maximum risk and full responsibility for all costs and 
resulting profit or loss. It provides maximum incentive for the 
contractor to control costs and perform effectively and imposes a 
minimum administrative burden upon the contracting parties. 

[41] Defense Contract Management Agency, Department of Defense Earned 
Value Management Implementation Guide, (Washington, D.C.: October 
2006). See also DOD Memorandum: Revision to DOD Earned Value Management 
Policy (Washington, D.C.: Mar. 7, 2005). 

[42] According to DOD guidance, EVM is generally not encouraged for 
firm-fixed-price and time-and-material contracts, except in certain 
situations, as determined by the complexity of the contracted effort 
and the inherent risks involved. 

[43] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP]. 

[44] Float time is the amount of time an activity can slip before 
affecting the critical path. 

[45] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004). 
Software Engineering Institute, Software Acquisition Capability 
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA: 
March 2002). 

[46] Work products are the contractor's standardized business 
procedures and related test cases, as well as customized design 
specifications and test cases for requirements that cannot be met with 
standardized products, such as for special reports and external 
interfaces. 

[47] Department of Defense, Risk Management Guide for DOD Acquisition, 
6th Edition, Version 1.0, [hyperlink, 
http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-
version.pdf] (accessed March 13, 2008). 

[48] Software Engineering Institute, CMMI for Acquisition, Version 1.2, 
CMU/SEI-2007-TR-017 (Pittsburgh, PA: November 2007). 

[49] The risk team consists of (1) a senior risk advisor who provides 
risk-related consultation to management, (2) a risk manager who 
provides general process and quality assurance support for risk 
management, and (3) a risk database administrator who issues risk 
reports and maintains a risk database. 

[50] The database includes those active risks that continue to require 
attention and closed risks that have either been addressed or have 
become actual problems. The database also provides information such as 
the risk owner, current risk level, likelihood of occurrence, and 
potential impact to the program's cost, schedule, and performance 
commitments. 

[51] Risk levels of high, medium, low are assigned using quantitative 
measurements of the probability of the risk occurring and the potential 
impact to the program's cost, schedule, and performance. Based on that 
assessment, a risk level is assigned to represent the risk's 
significance. High risks represent the greatest significance, medium 
risks represent moderate significance, and low risks represent the 
least significant risks. 

[52] This review determines whether the system's design is mature and 
stable. 

[53] The Program Manager's watch list is a report that captures the 
status, mitigation plans, and trends for all high risks and those 
medium risks that need to be elevated to senior managers. 

[54] GAO, Year 2000 Computing Crisis: A Testing Guide, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-10.1.21] (Washington, D.C.: 
November 1998); and IEEE Std 12207-2008, Systems and software 
engineering-Software life cycle processes (Piscataway, NJ: 2008). 

[55] See, for example, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-06-215]. 

[56] An integrated software suite used to facilitate collaboration, 
document management, and access to information between the government 
and contractor. 

[57] IEEE Std 12207-2008, Systems and software engineering-Software 
life cycle processes (Piscataway, NJ: 2008). 

[58] Ronald W. Reagan National Defense Authorization Act for Fiscal 
Year 2005, Pub. L. No. 108-375, § 332 (2004) (codified at 10 U.S.C. §§ 
186 and 2,222). 

[59] Office of Management and Budget, Circular No. A-94, Guidelines and 
Discount Rates for Benefit-Cost Analysis of Federal Program (Oct. 29, 
1992); GAO, Cost Assessment Guide: Best Practices for Estimating and 
Managing Program Costs, Exposure Draft, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP] (Washington, D.C.: 
July 2007). 

[60] Defense Contract Management Agency, Department of Defense Earned 
Value Management Implementation Guide, (Washington, D.C.: October 
2006). See also DOD Memorandum: Revision to DOD Earned Value Management 
Policy, (Washington, D.C.: Mar. 7, 2005). 

[61] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP]. 

[62] Software Engineering Institute, Software Acquisition Capability 
Maturity Model® version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA: 
March 2002). 

[63] Department of Defense, Risk Management Guide for DOD Acquisition, 
6th Edition, Version 1.0, [hyperlink, 
http://www.acq.osd.mil/sse/ed/docs/2006-RM-Guide-4Aug06-final-
version.pdf] (accessed March 13, 2008). 

[64] Software Engineering Institute, CMMI for Acquisition, Version 1.2, 
CMU/SEI-2007-TR-017 (Pittsburgh, PA: November 2007). 

[65] GAO, Year 2000 Computing Crisis: A Testing Guide, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-10.1.21] (Washington, D.C.: 
November 1998); and IEEE Std 12207-2008, Systems and software 
engineering-Software life cycle processes (Piscataway, NJ: 2008). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: