Reports and Publications

Policy and Guidance

Careers in Auditing

Contact Us

Audit Home

A Guide to Auditing Defense Acquisition Programs Critical Program Management Elements, September 1998

Prepared by: Acquisition and Technology Management Directorate, Office of the Assistant Inspector General for Auditing, Department of Defense

|What's New

TRAINING AID FOR DOD AUDITORS IN PLANNING,
EXECUTING, AND REPORTING FOR ACQUISITION AUDITS, August 2005

Table of Contents

Background   TOC

      In June 1988, the Inspector General, DoD, implemented a standard approach to performing audits of major acquisition programs. We refer to audits accomplished using that approach as Critical Program Management Elements audits or PME audits. Initially, the overall objective of a PME audit was to determine whether a major acquisition program was on the right track.

      In July 1993, we updated the PME audit guide to reflect changes to the DoD Acquisition Directives and Instructions (the DoD 5000 series), which were revised in February 1991. In March 1996, OUSD(A&T) again revised the DoD 5000 series. This revision to the PME approach reflects the latest issuance of the DoD 5000 series.

DoD 5000 Series   TOC The latest issuance of the 5000 series incorporates acquisition reform initiatives and new laws and policies that have been enacted since the last update. Those laws and initiatives include Federal Acquisition Streamlining Act (FASA) of 1994 and the institutionalization of Integrated Product Teams (IPTs). The revised 5000 series also incorporates automated information system life-cycle management policy and procedures, which were previously covered in the DoD Directive 8120.1 and DoD Instruction 8120.2.

      The revised 5000 series separates mandatory policies and procedures from discretionary practices. DoD Directive 5000.1 establishes guiding principles for all Defense acquisitions. DoD Regulation 5000.2-R specifies mandatory policies and procedures for Major Defense Acquisition Programs (MDAPs) and Major Automated Information Systems (MAISs), and specifically where stated, for other than MDAPs or MAISs. DoD Regulation 5000.2-R also serves as a general model for other than MDAPs and MAISs. The Defense Acquisition Deskbook describes the discretionary information to which program managers and other participants in the Defense acquisition process can turn for assistance in implementing guiding principles and mandatory procedures.

Objectives of a PME Audit. TOC As mentioned above, the objective of the original PME guide was to determine whether a major acquisition program was on the right track. While that is still an important objective, our objectives now revolve around identifying and focusing on the high risk areas or areas of special interest of a particular program. We no longer have the luxury of expending extensive audit resources on all critical program management elements of a program. In the past, we performed detailed audit work on all program management elements and reported on each element in the report. In an effort to streamline and make the acquisition audit process more responsive, the audit team must tailor its guide.

      Our objectives also include determining whether a program is implementing smart business practices and is making full use of the flexibilities available through acquisition reform initiatives. The Acquisition Reform Implementation section of this guide contains suggested audit steps specifically geared to gauging the status of implementation of Acquisition Reform efforts within a program. Those steps and other segments of this guide address many of the major focus areas that OUSD(A&T) set forth as underlying Defense Acquisition Reform. In addition, we continue to look for "best practices" or those things that a program is doing especially well that could be held up as a model or a lesson learned for other programs. We hope that this guide will help us in obtaining the above objectives and in further establishing credibility with our audit clients.

Preparing for a PME Audit TOC Thorough preparation is essential to the effective execution of a PME audit. Excellent preparation will save audit and client resources, and enhance our credibility with our audit clients. Before initiating a PME audit, the audit team should take several steps to ensure that the audit objectives will be accomplished effectively and efficiently. Those steps include meeting with key program officials, reviewing program assessments that various organizations have done on the program, and reviewing program reports and other information.

      Key Officials. TOC Several months before the initiation of the audit, the audit program director or audit project manager should meet with the program manager, the program executive officer, and the OSD action officer for the program. That meeting will start a dialogue on how the audit team can make the audit as useful as possible and focus on areas in which they can add the most value. The OSD Comptroller point of contact for the program is another excellent individual to meet with before or in the early stages of the audit survey. That individual frequently has an understanding of the key issues that the program is facing.

      Program Assessments. TOC Before initiating an audit, the audit team should become familiar with all assessments that other organizations have done. We discuss some of those key assessments below:

•   Defense Acquisition Board (DAB). The DAB is the DoD senior level forum for advising the USD(A&T) on critical decisions regarding ACAT ID programs. The auditors should determine when the next DAB review is planned and obtain the Acquisition Decision Memorandum (ADM) from the last DAB review. In addition, the auditors should obtain the "Blue Book" from the last DAB review. The "Blue Book" will set forth the baseline established for the program at the last milestone review and also issues and problems that were raised at the review.
•   OIPT Integrated Assessment. The OIPT leader for ACAT ID programs provides an integrated assessment to the DAB at major program reviews and milestone decision reviews using information gathered through the IPT process. It focuses on core acquisition management issues and takes account of independent assessments that OIPT members normally prepare. The auditors should obtain the OIPT integrated assessment from the last DAB or milestone review.
•   Joint Requirements Oversight Council Review (JROC) Procedures. The JROC reviews all deficiencies that may necessitate development of major systems before any consideration by the DAB at Milestone 0. The JROC validates an identified mission need, assigns a joint potential designator for meeting the need, and forwards the Mission Need Statement (MNS) with JROC recommendations to USD(A&T). The auditors should obtain a copy of the MNS and determine whether the JROC is playing a continuing role in validating key performance parameters in program baselines before a DAB.

 Periodic Reports. TOC Before initiating the audit, the audit team should also review key program documentation and reports. Part 6, DoD 5000.2-R, describes mandatory reports that must be prepared periodically to provide acquisition executives and Congress with adequate information to oversee the acquisition process and make necessary decisions. The Defense Acquisition Executive Summary (DAES) is a key report for gaining an understanding of program issues and concerns and is the principal mechanism for tracking programs between milestones reviews. The DAES is the vehicle for reporting program assessments, unit cost, current estimates of the APB parameters, status reporting exit criteria, and vulnerability assessments. The DAES and OSD DAES Assessments, which contains the OSD assessment of the DAES, will give key indicators of program progress and issues by function from both the Service Program Manager and OSD analyst perspective. Copies of these reports can be obtained at the initial meetings with key program officials. Auditors should analyze the DAES information and interview the functional OSD analysts concerning a program before visiting the program office during the audit. That will enable the auditor to focus on potential issue areas and a more "tailored" application of the audit approach. See Periodic Reporting section of this guide for other reports to ask for and audit steps relating to the reports.

      The discretionary section of the Defense Acquisition Deskbook has a table that lists mandatory as well as discretionary items for each milestone. Those tables will be helpful in determining what program information should be available. When assigned areas in a PME audit to review, the auditor should thoroughly review the relevant sections of the Deskbook. The Deskbook also provides access to the most current directives and references.

Engaging Integrated Product Teams    TOC

      Early in the survey phase of a PME audit, the audit team should become familiar with the IPT-structure within the program. IPTs are now an integral part of the Defense acquisition oversight and review process. Engaging the IPT early in the audit process will help the audit team identify critical issues and also issues that are already being effectively worked. Several types of IPTs exist. For each ACAT ID and IAM program, the Overarching IPT (OIPT) provides assistance, oversight, and review as the program proceeds through the acquisition life-cycle. The Working-Level IPTs (WIPTs) focus on a particular topic, such as cost/performance, testing, or contracting. An Integrating IPT (IIPT) will coordinate WIPT efforts and cover all topics not otherwise assigned to another IPT. The IPT section of this guide sets forth possible audit steps to use in evaluating how well the IPT process is working for a program.

Conducting a PME Audit   TOC

      This PME audit manual is a general approach for audits of acquisition programs. It should not be construed as a checklist or as the audit guide for a particular system. Rather, this manual is simply a tool that the audit team can use to facilitate identifying those key high risk areas of an acquisition program needing audit or review and in developing a "tailored" audit guide for a specific system. The audit team should develop a survey program and subsequent audit program if necessary, based on this manual information known about the program, prior audit experience or similar programs or topics, and any changes in DoD or Military Department policies, procedures and practices made since this manual was published.

      We grouped the following steps into categories that, for the most part, parallel the categories identified in the DoD 5000 Series. In order to streamline our audits as much as possible, we realize that we must apply "risk management" to the audit process. Auditors should use the information obtained about the program in the research and preannouncement efforts to assist them in determining whether significant problems may exist in a particular area and in determining how much effort should be put into the area. All audit steps are by no means required for all systems. We are no longer required to review and comment on all areas of a program. Instead, we want to focus on those areas of a program that are experiencing problems and in which we can add the most value. The auditor's judgment will come to play in determining what steps to perform for a particular program. Also, certain steps may only be applicable to Major Defense Acquisition Programs (MDAPs).

      Unlike previous editions of the PME guide, this guide does not have separate sections for the various acquisition phases. However, if a step is only applicable to a particular acquisition phase (e.g., engineering and manufacturing development), we have indicated that in parenthesis after the step. Auditors are cautioned to remember that the response to a particular audit step will be different depending on what acquisition phase the system is in. For example, a program in program definition will have a logistics plan that is not as detailed as a program in the production phase. This guide is also being maintained "on-line" so that it can be continually revised as auditors make suggestions for improving PME audits.

Reporting on a PME Audit   TOC

      The reporting process for a PME has also changed. In order to be more responsive to our audit clients, we want to strive to report out on significant issues as we find them so that management can take appropriate action as soon as possible. In the past, we frequently waited and issued a single report with multiple findings at the end of a 10 or 11-month time period. Just as DoD cannot afford a 15-year acquisition cycle when the comparable acquisition cycle in the commercial sector is 3 to 4 years, we can no longer wait a year before issuing an audit report that addresses significant issues.

Electronic Version of PME Guide  TOC

An electronic version of the PME Guide is available at the Inspector General, DoD website at www.dodig.mil/audit/pmeguide.html.

Lessons Learned and Suggestions for Updates, Changes and Corrections  TOC

Each audit will result in information, ideas, methodologies, etc that others can use in performing their audits. A lessons learned feedback form is available [temporarily off-line]. Anyone who has lessons learned or other ideas that might be of interest can electronically submit them. This can include suggested problem areas or other issues that for some reason were not in the report of draft policy changes. Also, suggestions to make this manual more responsive to the auditor's need can be submitted on the feedback form.

Other Audits   TOC

      Acquisition audit reports that have issues similar to ones being addressed during a PME audit are a good source of information. Links are being provided to existing audit reports with findings related to specific PME topics.

PROGRAM DEFINITION   TOC

The purpose of program definition is to translate broadly stated mission needs into operational requirements from which specific performance specifications are derived. Acquisition programs may be initiated in response to specific military threat, economic benefits, new technological opportunities or other considerations. The audit steps in this section will help to ensure that programs are well-defined and carefully structured in order to obtain a balance of cost, schedule, and performance; available technology; and affordability constraints. The purpose of reviewing program definition is to:

• Determine whether the program was initiated to satisfy a specific military threat, economic benefits, new technological opportunities, or for other considerations.
• Evaluate that deficiencies in the current capability exist and that non-material changes would not adequately correct the deficiencies.
• Validate that the program performed an analysis of alternatives.
• Review program stability through assessment of program affordability and determination of affordability constraints.
• Ensure that the system program definition clearly assessed the system's operational effectiveness, operational suitability, and life-cycle cost reduction.

Intelligence Support: TOC Intelligence support of Defense acquisitions will be prepared and maintained throughout the life of the system. Validation of intelligence documentation is required to ensure that a system is able to satisfy the mission in its intended operational capacity throughout its expected life. DoD Regulation 5000.2-R states that acquisition programs initiated to satisfy a specific military threat shall be based on authoritative, current, and projected threat information.

• Has the environment changed significantly - enough to call into question the need for the program?
• Was an assessment done identifying the operational threat environment, the threat to be countered, the system specific threat, reactive threat, and technologically feasible threat?

Requirements Evolution: TOC DoD Regulation 5000.2-R requires the DoD Components to document deficiencies in current capabilities and to develop a mission need statement (MNS) that the new capabilities will provide. System performance objectives and thresholds will be developed from and remain consistent with the initial broad statements of operational capability. DoD Regulation 5000.2-R states that requirements shall be refined at successive milestone decision points, as a consequence of cost-schedule performance trade-offs during each phase of the acquisition process.

Analysis of Alternatives: TOC DoD Regulation 5000.2-R requires that an analysis of alternatives be prepared and considered at appropriate decision reviews.

• ACAT 1 Programs - at program initiation DoD Component Head (usually Milestone 1)
• ACAT 1A Programs - shall be prepared by the PSA for consideration at Milestone 0, and
• ACAT 1D and ACAT 1AM Programs - the DoD Component Head or designated official shall coordinated with USD (A&T) or USD (C3I) staff, JCS, or PSA staff, and Director, PA&E.

Affordability: TOC Affordability is the ongoing assessment of a program to ensure that it is being executed within DoD planning and funding guidelines, has sufficient resources identified and approved in the Future Years Defense Program (FYDP), and is managed based on accurate cost and manpower data. Affordability decisions are made throughout the entire acquisition cycle. DoD Regulation 5000.2-R states the assessment of the program affordability and determination of affordability constraints is intended to ensure greater program stability.

Supportability: TOC DoD Regulation 5000.2-R states that supportability is far more than the logistics elements.

PROGRAM STRUCTURE   Table of Contents

Program structure identifies management elements that are necessary to structure a sound, successful program. The elements address what the program will achieve (program goals); how the program will be developed and/or procured (acquisition strategy); how the program will be evaluated against what was intended (test and evaluation); and what resources will be needed for the program (life-cycle resource estimates). Properly tailored program strategies form the basis for sound management and include innovative ways to achieve program success.

Defining the elements necessary to structure a successful weapons system program can be accomplished by:

• establishing program goals,
• formulating a program development and production acquisition strategy to execute goals,
• measuring program success and progress by evaluating test results, and
• estimating life-cycle resources that concurrently result in an effective and efficient product

Program Goals: TOC DoD Regulation 5000.2-R Policy states that program goals identified as thresholds and objectives shall be established for the minimum number of cost, schedule, and performance parameters that describe the program

For years the non-defense sector has successfully developed and produced high-quality products that fully meet or exceed customer needs, while also meeting specific, predetermined cost targets for those products. The thrust of "cost as an independent variable" (CAIV) adapts these successful practices to meet DoD needs.

The CAIV process recognizes that needed military capabilities are, in most cases, best expressed as end results, which actually are the aggregate combination of numerous more detailed, parameters. With rare exception, there are multiple sets of detailed specifications that can be combined to attain the desired end results, so that any one item can be varied significantly so long as compensating adjustments are made else where in the system.

• Determine who established the thresholds and objectives. Were they user sanctioned? Were the thresholds and objectives independently validated?

• Does the APB comply with the DoD Regulation 5000.2-R guidance for cost, schedule, and performance and is it supportable?
• How many times has the baseline been modified and determine the causes of the breach? What issues lead to the breach and have those issues been adequately addressed?
• Determine what the program has done in terms of risk management to eliminate further breeches. (See Page 12, Risk Management)

• Using the most recent Defense Acquisition Executive Summary (DAES) compare the extent of performance and progress with ADM exit criteria requirements.
• Validate the DAES data by determining how the reported information was derived.

• For a new program, is CAIV being used from the program's onset?
• For an existing program in the later acquisition stages, are CAIV concepts being retrofitted?
• Have cost, schedule, and performance trade-offs been made and where required have they been accepted by designated approving authorities

Acquisition Strategy: TOC The acquisition strategy serves as the evolving roadmap for program execution from program initiation through post-production support. Essential elements include, but are not limited to, sources, risk management, cost as independent variable, contract approach, management approach, environmental considerations, and source of support as well as other major initiatives as described in the acquisition regulation such as critical events.

• program stability for industry to invest, plan and bear risk;
• industrial capability to design, develop, produce, support, and if appropriate, restart the program; and
• preservation of industrial capabilities.

• Does the risk management program define high, moderate, and low risk?
• Identify those tasks that have been labeled as high risk
• Has a risk abatement plan been established for those tasks identified as high risks? Does the risk abatement plan appear reasonable?
• Review tasks that have been rated moderate risks to determine if they have been rated properly in accordance with the definitions of high and moderate risk (often, borderline high risk tasks will be labeled as moderate risks)

• How is the prime contractor maintaining competition in the program through its subcontractors?
• How is the prime contractor flowing down streamlining initiatives to the subcontractors?
• If other than full and open competition, was justification documented and approval obtained?

• Was there an extensive number of undefinitized contractual actions that remained undefinitized for an excessive amount of time?
• Was there an excessive number of changes to the requirements of the system?

• If the above is not the case and DoD will logistically support the acquisition rather than the contractor, was a waiver prepared by the program manager and approved by the Milestone Decision Authority for the exception?

• Have the developmental and operational testers been involved in the test and evaluation planning process?
• Are the measures of effectiveness and measures of performance consistent with the Operational Requirements Document and does the test and evaluation plan address MOEs and MOPs with the appropriate quantitative criteria, test event, and scenario description and resource requirement?
• Does the TEMP ensure that testing is accomplished against a realistic threat environment? If not, how has the program office assessed the risk and identified risk mitigation actions?
• Has the pass/fail criteria for the system been clearly identified?
• Determine whether the minimum acceptable performance specified in the ORD was used to establish test criteria for operational test and evaluation.
• Determine if the TEMP identifies the resources needed to execute the test plan.
• Does the test plan address all system components to include hardware, software and human interfaces that are critical to demonstrating performance against the specifications and requirements in the ORD?
• Does the TEMP focus on the overall structure, major elements, and test objectives that are consistent with the acquisition strategy?
• Has the test and evaluation program been approved in writing by the appropriate officials? Was a TEMP approved at Milestone I, II, III?

• Have additional resources been added if needed?
• Have risks resulting from changes been assessed and risk mitigation actions identified?
• If the mitigation actions affect how or when the system can be used in a threat environment (i.e. operational use), have the users approved?

PROGRAM DESIGN TOC

The purpose of program design is to establish the basis for a comprehensive, structured, integrated, and disciplined approach to the life-cycle design of major weapons and automated information systems.

Integrated Product and Process Development   TOC

Integrated Product and Process Development is a management technique that simultaneously integrates all essential acquisition activities though the use of multidisciplinary teams to optimize the design, manufacturing, and supportability processes. IPPD facilitates meeting cost and performance objectives from product concept through production including field support. The key IPPD tenet is teamwork through Integrated Product Teams (IPTs).

The Program Manager (PM) shall employ the concept of IPPD throughout the program design process to the maximum extent practicable. The IPPD management process shall integrate all activities from product concept through production and field support, using multidisciplinary teams to simultaneously optimize the product and its manufacturing and supportability to meet cost and performance objectives. It is critical that the processes used to manage, develop, manufacture, verify, test, deploy, operate, support, train people, and eventually dispose of the system be considered during program design.

1. Obtaining minutes and issue status listings from previous meetings
2. attending one or more IPT meetings
3. Interviewing selected IPT members

1. Hierarchy of IPTs
2. IPT mission and functions statements
3. Meeting Schedules
4. Permanent and "as needed" IPT members

• working as an effective means of communicating and reaching consensus,
• employing the appropriate organizations, functional disciplines and expertise,
• aggressively addressing major issues, and
• formulating and approving the critical acquisition documentation.

Systems Engineering: TOC The Program Manager shall ensure that a systems engineering process is used to translate operational needs and/or requirements into a system solution that includes the design, manufacturing, test and evaluation, and support processes and products. The systems engineering process shall establish a proper balance between performance, risk, cost, and schedule, employing a top-down iterative process of requirements analyses, functional analysis and allocation, design synthesis and verification, and system analysis and control.

• Assess configuration control plans. Do the plans tailor requirements, and incorporate subcontractor products? Do the plans outline procedures to adequately manage and control the system configuration?
• Is an integrated system been developed to capture and control the technical baseline?

Manufacturing and Production   TOC

The producibility of the system design shall be a priority of the development effort. Design engineering efforts shall focus on concurrent development of producible designs, capable manufacturing processes, and process controls to ensure requirements satisfaction and minimize manufacturing costs. The use of existing manufacturing processes shall be capitalized upon whenever possible. When new manufacturing capabilities are required, flexibility (i.e. sensitivity to rate and product configuration) shall be considered. Full rate production of a system shall not be approved until the system's design has been stabilized, the manufacturing processes have been proven, and the production facilities and equipment are in place (or are being put in place). (see DoD 5000.2-R, Part 4, Paragraph 4.3.1).

Significant and unresolved issues exist from government and contractor design reviews. Principle among these reviews are the Preliminary Design Review (PDR), Critical Design Review (CDR), Functional and Physical Configuration Audits (FCAs and PCAs), and Production Readiness Review (PRR) or similar production readiness assessment.

• Does the program office have access to the technical data package? What access is available?
• Is the required level of data ordered consistent with the program office's acquisition and logistics support plan?
• Was the supporting activity consulted when determining the technical data requirements?
• Were responsibilities for monitoring and closing these action items assigned?
• Will DCMC provide on site verification of action item closures?

• Determine whether the plan lays out the technical data requirements.
• If a plan was not developed, determine why not.

• design instability,
• changes in technical requirements,
• quantity and funding fluctuations,
• long-lead times to procure critical parts,
• availability of facility and equipment, and
• availability of special tooling and equipment?

• Will the reviews be performed incrementally? Incremental reviews allow for more timely discovery of problems.
• Will incremental reviews be performed in an event-based manner, i.e., incremental PRRs held after completion of Critical Design Review and prior to: long-lead procurement for Low-Rate Initial Production (LRIP), prior to start of LRIP, and prior to start of full-rate production.

• Were responsibilities for monitoring and closing action items assigned?
• Are action items being addressed?
• Will DCMC provide on-site verification of action items?

Quality: TOC Quality products and services are necessary for the success of military operations, as well as to successful system development and production. The quality of the product or service is determined by deciding how well they meet the requirements and satisfy the customers needs at a reasonable cost. The goal of an effective acquisition program is to acquire goods and services that meet or exceed requirements better, faster, and at less cost. The emphasis on quality has evolved dramatically over the years. The shift in thinking now emphasizes development of quality products through design and associated processes. The key to success is to prevent quality problems through sound processes up front and early on as opposed to finding them later and performing rework.

Acquisition Logistics: TOC DoD Regulation 5000.2-R requires acquisition programs to establish logistics support concepts early in the program and refine them throughout the development process. The regulation also requires life-cycle costs to play a key role in the overall selection process and support concepts for new and future weapon systems. The Program Office should conduct acquisition logistics management activities throughout system development to ensure the design and acquisition of systems can be cost-effectively supported and to ensure that these system are provided to the user with the necessary support infrastructure for achieving the user's peacetime and wartime readiness requirements. (Auditor's Note: for additional information on supportability issues, see page 8.)

Life-Cycle Cost: TOC For all ACAT I and IA programs, a life-cycle cost estimate shall be prepared by the program office in support of program initiation (usually milestone I) and all subsequent milestone reviews. For ACAT I programs, the MDA may not approve entry into engineering and manufacturing development or production and deployment unless an independent estimate of the full life-cycle cost of the program and a manpower estimate for the program have been completed and considered by the MDA. Historically, we pay a huge amount of attention to the initial cost of our acquisition program, but little attention to the sustainment costs. Between 50% and 75% of total program costs are in the sustainment area. (Auditor's Note: For additional information relating to Life-Cycle Cost, see Page 17, Life-Cycle Resource Estimates.)

Integrated Logistics Support Plan: TOC The Integrated Logistics Support Plan (ILSP) is the principal logistics document for an acquisition program and serves as a source document for summary information required in various milestone documents. This formal planning document is kept current through the program life and sets forth the plan of operational support, provides a detailed Integrated Logistics Support program to fit with the overall program, and provides decision-making bodies with the necessary logistics information to make sound decisions in system development and production.

Maintenance Plan and Implementation: TOC The maintenance plan is a description of maintenance considerations and constraints for the system under development. The plan is one of the principal elements of the ILS and should establish maintenance concepts and requirements for the lifetime of the system. A preliminary maintenance concept is developed and submitted as part of the preliminary system operation concept for each alternative solution candidate by the operating command with the assistance of the implementing and supporting commands.

• Did the plan show the responsibilities and requirements of the Program Office and the operating sites where the system will be deployed?

• Determine if simulation models were used in deciding on the maintenance concept. If models were used determine if the criteria used in the model is logical and meets the parameters of the maintenance plan.
• Determine if the maintenance concept is consistent with the deployment concept of the weapon system.

Deployment: TOC The deployment process is designed to turn over newly acquired or modified systems to users who have been trained and equipped to operate and maintain the system. All elements of the integrated logistic support must be in place at deployment with the exception of those for which interim contractor support is available. When properly planned and executed, deployments result in high unit readiness, reduced cost, less logistical turmoil, and help establish reliability for the new system.

Environmental, Safety and Health: TOC Environment, Safety and Health (ES&H) analyses cover a broad range of topics directly related to the well-being of persons, places and things that may be affected by a weapon system over its life cycle (concept to disposal). DoD 5000.2-R, part 4, section 4.3.7 provides specific requirements covering five areas which form the principle technical and management disciplines of ES&H:

• Compliance with the National Environmental Policy Act,
• Periodic review of environmental regulations for the purpose of analyzing the impact on the program's cost, schedule and performance,
• Establishing a system safety and health hazards program,
• Establishing a hazardous materials management program, and
• Establishing a pollution prevention program.

All programs, regardless of acquisition category, must comply with ES&H requirements. Further, Program Offices must conduct their ES&H program in accordance with applicable federal, state, interstate, and local environmental laws and regulations, Executive Orders, treaties, and agreements. These requirements constitute an external constraint, beyond the PM's control, on system design construction, modification, testing, operation, support, maintenance, repair, demilitarization and disposal. Often the environmental regulations prescribe what must be done and how to do it. These regulations are very costly to comply with during production and even more so later on during the operation and support of the system and because of this, environmental regulations should be fully evaluated early on in the program and then periodically re-evaluated to determine their impact on the program cost, schedule, and performance.

ES&H analyses must be conducted to integrate ES&H issues into the systems engineering process and to support development of the Programmatic ESH Evaluation (PESHE). The first audit step in evaluating any program is to determine which ES&H requirements apply to that particular program and whether those requirements have been incorporated throughout the life of the program (program documentation, requirements, and contractor support).

• Determine whether the ES&H program is an integral part of integrated product and process development teams.
• Does the Program Manger have an aggressive attitude towards ES&H management issues? For example, determine whether ES&H is an agenda item at contractor program status briefings.
• Does the system contractor have a strong ES&H management history or a structured ES&H management program?
• Are ES&H considerations integral components of the design process?

• Does the program help minimize environmental impact?
• Does the program consider life-cycle costs?

Open System Design TOC The objective of this area is to determine if program managers are effectively implementing the use of the open systems (OS) approach in systems acquisition. The requirements for opens systems architecture has been incorporated into DoD acquisition policy and guidance. In July 1996, the Open Systems Joint Task force for the Principle DUSD(A&T) surveyed 60 program offices. The results were:

• One half were well aware of open systems
• One third fully understood open systems
• One quarter were fully implementing open systems

Audit steps concerning open systems should be performed together with the steps annotated in the Software Engineering section since open system design criteria is a consideration of software engineering. See Change 3 to 5000.2-R for details.

• documentation on how the use of the open system approach affected system design;
• documentation on both open (non-proprietary) and closed (proprietary) interfaces and protocols within the system;
• a plan for certifying contractor compliance with agreed-to open systems interfaces.

• Did the request for proposal specify use of an open system approach?
• Has the program office documented their approach for measuring the level of openness at the system, subsystem, and component level?

Software Engineering: TOC Software shall be managed and engineered using best processes and practices that are known to reduce cost, schedule, and performance risks. It is DoD policy to design and develop software systems that are based on systems engineering principles to include:

1. Developing software systems architectures that support open systems concepts; exploit commercial off-the-shelf (COTS) computer systems products; and provide for incremental improvements based on modular, reusable, extensible software;
2. Identifying and exploiting software reuse opportunities, Government and commercial, before beginning new software development;
3. Selection of programming language in the context of the systems and software engineering factors that influence overall life-cycle costs risks, and potential for interoperability. Additional guidance is contained in ASD(C3I) Memorandum, April 29, 1997;
4. Use of DoD standard data. Additional guidance is contained in DoDD 8320.1;
5. Selecting contractors with the domain experience in developing comparable software systems, a successful part performance record, and a demonstratable mature software development capability and process; and
6. Use of a software measurement process in planning and tracking the software program, and to assess and improve the software development process and associated software product.
7. Ensuring that information operations risks have been assessed (DoDD S 3600.1).
8. Ensuring software is Year 2000 compliant.

• Does the program's acquisition strategy consider evolutionary and incremental models of software development?

• Do those individuals have sufficient background and training to perform that function?
• Have they taken the basic, intermediate or advanced courses in software acquisition management? (Note: several Defense Science Board studies have identified software acquisition management training as a weak spot in overall program office management.)

Year 2000 (Y2K) Issue. TOC The Year 2000 (Y2K) problem, also known as the Century Date Change presents problems because developers of computer systems and applications in the past have used two digits to designate the year. For the past several decades, systems have typically used two digits to represent the year, such as "98" representing 1998, to conserve electronic data storage and to reduce operating costs. With the two-digit format, however, the year 2000 is indistinguishable from 1900, or 2001 from 1901, and so forth. As a result, legacy systems and some raecent systems may abort or produce erroneous data once the year 2000 arrives. Y2K presents a challenge for management. A fix for updating the systems to deal with the new millennium will require placing those individuals who are able to make the corrections in the field where the systems have been deployed. This will require integrated planning by the acquisition managers and procurement personnel at all levels.

To assess a program's plans for addressing the Y2K problem, please refer to Appendix B for possible audit questions or steps that should be considered.

Reliability, Maintainability, and Availability: TOC Reliability, maintainability and availability have a direct impact on both operational capability and life cycle costs and therefore are important considerations for the warfighter. The ability to successfully complete a mission is directly dependent on the weapon or automated information system performing that mission without experiencing a mission critical failure (reliability). An item that cannot be repaired in a timely or efficient manner (isn't maintainable) will unnecessarily consume warfighting resources and can decrease that warfighter's ability to initiate and complete a mission. Warfighting capability is also dependent on the weapon or automated information system being ready to under take a mission when called upon to do so (availability). From the life cycle cost perspective, it has been universally recognized that the qualities of reliability, maintainability and availability result in reduced life cycle costs. Operating and support costs of a weapon or automated information system are major elements of life cycle cost and typically overshadow the development costs. The reliability, maintainability, and availability characteristics of a system are major drivers of operating and support costs.

PM shall ensure that reliability, maintainability, and availability activities are established early in the acquisition cycle to assure meeting operational requirements and reduced life-cycle ownership cost. The RAM requirements shall be based on operational requirements and life-cycle cost considerations; stated in quantifiable, operational terms; measurable during developmental and operational test and evaluation; and derived from and directly support system readiness objectives. Maintainability requirements shall address servicing, preventive, and corrective maintenance. Availability requirements shall address the readiness of the system.

The PM shall plan and execute RAM and availability design, manufacturing development and test activities such that equipment used to demonstrate system performance prior to production reflects the mature design. Demonstrations shall use production representative systems (or as near as possible) and actual operational procedures (e.g., actual technical orders, spare parts, tools, support equipment, and personnel with representative skill levels). (For general guidance and information on reliability see "RADC Reliability Engineer's Toolkit and PAC-3 Report findings C&D for examples of shortfalls in a reliability program.)

The objective of this section of the PME audit is to determine whether there are any indications that the RAM requirements will not be met or the objective and projected RAM growth cannot be achieved.

Human Systems Integration: Table of Contents A comprehensive management and technical strategy for human systems integration shall be initiated early in the acquisition process to ensure that: human performance; the burden the design imposes on manpower, personnel, and training (MPT); and safety and health aspects are considered throughout the system design and development processes. ( see Human Systems Integration DoDI 5000.2-R, Part 4, Paragraph 4.3.8).

Human factors engineering requirements shall be established to develop effective human-machine interfaces, and minimize or eliminate system characteristics that require extensive cognitive, physical, or sensory skills; require excessive training or workload for intensive tasks; or result in frequent or critical errors or safety/health hazards. The capabilities and limitations of the operator, maintainer, trainer, and other support personnel shall be identified prior to program initiation (usually Milestone I), and refined during the development process. Human-machine interfaces shall comply with the mandatory guidelines for all C4I systems, automated information systems, and weapons systems that must interface with C4I systems or automated information systems, as defined in the DoD Technical Architecture Framework for Information Management, Version 2.0. June 30, 1994.

Reports, plans, and program decisions made by the HSI communities outside the acquisition infrastructure (e.g., manning documents and personnel occupational specialty decisions) must reflect and, to every extent possible, be reflected in program design decisions, trade-offs, risk assessments, and test results.

Interoperability: Table of Contents Compatibility, interoperability, and integration are goals that must be addressed for all acquisition programs. These goal should be determined and validated during the generation of the requirements.

Survivability: TOC Unless waived by the Milestone Decision Authority, mission critical systems, regardless of ACAT, shall be survivable to the threat levels anticipated in their operating environment. System survivability from all threats found in the various levels of conflict shall be considered and fully assessed as early as possible in the program, usually during Phase I.

• Compare most recent System Threat Analysis Report (STAR) against the approved Operational Requirements Document (ORD).
• Determine whether the STAR has been updated to support the prior milestone review and if it has been validated by the Defense Intelligence Agency.
• Does the ORD include requirements for survivability against each of the various threats described in the STAR?
• Are survivability capabilities in the ORD expressed in terms of measurable and quantitative objective and threshold performance requirements that can be validated through test and evaluation?

Work Breakdown Structure: TOC The Work Breakdown Structure (WBS) is the basis for communications throughout the acquisition process. A WBS defines a defense system in product terms -- hardware, software, services, data, facilities -- and relates them in a family tree that displays the relationships of the product(s) to each other and to the end product.

The WBS is a product of the systems engineering process. As new concepts take form, the WBS helps program management organize, define and display the emerging program. Just as a program evolves during various phases, so too does its unique WBS. But all programs, however different their end products may be, share common elements, for example, program management, data, and training. After the Program WBS has been developed to reflect both its unique and common elements, it becomes the basis for the Contract WBS--the discretionary extension by the contractor from the Program WBS to include all the products for which a given contractor is responsible.

In summary, the work breakdown structure is a useful tool as program managers engage in planning and controlling the program. It relates the various work efforts or parts of the program to the overall products or whole system. It is the foundation for:

• Program and technical planning through Systems Engineering
• Contractor's work effort
• Schedule definition
• Cost estimation, accountability, tracking and budget formulation
• Progress status reporting and problem analysis

Program offices shall tailor a program WBS for each program using the guidance in MIL-HNBK-881. MIL-HNBK-881 shall be cited in solicitation and contracts "for guidance only" in extending the program WBS to develop the complete contract WBS.

Assessing the WBS is a good way for the auditors to understand the program. Generally, a finding in this area is rare, but review of the area may lead to a more significant issue such as failure to address a key program element.

Contractor Performance: TOC Unless waived by the Milestone Decision Authority or designated representative, compliance with the EVMS criteria is required on significant contracts and subcontracts within all acquisition programs. Significant contracts are defined as DoD development contracts and subcontracts with a value of $70 million or more and production contracts and subcontracts with a value of $300 million or more (in FY 1996 constant dollars).

The following exceptions apply:

Key Contractor Performance Documents that include contractor performance information include but are not limited to:

• Defense Acquisition Executive Summary
• Contractor Performance Reports
• Memorandum of Agreement between DCMC and the Program Office
• Work Breakdown Structure
• Statement of compliance with CSCS or EVMS criteria
• Surveillance Plan/Agreement

Standardization Documentation: TOC Preference shall be given to specifications and standards developed under the Defense Standardization Program. DoD 5000.2-R authorized the publication of DoD 4120.3-M that describes the Defense Standardization and Parts Management Program.

• Is physical uniformity a minimum essential requirement or necessary for ease of operation and safety?
• Will the end item be used in a variety of applications?
• Would training for operation, maintenance, or repair be improved by standardization?

• The technology is unstable and the Program Office would like to take advantage of technological advances?
• The primary goal of the system is to satisfy specific user preferences?
• Standardization will unacceptably inhibit design flexibility and innovation?

Metric System: TOC The metric system of measurement is to be used for all acquisitions requiring new designs.

Program Protection: TOC The program must identify those parts of the acquisition which require special consideration to prevent unauthorized disclosure or inadvertent transfer of important technology. Planning for program protection should begin early in the acquisition process.

Electromagnetic Environmental Effects (E3) and Spectrum Management: TOC Electric and electronic systems should be designed to be compatible with all other electric or electronic equipment with other systems within the operational environment.

Program Management TOC

Implementing Acquisition Reform TOC

Acquisition reform is a key focus within the Department of Defense. Although the following questions can be asked as a separate objective, many of the questions should be considered within the other sections.

Operational Requirements Document (ORD). TOC The operational requirements document provides performance requirements necessary to meet an operational need and defines the desired characteristics and capabilities for the proposed new system. It is first prepared by the user or a user representative during Phase 0 for use in Phase 1 and is used to update the program baseline and develop contract specifications during each subsequent acquisition phase. The ORD is required for all programs, validated and approved by the operational validation authority and is included the Acquisition Program Baseline. The ORD identifies required performance capabilities, provides the number of systems/subsystems needed, addresses integrated logistics support requirements and discusses infrastructure support and interoperability issues.

Analysis of Alternative (AOA). TOC The AOA is an evaluation of the advantages and disadvantages of alternatives being considered to satisfy a requirement, to include the sensitivity of each alternative to possible changes in key assumptions or variables. The analysis will aid decision makers in judging whether or not any of the alternatives offer sufficient benefit to be worth the cost.

Acquisition Strategy. TOC The acquisition strategy establishes a framework within which detailed acquisition planning and program execution are to be accomplished. It describes how DoD will acquire a major system. The acquisition strategy will describe the relationship of the essential elements of the program such as management, technical, resource, testing, training, deployment, support, safety, procurement and contracting.

Mission Need Statement (MNS). TOC The mission need statement documents deficiencies in current capabilities and opportunities for taking advantage of current technology. The MNS is required for all potential materiel acquisitions and describes the need in broad operational terms not in system specifics. It is submitted to the operational validation authority who is responsible for its validation and approval. The approved MNS is forwarded to a milestone decision authority for a Milestone 0 decision.

Acquisition Decision Memorandum (ADM). TOC The Acquisition Decision Memorandum documents the decision made by the Defense Acquisition Board during Milestone Reviews. It provides written direction to the services and is signed by the Office of the Under Secretary of Defense(Acquisition and Technology). Typically the ADM will:

• approve the initiation of a new program and entry of the program into Phase I;
• approve proposed changes or modifications to the acquisition strategy;
• establish program-specific exit criteria that must be accomplished before a program can move from one phase to the next; and
• identify affordability constraints derived from the planning, programming and budgeting system.

Test and Evaluation Master Plan. TOC The Test and Evaluation Master Pan (TEMP) documents the overall structure and objectives of the test and evaluation program. It provides a framework within which to generate detailed test and evaluation plans and it documents schedule and resource implications associated with the test and evaluation program.

Defense Acquisition Executive Summary (DAES). TOC The DAES is the vehicle for reporting to the Under Secretary of Defense (Acquisition and Technology) (USD(A&T)) program assessments, unit costs, current estimates of the Acquisition Program Baseline parameters, status reporting of exit criteria, and vulnerability assessments. Its purpose is to highlight potential and actual program problems before they become significant.

Selected Acquisition Report (SAR). TOC The SAR provides the status of total program cost, schedule, and performance, as well as program unit cost and unit cost breach information. In the case of joint programs, the SAR includes such information for all joint participants. Each SAR also includes a full life-cycle cost analysis for the reporting program and its predecessor.

Appendix B: Audit Questions/Steps Relating to the Year 2000 Issue TOC

a. Does the program have a plan for identifying, assessing, validating/testing for Y2K compliance?

b. What are the Y2K issues facing the program?

c. What resources are required for Y2K compliance? Is funding available?

d. Has the program identified interfaces with other systems and if those systems are Y2K compliant? Were MOAs obtained for all external interfaces?

e. Do system/program contracts have the Y2K clause? If not, are there actions being taken to have a clause incorporated?

f What is the certification process? Is the Y2K compliance certification conducted by an independent third party?

g. Depending on the status of the program/system, what contingency plans have been made?

Return to Audit Home Page

Please send comments or questions on this site to: auditnet@dodig.mil