7.9. Post-Implementation Review (PIR)

DEFENSE ACQUISITION GUIDEBOOK
Chapter 7 - Acquiring Information Technology

7.9. Post-Implementation Review (PIR)

7.9.1. Background

7.9.2. Overview

7.9.3. PIR Within the Acquisition Framework

7.9.4. PIR Implications for Evolutionary Acquisition

7.9.5. PIR Implementation Steps

7.9.5.1. Plan the PIR

7.9.5.2. Conduct the Post Implementation Review (PIR)

7.9.5.3. Conduct the Analysis

7.9.5.4. Prepare a Report and Provide Recommendations

7.9.6. PIR Further Reading

7.9.1. Background

The Government Performance and Results Act (GPRA) Modernization Act of 2010 requires that Federal Agencies compare actual program results with established performance objectives. In addition, Section 11313 of Subtitle III of title 40 of the United States Code (formerly known as Division E of the Clinger-Cohen Act (CCA) (hereinafter referred to as "Title 40/CCA") requires that Federal Agencies ensure that outcome-based performance measurements are prescribed for the Information Technology (including National Security Systems (IT/NSS)) to be acquired and that these performance measurements measure how well the IT/NSS supports the programs of the Agency.

DoD Instruction 5000.02, Tables 2-1 and 2-2, identify this information requirement as a Post-Implementation Review (PIR) and require a PIR for all acquisition program increments at the Full-Rate Production Review/Full-Deployment Decision Review (FRPDR/FDDR). To clarify this requirement, it is a plan for conducting a PIR that is due at the FRPDR or FDDR. The actual PIR is conducted, and a report is generated after Initial Operational Capability (IOC) and generally before Full Operational Capability. (Refer to section 7.9.5 of this guidebook for specific PIR Implementation Steps.)

The Office of Management and Budget (OMB) in OMB Circular A-130 Chapter 8 paragraph b.1(c and d) prescribes PIR procedures within the capital planning and investment control construct for measuring how well acquired IT supports Federal Agency programs.

OMB Guidance for IT Investment Reporting, Table C1 of Section C to Exhibit 300B. Performance Measurement Report requires operational data that measures the effectiveness of the investment in delivering the desired service or support level and measures the investment against its defined process standards or technical service level agreements (SLAs) (e.g., “Reliability and Availability”). The 300B is an annual report that is submitted twice annually, BES in September and PB in March. Since the 300B is a new report, procedures for coordination with PIR information requirement are TBD. Updates will be posted in the PIR section of the IT-CoP. (Readers who are members of ACC can receive notification of PIR process changes between DAG versions by going to IT-CoP and clicking on “Other Actions” and then clicking on “Subscribe”. To become a member of ACC, click on “Become a Member” in the left margin.)

7.9.2. Overview

This section provides guidance on how to plan and conduct a PIR for a capability that has been fielded and is operational in its intended environment. A PIR verifies the measures of effectiveness (MOEs) of the Initial Capabilities Document (ICD) or the benefits of a business plan and answers the question, "Did the Service/Agency get what it needed, per the ICD/Business Plan, and if not, what should be done?

Who is Responsible? The Sponsor is responsible for articulating outcome-based performance measures in the form of MOEs or benefits and ensuring they are reported in the ICD or Business Plan. The Sponsor is responsible for planning the PIR, gathering data, analyzing the data, and assessing the results. The Program Manager (PM) is responsible for maintaining an integrated program schedule that includes the PIR on behalf of the Sponsor. The PM is also responsible for supporting the Sponsor with respect to execution and reporting of the PIR.

What is a PIR? The PIR is a process that aggregates information needed to successfully evaluate the degree to which a capability has been achieved. Table 7.9.2.T1 represents potential sources of such data. Note that the information sources in this table represent a broad segment of acquisition, administrative, and operational activities.

Table 7.9.2.T1. Potential PIR Information Sources

7.9.3. Post Implementation Review (PIR) Within the Acquisition Framework

A useful way to view a PIR is that it is a doctrine, organization, training, materiel, leadership and education, personnel, facilities, and policy (DOTMLPF-P) assessment. As shown in Figure 7.9.3.F1, the capability-based assessment (CBA) (or business case for business systems) defines the need, provides MOEs, and analyzes the changes that may be needed to one or more Joint Capability Areas (JCA) (For definitions and a discussion of JCAs see the IT-CoP page on JCAs). The "materiel (M)" contribution to the need enters the defense acquisition framework, which coordinates with the remaining DOTLPF process and arrives at the FRPDR/FDDR. The final PIR plan is presented at this time. Following IOC the fielded system is integrated with the changes to process and culture implemented during DOTLPF, and becomes a recognizable and measurable capability. The PIR takes place at this time and informs the DOTLPF, acquisition, and future CBA processes.

Figure 7.9.3.F1. Identification, Development and Verification of Capability

7.9.4. Post Implementation Review (PIR) Implications for Evolutionary Acquisition

PIRs provide important user feedback and consequently are a fundamental element of evolutionary acquisition. Ideally, we want to understand how well a recently completed increment meets the needs of users before finalizing the requirements for a subsequent increment. In practice however, the opportunity for such feedback depends on the level of concurrency in the increment development schedule.

Additionally, changes in the environment may drive new requirements. The PIR gives both the Sponsor,PM, and other stakeholders such as DOT&E and CAPE, empirical feedback to better understand DOTMLPF issues with the completed increment. This feedback enables the acquisition principals to adjust or correct the Capability Development Document/Capability Production Document and/or the DOTMLPF Change Recommendation (DCR) for subsequent increments.

7.9.5. Post Implementation Review (PIR) Implementation Steps

7.9.5.1. Plan the PIR and submit plans and report

The final PIR Plan is due at the FRP/FD Decision Review.

When planning the PIR, consider the following:

• Timing of the PIR. The PIR should take place post-IOC after a relatively stable operating environment has been established and the data identified in the PIR plan has been collected. Time frame for the PIR varies with the solution deployment strategy, but the PIR is to be executed and a report submitted prior to Full Operational Capability. If an FOC is not planned, the PIR is to be executed within one year of IOC.
• Identification of Scope.
• Identification of Stakeholders. Remember to consider those who will be tasked to provide resources.
• Team Composition. The PIR team should include, at minimum, the following:
  • Functional experts with working knowledge of the business area and its processes;
  • People with relevant technical knowledge;
  • CIO representatives, functional sponsors, and Domain Owners;
  • Oversight representatives.
• Identification of information sources. The ICD or Business Plan that articulated the outcome-based performance measures, or MOEs, is a good place to start. Additional data can be gleaned from operations conducted in wartime and during exercises. The lead time for most major exercises is typically one year and requires familiarity with the exercise design and funding process. Sources to consider are found in Table 7.9.2.T1.
• Analysis approach. The analysis approach is key to defining the structure and metadata of the information to be collected. For example, the definition of return on investment (ROI) in the Economic Analysis will drive the analysis approach of achieved ROI and the data to be collected.
• Reporting. The report describes the execution of the PIR, addresses the capability gaps that the IT/NSS investment was intended to fill, addresses the degree to which the gaps were filled, and recommends actions to mitigate unfilled capability gaps.
• Resource requirements. Identify the sources of resources such as manpower, travel, analysis tools, communications and other needs unique to the program. Demonstrate agreement by the resource providers; including them in the chop page or citing existing agreements.
• Schedule.
• Routing of the draft and final PIR Plan should include the identified stakeholders. An information copy of the final PIR Plan and Report is sent to the Senior Military Advisor DOT&E and the Program's CAPE Action Officer.

7.9.5.2. Conduct the Post Implementation Review (PIR)

The PIR should be carried out according to the PIR planning that was reviewed and approved at the FRPDR/FDDR. Care should be given to quality of the raw data. Based on the PIR plan, the PIR should, at a minimum, address:

• Customer Satisfaction: Are the users satisfied that the IT investment meets their needs?
• Mission/Program Impact: Did the implemented capability achieve its intended impact?
• Confirmation that the validated need has not changed; or if it has, include as part of the course of action provided in the PIR report.
• A measure of the MOE found in the ICD.
• Benefits such as the ROI found in the business plan. Compare actual project costs, benefits, risks, and return information against earlier projections. Determine the causes of any differences between planned and actual results.

7.9.5.3. Conduct the Analysis

The analysis portion of the PIR should answer the question, "Did we get what we needed?" This provides a contrast to the test and evaluation measurements of key performance parameters that answer the question, "Did we get what we asked for?" This would imply that the PIR should assess, if possible, the extent to which the DoD's investment decision-making processes were able to capture the warfighter's/user’s initial intent. The PIR should also address whether the warfighter/user needs changed during the time the system was being acquired. The outputs of the analysis become the PIR findings. The findings should clearly identify the extent to which the warfighters got what they needed.

7.9.5.4. Prepare a Report and Provide Recommendations

Based on the PIR findings, the PIR team prepares a report and makes recommendations that can be fed back into the capabilities and business needs processes. The primary recipient of the PIR report is the Sponsor who articulated the original objectives and outcome-based performance measures on which the program or investment was based.

A copy of the PIR report is also forwarded to DOT&E and CAPE. DOT&E will use the results to confirm the effectiveness and suitability assessment made during IOT&E and possibly to improve the test planning and execution for follow-on increments or similar systems (see Section 9.7.9). CAPE will compare the benefits of selected programs to those presented in the Economic Analysis.

The results of the PIR can also aid in refining requirements for subsequent increments. Recommendations may be made to correct errors, improve user satisfaction, or improve system performance to better match warfighter/business needs. The PIR team should also determine whether different or more appropriate outcome-based performance measures should be developed to enhance the assessment of future spirals or similar IT investment projects.

For further guidance on PIRs, see the Information Technology Community of Practice Post Implementation Review Area. This contains the following additional guidance:

• PIR Measurement Framework
• Common Problems with PIR Implementations
• Example plan and report

7.9.6. Post Implementation Review (PIR) Further Reading

Both government and the commercial sector address the practice of conducting a PIR for materiel, including software IT investments. The Government Accountability Office and several not-for-profit organizations have written on the subject of measuring performance and demonstrating results. The Clinger-Cohen Act Community of Practice PIR Area lists a number of key public and private sector resources that can be used in planning and conducting a PIR.

Top of page