12.5 Technical Reviews and Audits

Technical reviews and audits are a systems engineering tool that provide a way to assess progress and maturity of the product as it moves through the various phases of the acquisition life cycle.  These reviews and audits are consistent with existing DOD and commercial best practices and form the backbone for effective systems engineering planning.  All reviews are or should be multi-disciplined that ensure all of the members of the integrated product team (IPT) have an opportunity to review the product and documentation in order to assess progress in their functional area towards achievement of phase goals.  These reviews provide a systematic process for assessing risk and easing the transition from development to production and beyond by:

• Assessing the maturity of the design/development effort;
• Clarifying design requirements;
• Challenging the design and related processes;
• Checking proposed design configuration against technical requirements, customer needs, and system requirement;
• Evaluating the system configuration at different stages;
• Providing a forum for communication, coordination, and integration across all disciplines and IPTs;
• Establishing a common configuration baseline from which to proceed to the next level of design and production; and
• Recording technical decisions and rational in the decision database.

Reviews are an important oversight tool that the program manager can use to review and evaluate the state of the system and the program, re-directing activity if necessary. Figure 12-2 shows the relative timing of each of the technical reviews, technically oriented program reviews, and technology readiness assessments.

Figure 12-2  Systems Engineering Technical Review Timing

The following business and technical reviews are held for most programs:

• Initial Technical Review (ITR),
• Alternative Systems Review (ASR),
• System Requirements Review (SRR),
• Technology Readiness Assessment (TRA),
• Integrated Baseline Review (IBR),
• System Functional Review (SFR),
• Preliminary Design Review (PDR),
• Critical Design Review (CDR),
• Test Readiness Review (TRR),
• System Verification Review (SVR),
• Functional Configuration Audit (FCA),
• Production Readiness Review (PRR),
• Operational Test Readiness Review (OTRR),
• Physical Configuration Audit (PCA), and
• In-Service Review (ISR).

OSD has developed a checklist for each of technical reviews. The checklist structure for many of the reviews is shown below and includes twelve focus areas to include the PQM community.  Questions can be segregated by focus area by enabling the macros and selecting PQM.  This will provide only those questions that have been identified as an interest area for that focus area. These checklists are available on the Systems Engineering Community of Practice (CoP) at the Defense Acquisition University (DAU).

Figure 12-3 Typical Format for a Technical Review

A note of caution.  The current DOD checklist contains questions that are relevant for personnel in the Production, Quality and Manufacturing (PQM) career field.  However, those questions need to be reviewed carefully for appropriateness.  Often there are more questions centered on Diminishing Manufacturing Sources and Material Shortages (DMSMS) than there are for Manufacturing, Quality and Environmental considerations combined.  For this reason it is recommended that PQM'ers review the Manufacturing Readiness Level questions to augment PQM questions in the existing technical reviews and audits.

The ITR is a multi-disciplined technical review to support a program's initial Program Objective Memorandum submission. This review ensures a program's technical baseline is sufficiently rigorous to support a valid cost estimate (with acceptable cost risk) and enable an independent assessment of that estimate by cost, technical, and program management subject matter experts (SMEs). The ITR assesses the capability needs and Materiel Solution approach of a proposed program and verifies that the requisite research, development, test and evaluation, engineering, manufacturing, logistics, and programmatic bases for the program reflect the complete spectrum of technical challenges and risks. Additionally, the ITR ensures the historical and prospective drivers of system life-cycle cost have been quantified to the maximum extent and that the range of uncertainty in these parameters has been captured and reflected in the program cost estimates.

The ASR is a multi-disciplined technical review to ensure the resulting set of requirements agrees with the customers' needs and expectations and then the system under review can proceed into the Technology Development phase. The ASR should be completed prior to, and provide information for, Milestone A. Generally, this review assesses the preliminary materiel solutions that have been evaluated during the Materiel Solution Analysis phase, and ensures that the one or more proposed materiel solution(s) have the best potential to be cost effective, affordable, operationally effective and suitable, and can be developed to provide a timely solution to a need at an acceptable level of risk. Of critical importance to this review is the understanding of available system concepts to meet the capabilities described in the Initial Capabilities Document (ICD) and to meet the affordability, operational effectiveness, technology risk, and suitability goals inherent in each alternative concept.

The SRR is a multi-disciplined technical review to ensure that the system under review can proceed into initial systems development, and that all system requirements and performance requirements derived from the Initial Capabilities Document or draft Capability Development Document are defined and testable, and are consistent with cost, schedule, risk, technology readiness, and other system constraints. Generally this review assesses the system requirements as captured in the system specification, and ensures that the system requirements are consistent with the approved materiel solution (including its support concept) as well as available technologies resulting from the prototyping effort.

The SFR is a multi-disciplined technical review to ensure that the system's functional baseline is established and has a reasonable expectation of satisfying the requirements of the Initial Capabilities Document or draft Capability Development Document within the currently allocated budget and schedule. It completes the process of defining the items or elements below system level. This review assesses the decomposition of the system specification to system functional specifications, ideally derived from use case analysis. A critical component of this review is the development of representative operational use cases for the system. System performance and the anticipated functional requirements for operations maintenance, and sustainment are assigned to sub-systems, hardware, software, or support after detailed analysis of the architecture and the environment in which it will be employed. The SFR determines whether the system's functional definition is fully decomposed to its lower level, and that Integrated Product Teams (IPTs) are prepared to start preliminary design.

The PDR is a technical assessment establishing the physically allocated baseline to ensure that the system under review has a reasonable expectation of being judged operationally effective and suitable. This review assesses the allocated design documented in subsystem product specifications for each configuration item in the system and ensures that each function, in the functional baseline, has been allocated to one or more system configuration items. The PDR establishes the allocated baseline (hardware, software, human/support systems) and underlying architectures to ensure that the system under review has a reasonable expectation of satisfying the requirements within the currently allocated budget and schedule.

The TRA is a regulatory information requirement for all acquisition programs. The TRA is a systematic, metrics-based process that assesses the maturity of critical technology elements (CTEs), including sustainment drivers. The TRA should be conducted concurrently with other Technical Reviews, specifically the Alternative Systems Review (ASR), System Requirements Review (SRR), or the Production Readiness Review (PRR). If a platform or system depends on specific technologies to meet system operational threshold requirements in development, production, or operation, and if the technology or its application is either new or novel, then that technology is considered a CTE.

The CDR is a key point within the Engineering and Manufacturing Development (EMD) phase. The CDR is a multi-disciplined technical review establishing the initial product baseline to ensure that the system under review has a reasonable expectation of satisfying the requirements of the Capability Development Document within the currently allocated budget and schedule. Incremental CDRs are held for each Configuration Item culminating with a system level CDR. This review assesses the final design as captured in product specifications for each Configuration Item in the system and ensures that each product specification has been captured in detailed design documentation. Configuration Items may consist of hardware and software elements, and include items such as airframe/hull, avionics, weapons, crew systems, engines, trainers/training, support equipment, etc. Product specifications for hardware enable the fabrication of configuration items, and include production drawings. Product specifications for software enable coding of the Computer Software Configuration Item. The CDR evaluates the proposed Baseline ("Build To" documentation) to determine if the system design documentation (Initial Product Baseline, including Item Detail Specs, Material Specs, Process Specs) is satisfactory to start initial manufacturing.

The TRR is a multi-disciplined technical review designed to ensure that the subsystem or system under review is ready to proceed into formal test. The TRR assesses test objectives, test methods and procedures, scope of tests, and safety and confirms that required test resources have been properly identified and coordinated to support planned tests. The TRR verifies the traceability of planned tests to program requirements and user needs. It determines the completeness of test procedures and their compliance with test plans and descriptions. The TRR also assesses the system under review for development maturity, cost/ schedule effectiveness, and risk to determine readiness to proceed to formal testing.

The SVR is a multi-disciplined product and process assessment to ensure the system under review can proceed into Low-Rate Initial Production and full-rate production within cost (program budget), schedule (program schedule), risk, and other system constraints. Generally this review is an audit trail from the System Functional Review. It assesses the system functionality, and determines if it meets the functional requirements (derived from the Capability Development Document and draft Capability Production Document) documented in the functional baseline. The SVR establishes and verifies final product performance. It provides inputs to the Capability Production Document. In some organizations the SVR is conducted concurrently with the Production Readiness Review.

The FCA is the formal examination of the as tested characteristics of a configuration item (hardware and software) with the objective of verifying that actual performance complies with design and interface requirements in the functional baseline. It is essentially a review of the configuration item's test/analysis data, including software unit test results, to validate the intended function or performance stated in its specification is met. For the overall system, this would be the system performance specification. For large systems, audits may be conducted on lower level configuration items for specific functional areas and address non-adjudicated discrepancies as part of the FCA for the entire system. A successful FCA typically demonstrates that Engineering and Manufacturing Development product is sufficiently mature for entrance into Low-Rate Initial Production.

The PRR examines a program to determine if the design is ready for production and if the prime contractor and major subcontractors have accomplished adequate production planning without incurring unacceptable risks that will breach thresholds of schedule, performance, cost, or other established criteria. The review examines risk; it determines if production or production preparations identify unacceptable risks that might breach thresholds of schedule, performance, cost, or other established criteria. The review evaluates the full, production-configured system to determine if it correctly and completely implements all system requirements. The review determines whether the traceability of final system requirements to the final production system is maintained.

At this review, the Integrated Product Team (IPT) should examine the readiness of the manufacturing processes, the quality management system, and the production planning (i.e., facilities, tooling and test equipment capacity, personnel development and certification, process documentation, inventory management, supplier management, etc.). A successful review is predicated on the IPT's determination that the system requirements are fully met in the final production configuration, and that production capability forms a satisfactory basis for proceeding into Low-Rate Initial Production (LRIP) and Full-rate production.

Typically performed incrementally, PRRs determine if production preparation for the system, subsystems, and configuration items is complete, comprehensive, and coordinated. A PRR formally examines producibility of the design, the control over the projected production processes, and adequacy of resources necessary to execute production.

The OTRR is a multi-disciplined product and process assessment to ensure that the system can proceed into Initial Operational Test and Evaluation with a high probability of success, and that the system is effective and suitable for service introduction. The Full-Rate Production Decision may hinge on this successful determination. The understanding of available system performance in the operational environment to meet the Capability Production Document is important to the OTRR. Consequently, it is important the test addresses and verifies system reliability, maintainability, and supportability performance and determines if the hazards and ESOH residual risks are manageable within the planned testing operations. The OTRR is complete when the Service Acquisition Executive evaluates and determines materiel system readiness for Initial Operational Test and Evaluation.

The PCA is conducted around the time of the Full-Rate Production Decision. The PCA examines the actual configuration of an item being produced. It verifies that the related design documentation matches the item as specified in the contract. In addition to the standard practice of assuring product verification, the PCA confirms that the manufacturing processes, quality control system, measurement and test equipment, and training are adequately planned, tracked, and controlled. The PCA validates many of the supporting processes used by the contractor in the production of the item and verifies other elements of the item that may have been impacted/redesigned after completion of the System Verification Review. A PCA is normally conducted when the government plans to control the detail design of the item it is acquiring via the Technical Data Package. When the government does not plan to exercise such control or purchase the item's Technical Data Package (e.g., performance based procurement), the contractor should conduct an internal PCA to define the starting point for controlling the detail design of the item and establishing a product baseline. The PCA is complete when the design and manufacturing documentation match the item as specified in the contract. If the PCA was not conducted before the Full-Rate Production Decision, it should be performed as soon as production systems are available.