17.8 The MRL Assessment

An assessment of manufacturing readiness is an important tool for evaluating manufacturing maturity and risk that is most useful in the context of a broader manufacturing risk management process.  These assessments should lead to actions such as:

• Setting goals for increased manufacturing maturity and reduced manufacturing risk;
• Creating action plans and funding estimates to reach those goals;
• Reaching decisions about the readiness of a technology or process to transition into a system design or onto the factory floor; and
• Reaching decisions on a system's readiness to proceed into the next acquisition phase.  

Therefore, an assessment of manufacturing readiness should compare the status of the key program elements to a nominal MRL appropriate for the stage of the program, describe the risk associated with elements that fall short of the goal, and lay the foundation for manufacturing risk mitigation planning and investment. An assessment should be able to answer the question "are you ready?" 

Too often many DOD acquisition programs did not ask the question "are you ready?"  As a result by not asking the question, and not doing an adequate risk assessment, the programs ran into problems.  As noted in several GAO reports, programs that did not have adequate knowledge of technology, design and production risk, ran into problems.  Those programs that included production considerations early had fewer problems. 

Figure 17-3 MRL Assessment Process

It is rarely feasible to visit every supplier of every material, component and assembly to examine the manufacturing risk.  Some elements should be assessed on-site and others may utilize alternative approaches.  The type and depth of the assessment is determined by the risk level of the element.  On-site evaluations are typically reserved for the locations where one or more of the following apply:

• The highest percentage of manufacturing cost is incurred;
• Final assembly and test is conducted;
• The most sensitive manufacturing tasks are accomplished;
• The materials, components or subsystems that are the least technologically mature are produced or availability issues exist; or
• Known significant problems or risks exist (low yields, high costs, immature manufacturing processes, etc.).

The assessment taxonomy encompasses what will be assessed, where the assessments will take place, and who will lead the assessment.

The government program/project office, in conjunction with the prime contractor, should make an early determination of potential issues by breaking out system, subsystem, or component level for analysis and then determining the applicability of components for evaluation.  Consideration should also be given to associated test and assembly processes.  The following questions have been developed to assist in the determination of elements to be assessed.  All Critical Technology Elements and other significant areas of the work breakdown structure or bill of materials should be subject to the following filtering questions.  Any "yes" responses imply that an assessment of manufacturing readiness may be needed for that element as a function of risk.

The assessments of manufacturing readiness are typically performed by teams and the government program/project office is responsible for forming them.  The government program/project office to lead the team at prime contractors and the prime contractor to lead the team for the sub-tiers. 

Team members should be experienced and knowledgeable in manufacturing engineering, industrial base, quality, supply chain, systems engineering, and production to identify potential manufacturing constraints, risks, and the capability of the technology and industrial base to execute the manufacturing efforts.  Subject matter experts (SMEs) may be required to identify specific manufacturing issues not expected to be uncovered by general production personnel. Representatives from DOD staff organizations may participate as well, if the assessment is being performed on an acquisition program approaching a milestone decision.  The program/project office should consider contacting the appropriate office of the Defense Contract Management Agency (DCMA) to gather information on the contractor's current and past performance. 

Team selection can begin once the scope and a rough schedule of activity is developed.  These teams will vary in size depending on the scope of the assessment.  Sub teams may be put together to focus on various subsystems or technologies.  Reviews should be tailored to the specific circumstances of the program. 

Team members will need to understand the purpose of the assessment as well as the following:

• Initial schedule;
• Format and timing of reporting their results to the team;
• Standards of behavior at the contractor's facility;
• Security clearances or nondisclosure agreements;
• Personal preparation;
• The need for a detailed understanding of their assigned area and the role of shop floor observations and off-line discussions with contractor personnel; and
• Responsibilities after the on-site review.

The leader of the assessment should orient the contractor to be assessed before the assessment occurs.  This orientation may involve including contractor personnel in planning meetings as well as providing the contractor with an orientation package that includes:

• The MRL definitions and threads;
• Self-assessment questions; and
• An initial identification of critical technologies or processes for self-assessment.

For on-site assessments, the orientation package should also include:

• The questions the assessment team will use;
• An agenda for the assessment visit;
• Identification of evidence to be provided at the onsite visit;
• High-interest areas where shop floor visits and/or discussions will be expected; and
• Expectations of resources, time, etc. required for the assessment.

Make arrangements with the contractor for an assessment team meeting room to be available where private discussions can be held and team members can record their observations.  Also, make arrangements with the contractor for assessment team members to bring computers into the facility to facilitate the capture of their observations in electronic format.

The leader of the assessment should ask the contractor(s) to conduct a self-assessment to address the following basic questions:

• What is the current MRL for each of the key technologies being developed and each key manufacturing process being used?
• If currently funded activities continue as planned, what MRL will be achieved for each key technology or process by the end of this acquisition phase or program?  What activities and schedules are required to achieve this MRL?
• In the case of an ATD or ACTD, what MRL would be sufficient for you or an OEM using your technology to commit to it in a product baseline design?

In the case of on-site assessments, the contractor should be prepared to brief the results to the assessment team when it is on-site.  For companies that provide key components or subassemblies and for which a site visit is not feasible, the contractor's written self-assessment should be analyzed by the assessment team.

Site visits are intended to provide a more detailed understanding than can be gained from briefings and documents.  Assessments of manufacturing readiness should be structured in such a way as to take maximum advantage of discussions with contractor experts and first-hand observations of the status of shop floor activities.  A balance must be struck between the time spent in briefing rooms and the time spent making observations in the contractor's facility and having discussions with individuals and small groups of the contractor's personnel.  A typical agenda for a review may contain the following elements:

• Contractor welcome, review of agenda, assessment schedule. and orientation to the facility;
• Introduction of assessment team and contractor personnel;
• Briefing to contractor describing objectives and expectations for the on-site visit;
• Contractor overview and discussion of the results of their self-assessment;
• Shop-floor visits to key areas by individuals or small groups;
• Small group discussions between assessment team members and contractor SMEs;
• Private meeting of assessment team to record and discuss observations; and
• Out-briefing by assessment team to contractor.

When conducting an assessment of manufacturing readiness, there should be a well-defined hierarchy among the elements assessed.  The hierarchy should start at the system level and flow down to the lowest component that forms the smallest unit for examination.  The assessment team should determine the MRL threads applicable to each element in the hierarchy and identify the needed system level test and assembly processes that require an MRL assignment.  This includes test and assembly steps that would be included in a subsystem or component fabrication. 

During the assessment process, a component or subsystem may be found to be more complex than originally thought, so an even more detailed analysis or 'deep dive' may be warranted.  If the assessment team determines further examination of critical components is necessary, the MRL threads should be applied at that level.  Sub-components are examined along with process steps, and an MRL is determined for this final sub-tier element.  Team members should seek existing, objective documentation that supports assessment results in key areas (e.g., plans, yield data, reports, briefings, work instructions). 

In determining the manufacturing readiness of a component or subsystem, the key emphasis is on the manufacturing risk.  Utilize the MRL Matrix to structure the review and establish target criteria for each thread/sub-thread.  If the target criteria are not met, utilize the risk matrix approach in the "DOD Risk Management Guide for Acquisition" to characterize the risks.  The team assesses the number and severity of the risks to determine the manufacturing readiness of the component or subsystem.

Finally, the assessment team should include the actions necessary to bring readiness up to the target level in time to transition a technology or support a milestone decision with manageable risk.

At the end of each day, DCMA personnel should be asked to provide their perspective and insight on the contractor's presentations and status.  If the contractor was unable to provide adequate information to support an assessment in a key area, assign an action item for the contractor to provide the information by a specific date.

Near the end of the assessment, the team should meet at the contractor's facility to discuss its observations and capture its impressions in electronic format.  The team should also provide an out-brief to the contractor highlighting strengths and risks, MRL achievements compared to targets, and action items.  Finally, the contractors' hospitality and cooperation should be recognized.

MRL assessments are not a simple go/no-go gauge.  Therefore, assigning a single MRL to an entire technology or weapon system has little value.  Even in a relatively simple case, where an assessment is being accomplished on a single technology with perhaps a half-dozen hardware components, it is likely the MRL will vary widely from component to component and perhaps even manufacturing process by manufacturing process for a specific component.  Some components may be off-the-shelf, standard hardware, or made with well-established materials and processes from reliable suppliers, thus perhaps having an MRL in the range of 8 to 10.  Other components may incorporate new design elements that move well beyond the proven capabilities of a key manufacturing process and perhaps are at MRL 4. 

Using a 'weakest link' basis, a technology or system would have to receive an overall MRL that reflects the element of that technology that had the lowest level of readiness, in this case, MRL 4.  In many instances, this approach could be misleading and give the impression of an overall level of risk greater than the actual situation.  For assessments of more complex subsystems and systems, this simplification becomes even less useful since it is unlikely that every element is going to be, for example, at MRL 6 by Milestone B.  

Therefore, the assessment report (as described in section 4.9), should contain a bottom-up assessment of the relative manufacturing readiness at the system, sub-system and component level.  Findings for lower level components can be fit into a format for analysis and decision making at higher levels of the program as shown in Table 4-1.  Each MRL (at any level) should be identified to provide insight into specific risks.

The results should be documented by team members in a format agreed to in advance.  This could be in the form of a briefing or a written report.  Often the outbriefing is an interim report.  Usually some analysis is required by the assessment team after site visits are complete to clearly define the manufacturing readiness and risk status of the key technologies and manufacturing processes and to put the identified risks into a program context.  These final results are then typically documented in a written report or out-brief containing the following:

• A description of the technology, component, subsystem or system which identifies the elements that were assessed; the key objectives of the development effort; and a discussion of the current state of the art;
• A discussion of the companies which are responsible for the elements that were assessed;
• A list of team members;
• Dates and locations of site visits;
• A description of the manufacturing processes for the elements that were assessed;
• The MRL for each element that was assessed;
• Areas where manufacturing readiness falls short of target MRL;
• Plans to reach target MRL;
• Assessments of the type and significance of risk to cost, schedule or performance; and
• Assessments of the effectiveness of current risk mitigation plans.

The government program/project office is the primary audience for the report since it forms the basis for managing manufacturing risk.  In general, the report establishes a manufacturing maturity baseline that should be used to either create a Manufacturing Maturation Plan to increase manufacturing readiness/maturity sufficiently to support transition to the next phase of acquisition or to demonstrate that the technology is ready for transition.  The report may also provide information to an MDA determination of whether the level of manufacturing risk supports Milestone approval.