11.3.3. Quality Management
According to American National Standards Institute (ANSI), International Organization for Standardization (ISO), and American Society for Quality (ASQ), international standard ANSI/ISO/ASQ Q9000-2005 (ISO 9000), Quality Management Systems—Fundamentals and Vocabulary:
- Quality is the degree to which a set of inherent characteristics fulfils requirements. It may apply to a product or process. Inherent characteristics may be physical, sensory, behavioral, temporal, ergonomic, functional, etc.
- Quality management represents the organized activities to direct and control an organization with regard to quality.
- Quality assurance is the part of quality management focused on providing confidence that quality requirements will be fulfilled.
Effective quality management activities are important for reducing process-related risks to programs. Such risks include:
- Ill-defined or omitted requirements;
- A breakdown in requirements flow down;
- Uneconomically producible designs as a result of inappropriate application of technical processes;
- Inadequate procedures to implement contract requirements;
- Suppliers with inadequate capabilities;
- Decreasing leverage with sub tiers as a result of ineffective supplier management;
- Dissatisfied customers as a result of ineffective customer engagement; and/or
- Undetected product defects resulting from unidentified verification technologies or failure to implement existing ones.
If not managed and mitigated, these risks may start a chain of events leading to undesirable outcomes such as:
- Product defects discovered in production or testing that may require expensive and time-consuming rework
- Products that may not meet customer needs
- Product deficiencies discovered in the field that may lead to degraded mission effectiveness, early wear out or mishaps
- Cost overruns or delays for current contracts and
- Cost escalation for future contracts
- Parts shortages at the wholesale and retail levels
The later these risks are identified, the greater the cost of corrective action and the greater the delays in schedule. Early identification, management, and mitigation of important process-based risks to a program lead to less expensive and less disruptive corrective actions that break the chain of undesirable outcomes.
While the DoD program manager should encourage and support the contractor's efforts to assure quality, ultimately, the prime contractor is responsible. Therefore, from a DoD perspective, a key program success factor is selecting contractors that can demonstrate effective quality management. This subject is discussed in section 11.3.3.1.
The contract should provide incentive to the contractor to deliver products or services that provide value beyond the basic requirement. Without additional incentives, the systems engineering process will normally lead to decisions that satisfy requirements at the lowest cost. It may however be possible to incentivize the contractor to (1) exceed a basic requirement such as mean time between failures or (2) generate a higher level for an important derived requirement (e.g., one that affects operational flexibility, maintainability, supportability, etc.). Section 11.3.3.2 discusses this topic.
Applying best practices as described in Sections 11.3.3.1 and 11.3.3.2 may not be sufficient to manage and mitigate the process-based risks list above. Section 11.3.3.3 discusses how encouraging a quality focus can also contribute.
Government Contract Quality Assurance (GCQA) determines if contractual requirements have been met prior to acceptance of supplies and services. GCQA is conducted by the program manager and Defense Contract Management Agency (DCMA) as identified in contract administration delegations to DCMA by the Contracting Officer. Section 11.3.3.3 discusses some best practices for setting quality assessment and oversight requirements for the GCQA function, tailored to the expected risks.
11.3.3.1. Differentiating Among Offerors on the Basis of Quality
A contractor's quality management system is used to direct and control the organization with regard to quality. Quality management is an enterprise level process, driven by senior leadership involvement, to support the delivery of high quality products and services by ensuring that all aspects of quality are considered and acted upon by every element of the organization. The fundamental goal is to provide objective insight to assure that: customer requirements are thoroughly analyzed and understood; processes are defined and capable; and the resulting product meets the customer's needs. It interacts with systems engineering technical processes and technical management processes by focusing on both the quality of the system and the quality of the processes being used to create the system. Quality management provides objective insight into processes and work products for all stakeholders including program team members, management, suppliers, customers, and users involved with the development, manufacture, operation, and support of a system.
The quality management process begins early in the life cycle and continues throughout. The principal elements of the quality management process include:
- Objectively evaluating performed processes, work products, product/process design and services against the applicable process descriptions, standards, procedures, policies, and documented expectations;
- Understanding the full scope of customer requirements, assessing risks associated with meeting those requirements, and verifying that they are satisfied;
- Identifying and documenting noncompliance issues, especially those affecting cost, schedule, productivity, and performance;
- Using tools and techniques in a disciplined manner to determine root causes of noncompliance issues;
- Addressing noncompliance issues by initiating and tracking corrective and preventative actions to assure the root cause(s) of the defect/deficiency has been identified and removed; and
- Providing feedback to program managers, their staff, and corporate managers to identify lessons learned, improve process robustness for future projects, and evaluate trends.
While the quality management focus is on the key aspects of the product realization process (e.g., requirements, design, make/buy decisions, supplier management, production), it also encompasses supporting processes such as contracting and training. Both value-added activities and continuous process improvement should be stressed and encouraged.
Further information about quality management may be found in ISO 10005 Quality Management - Guidelines for Quality Plans (available for purchase), AQAP-2000 NATO Policy on an Integrated Systems Approach to Quality through the Life Cycle, AQAP-2009 NATO Guidance on the Use of the AQAP 2000 Series, and at Process and Product Quality Assurance in the CMMI for Development (CMMI-DEV) v1.2 or the CMMI for Acquisition (CMMI-ACQ) v1.2.
Program managers should allow contractors to define and use their preferred quality management system as long as it meets the needs of the program. International quality standard ISO 9001-2008, Quality Management Systems - Requirements, AQAP-2110, NATO Quality Assurance Requirements for Design, Development and Production, and AS 9100C:2009, Aviation, Space and Defense Quality Control Management System Standard, define process-based quality management systems and are acceptable for use on contracts per FAR 46.202-4, Higher-Level Contract Quality Requirements. AQAP-2110 and AS 9100 contain additional requirements beyond ISO 9001. AS 9100 is applicable to most complex DoD systems. The AQAP 2000 series should be considered for complex DOD systems, when the supply chain or the end products have NATO or international implications. Program managers should consider the use of additional requirements (such as those contained in the Missile Defense Agency Assurance Provisions) beyond ISO 9001 as appropriate.
Other sector specific quality management systems acceptable under FAR 46.202-4 include:
- TL 9000, Quality System Requirements for the telecommunications industry
- ISO/IEC 90003:2008, Software engineering -- Guidelines for the application of ISO 9001:2000 to computer software (available for purchase)
- QS-9000 or ISO/TS 16949:2009 (available for purchase), ISO 9000 harmonized standards for automotive suppliers of production materials and service parts in North America
To improve a contractor's quality management system, standards bodies encourage registration based upon an impartial third party evaluation. The Department of Defense does not require registration of a contractor's quality management system because registration does not guarantee product or service quality. Reasons why the Department of Defense does not require registration include the following:
- Registrars (auditors) do not look at the product;
- There have been instances where a registered contractor delivered a deficient product;
- Many companies pursue registration of their quality management system as a goal in itself or as a marketing tool; and
- Some registrars are less demanding.
Compliance to a standard such as ISO 9001 (available for purchase), AQAP-2000, AQAP-2009, or AS 9100, does not, in itself, guarantee product or service quality. These standards are management system standards that identify requirements for processes within an organization, describe expected tasks and outcomes, and explain how the processes and tasks integrate to produce required inputs and outputs. Standards are meant to enable the organization to develop a set of processes that, if done by qualified persons using appropriate tools and methods with appropriate leadership involvement, will enable a capability for delivering high quality products or services.
Product or service quality is achieved through the implementation of a strategic plan to integrate all business and technical functions that result in the consistent application of proven, capable processes within an organization. Managers must ensure that all management systems are working toward the same goals and are not creating conflicting or dysfunctional behavior. Implementing a standard is of little use if the financial system rewards individuals for delivering non-conforming products/services. Because everything a contractor does should be related to the quality of its products or services, a contractor's quality management system should be the basis for integrating all other management systems within an enterprise. Therefore, include quality management as a selection factor and look for the following elements of a quality management system in proposals:
- Effective policies and procedures that encourage the use of the system;
- Organizations with defined authorities and responsibilities;
- Objectives to drive people, processes, and the system;
- Method to analyze and resolve quality problems;
- Metrics that reflect desired outcomes;
- Interacting processes to transform inputs into outputs; and
- Records as evidence of what happened.
Furthermore, to the extent that they are available, metrics that show the effectiveness of the contractor's quality management system and processes over time should also be used to differentiate among offerors.
The following subsections describe several broad areas that have had a significant impact on quality. Topics include Customer Satisfaction, Supply Chain Quality Management, Top Management Involvement, and Continual Improvement of Performance. They provide additional guidance on items the program office and the contracting office should ask for in Requests for Proposals and evaluators should look for in proposals to make a better assessment of a contractor's quality. These items may be used to differentiate among offerors. Depending on the specific situation, there may also be other areas (e.g., competent personnel for special processes) where information should be sought.
11.3.3.1.1 Customer Satisfaction
Customer satisfaction, when quantified, is a valuable enterprise-level outcome metric. The Department of Defense has recognized the importance of customer-satisfaction performance measures. Since the passage of the Federal Acquisition Streamlining Act of 1994, all Federal Departments and Agencies have initiated procedures to record contractor performance on in-process contracts and to use past contractor performance information in source selection.
Too often in the past, the Department of Defense relied heavily upon detailed technical and management proposals and contractor experience to compare the relative strengths and weaknesses of offers. This practice often allowed offerors that could write outstanding proposals, but had less than stellar performance, to "win" contracts even when other competing offerors had significantly better performance records and, therefore, represented a higher probability of meeting the requirements of the contract. Emphasizing past performance in source selection, can help ensure that the winning teams (prime contractors and major subcontractors) are likely to meet performance expectations. When evaluating past performance data, consideration should be given to the relevancy, complexity and ultimate mission success of the contract.
Beyond the Department's past performance information, a Request for Proposals may ask for further evidence of customer satisfaction such as data tabulated from customer surveys or from complaints and equally important, how changes were made because of the results.
Supplier assessment programs may also be helpful in understanding how well a company is able to satisfy its customers. Suppliers have demonstrated some degree of customer satisfaction when they are accredited by a group of companies, in a particular sector, that joined together to agree on criteria and a process for assessing, exchanging and publishing supplier data to facilitate business relationships. For example, Nadcap is a worldwide cooperative program of major companies designed to manage a cost effective consensus approach to special processes and products and provide continual improvement within the aerospace industry; the Coordinating Agency for Supplier Evaluations (C.A.S.E.) exchanges and publishes non-prejudicial supplier data to help make informed supplier selections. Reports from consumer organizations or the media may also be useful.
11.3.3.1.2 Supply Chain Quality Management
Because quality deficiencies for non commercial-off-the-shelf (COTS) products often occur in the lower tiers, prime contractors should have insight at least two levels down their supply chain. Prime contractors, in addition to having approved vendor (i.e., subcontractor) lists, should ask their subcontractors' about planned suppliers. These subcontractors should also have insight two levels down their supply chain and flow the same requirement down to their suppliers, etc. For COTS products, all contractors should use approved sources.
It is important for DoD program managers to inform their prime contractors of their interest in quality throughout the supply chain. Therefore, through requests for proposals and corresponding proposal evaluation factors, the program office and the contracting office should request and evaluate evidence of effective supply chain management. The evidence should reflect the following characteristics:
- Relationships with suppliers that promote and facilitate communication to improve the effectiveness and efficiency of processes that add value;
- The use of supplier development programs focused on continuous improvement;
- Strategic partnerships with suppliers, over the product life cycle, that are based on a clear understanding of the partners' and customers' needs and expectations in order to improve the joint value proposition of all stakeholders;
- Processes that effectively and efficiently monitor, evaluate, verify, and improve the suppliers' ability to provide the required products with a focus on defect prevention rather than defect detection;
- Right of access for both the prime contractor and the Government to supplier facilities and documentation where applicable; and
- Requirements for the supplier to flow down analogous quality management system provisions to its subcontractors.
Because quality deficiencies often occur in the lower tiers, prime contractors, in addition to having approved vendor (i.e., subcontractor) lists, should ask their subcontractors' about planned suppliers. These subcontractors should flow the same requirement down to their suppliers, etc. For critical and complex commercial-off-the-shelf (COTS) products, the prime and its subcontractors should use their own internal processes and controls to ensure that the COTS product meets its critical attributes.
11.3.3.1.3 Top Management Involvement
Quality will permeate all levels of a company only if top management provides the leadership necessary to drive and reinforce that behavior. Requests for Proposals should also ask for evidence of top management support for quality. The following list identifies important factors in evaluating the effectiveness of top management support:
- Establishing a corporate strategic vision, objectives, policies and procedures that reflect a commitment to quality both in-house and in suppliers' facilities;
- Communicating, at every opportunity, organizational direction and values regarding quality;
- Providing structures and resources to support full implementation of a quality management system;
- Soliciting quantitative and qualitative feedback on the effectiveness and efficiency of quality management and taking actions based on that feedback, even when change may be difficult;
- Establishing a quality policy, at the highest level in the company, that commits to continuously improving processes and exceeding customer expectations;
- Reviewing the quality management system periodically with particular attention paid to achieving goals and objectives throughout the organization, customer satisfaction, and the exchange of ideas for continuous improvement;
- Setting ambitious quality objectives and promulgating them through quality policy;
- Demonstrating importance put on quality functions by providing for independent reporting channels; and
- Establishing management accountability with emphasis on quality results and customer satisfaction.
11.3.3.1.4 Continual Improvement of Performance
An offeror with effective quality management will seek continual improvement of its processes, product designs, and thereby products by improving its overall performance, efficiency, and effectiveness. Such behavior increases the likelihood of increasing customer satisfaction and enhancing an organization's competitive posture.
More specifically, all processes have defined inputs and outputs as well as the required activities, actions and resources. Therefore, process improvement encompasses both:
- Improving conformance to the defined process and
- Improving the defined process itself to add value and eliminate waste.
Such process improvement invariably leads to (work and end) product improvement and consequently increased customer satisfaction.
When asking for evidence of a strong commitment to continual improvement in a request for proposal, the following list provides considerations for evaluating a response.
- How conditions are created to promote innovation,
- How open two-way communications are encouraged,
- How corrective actions are treated as an improvement tool,
- How change is approached on a systematic, consistent basis, to include follow-through implementation, verification and documentation,
- How people are provided with the authority, technical support and necessary resources for change,
- How continuous improvement process tools are deployed company-wide,
- How self assessments, benchmarking, competitor analysis, and other metrics are used to evaluate process performance and drive improvement, and
- How capability and maturity models or reviews support an effective continual improvement process and provide both insights to the improvement process itself and objective evidence of success.
11.3.3.2 Incentivizing Higher Quality in Contracts
Contract incentives can be structured to ensure quality by contributing to the contractor's value proposition. Factors that are typically important aspects of a contractor's value proposition include:
- Customer satisfaction;
- Planning stability;
- Good financial performance; and
- Improved cash flow.
Listed below are examples of contract incentives that can be made available to the prime contractor and the prime contractor can in turn make available to subcontractors under the appropriate conditions:
- Increased fee;
- Extended contract length;
- Follow-on contracts awarded;
- Accelerated progress payments;
- Shared savings; and
- Opportunities for return on investments (some of which may increase the contractor's competitiveness on other contracts).
The following are some potential ways to use these contract incentives to improve quality, and at the same time, improve other product characteristics that are of value to DoD. Their applicability depends on the specific situation.
- Warranties. The program manager could treat the warranty as a fixed price option per item. If there are no failures, the contractor keeps the money that DoD paid for the warranty. To reduce the price of the warranty, the program manager could consider a situation where DoD pays to repair the first failure and the contractor warranties the next "n" failures. Typically the warranty should exclude combat damage, abuse, misuse, and other factors out of the contractors' control.
- Award Fee for Product Support Contracts. The program manager could make the fee a function of operational availability.
- Award Fee for Product Development Contracts. The program manager could make the fee a function of successful operational test and evaluation.
- Progress Payments. The program manager could make payments contingent on successful corrective actions taken to alleviate quality deficiencies. The program manager could also establish an agreement with the contractor to repay the fee with interest if future measurements do not meet the conditions necessary for the entire amount of the fee to be awarded.
- Share of Savings. The contract could encourage the contractor to invest in facilities, non recurring engineering, technology insertion, etc. that will result in improved performance and reduced costs. The program manager could then use the value engineering clause to repay the investment and give the contractor a share in the savings generated.
In building such relationships, the program manager should avoid actions that encourage risky behavior by the contractor. For example, by insisting on reducing cost, accelerating the schedule, improving performance beyond demonstrated technology limits, etc. the contractor may be forced to forgo quality-related processes. This may not only defeat the purpose of contractual incentives but also negate the other quality activities discussed in this section.
DAG Tutorial 
