United States
Environmental Protection
Agency
Office of Wastewater EPA/832-D-00-001
Management July 2000
(4204)
Office of Wastewater
Management
Common Sense Guide to
Quality Management
QM
QM
QM
QM
QM
PILOT TEST DRAFT: July 2000
-------
-------
EPA/832-D-00-001
July 2000
Office of Wastewater Management
Common Sense Guide to
Quality Management
U.S. Environmental Protection Agency
Office of Wastewater Managment
Washington, DC 20460
-------
Foreword
The U. S. Environmental Protection Agency (EPA) has an Order 5360.1 CHG 1, July 1998, entitled
"Policy and program requirements for the Mandatory Agency-Wide Quality System ", directs that all
Agency organizations to develop and implement a quality system to support its mission and that conforms
to requirements in the Order and in applicable extramural agreement regulations. All Offices within EPA
must comply with this order.
As part of the Office of Wastewater Management's (OWM's) commitment to providing high quality
products and complying with this order, we have developed a guide that provides a common sense approach
to quality management for managers and staff. This guide contains examples from OWM's ongoing
activities showing how they can be improved by adhering to sound principles of quality management. The
guide also contains check lists that if followed should help individuals provide improved management of the
quality of their activities. These check lists should also help document these activities in a manner that is
not overly burdensome.
We hope that you will use this guide to improve the quality of all your activities and encourage your
feedback so that we might improve the usefulness of this document in the future.
ichael B. Cook, Director
Office of Wastewater Management
n
-------
Acknowledgments
This document represents the efforts of several individuals. Gratitude is extended to each person involved
in preparing and reviewing this guide.
The authors are Dr. John Walker. Municipal Technology Branch, Office of Wastewater Management,
U.S.EPA, Lynn Riddick and Kim Conmy, DynCorp, Alexandria, VA. Special thanks go to the principal
reviewers who were and Ben Lesser and Pat Bradley of OWM and Glenn Nestel of Hagler Bailly.
in
-------
Contents
Chapters
Chapter 1. Purpose and Overview of the Guide 1.1
Applies to Virtually All OWM Activities 1.1
What Is Quality management? '. 1.2
What Level of Effort Must I Use to Obtain the Quality Needed : 1.3
What Is Quality? 1.4
What's In The Rest of the Guide? 1.4
This Document Is Meant to be Dynamic 1.6
Chapter 2. Quality Tools 2.1
Stages of a Quality Management Effort 2.1
Introduction to quality and Peer Review 2.1
Implementing Quality Review 2.3
More About Quality Review 2.3
More About Peer Review 2.4
Quality Review and Peer Review Mechanisms 2.6
Other Quality Management Tools 2.8
Standard of Widely Accepted Tools 2.8
Quality Assurance Measures and Acceptance Criteria 2.9
Chapter 3. Applying Quality Management Tools to OWM Projects 3.1
Planning Is One of the Most Important Aspects of QM 3.1
QA Tools Are Also Needed for Project Documentation 3.5
Chapter 4. QM Documentation 4.1
Document as You Go , 4.1
Documentation Benefits 4.1
Documentation Aids 4.2
IV
-------
Checklists
Optional OWM Checklist for Documenting QM Activities 4.5
Optional OWM Project Planning Documentation Form 4.8
Optional OWM Peer Review Documentation Form 4.9
Chapter 5. Examples of Quality Management 5-!
Examples
1: Response to Congress on Use of Decentralized WW Tmt. Systems .. 5.3
2: Investigation of BMP for Photo Processors 5.5
3: Storm Water Phase 1 Model for 1996 Clean Water Needs Survey . . 5.7
4: Assessment of "Solar Aquatics" Technology of Treating WW 5.9
5: 40 CFRPart 503 Biosolids Rule 5.11
6: 1998 Biosolids Quality Survey 5.13
7: CSO Monitoring and Modeling Guidance 5.15
Chapter 6. Answers to Commonly Asked Questions 6-l
"Needed Quality Requirements 6.1
Controlling Quality 6.2
Quality Control Tools 6.2
Quality Management Applies to 6.4
Documenting Quality Management 6.4
Why Bother with Quality Management 6.4
Quality Control, Quality Requirements, Quality Assurance and QM 6.5
Primary and Secondary Data 6-5
Hey! I Just Got Someone Else's Project 6.6
Quality Review 6.7
Peer Review 6-8
How Is Peer Review Different From? 6.11
Federal Advisory Committee Act as a Quality Management Tool 6.12
Appendices
A EPA Quality Assurance Requirements and Guidance Documents
B EPA Peer Review Policy
C Example Project File Contents List, ABC Project
Example Documentation for the OWM Common Sense Guide to QM
C.1
-------
-------
Chapter 1
Purpose and Overview of the Guide
Applies to Virtually All Office of Wastewater Management (OWM) Activities
This Common Sense Guide to Quality Management is for OWM
managers and staff. It provides simple, clear guidance on how each
person can use quality management (QM) principles to improve his or
her work. It also provides practical, straightforward guidance on how to
comply with the U. S. Environmental Protection Agency (EPA) and the Office
of Water (OW) quality assurance policies and how to document the steps taken
to achieve quality outputs. The procedures provided in this guide apply to
virtually all OWM activities. These activities include, but are not limited to:
Developing policies
Developing regulations
Developing and using database management systems
Evaluating technologies
Preparing strategies
Sampling and analysis activities
Managing primary or secondary data
Office or Division-level strategic planning
, Preparing fact sheets
Preparing guidance documents
Preparing briefing documents
Preparing scientific documents or documents for the public
Preparing reports for Congress
Preparing correspondence for your signature or that of
management, and
Serving as project officer, task manager, or work assignment
manager for contracts or grants
1.1
-------
What Is Quality Management?
Quality management (QM) can be thought of as an organization's "culture"
with respect to managing the quality of its products. More specifically, it is
that aspect of OWM's overall management system that determines the
requirements for quality up front and implements the policies and procedures '
needed to ensure mat the quality requirements for its products are continuously
met. OWM's commitment to quality management includes:
• Development of policy regarding quality,
• Development of this guidance, and
• Commitment of funds to train staff and implement quality
management.
Quality management as outlined by Deming's1 Total Quality Management
model is to Plan -Do- Check and Act. ;
• Planning - Define Hie quality requirements as well as all other project
requirements up front including objectives and goals, budget, schedule,
milestones, and work product application.
• Doing - Implement the project with the appropriate documentation and
document control for quality assurance; including other critical quality
management implementation requirements such as
roles/responsibilities, training, communication, operational
procedures/protocols and contingency.
• Checking - Periodically measure progress and evaluate conformance
with requirements, including quality requirements of the project; take
corrective and preventive actions as required to meet project quality
requirements.
• Reviewing - At the end of the project, review how well the quality
requirements were satisfied, apply lessons learned to continuously
improve the quality management of future projects.
A COMPANY CAN NOT BUY ITS WAY INTO
QUALITY - IT MUST BE LED INTO QUALITY BY
TOP MANAGEMENT - Deming
Qualify assurance (QA) can be thought of as the entire set of management
strategies, controls, and activities used to deliver a product that meets user
expectations regarding quality. Quality control is no longer used in modern
Deming, W.E. Out of Crisis, MIT Center for Advanced Educational Services, Cambridge,
MA, 1986.
1.2
-------
quality management systems because of its negative connotations. As used
historically, quality control assumed that a certain level of defects (mistakes and
errors) were unavoidable and needed to be caught at the end of the factory,
project, etc. Today the quality control concept has been replaced with quality
management and quality assurance that defines quality requirements up front and
manages the quality throughout so quality requirements are met at the end.
Quality assurance measures and checks are used throughout and at the end to
provide this assurance.
The purpose is not to ensure OWM staff can recite quality definitions, rather
THE PURPOSE OF THIS GUIDE IS TO
HELP OWM MANAGERS AND STAFF
IMPROVE THE QUALITY OF THEIR
WORK.
What Level of Effort Must I Use to Obtain the Quality Needed for Each
Activity?
The level of effort needed to control quality depends on the project's purpose,
complexity, and nature.
In general, the highest level of quality is needed for those work products that (1)
support regulatory decisions, (2) will have widespread use or widespread impacts,
(3) are highly innovative in approach or design, (4) involve the work of multiple
organizations, or (5) span many months or years. Another way of saying this is
that the activities required to control quality are likely to be more intense if your
project fits any of these criteria.
What is Quality?
Lower quality would likely be acceptable for an E-mail note. Li this case, the
note sender would need to think about what he or she is saying in the note, and
would likely provide the onlyreview.
It is important to say a bit more about what quality is before proceeding with a
discussion of the tools that are useful for achieving quality. A dictionary
definition of quality is the "degree of or grade of excellence." Another
definition is conformance with requirements.
Chapters 2 and 3 of this guide provide a series of procedures and tools to help
make and implement the determination of the degree of excellence or quality
needed, i.e., the quality requirements for any given effort.
1.3
-------
What's in the Rest of the Guide?
Documentation Checklists
Chapter 4 of this guide provides a documentation checklist and a
straightforward discussion of simple, effective ways OWM staff can document
their QM activities without adding cumbersome paperwork. Examples of QM
documentation in actual OWM projects are provided in Appendix C.
Primary and Secondary Data Management
This document also suggests approaches for managing different forms of data
that normally are used to develop OWM work products. It will, by example
and reference, discuss ways of managing the quality of primary data (original
new data gathered by or for the project manager to address specifically an issue
at hand) and, similarly, it will discuss the management of secondary data (data
that were originally gathered by others for one purpose that are now being used
by the project manager, his or her contractor or grantee for a different, or
secondary, purpose). Examples that illustrate how QM has been applied to
different projects and different forms of data within OWM are included in
Chapter 5 of this Guidance.
Questions and Answers
Answers to commonly asked questions regarding QM are provided in
Chapter 6.
1.4
-------
References to QM . - >.
A list of suggested references is provided in Appendix A for those readers who
wish to learn more about QM.
Peer Review Policy N
EPA has a Handbook on Peer Review (EPA-1OO-B-98-001). Important parts
of the Agency's Peer Review policy are provided in Appendix B.
An Example Documentation File
An example Documentation File comprises Appendix C.
Finally, this guidance provides a straightforward discussion of the steps needed
to comply efficiently and effectively with EPA and Office of Water (OW) QA
policies.
This Document Is Meant to be Dynamic
This is a dynamic document intended to provide you with up-to-date
information and examples that will help you and other OWM staff implement
QM strategies in your daily activities. Therefore, we strongly encourage you to
provide feedback and examples of QM to your QA Coordinator for inclusion in
future updates to this guide.
1.5
-------
1.6
-------
Chapter 2
,--#•,
Quality Tools
Stages of a Quality Management Effort
tages of QM that should apply to any OWM activity or effort include:
S
planning the effort
• determining the quality requirements
» producing the product
• assuring that the product meets the quality requirements
• documenting how the quality was assured
Several tools are available to aid OWM staff in ensuring that the desired results
are achieved. These tools, which are summarized below, may be used by
themselves or in combination with others, depending on the nature, size,
importance, and visibility of the project. Additional information about these
tools can be found in the sources cited in the footnotes and the remainder of
this Chapter. Chapter 3 describes more fully the application of these QM tools to
OWM projects.
Introduction to Quality and Peer Review
Two of the most useful tools to define, ensure, and improve the relevance and
quality of your .work products are quality review and peer review. The phrase
"quality review" has been used in this guide in addition to the phrase "peer
review" to facilitate a more comprehensive description and thorough
understanding of the steps that you can take to meet needed quality goals.
Quality Review
Quality revie\v entails the use of knowledgeable individuals to assist in
determining, for all OWM work products and processes, the relevance to customer
needs and the level of quality needed, and later, to evaluate the work product to
determine if the level of quality is being obtained. Quality review, as it is defined
2.1
-------
in this guide, includes both technical or nontechnical review and input by all
stakeholders in OWM activities.
Peer Review
Peer review is a more formal Agency process that uses technically qualified
peers (persons of equal or greater skill to your own) to ensure independently
the quality of all major, technical work products (the criteria for "major"
products are listed on Pages 2.4 and 2.5.)
Exhibit 1 highlights the similarities and differences between quality and peer
concepts. Additional descriptions are provided below.
Exhibit 1
Quality Review vs. Peer Review
Quality Review
Peer Review
•A- Can be performed at any phase of a project?
•Ar Involves active outreach to and participation of
scientific and technical experts inside and outside EPA
•k Reviewers may or may not be independent of the work
product
* Recommended for all work products or activities
In-depth assessment of the quality objectives, quality
strategies, or final product associated with the project
* Non-technical stakeholders may be used to evaluate if
a work product is clearly presented and meets user
needs.
* The focus is on planning, monitoring, and/or
evaluating quality
•A- Can be performed at any phase of a project?
* Involves active outreach to and participation of scientific
and technical experts inside and outside EPA
•k Reviewers are independent of the work product
•Ar Required for all "major, technical work products" (defined
on pages 2.4 & 2.5.)
* In-depth assessment of assumptions, calculations,
extrapolations, alternate interpretations, methodology,
acceptance criteria, and conclusions pertaining to the work
product
* Most peer reviewers are technical experts
•A- The focus is to ensure that activities are technically
adequate, competently performed, property documented,
and satisfy established quality requirements.
2.2
-------
Implementing Quality Review
Exhibit 2
Implementation of Quality Review
At the beginning! f~
TS
A quality review of planned activities is especially important at the beginning of the activity. £
It helps determine:
1. The appropriateness and relevance of the task. If the planned task is not relevant, changeitso
that it is or do not do it.
2. The quality requirements for each of the activities required during the project.
3. The ability to meet the quality requirements, given the resources and time available.
4. What to do if the quality needed is greater than available resources allow: (i) change the
approach which might include ways to leverage and/or change the level of resources; (ii) tell
managers that the level needed cannot ~be attained and identify what level can be attained with
what shortcomings, or (iii) do not do the task.
In the middle! ¥
£
Quality review by peers during the task will help assure that the level of quality you
determined was necessary and possible is being attained. When coupled with peer input, it also can
provide an opportunity to obtain suggested midstream corrections to address unforeseen problems.
(Note: Although this is a helpful strategy in some projects, please remember that any consolidation of
peer input and quality review functions during project implementation may preclude the quality
reviewers from serving as independent peer reviewers of the final product). This could result in
project delay or retraction.
X % /
And at the end! * -
•3T
•&•'
Quality review at the completion of the final draft of a report or project further helps ensure
that a relevant quality product has been attained and may also be useful in interpreting peer
reviewers' comments.
More About Quality Review
Quality Review Involves Peer and/or Stakeholder Input and Consultation
Quality review during the planning, development, and at the completion of a
project requires the assistance of recognized, qualified peers who also
adequately represent stakeholder concerns. This consultation or input of peers
during the development of an evolving Agency work product, which also includes
2.3
-------
the open exchange of data, insights, and ideas, is sometimes referred to as peer
input or peer consultation. :
• A good example of peer input is the input received from workgroup
members during development of a product. These workgroup members
have an active, ongoing participation in developing the work product.
• Another example of peer input is that received from stakeholder
representatives who have been asked for general comments on a draft
work product by an OWM project manager. (Stakeholder representatives
often include experts who could be considered "peers.")
• Peer input or peer consultation can be thought of as a subset of the quality
review process. ,
Eligibility of Quality Reviewers for Peer Review
Peers or stakeholders who provide active ongoing input and participation
during the development of a work product are not eligible to undertake a peer
review of that work product because of their lack of independence from its
development.
More About Peer Review (See also Appendix B).
Peer review is an essential Agency requirement for review of technical products
and the scientific and technical aspects of major products. This documented
critical review is conducted to ensure that activities are technically adequate,
competently performed, properly documented, and satisfy established quality
requirements. The peer review is an in-depth assessment of the assumptions,
calculations, extrapolations, alternate interpretations, methodology, acceptance
criteria, and conclusions pertaining to the specific major scientific and/or technical
work product and of the documentation that supports them.2
Peer review is a critical and, most often, final evaluation of the work product.
Like quality review, it may also be used as an interim evaluation of 3; work plan,
preliminary draft or the like, and it constitutes active outreach to and; participation
of the broad scientific, engineering, and economics communities external to as well
as within the Agency.
Major Product
The determination that a scientific or technical product is major is based on
consideration as to whether it meets at least one of the following criteria:
Defined in the Science Policy Council Handbook for Peer Review,
EPAUOO-B-98-001, January 1998.
2.4
-------
• Does it support major regulatory decisions or policy/guidance of major
effect? •'-;-
• Does it establish a significant precedent, model, or methodology?
• Does it address controversial issues?
• Does it focus on significant emerging issues?
• Does it have significant cros's-Agency/inter-Agency implications?
• Does it involve a significant investment of Agency resources?
• Does it consider an innovative approach for a previously defined
problem/process/methodolpgy?
» Does it satisfy a statutory or other legal mandate for peer review?
The Assistant Administrators and Regional Administrators are the ultimate
decision makers and are accountable for implementing Peer Review Policy within
their organizations. They may designate Office directors and Division Directors
of other appropriate level line-managers as the front line decision makers. The
principal agency staff are decision makers and their line-managers - peer review
leaders, and peer review coordinators and certain staff within ORD. The decision
makers decide whether a work product is major and needs peer review, and what
peer review mechanism to use. They also commit resources and ensure that peer
reviews are properly performed and documented. The decision makers also
designate a peer review leader to organize the peer review.
Who Are Peer Reviewers?
The peer review is conducted by qualified individuals (or organizations) who are
independent of those who performed the work, but who are collectively equivalent
to them in technical expertise (i.e., peers). These subject-matter experts will have
had no previous substantial participation in the development of the work product.
These individuals may be both from inside and outside of EPA.
Caution!
"Peer review" is not the same as quality review, stakeholder review,
or public comment. Neither stakeholder involvement nor public
comment are considered adequate substitutes for formal peer
review. '
Although independent peers can be employed for review during the initial planning
stages in addition to the final review, considerable care is needed to avoid the
perception or fact that these peers have become an actual part of the planning and
production process. If this happens, these peers run the risk of being perceived as
incapable of giving an independent final review, and the final product could be
delayed or retracted.
2.5
-------
Other Benefits of Peer Review
Peer Review helps to ensure better acceptance of your document by improving the
way it is presented and the conclusions and recommendations it reaches. Peer
review allows the Agency to determine if the work is judged as credible by
relevant experts who deal with the Agency and that the scientific and technical
underpinnings of its decisions are based upon the best current knowledge from
science, engineering, and other domains of technical expertise.
WHETHER "OFFICIAL AGENCY PEER REVIEW" IS
REQUIRED OR NOT, YOU WILL HAVE A MUCH MORE
RELEVANT, USEFUL AND ACCEPTABLE PRODUCT-
USUALLY WITH LESS PERSONAL STRESS AND MORE
SELF-SATISFACTION -- IF YOU FOLLOW THE QUALITY
REVIEW PROCESS OUTLINED ABOVE.
Additional information regarding the official Agency peer review process can
be found in Appendix B.
Quality Review and Peer Review Mechanisms
Quality review and peer review support can be provided by groups or individuals,
or a combination of the two. The most common approaches are to use
workgroups, committees, individual experts from universities, contractors, or
professional organizations in addition to experts within the Agency.
Workgroups
Workgroups are interdisciplinary teams of individuals with an interest and
experience in aspects of a project and who work together to plan and conduct a
particular project. The participants in the workgroup typically represent a variety
of program offices and missions. Workgroups can be used as a tool for obtaining
peer input, quality review, arid/or peer review services from more than one
individual. Note, however, that if a workgroup has had substantial involvement in
the development of the product, neither the workgroup nor its members can serve
as formal Agency peer reviewers. Therefore, it may be desirable to create more
than one workgroup for OWM projects that require both cross-program
development and cross-program review.
Teams
The use of cross -functional, multi-skilled teams is a strong quality management
concept and has become increasingly important at EPA. Teams are often being
used to plan and carry out specific tasks. However, teams can also be a good
mechanism for improving communication and coordination and for managing
quality. Some teams are fairly similar to workgroups while others are more
2.6
-------
formally established and tend to take on tasks as a group rather than as
individuals. Similar to workgroups^ the team participants are often
interdisciplinary in nature and sometimes are structured to represent a variety of
program offices and missions. Teams can also be used as a tool for obtaining peer
input, quality review, and/or peer review services from more than one individual.
As with work groups, if the team has had substantial involvement in the
development of the product, neither the team nor its members can serve as formal
Agency peer reviewers.
Federal Advisory Committee Act (FACA) Committees
The Federal Advisory Committee Act of 1972 established a system governing the
creation of advisory committees in the executive branch of the federal government.
Advisory committees are to be established only when necessary and must be
established under standard procedures. These committees also must be reviewed
on a routine basis and terminated when they are no longer accomplishing their
purpose. Examples of FACA committees include the statutorily required National
Drinking Water Advisory Committee, the discretionary Urban Wet Weather Flows
Advisory Committee, the Science Advisory Board, and the FIFRA Science
Advisory Council.
2.7
-------
Federal Advisory Committee Act (FACA) committees, like workgroups, may be
used to provide peer input, peer review, or quality review services from more than
one individual. Because rules regarding FACA committee meetings and
membership are typically more cumbersome than those involving workgroups, use
of a workgroup instead of a FACA committee is recommended for most OWM
projects. Use of a FACA committee is generally limited to large, highly complex
or controversial projects, or when it is desirable to obtain a consensus
recommendation from a group of individuals rather than a variety of individual
perspectives. There are very specific rules that apply to the need for and operation
of FACA committees and advice from the Office of General council might be
helpful to resolve any uncertainty.
Other Quality Management Tools
The remainder of this chapter describes other tools that OWM staff and managers
can use to plan, implement, monitor, and improve the quality of their; activities.
Specific ways in which these other tools can be used are also described in
Chapter 3.
Standardized Processes
Standardized processes, which can range from well-documented methods, to
standard operating procedures, to flow charts, are designed to reduce the
variability in activities that are performed repeatedly by numerous staff.
Standardized and documented procedures are particularly useful tools for data
gathering activities (such as sampling and analysis) or data management activities
(such as data review, database access, storage, and retrieval) that will be
performed by more than one individual or organization. In such cases, the up-
front investment of resources to document and apply these procedures can offer
the following benefits to OWM project managers: ;
• Consistency in performance
• Improved data comparability, credibility, and defensibility
• Reduced errors
» Increased efficiency in performing tasks, thus lowering costs
• A historical record of the processes used.
Standard or Widely Accepted Tools
Widely accepted or standardized tools, such as models, statistical analysis
software, data management software, or even word processing software can
greatly improve the quality of OWM products because these tools have been tested
already and evaluated by other organizations performing similar activities. Some
questions to ask when selecting such tools are:
• Is it readily available to all project participants?
2.8
-------
• Will it meet the needs of the project?
• Do all project participants have access to the same version of the
tool?
• If an update is issued during the lifetime of this project, will
participants begin using the updated version?
Because standardized tools can provide valuable benefits, it pays not only to
consider the use of standardized tools but also the ways in which these tools will
be used. Word processing software is the most obvious (and frequently
overlooked) example of the benefits of using standardized tools in standardized
ways. For example, newer versions of WordPerfect offer extensive document
management capabilities that, when used properly by all parties, can be a valuable
resource for controlling document quality, tracking document revisions, and
tracking workgroup comments. However, these same features can become a
burdensome nightmare if used improperly or inconsistently.
Quality Assurance Measures and Acceptance Criteria
Within EPA, QAis traditionally thought of as a series of procedures used to
monitor and control the quality of sampling and analysis data. In reality, QA
measures can be applied to all projects, including those that do not involve
sampling and analysis. Exhibit 3 highlights ten QA measures that allow full
characterization of the quality of primary data that are being collected, and, if
appropriate parameters are available, which can be used to help determine the
quality of data that have previously been collected for similar or different
purposes by other persons.
Exhibit 3
Quality Management Requirements Applicable to Sampling and Analysis Activities
Calibration linearity
Calibration verification
Absolute and relative retention time precision (for chromatographic analyses)
Initial precision and recovery
Ongoing precision and recovery
Analysis of blanks
Surrogate or labeled compound recovery
Matrix spike and matrix spike duplicate precision and recovery (for non-isotope dilution analyses)
Method detection limit demonstration
Analysis of reference samples _____ -
2.9
-------
Primary and Secondary Data
There are different ways of controlling and evaluating primary and secondary
data. If you are conducting any sampling and analysis activities for your
project, (i.e., collecting primary data), you should consider using all the QM
elements shown in Exhibit 3. Reasons for omitting any of these measures
should be documented in your QM file.
If you are gathering secondary sampling and analysis data to support your
project (e.g., analytical results that have been generated for another project or
purpose), you should attempt to obtain and evaluate any and all QA results
associated with those samples. If QA results are unavailable with secondary
data, it may be difficult, if not impossible, to define the quality of your data.
Several options are available to help determine the relevance and
appropriateness of the secondary data. These are:
• Evaluate the data against historical results and other data sources in
order to determine if they are within the "realm of reasonable
expectations"
• Consult with experts (e.g., peer input or quality review): who are
familiar with your type of data,
• Request results from any performance evaluation samples that may
have been performed by the involved laboratories during the initial data
gathering time frame, or '
• Audit the facilities and staff involved in gathering the data to assess the
quality of their data gathering (if the same facilities, staff, and
processes still exist).
Evaluate Sampling and Analysis Activities Against Project Requirements
QA results from sampling and analysis activities should be evaluated against
your project requirements. As a rule, it is simplest to rely on method-defined
QA acceptance criteria (i.e., performance specifications) for each of the
measures shown in Exhibit 3. However, it may be worthwhile to relax these
acceptance criteria when significant savings can be realized by doing so and
when such changes do not compromise project objectives. Similarly, it may be
worthwhile to relax your criteria if the secondary data you obtain do not meet
method-defined QA acceptance criteria but are sufficient to meet your needs.
Non-Sampling and Analysis Activities
All projects, including those that involve sampling and analysis, can benefit
from such simple QA measures as defining project timelines, milestones, and
budgets (see Exhibit 4.) More complex projects also can benefit from early
2.10
-------
testing or review (e.g., alpha testing, quality review, and peer review) and
testing or review of final drafts (e.g., beta tests, pilot tests, and peer review).
Exhibit 4
Quality Management Requirements Applicable to Non-Sampling and
Analysis Activities
Identification of key project milestones
Identification of target dates and budgets for completing each milestone
Monitoring project status against project schedules and budgets
Alpha testing to provide early assessment of the project
Beta testing of the final draft product
Pilot studies
Surveys
Surveys can be designed and implemented for several reasons. For example,
they are useful for collecting data needed to complete a project (e.g., a survey
of data resources available) and for determining the project scope (e.g., a
survey of customer needs). They can also be useful for evaluating project
quality (e.g., a survey of product users to determine if the planned task will
adequately meets the users needs) and to identify areas for improvement.
Alpha Testing (analogous to Quality Review)
Alpha testing is a quality control process originally designed to test computer
hardware or software; it is increasingly being used for initial testing of any type
of project. During an alpha test, a limited number of experienced users, both
internal and external to the developer's (project manager) organization, advise
the developer on the needs of the customer, the design of the interface, and the
selection of technology.
2.11
-------
Beta Testing (more like Peer Review)
Beta testing is typically applied to software products prior to public release.
Beta testing is the last stage of testing, and normally involves sending the
product to a number of users external to the developers organization for real-
world testing to determine if the product meets user requirements and
expectations/Beta testing is often preceded by a round of testing called alpha
testing.
Pilot Studies
Pilot studies are similar in concept to beta testing, but normally apply to non-
software related products, such as guidance documents and pollution control or
treatment techniques. Pilot studies typically involve implementation of the
product on a small scale. Pilot studies should be designed to capture as many
variables as possible within a small budget and time frame.
Quality Assurance Project Plans (QAPPs)
Although quality assurance project plans (QAPPs) have been used historically
for primary data gathering activities, they, like most other QM tools, can be
valuable in improving the quality of any project. This is particularly true of
large projects that involve multiple organizations and activities because the
QAPP describes the purpose of the project, quality objectives, roles and
responsibilities in implementing the project, resources needed, strategies for
implementing the project, and procedures for evaluating the quality of any data
used in the project. As a result, the QAPP provides a mechanism for
documenting the results of the quality planning process and serves as a tool to
ensure that all project participants understand project requirements, (i.e., that
., .,,
1 " -.-:"•."•. Y,v>\v ••• •••"* V' - "
#.,.-. *-. »' ,' ,,:,„//'" >
2:12
-------
everyone is following the same procedures and that the procedures are clear).
Requirements for ff QJfproject Plaff ((fkPP) are outlined in EPA QA/R-5,
EPA Requirements for Quality Assurance Project Plans, Draft Final, October
1997. Guidance for preparing a QAPP is outlined in EPAQA/G-5, Guidance
on Quality Assurance Project Plans, EPA/600/R-98/018, February 1998.
2.13
-------
-------
Chapter 3
Applying QM tools to OWM Projects
Planning is One of the Most Important Aspects of QM
Careful planning can save you thousands of dollars as well as weeks,
months, and even years of wasted effort. Although projects vary greatly
in scope and importance, each project should be started in essentially
the same way—by determining the relevance of the project, the level of quality
required, and by planning accordingly. The nature of each OWM project
drives the level of quality needed on the project. Two very important things to
realize are:
(1) The same level of effort is likely to be involved in doing a relevant or a
non-relevant task. However, the usefulness to your customers and your
ultimate self-satisfaction will be much greater for the relevant task,
even if the relevant task was not conducted in an optimum manner.
(2) Poorly planned projects often have flawed results and are more difficult
to do than projects that are carefully planned.
Planning Questions
The planning process should address the following questions:
• What are the quality requirements for my project?
• Can I achieve these requirements within a reasonable time frame with
available technical, financial, and staffing resources?
• Can I modify my project design or scope to ensure that my product will
meet quality requirements?
Additional issues that should be considered when addressing each of these
questions are shown in Exhibit 5.
3,1
-------
Exhibit 5. Quality Management Issues to Consider When Planning Projects
What level of quality do I need? Is this level of quality achievable? Should I modify my
project design?
• What is the primary purpose of the project?
• How is the project relevant to OWM's mission, and why is it important to proceed?
• Who is the project customer (e.g., Senior EPA management, the public, Congress, the
regulated community, etc.)?
• What are the customer's requirements? '
• What are the project goals?
• How can I measure the success of my project (e.g., through quantitative measures, surveys,
peer review, etc.)?
*
• What activities must be performed to meet project goals?
• What staff members are needed to complete these activities? Are these staff available? If
not, what other options exist (e.g., will staffing limitations dictate achievable project quality
or project design?)
• What resources and materials are needed to complete project activities? Are these
resources/materials available? If not, what other options exist (e.g., will resource limitations
dictate achievable project quality or project design)?
What is the required project completion date (e.g., will the targeted completion date dictate
project quality or project design?)
• What QM tools and requirements are appropriate for the project?
• What possible problems might arise during the project and what up-front actions can I take
to mitigate them?
If time, budget, staffing, or other resource limitations do not allow the required project
quality to be achieved, can I redefine the study, scope, or design? Can I postpone'the
project? What other solutions can I take to address the problem?
3.2
-------
Expert Assistance
In many cases, OWM Project Manager! may find it helpful to seek expert
assistance in answering these questions. Expert assistance may be provided
through informal meetings and discussions with staff. These persons may be
within or outside of EPA. Expert assistance may also be gained from
appropriate individuals in trade associations, environmental groups, interoffice
workgroups or FACA committees assembled to provide quality review,
technical oversight, and/or peer review support to the project.
Q: Who are the experts, and how do I find them?
A: Suggestions for finding experts include: seeking recommendations
from fellow co-workers, reviewing the public literature for widely
published authors, your quality assurance coordinator, or contacting
stakeholder organizations for recommendations.
Quality Gaps
Quality review at the planning stage, can help identify any potential gaps in
quality requirements for each project. To address these gaps, quality reviewers
may provide solutions or suggest advisors to the Project Manager. Typically,
such gaps occur when insufficient resources are available to meet project
objectives, but they also may occur when technology is not sufficient to support
project needs. The Project Manager can address these problems by:
" leveraging additional resources,
» modifying the project design and scope to reduce the level of
resources or type of technology needed, or
• clearly identifying to senior management that the level of
quality needed cannot be attained. Managers can then decide
to accept the level achievable with limited resources and
technology or to cancel the project.
A real-life example of this situation is Example 6 "1998 Biosolids Quality
Survey" in Chapter 5. Quality review meetings during the planning stage of
this activity convinced the Project Manager that the original project objectives
could not be met within the predefined budget. In subsequent meetings, the
quality review team assisted the project manager in identifying alternative
solutions that could be presented to the Assistant Administrator (AA). Later,
they assisted in designing a study to meet refined project objectives that were
approved by the AA after having learned of the resource constraints.
3.3
-------
Additional Quality Tools During Planning
Depending on the level of quality needed for the project, the OWM Project
Manager may want to consider the use of these additional tools during the
planning phase: i
« Use of internal or external peer reviewers to review the final
project plan, quality assurance project plan, grant proposal, or
other planning document. This is advisable when the project is
likely to be costly, highly controversial, or highly innovative.
« Assembly of a PACA committee or an interoffice workgroup to
provide oversight, recommendations, or peer input into the
overall project.
• Development of a formal QA Project Plan (QAPP) in
accordance with requirements.
3.4
-------
TIP: WHEN PLANNING LONG TERM PROJECTS,
PLAN FOR MID-STREAM ASSESSMENTS AND
CORRECTIONS.
ffljjgfflgftffiffi^^fe^^^^
A QA Project Plan (QAPP) is Required for Collection of Primary
Environmental Data. EPA policy requires that all work performed by or on
behalf of EPA that involves the collection and use of primary environmental data
be implemented in accordance with an Agency approved QAPP. The QAPP is a
critical planning document used to define and record how QA activities will be
implemented and assessed on the project. Requirements for a QAPP are outlined
in EPA QA/R-5, EPA Requirements for Quality Assurance Project Plans, Draft
Final, October 1997. Guidance for preparing a QAPP is outlined in EPA
QA/G-5, Guidance on Quality Assurance Project Plans, EPA/600/R-98/018,
February 1998.
QA Tools Are also Needed for Project Documentation
Effective management of your product's quality does not end with project
planning. To make the most of your resources, you should implement QM tools
throughout the duration of your project, evaluate the success of your product
against your quality objectives, and, where appropriate, refine your management
strategy to improve the quality of your product. For example, a long-term
regulation development effort involving the use of highly innovative data could
benefit from continuous involvement and oversight of external reviewers who
review the project plan during the planning stage, review the quality of data
gathered by OWM to develop the regulation, and OWM's technical methods and
assumptions for analyzing and interpreting the data, and
review OWM's presentation of its findings (e.g., its <_^ ^^,
technical support document). Examples 1 through 5 in ^ •-,
Chapter 5 all illustrate these principles.
*%/? O°
QM Tools During Task
QM tools that can be used to monitor and evaluate project quality during project
implementation include:
» Workgroup or committee oversight of technical activities
» Internal or external peer review of interim products and drafts
• Using widely accepted, peer-reviewed data, models, approaches, or other
tools when developing the product
3.5
-------
• Verification that assumptions during project planning were valid
• Monitoring of actual schedules and budgets against planned schedules and
budgets.
• Alpha testing (defined in Chapter 2) to obtain advice on the needs of the
customer, the design of the interface, and the selection of technology.
QM Tools at End of Task !
The last component of the QM cycle is to improve the quality of your current
product, or even of your next product by incorporating comments,
recommendations, and other improvements into your project activities. Midstream
improvements designed to address deficiencies or problems identified during the
course of project implementation can often save the Agency months or years of
wasted efforts and needless embarrassment or litigation.
Some quality management tools you might use to evaluate and improve the quality
of your final work product or your next work product include:
• External peer review to ensure that your final product meets ;original
project objectives, is clearly presented, and is technically accurate
• User/reader/stakeholder surveys designed to determine if the product is
useful, if it meets its intended purpose, and if additional activities,
guidance, clarification, are needed
3.6
-------
• Formal and informal.public comments about the usability of the product,
the technical approach to developing the product, and recommendations
regarding further activities.
• Beta testing (defined in Chapter 2) or pilot studies to determine if the
product meets user requirements and expectations.
Example 5, the 40 CFR Part 503 Biosolids Rule case study in Chapter 5,
illustrates the concept of quality improvement for a situation in which a second
formal peer review process was undertaken two years after project completion. In
this case, the second peer review was used to further verify the extent to which the
science and approach used in the rule-making process was valid, (i.e:, as a means
of evaluating the performance of the final product and identifying further areas of
improvement.)
3.7
-------
-------
Chapter 4
Documenting Your Quality Management
Document as You Go
D
ocumentation is not a distinct phase of quality management. It is an
ongoing requirement throughout all phases of the quality management
process.
IT IS OFTEN ARGUED THAT IF YOU DID NOT
DOCUMENT YOUR QUALITY MANAGEMENT
ACTIVITIES, YOU DID NOT PERFORM THEM.
Documentation Benefits
Documentation helps you:
Inform managers, team members, and stakeholders of project objectives,
design, and limitations
Defend the quality of your data against criticism
Defend, your work product against litigation
Ease the transition of new project managers and staff brought into the
project midstream
Demonstrate that you fulfilled Agency requirements concerning quality
management
Identify areas for future improvement and build on lessons learned
4.1
-------
Documentation is also a technique for monitoring and measuring technical and
financial progress. It also serves as a source of good backup information on
performance—past and present, and can ease transitions of new project managers
and staff brought into the project midstream.
Tip: Protect your project and ensure all staff members understand
requirements by documenting your activities throughout
project duration!
"It is characteristic of committee discussions and decisions
that every member has a vivid recollection of them and that
every member's recollection differs violently from every
other member's recollection" — Jonathan Lynn and
Anthony Jay, Yes, Prime Minister
Documentation Aids
Documentation does not have to be difficult. This chapter is designed to provide
you with several options for documenting quality management activities without
adding to your paperwork burden. Several different approaches and checklists
are provided in Exhibit 6-9. You may use some, all, or none of these checklists,
but you are required to document your QM activities in some way. Three possible
documentation options are: :
• A checklist designed to help you reference documentation that already
exists in other sources
• A checklist designed to help you document, in brief bullets, answers to the
quality management planning, implementation, assessment, and
improvement questions raised in Chapter 3, and/or
• A checklist designed to aid you in planning and documenting your peer
review activities.
These checklists were created to assist you. Exhibit 6 also provides
recommendations regarding minimum documentation requirements for specific
QM activities. Finally, Appendix B provides examples of QM documentation for
some of the case studies provided in Chapter 5, including completed versions of
the checklists provided in this chapter. [It is important to document pertinent
information about roles, responsibilities, training, communications,
operational controls, and inspections.}
4.2
-------
Exhibit 6. Documentation of QM Activities
Quality Review
1. A list of peers and their affiliations that helped plan the task with several words about why selected.
2. A brief statement of the quality needed and attainable with available resources.
3. A list of peers that are involved in oversight plus affiliations and why they were selected (may be the
same as the planner peers).
4. (Optional). A copy of the written "charge" given to reviewers.
Peer Review
1. A list of knowledgeable peers who have reviewed the product. Give peer reviewer affiliations and
several words about why they were selected.
2. A copy of the "charge" given to peer reviewers. The charge provides specific instructions for the
areas which the reviewer should focus on during the review.
3. The major points made by the reviewers and how these were addressed.
4. Written justification for any major work products that did not undergo peer review, including
signature approval from the AA. ___
Other QM Tools
1. Copies of any standardized processes (e.g., methods, flow charts, SOPs, etc.) used in the project.
2. Copies of any surveys and survey results used to determine project scope or evaluate project quality
3. Description of any pilot testing, alpha testing, or beta testing performed on an interim product.
Include names of testing participants, test results, and changes made as a result of test.
4. Copies of any QAPPs prepared for the activity.
4.3
-------
Contractor/Grantee Support
1. Document all important events included in the pre-award, award, management, and closeout phases of
the agreement and include this information in the project file.
Identifying Information: agreement identification number, title, program office, project manager
and their telephone number and mail code; and
Management Personnel: names, title, telephone number, and mail code of the project officer's
management chau>of-command through the approval official.
2. OWM project managers should establish a routine practice of reporting to the file as soon as the
agreement has been activated. This reporting may take the form of a memorandum to the file or a
brief note. The key to effective post-award assistance management is documentation; a paper trail in
case of a problem, or in other unforseen situations. A good post award management plan as required
by OWM for grants should meet these documentation requirements.
3. Include information regarding contractor or grantee performance in the file. Examples of such
documentation include: award fee evaluation results, commendations and awards for superior
performance, and cure notices or other written correspondence to direct specific corrective actions for
performance problems. Other examples include records of all meetings, conferences, site visits, and
phone calls related to the quality of the contractor or grantees performance.
4. Maintain all documentation regarding the financial performance. This includes documentation of
original and revised cost estimates, as well as documentation of actual costs. Documentation that
explains any significant deviation between estimated and actual costs also should be included in the
file.
4.4
-------
UJ
hj
>
I-
O
O
z
I-
LU
o
o
o
X.
o
U-
co
o
UJ
O
O
Q.
O
o
Is
c a.
f«
O6
Grant Application Peer Review Charge
Plan Contract Work Assignment Task
Written Correspondence (cite date, parties) _
Other
o
*•»
State the relevance of this project to
program mission
CM
Status Report QAPIan Peer review Charge
Quality Review Charge Study Plan
Written Correspondence (cite date, partles)_
Other
in
"S
CL>
State customer (internal or external
and specific requirements-
CO
Q>
2>
Grant Application Peer Review Cha
Quality Review Charge Study Plan
Written Correspondence (cite date, parties) _
Other
State the goals of the project
* Are they measurable?
•t
S
S
1
i
Organizational Chart (cKe date andsource)_
Grant Application QAPIan Study Plan
Other
^_
£
±;
State specific staff roles/responsibi
this project
»• Are in-house staff and staff
responsibilities identified?
(/)
0)
«
O in
•• Are staff and corresponding
responsibilities from other offic
identified?
* Are contractor staff and respon
identified?
»
Grant Application Study Plan WorkPlai
Other
in u
"5 Q.
State specific resources and materi
necessary for this project; for exam
> Data to be used
>• Hardware & software
-------
CO
111
a
o
UI
o
u_
CO
O
HI
O
§
ion (cite
date, par
eview
ponde
e-
fo
to
red
ed
it
u
req
d?
equ
re
r
titify appropriate
mple,
Are peer reviews
Are teams requir
Are workgroup
JSi HIi A A A
1*1
!s||
n
d
P-S'
Gr
St
Co
W
r this
ealisti
for
re
-1 "
= 1 a
g
la
and relev
assessed
qu
ca
.c -a
S &
.£ «
E c
L. c
a) o
•K u
4) (U
O u
2 o'c-c
*i Or "^ w
0§k O
v
-------
(0
LU
a
o
F
z
HI
O
o
Q
01
o
u.
H-
cn
o
UJ
O
O
O
Q.
O
C
o
*: ^
5 >:
C Q.
0) Q.
E «
ii
Q —
•s«
§ £
3"
|
lls
ll2
c o
O
*J
c
o'
z
1
i
1
Explain or cite location of
£
Assess the plan's achievability wit
existing resources
>. Staff
> Funds
* Time
CM
T"
0)
a .
£
3
U
i
l
t
§
[3
1
O
!
"o
s
Cite any written documen:
with QAC
*r
Discuss project plans with your Q/
Coordinator
CO
*~
1
"g
1 «
Final Report (document d,
Status Report (cite referel
Other
O)
c
•o
V)
Q)
"5.
E
c»-
Describe the use of quality review
>• Were quality reviews used duri
project planning?
* Was quality review used on co
drafts or on parts as produced
?
i
"c
S
p-
|
1
Final Report (document d
Status Report (cite referei
Annual Peer Review Subn
Other
§
,—
Describe the use of peer review
> How was the extent of peer rev
needed determined?
m
T~
i
c
1
Final Report (document d
Other
a
j:
*
1
Describe the quality assessment c
data used
(O
T"~
i
1
1
"E
Final Report (document d
Status Report (cite refere,
0
&
t
^
o>
£•
*c
Describe project technical monito
example,
»• Site visits
> Conference calls
>• Review of progress reports
T-
t
53
1
a ?
Final Report (document o
Status Report (cite refere
Other
V
••
c
0
Describe any problems that arose
project
CO
P3A311PV U33S SBH ^}!IBn^) }Eqj JUQUISS3SSV
-------
CO
LU
a
o
111
S
ID
O
o
Q
0^
o
u.
CO
o
LU
O
o
o
Q.
O
c
o
Is
C Q.
(U Q.
E w
11
Location of Di
(circle all f
1=3
ii
if2
|o
u
_«)
1
15
y
o"
z
nded Work Plan
o
E
c
Amended QA Pla
Other (describe)_
•g
to
8.
2
o.
c
to
0
i
!*3
(/)
3
W
^\
c
TO
1
O *i
cn "5
0) JS
a 1o
a>
*™
nc/ed Wor/c P/an
ID
C
Amended QA Pla,
Other (describe) _
•o
, C
C TO
« to
0) 0) v
u 3
ll
o
Describe:
O)
,E
o
w
'E
o
to
O
u
8
*Q*
k.
O.
0)
u
10
(U
Q
^_
CM
o
-------
OPTIONAL OWM PROJECT PLANNING DOCUMENTATION FORM
Project Title:
OWM Contact:
Primary purpose of project
Relevance of project to program mission
Project Goals
Measures used to determine if goals can be met
Staff Roles and Responsibilities (list or attach org chart)
Does the project rely on data? (Circle all that apply)
Primary data Secondary data No data
Describe how quality and relevance of data will be
assessed
Other resources and materials necessary for project (internal and external)
Project Budget
Anticipated problems meeting project objectives
or schedules
Project time line (list key milestones or attach project
schedule)
Possible solutions to mitigate problems
Anticipated Quality Management Tools (circle all that apply)
Quality review Internal peer review External peer review QA Plans
Data review and data quality assessment
Other (describe):
Discussion of Project Plans with QA Coordinator (document dates of discussion and major
issues/recommendations)
4.9
-------
OPTIONAL OWM PEER REVIEW DOCUMENTATION FORM
PART I - DOCUMENT IDENTIFICATION
Title of Material to be Reviewed: ;
Primary Author or Manager Responsible for Document:_
Type of Document (circle one): Guidance Report Regulation
Other (if other, explain)
PART II - PEER REVIEW OBJECTIVES
Type(s) of Peer Review to be Conducted (circle one or more):
Internal Agency Expert(s)
External Panel of Experts
Science Advisory Board
Interagency Committee
National Academy of Science
Committee of Another Agency
Internal Panel of Experts External Expert(s)
Agency-Appointed Special Board or Commission
Other Agency-Based Federal Advisory Committee
EPA-Sponsored Peer Review Workshop •
National Research Council
Other (please explain)
Summarize Mission/Charge to be Provided to Peer Reviewers (attach if desired):
Name of Person Approving Mission/Charge:_
Peer Reviewer Qualifications and Diversity (Describe the minimum qualifications and diversity required of
individuals selected to review the document Attach if necessary):
Are Funds Available for Peer Review? Yes No
If yes, describe (e.g., contract funds, travel funds, grant funds, etc.) _
4.10
-------
OPTIONAL OWM PEER REVIEW (DOCUMENTATION FORM (CONTINUED)
PART III - PEER REVIEWER INFORMATION
Peer Reviewer Name(s) and/or Organization:
Name
Name
Name
Name
Name
Name
Affiliation Date of Review
Affiliation Date of Review
Affiliation Date of Review
Affiliation Date of Review
Affiliation Date of Review
Affiliation Date of Review
PART IV - Peer Review Recommendations
Was peer review conducted throughout document development? Yes
If yes, were comments incorporated into document before proceeding to next level? Yes
Summarize nature of peer review comments and recommendations (attach if necessary):
No
No
Was a response written to address peer review comments?
If yes, please cite document and author or attach. If no, please explain.
Summarize overall benefit/improvements of the peer review process below:
Yes No
4.11
-------
-------
Chapter 5
Examples of Quality Management
This chapter provides examples of actual OWM projects in which the
quality management tools and techniques described in Chapters 2
through 4 were applied. These examples are intentionally diverse in
scope and quality management approach, and were selected to help
OWM project managers understand the importance of quality
management and its application to OWM projects. Some of the tools
presented in these case studies were identified at the beginning of the
project; others were incorporated into the project as it progressed.
Each example is divided into the following sections:
What can be learned from this example? This section summarizes
the reason we have included the example in this guide.
Project Background. This section provides a brief history of the
project, including the parties involved, major project objectives, and
final work product.
Quality Management Challenges. Each project is unique in terms of
potential obstacles for successful completion while accomplishing its
goals. Possible difficulties are listed in this section.
OWM's Quality Management Strategy. The basic processes for
implementing the use of quality management tools are summarized in
this section. Description of any workgroups formed and plans for peer
input, quality review, and peer review will be found here.
Final Work Product/ Current Project Status. Depending on
whether or not the project has reached completion, the title of this
section will differ. If it has been completed, then the final work product
5.1
-------
will be described briefly, and any future plans resulting from the
product will be described briefly. If the project has not yet reached
completion, future project plans will be noted, along with anticipated
work products and outcomes. ;
Quality Management Tools Used to Implement the Strategy. The
tools utilized in implementation of the strategy described in a previous
section are organized in this section by the type of tool used. Three
general categories are used to ease readability: Quality Review, Peer
Review, and Other QM Tools.
Documentation Notes. This section summarizes the type of quality
documentation maintained in the project file. These items may include,
but are not limited to drafts of project plans, sampling plans, peer
reviewer charges, comments from reviewers, and responses to
omments documents.
Quality Management Lessons Learned (if applicable). Continuous
improvement, by building on lessons learned, is a key component of
quality management. This section summarizes any comments made by
Project Managers concerning activities or processes they would like to
apply to future projects or wish they had considered during project
implementation to improve the quality or efficiency of their project.
5.2
-------
5.3
-------
Example 1
Response to Congress on
Use of Decentralized Wastewater Treatment Systems
What can be learned from this example?
An interoffice workgroup can be utilized successfully for quality review and peer input.
Project Background
In 1996, the Congressional House Appropriations Committee requested that EPA analyze and report on
the use of decentralized wastewater treatment systems as an alternative to current centralized systems.
EPA's analysis was to include an assessment of 1) the benefits of the alternative approaches, 2) the
potential savings and/or costs associated with these, approaches, 3) the Agency's ability to implement
these alternatives within the current statutory and regulatory structure, and 4) any Agency plans to
implement such alternatives using funds appropriated in FY97.
Quality Management Challenges
• Completing project goals within established time constraints
• Determining what information was necessary and whether it could be obtained with available
resources ',
• Finding appropriate experts
• Presenting technical information to a Congressional audience with varied levels of technical
understanding
OWM's Quality Management Strategy
OWM formed a workgroup comprised of experts throughout OW and ORD. The workgroup assisted in
planning the project and implementing project activities. OWM submitted the draft product for external
peer review to evaluate the adequacy of the project against project objectives and recommend product
improvements.
Final Work Product
OWM coordinated the required analyses, report development, and submission to the Congressional
committee. As a next step, OWM plans to use the analyses and recommendations described in the report
to prepare an implementation plan and guidance to states and municipalities regarding the use of
alternative wastewater treatment technologies.
5.4
-------
Quality Management Tools Used to Implement the Strategy
Quality Review
At the Beginning:
An initial brainstorming session with workgroup members
was used to:
• Determine quality needed for the project
• Develop a strategy for project implementation
• Identify data and data quality needs
• Recommend existing sources of relevant data
• Identify tasks for a contractor
During:
• The workgroup completed quality review of data and first
draft from contractor
• Plan for peer review of final drafts
At the End:
The workgroup evaluated and integrated, where
applicable, all comments from the peer reviewers
Peer Review
At the End:
The workgroup submitted the final
draft report from the contractor for
external peer review by:
• Technical staff from EPA Regional
Offices
• Several states and universities
• Water Environment Federation
• Small Flows Clearinghouse
• Association of State and Interstate
Water Pollution Control Agencies
Where applicable, all peer reviewer
comments were integrated into the
report before submitting it to Congress.
Other QM Tools
• The OWM PM maintained control of the electronic drafts to ensure all relevant comments were
integrated into the draft
• Most data used were from published, peer reviewed sources, and all secondary data were evaluated
by a contractor for relevance to project needs
• A widely accepted computer cost model described in peer reviewed journals was used for cost
analyses
Documentation Notes
The Project Manager retained copies of:
• Documentation of the initial workgroup meeting, including a basic strategy and report outline
• The written charge with specific instructions for each peer reviewer
• All internal and external peer reviewer comments
• All drafts of the report
5.5
-------
Example 2
Investigation of Best Management Practices for Photo
Processors via Draft Code of Management Practices
What can be learned from this example?
Stakeholders can offer technical and resource support in defining and managing the quality of the
project.
Project Background
OW is streamlining the National Pretreatment Program by simplifying regulatory and administrative
requirements applicable to POTWs and the facilities that discharge to them. As part of this initiative,
EPA has funded the Association of Metropolitan Sewerage Agencies (AMSA) and The Silver Council
(TSC) to demonstrate that a newly developed Code of Management Practice (CMP) can be used by the
photo industry as an alternative to existing compliance mechanisms for controlling wastewater
discharges. The CMP was developed by AMSA and TSC. This project seeks to demonstrate to all
stakeholders that the CMP alone or in combination with innovative regulatory options can help attain
environmental goals more effectively and efficiently than traditional approaches. More specifically, the
purpose of the demonstration project is: 1) demonstrate continuous progress toward meeting program
goals for clean water and efficient operation of POTWs by reducing the headworks loadings of silver, 2)
deliver cost savings (silver recovery, operating efficiencies) and generally mitigate the cost to industrial
users of achieving environmental objectives, and 3) simplify POTW program administration.
Quality Management Challenges
• Definition of project objectives that met the needs of both government and industry representatives
• Designation of proper representatives of the various organizations to the three committees
• Determination of the data collection and data quality needs, in order to design a statistically suitable
sampling and analysis plan
OWM's Quality Management Strategy
Three groups comprised of both government and industry were formed. The Steering Committee was
developed to oversee the project as a whole. A Technical Project Committee was responsible for
development of the work plan and final report and for responding to technical inquiries arising during
data collection and analysis. A Quality .Review Team was developed to provide external review of the
work plan and final report. Both the Steering Committee and Quality Review Team will review and offer
improvements for the final report.
Current Project Status >.
At this point, baseline data are being collected. Data collection, analysis and model development will
proceed as designated in the work plan. The final report will be used as a demonstration to states and
cities of options available and the corresponding success rates and cost analyses. ;
5.6
-------
Quality Management Tools Used to Implement the Strategy
Quality Review
At the Beginning:
A meeting with Steering Committee members was used to:
• Define clear project objectives
• Determine the quality needed for the project
• Determine data and data quality needs
• Develop a statistically-based sampling and analysis plan
« Identify tasks for a contractor
• Plan for peer review of the work plan and final report
During:
• The Technical Project Committee completed quality
review of initial contractor data
• The Technical Project Committee will perform a quality
review of the draft report from contractor
At the End:
• The Steering Committee will perform a quality review of
the final report
The Technical Project Committee will evaluate, and
integrate, where applicable, all comments from the
Steering Committee and Quality Review Team
Peer Review
During:
The Technical Project Committee
submitted the work plan for external
peer review by the Quality Review
Team, comprised of members of:
• EPA Offices
• State and City Offices
• Industry
At the End:
• The Technical Project Committee
will submit the final report for
external peer review by the Quality
Review Team
Other QM Tools
• A cost analysis model will be
developed for estimating financial
implications among approaches.
This model will be tested and refined
based on real world data and
experience.
Documentation Notes
The following have been documented by the Project Manager:
The charge given to each of the three committees
All committee meeting minutes
Draft workplan, comments on draft, and final workplan
Quarterly progress reports
Quality Management Lessons Learned
The Project Manager realized the importance of defining clear data quality requirements, especially
when numeric data are being generated. At the outset of the project, some participants did not see a
need for a statistical sampling and analysis design as a part of the work plan. Once the data quality
requirements were defined, however, it was realized that in order to obtain the specified level of quality
in the data and its interpretation, a clear sampling and analysis plan must be defined before sampling
begins. The plan was developed and incorporated into the work plan.
5.7
-------
Example 3
Storm Water Phase I Model for 1996 Clean Water Needs Survey
What can be learned from this example?
Quality and peer review of the data and the models in the Clean Water Needs Survey assisted in the
ability to estimate costs of the storm water treatment required by Storm Water Phase 1. Review of the
the up-front planning of budget, and timelines facilitated full completion with quality needed.
Project Background \
EPA's Clean Water Needs Survey (CWNS) is required by Sections 205(a) and 516(b)(l) of the Clean
Water Act. The CWNS is a summary of the estimated capital costs for water quality projects and other
activities eligible for State Revolving Fund (SRF) support. In 1987, Congress amended the CWA to add
Section 402(p), which directs EPA to establish phased National Pollutant Discharge Elimination system
(NPDES) permit requirements for Storm Water (SW) discharges. EPA was directed to prepare a Report
to Congress detailing SW management costs, with special attention to 1) costs for structural controls and
BMPs that might be eligible for SRF funding, and 2) the costs of implementing NPDES municipal storm
water programs on a national basis.
Quality Management Challenges ;
• Obtaining the desired work product while working under time and financial constraints
• Finding persons knowledgeable in SW management for input in project planning
• Determination of data and data quality needs, so a model could be properly developed
• Presenting technical information in way that could be understood by Congressional staff
OWM's Quality Management Strategy
A contractor held several workgroup sessions around the country, inviting persons knowledgeable in SW
management. The contractor used recommendations and concerns voiced in these sessions to develop
appropriate BMPs and controls to define criteria for categorizing the cities where data would be
collected. All data were collected and reviewed by the contractor. A model was developed and refined,
and a draft report was prepared to discuss the modeling process and its results. A group of three external
reviewers conducted a peer review of both the modeling process and the report. The comments were
reviewed by the Project Manager and the contractor, and where applicable, the Project Manager
incorporated the comments into the report. The ultimate decision to incorporate a comment was made by
the Project Manager.
Final Work Product
The results of the model have been released. Also, a draft final report was written. Because of financial
constraints, the report will remain in its current draft form. One chapter remains unwritten, and funds
are not likely to be allocated for completion of this chapter. ;
5.8
-------
Quality Management Tools Used to Implement the Strategy
Quality Review
At the Beginning:
Several workgroup sessions were held to:
• Determine what processes were available to
control SW discharges .
• Develop an implementation strategy for testing
the chosen BMPs and controls
• Identify data and data quality needs for the
model
• Identify data collection and other tasks for the
contractor
During:
• Initial model outputs were reviewed by the
contractor. Questionable data were reviewed
with the corresponding community In order to
determine the correct data.
At the End:
• Peer review comments were evaluated by the
contractor and the PM, and incorporated where
applicable.
Peer Review
At the End:
• The modeling process and draft report were
submitted for external peer review by three
individuals: one academic, one practitioner,
and one designer/engineer.
Other QM Tools
• The PM maintained control of the drafts to
ensure that the appropriate comments were
integrated into the final draft report.
« A model analyzing SW discharges was
developed, tested, and refined based on real
world data.
Documentation Notes
• Copies of the charge to peer reviewers, names, and affiliations of the peer reviewers.
• All peer reviewer comments were addressed in a Response to Comments document, available through
the Freedom of Information Act.
Quality Management Lessons Learned
Peer review was not a planned part of the project. The decision to peer review was made late in the
project. In retrospect, the PM would have chosen to plan for peer review at the outset, and to conduct
peer review as the project progressed. In this manner, time would have been available to review
thoroughly all candidates and choose among them. Also, time and financial constraints forced the
publication of the model without completion of the final report. Once the model had been published, less
interest in the discussions of the final report remained. A time line and budget planning for other options
when obstacles arose may have avoided this outcome.
5.9
-------
Example 4
Assessment of "Solar Aquatics" Technology for
Treating Wastewater
What can be learned from this example?
1) Application of quality management strategies to a primary data collection effort ;
2) Extensive planning, quality review, and peer review throughout all stages of this project produced
defensible results in a politically sensitive project. ;
Project Background
This project provided annual grant proposals to fund the demonstration of the benefits of a wastewater
treatment technology involving the use of natural processes that occur in ponds, wetlands, and land
treatment systems under controlled conditions by establishing natural ecosystems in a series ;of tanks in a
greenhouse. The technology is generally referred to as "Solar Aquatics", "Advanced Ecological
Engineered Systems", or "Living Machines". This technology has been supported by Congressional
add-on funds to EPA's budget to demonstrate the technology at four sites. The project involves full-scale
demonstration facilities to evaluate the grantee's technical and cost claims regarding their biological
wastewater treatment process while operating at full design capacity and under steady state conditions
Quality Management Challenges
• Development of procedures to verify that project outputs and work products were of the desired
quality.
• Identification of the appropriate personnel to provide technical assistance
• Mid-stream technological improvements made by the grantee that could affect data interpretation
OWM's Quality Management Strategy
Two technical advisory committees were formed, one by EPA and one by the grantee. EPA's technical
committee reviewed the grantee's proposal package, which included a QAPP, against project objectives.
During the project, and independent evaluation effort was conducted by an EPA contractor with funding
connected with the Congressional add-on at the request of the grantee involving separate samples and
data collection as well as split samples that were analyzed by an EPA contract laboratory, the grantee's
laboratory, and the demonstration site laboratory to verify that the data produced by the grantee were
reproducible and defensible. The quality of the grantee's work products were also evaluated
independently. EPA's draft reports were submitted for quality review and peer review by EPA's
technical advisory committee, the grantee's technical advisory committee, and other individuals
expressing an interest in the project.
Final Work Product
Comments from the external peer review process were integrated, where applicable, into EPA's Interim
Report to Congress, and Final Technology Assessment Report. The grantee used OWM's feedback to
refine its system and adjust some of its claims for the system. This allowed OWM to focus on what the
technology could do rather than on what it could not do.
5.10
-------
Quality Management Tools Used to Implement the Strategy
Quality Review
At the Beginning:
Two Technical Advisory Committees, one formed by EPA and one
formed by the grantee, communicated to:
• Review the grantee's proposals, project objectives, and study
design
• Determine data and data quality needs
• Review grantee's sample collection plan, QAPP , data and draft
reports
• ORD reviewed the grantee's QAPP in accordance with ORD's
approach for research and development (R&D) data gathering
projects
• EPA's technical advisory committee reviewed the grantee's
proposal package against project objectives
During:
• Split samples were analyzed by an EPA contract laboratory and
the demonstration site laboratory, in order to verify that the data
produced by the grantee's laboratory were reproducible and
defensible
Peer Review
During:
• OWM sought an independent
evaluation of the cost data,
biological and chemical data
submitted by the grantee
At the End:
• Draft reports were submitted
for peer review to individuals
expressing an interest in the
project
• Both EPA's and the grantee's
technical advisory committees
completed assessments of the
draft reports and offered
improvements
« Comments from the peer
review and quality review
processes were evaluated and
integrated, where applicable,
into EPA's Interim Report to
Congress, and Technology
Assessment Report
Documentation Notes
The Project Manager has retained documentation of the following, both for the overall demonstration
project and the independent field sampling/analyses activities:
Development, submitted, review, and approval of the QA/QC plan
Memo to outside reviewers requesting their review of the above mentioned plan
Outside reviewers comments
Memo providing feedback on the draft QA/QC plan
Final QA/QC plan
Progress reports, draft and final reports, and data generated
Quality Management Lessons Learned
Because the project was based on a technology that still was being developed and improved, the PM
believes that the project should have been categorized as an R&D project rather than a demonstration
project. Realizing during the planning stage that some changes were possible; however, the PM did
allow for the planning of R&D type QC requirements that were a key success for the project. Also,
using quality and peer review throughout the project allowed for the grantee to receive feedback and to
refine its technology and its claims.
5.11
-------
Example 5
40 CFR Part 503 Biosolids Rule
What can be learned from this example?
The value of peer input and peer review in managing secondary data.
Project Background
In February 1989, EPA proposed regulations concerning the use and disposal of municipal sewage.
Development of these regulations was an extensive effort involving several EPA divisions and staff, the
collection and analysis of secondary data (e.g., data originally collected for other purposes), and the
collection of primary data through a long-term survey of statistically-selected POTWs. The proposed
rule included an Agency commitment to submit the proposal through a formal peer review process. Peer
reviewers and stakeholders commenting on the proposed rule criticized specific aspects of the Agency's
use and interpretation of secondary data. EPA then embarked on an extensive effort to revise the
proposed regulations to address these concerns. ;
Quality Management Challenges
• Evaluating the technical validity of the public comments received (i.e., distinguishing comments that
were "politically motivated" from those that were "technically valid").
• Ensuring that the quality of secondary data used to develop final regulations would be sufficient to
withstand legal challenge
• Ensuring that the Agency's use and interpretation of secondary data was technically valid.
OWM's Quality Management Strategy
A three-day peer review meeting involving 35 peer reviewers was convened to fulfill EPA's commitment
to have a formal peer evaluation of the proposed rule. This peer review meeting was facilitated and
financially supported by the Association of Metropolitan Sewerage Agencies, the Association of State
and Interstate Water Pollution Control Administrators, the National Association of Counties, the
National League of Cities, the U.S. Conference of Mayors, and the Water Environment Federation. Peer
reviewers attending the meeting represented diverse cross-section of interests, expertise, and perspectives
on the diverse and highly technical issues relating to sludge management and disposal practices. In
response to peer review comments that inappropriate use of the secondary data in EPA's model created
an unrealistic worst-case scenario for a protection benchmark. Twelve experts (including some of the 35
peer reviewers) were asked to provide assistance (peer input) in revising the rule, especially in the
appropriate selection and use of secondary data, models, and in the simplification of the rule, based on
knowledge from research involving protection against potential harm. A smaller number of independent
xperts, primarily representing various EPA offices, reviewed the final draft of the rule. A third peer
review was initiated two years after promulgation of the final regulations as a means of verifying that the
regulations met expectations and for identifying areas for further improvement.
Current Project Status
The final rule was published after a two year effort to revise the proposed rule. OWM is now assisting
Regions, states, and municipalities with implementation of the final rule.
5.12
-------
Quality Management Tools Used to Implement the Strategy
Quality Review
During:
Twelve experts, some of whom were involved in a
peer review of EPA's proposed rule, were asked to
assist EPA in revising the rule to address peer
review comments.
Experts provided assistance in selecting
appropriate secondary data for use in the rule,
appropriate use of these data in EPA models, and
simplifying the rule
Peer Review
At the End:
• The revised draft of the final rule was
reviewed by several experts who were
completely independent of the quality
review process activities or the regulatory
development process. Most of these
experts represented various EPA offices.
• An internal review of the final rule was
coordinated in accordance with the formal
peer review procedures used by the
National Academy of Science.
• A follow-up peer review was performed
two years after promulgation of the rule.
Comments from this review were used to
identify further OWM priorities related to
management and regulation of biosolids.
Documentation Notes
A formal docket was assembled to support both the proposed and final rulemaking activity. The
docket included copies of all data used to support the regulation, all assumptions used to interpret
the data, all comments received, a formal response to those comments, and records of the peer
review process.
Quality Management Lessons Learned
• The inclusion of expert quality and peer reviewers earlier in the regulatory development process
would have prevented EPA from developing a proposed rule that was based, in part, on a flawed
interpretation of the data.
5.13
-------
Example 6
1998 Biosolids Quality Survey
What can be learned from this example?
The importance of up-front planning and involvement of experts when project budgets are not sufficient
to meet project goals.
Project Background
In FY 1998, the Assistant Administrator for Water directed OWM staff to gather data necessary to
evaluate the quality of biosolids (municipal sewage sludge) that are being beneficially used in
accordance with the 40 CFR 503 regulations. The AA provided a five year budget of $350,000/year to
address the specific question, "Is the quality of biosolids generated today the same, better, or worse than
the quality of biosolids generated in 1988?" (EPA conducted a National Sewage Sludge Survey and
other data gathering activities in 1988.)
Quality Management Challenges
• Designing a study that answers the AA's question within the specified budget and time period.
• Including in the planning process an awareness of new sampling and analysis methodologies that
allow determination of pollutants at lower concentrations than could be measured in 1988
• Finding appropriate experts
• Building on lessons learned in the 1988 data gathering activities
OWM's Quality Management Strategy
OWM consulted with statisticians, chemists and other staff from OST regarding sampling and analysis
costs, statistical design, and methodology that might be appropriate for the study. The statisticians
cautioned that it would not be possible to answer statistically the question posed with the funds provided.
The statisticians indicated it was necessary to survey 19,000 entities to determine whether a statistically
significant difference exists between biosolids today and in 1988, whereas cost estimates suggested that
only 200 entities could be surveyed.. OWM then conducted several meetings with technical and
statistical experts to determine options. Options included re-framing the project objectives, harrowing
the scope of the question to address self-reported data, and requesting a larger budget. After briefing the
AA on the challenges, the AA re-directed OWM to eliminate the need for statistical comparison to 1988
and instead answer the question, "Are biosolids generated today still of sufficient quality to meet the
risk assessment objectives used to develop the current regulations?"
Current Project Status
OWM staff are working with a group of internal and external experts to design a study that meets new
project objectives within the allotted budget.
5.14
-------
Quality Management Tools Used to Implement the Strategy
Quality Review
At the Beginning:
Meetings were held to:
• Determine project objectives
• Develop a strategy for project implementation
• Identify financial resources
• Determine data and data quality needs
• Develop a statistically-based study design based on .
those needs
During:
Because financial constraints prohibited the
implementation of a statistically-based study design,
additional meetings were held to:
• Develop alternatives to the initial project objectives
Decide if implementation of alternatives would offer
significant, useful information
Determine data and data quality needs for
implementation of options generated
Determine if alternative approaches could be
implemented with available resources
Peer Review
• The initial draft of the study plan was
submitted for formal review by external
experts experienced in the collection and
management of biosolids. Results of this
review are being used to revise the study
design.
• The final study plan also is scheduled for
external peer review.
Other QM Tools
• Standardized sampling and analysis
methods will be used to ensure consistency
across sampled facilities and analytical
laboratories.
• All of the QC measures shown in Chapter
2, Exhibit 2-2 will be applied to sampling
and analysis activities.
• State and Regional sampling personnel
will be provided with sampling guidance
and sampling kits to control variability
between sample collection processes and
sampling containers.
Documentation Notes
• Documentation of initial charge issued (verbally) by the Assistant Administrator
• Notes, agendas, and internal correspondence from the initial project planning meetings (including
cost estimates, statistical scenarios, and information on data needs and existing data,)
• Copies of materials used to brief the AA on the difficulty answering the initial question and notes
from that meeting that document the revised charge
• Notes, agendas, and internal correspondence from the subsequent planning meetings being held to
discuss study designs needed to answer the new question.
Quality Management Lessons Learned
The process of determining project objectives and resources available at the outset may sometimes
appear to be burdensome, but it can save a large tremendous amount of time and money. In this case,
OWM's 'investment of up-front planning ensured that a $350,000 annual budget would not be spent in
gathering data that failed to answer scientifically the question these funds were directed to address.
5.15
-------
Example 7
CSO Monitoring and Modeling Guidance
What can be learned from this example?
It is never too late to add new QM strategies to a project. In this example, an unplanned peer review
phase was added, and dramatically improved the quality of the final work product.
Project Background
In 1994, EPA issued the final Combined Sewer Overflow (CSO) Control Policy. The policy establishes
a consistent national approach for controlling discharges from combined sewer systems to the Nation's
waters through the National Pollutant Discharge Elimination System (NPDES) permit program. Lit
support of EPA's commitment to providing the required tools for effective implementation of the policy,
EPA has issued several guidance documents. The CSO Monitoring and Modeling Guidance was
developed to provide a set of guidelines that provides flexibility for municipalities to develop site-
specific strategies for characterizing their combined sewer system operation and impacts, and for
developing and implementing comprehensive CSO control plans. The guidance includes low cost
monitoring and modeling techniques that should be useful to communities of all sizes.
Quality Management Challenges
• Defining project goals and scope of the document to be produced
• Completing project goals within established resource constraints
• Identifying available relevant models
• Ensuring the guidance was applicable to communities of all sizes
• Assignment of a new Project Manager mid-way through the project
• Finding appropriate experts to peer review draft manual, and to address comments from peer
reviewers and stakeholders
• Determining the level of peer review to utilize in the project
OWM's Quality Management Strategy
Because he took this multi-year project over midstream, the project manager was faced with the
challenge of reconciling initial project plans with more recent developments in CSO monitoring and
policy. The project manager also was faced with a challenge of applying the Agency's newly refined
peer review policy and procedures to this guidance document. The project manager addressed these
challenges by 1) seeking multiple rounds of internal review prior to external review and public release,
2) consulting with other EPA and Water Environment Federation staff to identify appropriate peer
reviewers, 3) submitting the final draft to three external organizations for formal peer review, and 4)
releasing the document to stakeholders for informal comment concurrent with the formal peer review
process. All comments received from the stakeholders and from formal peer reviewers were documented
and addressed in a formal "response to comments" document. In addition, all comments were evaluated.
by three people and integrated, where applicable,, into the manual.
Current Project Status
A draft final document has been produced submitted to the Branch Chief, Division Director, and Office
Director for final review. Once comments are received and addressed, the final document will be
published, providing guidance to all communities, regardless of size, on development of
monitoring/modeling programs commensurate with their CSO problems.
5.16
-------
Quality Management Tools Used to Implement the Strategy
Quality Review
During:
• Drafts of the guidance were reviewed several
times internally before release to public.
Discussions were held to find resources for
" potential peer reviewers. They were chosen
based on the recommendations of EPA staff and
the Water Environment Research Federation.
At the End:
Concurrent with formal peer review, the draft
manual was released to stakeholders for review
and informal submission of comments.
Stakeholder comments were evaluated by three
people and, where applicable, integrated into the
manual.
Peer Review
At the End:
• The draft manual was submitted for external
peer review by three independent
organizations.
• Peer review comments were reviewed by
three people and, where applicable, integrated
into the manual.
Other QM Tools
• Existing data models were referenced
planning data gathering and analysis.
Documentation Notes
The Program Manager has retained copies of the following:
The charge sent to external peer reviewers
Comments received from reviewers
Decisions and explanations regarding the acceptance or rejection of comments
A Response to Comments document
.Quality Management Lessons Learned
Although the Project Manager had not planned on peer review of the draft work product, it was selected
for peer review by OWM management in response to EPA's new peer review policy. The Project
Manager was pleased with the results of the review and would incorporate it during the planning stages
of future projects. Also, the manual changed dramatically in scope and organization as the project
proceeded. In future projects, the Project Manager believes that it would be beneficial to define clearly
I the scope and objectives of the project; and a basic outline of the final work product.
5.17
-------
-------
Chapter 6
Answers to Commonly Asked QM Questions
A number of questions are often asked about quality management. This
chapter poses many of these questions and answers them. Additional
discussion concerning many of these issues may be found elsewhere in
this guide.
Needed Quality Requirements
Q: How do I determine the quality requirements that I need for the project?
A: All EPA work products should be of high quality, but in general, the
highest quality requirements are needed for those work products that: (1)
support regulatory decisions, (2) will have widespread use or widespread
impacts, or (3) are highly innovative in approach or design. OWM staff
members should attempt to design all their activities in a way that maximizes
quality within given resource budgets and project time lines, and are strongly
encouraged to incorporate quality review into the planning stage of any project
that demands the highest level of quality. Quality reviewers can then assist in
determining the overall level of quality needed and capable of being achieved.
6.1
-------
Controlling Quality
Q: What are some of the ways I can control quality?
A: The most valuable ways to control quality are:
»• to consider your quality needs up-front during the project planning
process and design your project accordingly,
* to follow through on your planned project design,
> to monitor your project continuously to ensure you are meeting your
planned schedules, budgets, and objectives, and
> either to revise your plan or to seek additional resources as soon as you
encounter unanticipated difficulties that may impact your schedules,
budgets, or objectives.
> ' to document how the quality was assured.
Q: How do I control the quality of primary data?
A: Specific ways of controlling the quality of primary data include: defining
data needs, using standardized methods to collect data, and using QA measures
such or those shown in Exhibit 3 of Chapter 2. Detailed information regarding
the control of primary data may be found in EPA's guides pertaining to Data
Quality Objectives, listed in Appendix A.
Q: How do I control the quality of secondary data?
A: Specific ways of controlling the quality of secondary data include: (i) using
experts to assist you in finding the most relevant data available, (ii) using
experts to assist you in determining if the data obtained are, in fact, suitable for
your project, (iii) using experts to assist you in determining data limitations,
and (iv) testing any conclusions you reach with secondary data against "real
world expectations." (This might include, for example, performing a pilot
study or bench-scale test to determine the accuracy of model outputs resulting
from assumptions made with secondary data).
Quality Control Tools
Q: What are some of the tools I can use to control quality?
A: Some examples of quality management tools include the use of: (see also
chapter 2)
«• Experts to assist in defining quality needs and devising plans to
obtain the desired level of quality (quality review and peer
input);
6.2 ,
-------
Experts to review interim and final work products (quality
review and peer review);
Standard operating procedures or methods to ensure that a
process is applied consistently over time and by multiple
individuals;
Peer review or widely accepted data management tools or
strategies (e.g., a cost/benefit model, a statistical approach, or
an analytical database);
Pre-defined data quality objectives, stringent QM
measurements, and QM acceptance criteria for data gathering
projects;
Data whose quality already has been defined for other projects;
and
Centralized process control strategies (e.g., Word Perfect
document management features that track all edits made and by
whom, or strategies to control decision making by establishing
a hierarchy of decision makers).
6.3
-------
Quality Management Applies to
Q: To which OWM activities does QM apply?
A: Everything! QM is not limited to activities involving the collection of data.
It applies to the development of regulations, guidance documents, briefing
packages, and more. See Chapter 1 for a list of other examples to which QM
principles apply.
Documenting Quality Management
Q: If I do an excellent job of quality management, but I did not document
anything, did I really do Quality Management?
A: Not really. Without documentation, you cannot demonstrate that you have
considered aspects of project or data quality. A lack of documentation also
deprives other OWM staff of the opportunity to build on your products, your
processes, or your activities. This contradicts a key concept of quality
management-continuous improvement.
Why bother with Quality Management?
Q: I already have more than enough work to do. Why should I bother with
quality management?
A: See Exhibit 7.
Exhibit 7
Top 10 reasons why you should practice quality management
10. Because it can save hundreds, thousands, and even millions of taxpayer
dollars.
9. Because it could save you weeks or months—you'll have time to go on
vacation.
8. Because your QM documentation will facilitate transferring the project to
someone else so you can move on to bigger and better things.
7. So you can survive a QA audit.
6. Because it helps defend against adverse comment or litigation. <
5. To ensure your data are available and usable for the future.
4. So you can achieve Agency recognition for a well-managed project.
3. Because your QA Coordinator says so.
2. Because your QA Manager says so. And if that's still not enough . ..
1.- Because the Administrator says so! ;
6.4
-------
Quality Control, Quality Requirements, Quality Assurance, and Quality Management
Q: What is the difference between quality control (QC), quality requirements,
quality assurance (QA), and quality management (QM)?
A: QC is the term now generally associated with end of the pipe inspections to
identify any defective products. It is no longer used by most quality managers
because it implies that there will be defects. The current quality management
system is designed to control quality from the beginning
Quality requirements are the standards that are necessary for the work
product/effort under consideration.
QA is the collective system of processes used to control quality. The level of
effort needed to control quality depends on the project's purpose, complexity,
and nature.
QM describes an organization's commitment to quality by its management
QM involves the development of policies and the provision of resources and
rewards to ensure that have been met. QM as outlined by Demming's Total
Quality Management model is to Plan -Do- Check and Act.
Because "quality management" is a more recent term that was developed to
equate QA with total quality management (TQM) principles, it is widely
confused with QA, and the words are often used interchangeably. This guide
seeks to promote OWM's QM culture by illustrating the ways in which QM
can benefit OWM staff and products and by providing straightforward ways
(and examples) of implementing QM principles into projects through
development of QA systems and tools.
Primary and Secondary Data
Q: What is the difference between primary data and secondary data!
A: Primary data are original new data actively collected by or for the project
manager to address specifically an issue at hand. Examples of primary data
include sampling and new analytical measurements, new survey results, and
comments to proposed regulations collected by EPA or by EPA's contractors to
address an immediate need.
Secondary data consist of data that were originally gathered by others for one
purpose and that are now being used by the project manager, and his or her
contractor or grantee for a different, or secondary, purpose. This may include
previously collected analytical data from scientific literature and state program
monitoring programs, historical information, etc., that is now being assembled for
6.5
-------
use in a current EPA project.
Q: Does Quality Management apply only to the collection of primary data?
A:. No. Quality Management applies to the collection of both priimary and
secondary data. OWM staff using secondary data must assess the quality and
appropriateness of the secondary data, including how it was collected originally,
the purpose for which it was collected, and the limitations on its use.
Q: If we have had no opportunity to plan for the quality of the secondary data
that we are using, how do we assure that the data are appropriate for our
projects?
A: Although you may not have been involved in planning the quality of data
gathered by others, you can plan how you -will use such secondary data in your
study. When using secondary data, it may be advisable to consult experts
familiar with the generation and use of the data (or similar types of data) to
help evaluate the applicability of the data to your project, identify its limitations,
and determine how it can be used properly in your project. See Example 5 in
Chapter 5 as a case study in which problems occurred because experts were
not consulted prior to using secondary data The quality review and peer
review tools are particularly well-suited to evaluating the adequacy of secondary
data.
Q: Who are the experts, and how do I find them?
A: Suggestions for finding experts include: seeking recommendations from fellow
co-workers, reviewing the public literature for widely published authors, your
quality assurance coordinator, or contacting stakeholder organizations for
recommendations
Hey! I Just Got Someone Else's Project
Q: Can quality management techniques that are not planned for and
implemented at the start of a project be added part way through thie project?
A: Yes. Quality management is a highly dynamic process that can be thought
of as a cycle involving planning of the project and project quality,
implementation of the plan, assessment of whether the plan is working or has
worked, and refinement of the plan to improve the product or project process.
If a project becomes technically more complicated than originally anticipated,
for example, a workgroup of additional experts may be formed to work through
the issues. Or the project may become more controversial than at the start, and
more extensive peer review may be added.
6.6
-------
Example 7 in Chapter 5 illustrates a situation in which a requirement for formal
peer review was added just prior to Completion of the final work product.
Example 5 in Chapter 5 illustrates a situation in which unanticipated
complexities of the project led to difficulties and required previously unplanned
levels of quality and peer review.
Quality Review (See also Chapter 2)
Q: What is Quality Review?
A: Quality review entails the use of knowledgeable individuals to assist in
determining the level of quality needed of all OWM work products or
processes, and later, to evaluate the work product to determine if the level of
quality is being obtained. Quality review, as we are defining it in this guide,
includes both technical or nontechnical review and input by all stakeholders in
OWM activities.
Q: What is one of the most important parts of quality review?
A: A quality review of planned activities is especially important atthe
beginning of the activity. It helps determine:
* The appropriateness and relevance of the task. If the planned task is
not relevant, either change it so that it is or do not do it.
> The level of quality needed for each of the activities required during the
project.
> The level of quality that is possible, given the resources and time
available
* What to do if the quality needed is greater than available resources
allow: (i) change the approach which might include -ways to leverage
and/or change the level of resources; (ii) identify clearly to managers
that the level needed cannot be attained and identify what level can be
attained with what shortcomings, or (Hi) do not do the task.
Q: What are other important times for quality review?
A: Quality review by peers during the task will help assure that the level of
quality you deemed necessary and possible is being attained. When coupled
with peer input, it also can provide an opportunity to obtain suggested mid-
stream corrections to address unforeseen problems. (Note: Although this is a
helpful strategy in some projects, OWM staff members need to remember that
any consolidation of peer input and quality review functions during project
implementation may preclude the quality reviewers from serving as
independent peer reviewers of the final product).
6.7
-------
Peer Review
Quality review at the completion of the final draft of a report or project further
helps ensure that a relevant quality product has been attained and may also be
useful in interpreting peer reviewers' comments.
Q: Whom do I go to for peer review?
A: Peer review should be performed by individuals who are technically
qualified to perform the work, are free from real or perceived bias concerning
the work product, and are independent of the work product (i.e., they were not
substantially involved in its development). Qualified peer reviewers may be
found within OWM, within OW, within other EPA offices or EPA Regions, or
external to EPA (e.g., states, municipalities, stakeholder organizations, private
citizens, or contractors).
Q: Do all peer reviewers have to be "technical" people?
A: Generally, peer reviewers should "be selected to meet the objectives of your
peer review. If your work product involves guidance or analysis of engineering
practices, for example, your work plan or product should be reviewed by a
qualified engineer. If the work product is intended for a wide, non-technical
audience, however, it also may be desirable to include a non-technical peer
reviewer whose purpose is to evaluate whether the product is clearly presented
to its intended audience.
6.8
-------
Q: How much will I have to pay for external peer review services?
A: In a great many cases other technical experts will serve as peer reviewers as
a professional courtesy to those persons who have produced the product. They
will often do this because they also have an interest in having and using a
product like that being produced and/or they have an interest in assuring that it
is a good product as it may ultimately apply to them. They are more likely to
come if they do not have to pay their own way, but instead have their travel and
lodging expenses paid for them.
When you find that you will need to pay for peer review — either because of the
short time frame in which you need to get the work done, or because of the
considerable amount of work that needs to be done - you should estimate the
time involved in conducting a peer review of each of your products. Then
multiply that time figure by the number of reviewers who will be required to
review your product and multiply that by the estimated hourly rates for
individuals with expertise in your subject area. Arrangements for pay are
generally made by contract.
Q: How do I initiate the peer review process?
A: You begin by writing a "charge" to peer reviewers that documents specific
issues or questions you would like them to address and invites comments on
the entire product. Additional information can be found in the Science Policy
Council Handbook for Peer Review, EPA 100-B-98-001, January 1998.
Q: Once peer review comments are received, what is the procedure for
incorporating them into my final work product?
A: All peer review comments received must be evaluated. This evaluation
may include consultation with other experts or staff within the Agency. All
comments must be either addressed in the revised work product, or must be
documented in the QM file to indicate why the comment was not addressed.
Q: Does peer review have any role in regulatory negotiations?
A: Regulatory negotiations are not candidates for peer review. However, to
ensure that decisions are based on sound and credible science, the major
scientific and technical work products that support the negotiation need peer
review before the negotiation takes place.
Q: Are there any circumstances when a major work product may not be
subjected to a peer review?
6.9
-------
A: Yes, there may be situations when a decision to eliminate the peer review
process can be justified. For example:
• Additional peer review is not required with work that has been
previously reviewed by recognized experts or an expert body.
• Additional peer review is not required when the scientific and/or
technical methodologies or information being used are commonly
accepted in the field of expertise.
• A major work product would not receive peer review when the
regulatory activity or action which the work supports is terminated or
canceled. No further action is necessary
• Statutory and court ordered deadlines may limit or preclude peer
. review.
• Rarely, resource limitations may restrict peer review. These should be
evaluated on a case by case basis.
If peer review of a major work product is not conducted, written justification,
signed-off by the appropriate decision maker, must be documented.
Q: Can work products that are not determined to be major still be subjected to
peer review?
A: Yes, peer review may be warranted if it adds substantial value to the work
product. Another alternative is to use the less formal "quality review" process
for such final work products because:
• the formal Agency "peer review" requires each office annually to
prepare and submit to the Administrator records of its peer review
activities, and
• the peer review process requires that reviewers be completely
independent of the work product. The quality review mechanism
imposes neither the added level of documentation nor the added level
of independence.
Q: How does one determine the appropriate Peer Review Mechanism?
A: Arranging for the most appropriate and feasible peer review will involve
consideration of substance, time, and resource trade-offs. Essentially, the level
of peer review matches the impact and complexity of the major work product.
The greater the possible impact or complexity, the stronger the indication for a
more extensive and involved peer review and for external peer review, in
particular. The choice of mechanism will depend upon the experience and
assessment of the Decision Maker.
6,10
-------
How Is Peer Review Different From?
Q: How does peer review different from quality review?
A: Peer review is more formal than quality review and involves the use of
qualified peers who have had very little involvement in the activity, (e.g.,
except perhaps for review of the project's plans). Peer review is a required
Agency process that uses technically qualified peers (persons of equal or
greater rank) to ensure the quality of all major, technical work products.
Quality review does not require the independence of peers who assist in the
conducting and review of an activity.
Q: How does peer review different from stakeholder involvement?
A: Stakeholder involvement occurs when the Agency works with external
interest groups that have some stake or concern over the outcome of the
technical work product or regulatory position. This is an interactive process,
working with other agencies, industry groups, regulated-community experts,
environmental groups, and other groups that represent a broad spectrum of the
regulated community. This process usually strives for a consensus approach.
The goal of peer review, on the other hand, is to obtain an independent, third-
party review. Stakeholder involvement does not substitute for peer review even
though it adds value to the work product.
6.11
-------
Q: How does public comment differ from peer review?
A: The critical distinction is that public comment does not necessarily draw
the kind of independent, expert information and in-depth analyses expected
from the peer review process. Public comment is open to all issues, whereas
the peer review process is limited to the consideration of technical issues.
While it may be an important component of the review process, public
comment does not substitute for peer review.
Federal Advisory Committee Act (FACA) as a Quality Management Tool
Q: I have heard that there are a lot of rules regarding the use of Federal
Advisory Committee Act (FACA) committees and that workgroups are
considered FACA Committees. Where do I find more information about these
requirements, and when would I want to use such a cumbersome tool?
A: Peer review carried out by formal and established (chartered) Federal
advisory committees (such as the Science Advisory Board) is always subject to
FACA requirements. If, however, EPA conducts a peer single review meeting
for the purpose of obtaining advice from the individual attendees (as opposed
to consensus advice from the group), the meeting would not be subject to
FACA. Questions concerning the applicability of FACA to peer review
meetings should be addressed to the FACA experts in the Cross-Cutting Issues
Division of Office of General Council.
6.12
-------
Often, the choice to use a F AC A .committee is dictated through a
Congressional, Presidential, or Administrator's directive to address complex or
controversial projects and issues. At other times is may be a recommendation
of the program office.
6.13
-------
-------
Appendix A
EPA Quality Assurance Requirements and Guidance
Documents
[NOTE: Many of the following documents are available to download from EPA's Quality
Assurance Division Internet pages at the following address:
http://es.epa.gov/ncerqa/qa/qa_docs.html\
EPA QA/G-0, EPA Quality System Description
QA/G-0 provides an overview of the EPA Quality System and describes the various quality
assurance (QA) and quality control (QC) policies and practices comprising the Quality System.
This document provides a brief history of the evolution of the EPA Quality System and why it
plays a critical role in Agency operations. This document was placed into the Agency's
directives clearance process in December 1997.
EPA QA/R-1, EPA Quality Systems Requirements for Environmental Programs
QA/R-1 is the external policy document by which EPA will announce its implementation of the
American National Standard ANSI/ASQC E4-1994, Specifications and Guidelines for Quality
Systems for Environmental Data Collection and Environmental Technology Programs. An
internal preliminary draft has been completed and is awaiting formal adoption of the standard by
EPA. The same information will be part of the EPA Quality Manual for Environmental
Programs, an internal policy manual. When E4 has been formally adopted by EPA, the draft will
be distributed for comment.
EPA QA/G-1, Guidance for Developing Quality Systems for Environmental Data Operations
QA/G-1 provides non-mandatory guidance to help organizations develop a QA program that will
meet EPA expectations and requirements. There is no draft currently available.
EPA QA/R-2, EPA Requirements for Quality Management Plans
QA/R-2 is the policy document containing the specifications and requirements for Quality
Management Plans (QMPs) for organizations with which EPA has extramural agreements. A
Draft Final version is awaiting Agency approval for release. QA/R-2 is the intended replacement
-------
for QAMS-004/80. The same information contained in this document is found in the EPA
Quality Manual for Environmental Programs, an internal policy manual. Current Draft Version:
August 1994.
EPA QA/R-2A, EPA Requirements for Quality Management Plans for Analytical Laboratories and Facilities
QA/R-2A will provide detailed requirements for environmental analytical labs. Since there may
be a national consensus standard for labs, the content of this document is unclear at present.
EPA QA/G-2, Guidance for Preparing Quality Management Plans
QA/G-2 provides non-mandatory guidance to help organizations develop Quality Management
Plans (QMPs) that will meet EPA expectations and requirements. The document will contain
tips, advice, and case studies to help users develop improved QMPs. There is no draft currently
available.
EPA QA/G-2EA, Guidance to Implementing Quality Assurance in Extramural Agreements
The G-2EA document is intended for use by EPA project officers and contains one section each
on steps needed to determine and implement QA requirements for contracts, work assignments,
cooperative agreements, and interagency agreements. The text will be written from the
perspective of the project officer and will present the step-wise process for each extramural
agreement mechanism in one document. There is no draft currently available.
EPA QA/G-2C, Guide to Satisfying EPA Quality Assurance Requirements for Contracts and Work
Assignments
The G-2C document is intended for use by contractors performing work on behalf of EPA and
contains sections on the steps needed to identify, document, and implement QA requirements for
proposals, awarded contracts, work assignments, and technical directives (or similar
mechanisms). The text will be written from the perspective of the contractor and will present the
step-wise process for each contract mechanism in one document. There is no draft currently
available.
EPA Q A/G-2F, Guide to Satisfying EPA Quality Assurance Requirements for Financial Assistance Agreements
The G-2F document is intended for use by organizations receiving financial assistance from EPA
and contains sections on the steps needed to identify, document, and implement QA requirements
for research grants, state and local government grants, cooperative agreements, and other
financial assistance agreements. The text will be written from the perspective of the applicant or
awardee for assistance and will present the step-wise process for each extramural agreement
mechanism in one document. There is no draft currently available.
EPA QA/G-3, Guidance for the Management Systems Revie\v Process
QA/G-3 provides non-mandatory guidance to help organizations plan, implement, and evaluate
management assessments of their quality systems. The guidance will present a step-by-step
description of the MSR process. Current Draft Version: January 1994.
-------
EPA QA/G-4, Guidance for the Data Quality Objectives Process, EPA/600/R-96/055, September 1994
QA/G-4 provides non-mandatory guidance to help organizations plan, implement, and evaluate
the Data Quality Objectives (DQO) process, with a focus on environmental decision-making for
regulatory and enforcement decisions. The guidance presents a step-by-step description of the
DQO process.
EPA QA/G-4D, Data Quality Objectives Decision Errors Feasibility Trials (DEFT) Software,
EPA/600/R-96/056, September 1994
QA/G-4D provides non-mandatory guidance for using the Decision Error Feasibility Trials
(DEFT) software to help organizations plan, implement, and evaluate the Data Quality
Objectives (DQO) process. The guidance presents a step-by-step description of the use of the
PC-based DEFT software DQO process.
EPA QA/G-4R, Guidance for the Data Quality Objectives Process for Researchers
QA/G-4R provides non-mandatory guidance on the application of the Data Quality Objectives
(DQO) Process for researchers and experimenters. The guidance integrates the DQO Process
with statistical design of experiments. There is no draft currently available.
EPA QA/G-4HW, Guidance for the Data Quality Objectives Process for Hazardous Sites
QA/G-4HW provides non-mandatory guidance to help organizations plan, implement, and
evaluate the statistics-based Data Quality Objectives (DQO) process as applied to hazardous
waste sampling activities. The guidance will present a step-by-step description of the DQO
process and its application to environmental remediation and waste management activities.
There is no draft currently available.
EPA QA/G-4CS, The Data Quality Objectives Process: Case Studies
QA/G-4CS will be a collection of case studies that illustrates applications of the DQO process to
environmental situations. These case studies will be adapted from actual investigations and
several lead to complex applications of the DQO process. There is no draft currently available.
EPA QA/R-5, EPA Requirements for Quality Assurance Project Plans
QA/R-5 is the intended replacement for QAMS-005/80. This external policy document will
establish the requirements for QA Project Plans prepared for activities conducted by or funded by
EPA. It is intended for use by organizations having extramural agreements with EPA. Current
Draft Version: October 1997. This document was placed into the Agency's directives clearance
process in October 1997.
EPA QA/G-5, Guidance on Quality Assurance Project Plans, EPA/600/R-98/018, February 1998
QA/G-5 provides non-mandatory guidance to help organizations develop a Quality Assurance
Project Plans (QAPPs) that will meet EPA expectations and requirements. The document
provides a linkage between the DQO process and the QAPP. It contains tips, advice, and case
studies to help users develop improved QAPPs.
-------
EPA QA/G-5S, Guidance on Sampling Designs to Support QA Project Plans
QA/G-5S provides non-mandatory guidance on the practicality of constructing sampling plans to
meet the guidelines outlined in the statistics-based DQO Process (QA/G-4), and the QAPP
(QA/G-5). Different sampling schemes are discussed and the relative strengths and weaknesses
outlined. There is no draft currently available.
EPA QA/G-5T, Guidance on Specialized Topics in Quality Assurance
QA/G-5T is a collection of discussion papers on some of the more advanced topics in Quality
Assurance. These papers reflect some of the latest research results in QA and is also a forum for
discussion of QA policy issues. There is no draft currently available.
EPA QA/G-6, Guidance for the Preparation of Operating Procedures for Quality-Related Operations,
EPA/600/R-96/027, November 1995
QA/G-6 provides non-mandatory guidance to help organizations develop and document Standard
Operating Procedures (SOPs). The document contains tips, advice, and case studies to help users
develop improved SOPs. !
EPA QA/G-7, Guidance on Technical Assessments for Environmental Data Operations
QA/G-7 will provide non-mandatory guidance to help organizations plan, conduct, evaluate, and
document technical assessments for their programs. Such technical assessments include
Technical Systems Audits (TSAs), surveillance, readiness reviews, and Performance Evaluations
(PEs). The document will contain tips, advice, and case studies to help users develop improved
processes for conducting technical assessments. This is currently a planning item. QAD expects
to use a Work Group process to develop this guidance.
EPA QA/G-8, Guidance on Environmental Data Validation and Verification
QA/G-8 will provide non-mandatory guidance to assist users in validating and verifying
environmental data. The guidance will discuss alternative approaches to data validation,
including examples of statistics-based validation techniques, and options for verifying that the
validated data satisfy relevant claims about their authenticity and quality.
EPA QA/G-9, Guidance for the Data Quality Assessment Process: Practical Methods for Data Analysis,
EPA/600/R-96/084, January 1998
QA/G-9 provides non-mandatory guidance for planning, implementing, and evaluating
retrospective assessments of the quality of the results from environmental data operations. DQA
is a statistically-based, quantitative evaluation of the extent to which a data set satisfies the user's
needs (or DQOs). This particular document is aimed at the project managers who are responsible
for conducting the environmental data operations and assessing the usability of the results.
EPA QA/G-9D, Data Quality Evaluation Statistical Toolbox (DataQUEST), EPA/600/R-96/085, December
1997
QA/G-9D provides non-mandatory guidance for planning, implementing, and evaluating
retrospective assessments of the quality of the results from environmental data operations using
the PC-based software, DataQUEST.
-------
EPA QA/G-10, Guidance for Determining Quality Training Requirements for Environmental Data Operations
QA/G-10 will provide non-mandatory guidance to help organizations determine and develop
training requirements for their programs. The document contains an overview of all of QAD's
QA publications together with how the documents interact with the Agency's QA System. In
addition the document outlines all the training courses in QA developed by QAD, a description
of all the individual modules constructed by QAD, and directions for downloading these from
Agency's QA home page. This document has not been reviewed internally by all Agency
program or regional offices and is subject to change or revision. External Review Draft versions
were released at the EPA National Quality Assurance Meeting held in August 1997.
EPA QA/G-11, Guidance on Decision Quality Planning for Project Managers
QA/G-11 will provide non-mandatory guidance for assuring quality in the planning of
environmental programs and projects. Its intention is to help project managers integrate quality
management principles and practices into all of their project activities. There is no draft
currently available.
EPA QA/G-12, Quality Assurance for Environmental Technology Design, Construction, and Operation
QA/R-12 will provide non-mandatory guidance for assuring quality in the design, construction,
and operation of environmental technology. This guidance will describe how to apply Part C
requirements of ANSI/ASQC-E4-1994 to various environmental technology scenarios and how to
demonstrate conformance to that American National Standard. There is no draft currently
available.
EPA 100-B-98-001, Science Policy Council Handbook on Peer Review
Denting, W.E. Out of Crisis, MIT Center for Advanced Educational Services, Cambridge, MA, 1986.
-------
-------
Appendix B
Peer Review
Policy Overview
Major scientifically and technically based work products related to Agency decisions
should normally be peer reviewed. Agency managers within headquarters, Regions,
laboratories, and field components determine and are accountable for the decision
whether to employ peer review in particular instances and, if so, its character, scope
and timing. These decisions are made in conformance with program goals and priorities,
resource constraints, and statutory or court-ordered deadlines. For those work products
that are intended to support the most important decisions or that have special importance
in their own right, external peer review is the procedure of choice. Peer review is not
restricted to the penultimate version of work products; in fact, peer review at the planning
stage can often be extremely beneficial. (From the forward of the Science Policy Council
Handbook on Peer Review EPA 100-B-98-001)
Why Do We Do a Peer Review?
The Agency strives to ensure that the scientific and technical underpinnings of its decisions
meet two important criteria: .
• They should be based upon the best current knowledge from science, engineering,
and other domains of technical expertise, and
• They should be judged credible by relevant experts who deal with the Agency.
What Products Need Review?
• Major scientific and technical work products as defined in Chapter 2.
• A rule or regulation itself is not subject to Peer Review Policy. However, if the
rule is supported by major scientific and/or technical work products, those work
products should be peer reviewed, prior to use in the rule. Tier land 2
rulemakings are significant because they have an economic impact of $ 100 million
or more. Analytical blueprints are required for Tier 1 and 2 rulemakings. The
analytical blueprint is the process whereby the project manager identifies the
supporting scientific and technical work products and identifies needed peer
1
-------
-------
Appendix B
Peer Review
Policy Overview
Major scientifically and technically based work products related to Agency decisions
should normally be peer reviewed. Agency managers within headquarters, Regions,
laboratories, and field components determine and are accountable for the decision
whether to employ peer review in particular instances and, if so, its character, scope
and timing. These decisions are made in conformance with program goals and priorities,
resource constraints, and statutory or court-ordered deadlines. For those work products
that are intended to support the most important decisions or that have special importance
in their own right, external peer review is the procedure of choice. Peer review is not
restricted to the penultimate version of work products; in fact, peer review at the planning
stage can often be extremely beneficial. (From the forward of the Science Policy Council
Handbook on Peer Review EPA 100-B-98-001)
Why Do We Do a Peer Review?
The Agency strives to ensure that the scientific and technical underpinnings of its decisions
meet two important criteria:
• They should be based upon the best current knowledge from science, engineering,
and other domains of technical expertise, and
• They should be judged credible by relevant experts who deal with the Agency.
What Products Need Review?
• Major scientific and technical work products as defined hi Chapter 2.
• A rule or regulation itself is not subject to Peer Review Policy. However, if the
rule is supported by major scientific and/or technical work products, those work
products should be peer reviewed, prior to use in the rule. Tier 1 and 2
rulemakings are significant because they have an economic impact of $ 100 million
or more. Analytical blueprints are required for Tier 1 and 2 rulemakings. The
analytical blueprint is the process whereby the project manager identifies the
supporting scientific and technical work products and identifies needed peer
-------
Comments Received and Addressed
Most individuals that received the document for peer review did not respond.
Substantial review was received from Pat Bradley, Ben Lesser, and Glen Nestel.
Less comprehensive reviews were received from Mike Cook and Jim Wheeler.
Mike Cook indicated his belief that the document would be very useful to OWM,
Individual staff corrected the examples in Chapter 5. Ben Lesser and Pat Bradley
gave comments that were useful in improving the document and were for the most
part incorporated. Glen Nestel, gave the most thought provoking comments and
indicated the need to drop the term Quality Control. He stated that the term
quality control has a negative connotation because it is based upon the concept
that control is needed in the end of an activity because there will be mistakes.
This is in contrast with the more modern concept of Quality Management that
takes place throughout the entire development of the work products. Glen Nestel
also indicted the desirability bf including TQM concepts in the draft in addition to
the considerable focus already present on quality and peer review tools. His
comments were incorporated. Copies of those reviews are included in a
supplemental file. The peer review comments were incorporated into this final
draft for pilot testing by the peer review leader.
Contract Management '[...-.
The peer review leader served as the Task Manager for the development of the
Guide. Wendy Blake-Coleman was the Project Officer for the DynCorp contract.
My control of the project was predominately through having submitted chapers
and parts of chapters throughout the development of the draft document so that
they could be corrected and placed in a user friendly format.
Supplemental Project File
Additional quality and peer review information is contained in a supplemental
project file. This information includes a number of draft pieces of the report as
well as communication with the OW Project Officer. Also present in this
supplemental file are four peer reviewed copies of the guide and a copy of the
revised OWM pilot test copy.
{Propose to Add about two more example documentation flies]
-------
Appendix B
Peer Review
Policy Overview
Major scientifically and technically based work products related to Agency decisions
should normally be peer reviewed. Agency managers within headquarters, Regions,
laboratories, and field components determine and are accountable for the decision
whether to employ peer review in particular instances and, if so, its character, scope
and timing. These decisions are made in conformance with program goals and priorities,
resource constraints, and statutory or court-ordered deadlines. For those work products
that are intended to support the most important decisions or that have special importance
in their own right, external peer review is the procedure of choice. Peer review is not
restricted to the penultimate version of work products; in fact, peer review at the planning
stage can often be extremely beneficial. (From the forward of the Science Policy Council
Handbook on Peer Review EPA 100-B-98-001)
Why Do We Do a Peer Review?
The Agency strives to ensure that the scientific and technical underpinnings of its decisions
meet two important criteria:
• They should be based upon the best current knowledge from science, engineering,
and other domains of technical expertise, and
• They should be judged credible by relevant experts who deal with the Agency.
What Products Need Review?
» Major scientific and technical work products as defined in Chapter 2.
• A rule or regulation itself is not subject to Peer Review Policy. However, if the
rule is supported by major scientific and/or technical work products, those work
products should be peer reviewed, prior to use in the rule. Tier 1 and 2
rulemakings are significant because they have an economic impact of $ 100 million
or more. Analytical blueprints are required for Tier 1 and 2 rulemakings. The
analytical blueprint is the process whereby the project manager identifies the
supporting scientific and technical work products and identifies needed peer
1
-------
review. It also shows the schedule of peer review. If significant change based
upon public comment, additional peer review may be required.
Work products supporting Tier 3 rulemaking may also be considered major and
thus candidates for peer review.
• Economic work products that need peer review include:
Internal Agency guidance for conducting economic analysis.
New economic methodologies or unique or novel applications of existing
economic methodologies, particularly those that are path breaking.
Broad-scale economic assessments of regulatory programs, such as the
Congressionally-mandated study of the costs and benefits of the Clean Air
Act.
Is Peer Review by a Refereed Scientific Journal Adequate?
Peer review by a refereed scientific journal is a satisfactory form of peer review for EPA
purposes. EPA employees are encouraged to have an article they produce internally peer
reviewed prior to submission to a peer reviewed scientific journal. However, journal peer
review may not cover issues and concerns that the Agency would want peer reviewed to
support an Agency action.
SOMETIMES MORE THAN ONE PEER
REVIEW OF A PRODUCT MAY BE DESIRABLE
Roles of People and Organizations in Peer Review
• The Assistant Administrators and Regional Administrators are the ultimate
decision makers and are accountable for implementing Peer Review Policy within
their organizations.
They may designate Office directors and Division Directors of other
appropriate level line-managers as the front-line decision makers.
• Decision makers
- Decide whether a work product is major and needs peer review, and what
peer review mechanism to use.
- Commit resources and ensure that peer reviews are properly performed
and documented.
— ' Designate a peer review leader to organize the peer review.
-------
- By signature document the decisions made that are reported in the peer
review annual report including the peer review summary report
- Certify any decision not to peer review.
• Agency staff involved in peer review are the decision makers and this includes
line-managers, peer review leaders, peer review coordinators and certain staff
within ORD.
• The peer review leader
— Organizes, conducts, and completes the peer review.
- Selects the peer reviewers unless there is a contract or some other group
be available for conducting peer review.
- Keeps the record and initiates the peer review summary.
- Advises the peer reviewers of their responsibilities.
- Provides decision-makers information on the charge to the reviewers, their
profile, their comments and how they are to be used.
— The peer review leader must carefully evaluate and analyze the objectivity
of all peer review comments and recommendations and document how
they have been used.
— Appropriately archives the peer review record.
- Advises the peer review coordinator regarding the status of the peer
review for purposes of the annual peer review report.
• The peer revie\v coordinator is the main contact peer review matters. The peer
review coordinator for OW is currently Arnold Kuzmack and for OWM is Barry
Benroth. The peer review coordinator:
Provides advice, guidance and support to the various peer review leaders
and helps mediate difficult peer review issues
Participates in Agency peer review training, workshops, etc., as requested
and disseminate this information to the organization
Coordinates and/or presents training within the organization
Establishes procedures to assure that the required work product peer
review documentation (peer review record) is filed and maintained in an
appropriate manner and keeps the annual peer review report.
Where do you find Peer Reviewers?
• References from special and public interest groups, professional societies, trade
organizations, professional societies, tribes, colleges and universities, the national
Research Council, other Federal Agencies, Science Advisory Board and Regional
and Program Offices.
• Ask them if they have a real or perceived conflict of interest. If needed, speak
with the Office of General Council (OGC ) for assistance.
-------
Peer Reviewer Selection
Selection'criteria for peer reviewers include that the persons have expertise in the scientific
and technical subject matter and that the persons disclose conflicts of interest, real or
perceived. Independence (i.e., freedom from institutional, ideological, or technical bias) is
necessary for objective, fair, arid responsible peer review. Some peer reviewers may have
conflicts of interest. This is okay as long as this fact is disclosed up front and in the peer
review, and that the peer reviewer tries to be objective as possible. It is also important
that the resulting group of peer reviewers be balanced in terms of differing expertise,
backgrounds, and perspectives on the subject matter.
The person or group doing the selection of the peer reviewers depends upon how the peer
review is handled. If it a contractor, the contractor selects the peer reviewers. Likewise if
it is done by the Science Advisory Board (SAB), the SAB makes the selection, or it could
be the peer review leader.
Peer Review Funding
Senior managers (including decision-makers and budget officers) need to be sure that
budget requests include anticipated resources for peer review. Peer review needs to be
considered a normal part of doing business.
Peer Review Charge
The peer revie\v charge is prepared by the peer review leader. This very important
element of peer review should include:
An introduction saying what work product is, how it was developed, and how it is
to be used.
A brief listing and description of background materials provided to the reviewers,
and
The issues or questions to be addressed by the peer reviewers.
A statement requesting that draft materials not be disclosed to public
A request for written comments
A request that the prospective peer reviewer advise the Agency of any real or
perceived conflict of interest.
Note: If doing the peer review without charge, the peer reviewer should sign a statement
saying that he or she has no expectation of compensation from EPA.
How Do I Incorporate Peer Review Comments into the Final Work Product?
The peer review leader must carefully evaluate and analyze the objectivity of all peer
review comments and recommendations. Adequate documentation is required to show that
comments are accepted or rejected. Documentation can be brief, bust must address the
legitimate, valid comments. A carefully crafted charge to the peer reviewers helps simplify
organizing and analyzing comments.
Documentation
• The peer revie\v record contains the type of peer review and a summary of the
outcome of the peer review. It includes sufficient documentation for an
-------
uninformed person to know what actually happened and why. The peer review
leader is responsible for keeping the peer review record. The record should
contain:
The draft work product submitted for peer review whether a near final
document or and early draft of plans.
Materials and information including the charge given to the peer
reviewers.
Information about the peer reviewers - names, affiliations, and identified
potential conflicts and their resolution.
Times and locations of meetings.
A memorandum or other written record responding to the peer review
comments specifying acceptance or, where appropriate, rebuttal and
nonacceptance.
The final product.
The summary report on the outcome of the peer review.
Peer review leaders should maintain the peer review record with themselves until
the peer review is totally completed. Minimally, the record should be kept one
year after the completion of the peer review, for example in an archive.
-------
-------
Appendix C
Example Project File Showing the Kind of QA
Documentation that Should Be Retained
Example Documentation for the Production of the Office of Wastewater Management (OWM)
Common Sense guide to Quality Management
«
Project Background
In 1996, during the preparation for and the Office of Research and Development
(ORD) Management System Review of OWM, it became readily apparent that a
Commonsense Guide to QM could be-very helpful to OWM management and
staff. During that time period, there also was considerable emphasis on the need
for peer review within the Agency. In response to this need, OWM prepared and
conducted training on both quality and peer review. The training featured
presentations by individual staff members regarding their projects and their
review activities. The development of the Guide was envisioned that would.
" incorporate important elements of the quality and peer review training and
provide user-friendly quality management guidance dealing with all OWM efforts
including the collection and use of both secondary and primary data.
A limitation of the existing ORD QM guidance for OWM is its focus on the
collection of primary data and lack of applicability to all aspects of OWM effort.
The hope for this guide was to include examples that are relevant to OWM and to
provide check lists to help persons do a good job of QM including documenting
their QM efforts. The OWM Office Director endorsed the need for the
development of this Guide.
Purpose
The purpose of this guide is to help OWM managers and staff improve the
quality of their work. It provides simple, clear guidance on how each person
can use quality management (QM) principles to improve his or her work. It
also provides practical, straightforward guidance on how to comply with the
U. S. Environmental Protection Agency (EPA) and the Office of Water (OW)
quality assurance policies and how to document the steps taken to achieve
-------
quality outputs. The procedures provided in this guide apply to virtually all
OWM activities.
Quality Review
A series of meetings were held to gain the input of a broad spectrum of Agency
QM expertise. This included meetings with Wendy Blake-Coleman, Quality
Assurance Officer for the Office of Water (OW) and the Quality Assurance
Officers from each of the OW offices. In addition, members of the OW QM
contractor DynCorp (Jim .King, Karen Elm, and Maggie Jones) were present and
provided assistance for the planning of this document.
Planning for this document proved to be no easy task. Several initial partial
drafts of outlines and skeletal development of the document were reviewed and
rejected by the peer review leader for this effort (John Walker). Part of the initial
planning involved the need to determine just who the document was being
developed for - all OW offices or just OWM. Finally, with $20,000 in contract
dollars being provided by OWM with some additional funding by OW, it was
decided that this guide would be developed as an OWM document, and that other
offices might use or adapt parts of it later if it was found to be useful to them.
Another possible output subsequent to the issuance of the Guide was the
development of a training program for OWM by ORD and OWM.
During this development period, it was decided that Lynn Riddick and Kim
Conmy of Dyncorp would work with OWM to prepare the draft Guide. Drafts of
the various chapters and of the examples were reviewed by the peer review leader
and other selected Agency QM and management staff to obtain user-friendly
format. Ultimately, each chapter was reviewed by the peer review leader and
individual sections were reviewed by the staff responsible for each example
and/or familiar with concepts in individual chapters. Once this level of quality
review had been obtained, a peer review draft was put together by the Peer
Review Leader for peer review. This effort in preparing the draft involved
putting the guide into a format that would be used in the publication.
Peer Review
Peer Reviewers were selected and a charge letter prepared. A copy of this letter
is given below.
-------
-------
Charge :
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
September 2, 1998
Office of
Water
MEMORANDUM
SUBJECT: Request for Review of Common Sense Guide to Quality
Management
FROM: John Walker (4204)
Quality Assurance Officer for the
Office of Wastewater Management
TO: Reviewers (see list below)
Please find attached a draft copy of a new Guidance of Quality
management for the Office of Wastewater Management (OWM) at the U.S.
Environmental Protection Agency (EPA) which I have been developing with the
assistance of a contractor (DynCorp). This draft has resulted after a number of
iterations of guidance with all parts present except Appendix C. I took the last
draft from the contractor and changed its wording and some of its order in the
text. In addition, I have put it into what I hope is a user-friendly format into
which I have inserted a lot of headings and subheadings. I also put the text into a
three-quarter-width page format. The ultimate plan is to insert additional
illustrative material. This would include several photographs and even a cartoon
if we knew of one that is appropriate and one that the cartoonist would give
permission for use. There are several quotes, some of which may not be very
appropriate. Please comment if you like and suggest any additional such
information you might like.
Your assistance in providing a peer review or quality
review (whichever is checked) of this draft guidance would be most
appreciated. I specifically would like your review to see if the material is
presented in a way that is easy to read and understand. I would like you to see if
anything is incorrect. I would like for you to suggest, where you think
appropriate, the insertion of any additional topics of material. I would like you to
look at the questions and answers to see if they are correct and if there are any
that you think would be useful to add. I would like you to look at the examples
of OWM quality and peer review in Chapter 5 to see if they are understandable
and if they add useful information and in the process make the ability to do
quality management more easily.
-------
I have asked you to do a number of things in a generic sense. Please
actually only do those parts of the review for which you have time and expertise.
Again, I greatly appreciate your willingness to assist us in making this review. I
request that you send me you comments by September 15. If you are unable to
do the review by that time, please just let me know. I believe that you will find
that this guidance is really not very long and I would recommend that you
approach it by reading Chapter 1 first followed by Chapters 2 and 3. Then, the
fourth Chapter contains documentation checklists. Please look these over briefly
to see if they are helpful, and if not, any comments for improvement. Chapter 5
contains examples of QM within OWM. Only spend time on this chapter to the
extent you see if the examples add useful meaning and.or if they are presented in
an understandable form, and/or if some other form might be more helpful. The
plan is to have each example on facing pages in the printed document. I have
packaged Appendices A and B separately. Please glance at these and comment
on the ease of understanding the peer review memo (App. A) and for those that
know, the relevance of the cited references (App. B).
My plan is to do this round of peer or quality review and work to address
any issues and questions, etc. that you raise. Then, after this revisipn possibly a
second peer review by others.
List of Reviewers: (Internal and external to EPA)
Barry Benroth, Quality Assurance Coordinator, Municipal Support Division
Tony Smith, Quality Assurance Coordinator, Permits Division
Ben Lesser, OWM
Pat Bradley, PD
Jim Wheeler, MSD
Bob Lee, Chief, Municipal Technology Branch
Fred Lindsey, Deputy Director OWM
Wendy Blake-Coleman, Quality Assurance Officer for the Office of Water
Arnold Kuzmack, Office of Water Peer Review Officer
Tom Dixon, Quality Assurance Division, ORD
Mary Fraker, Powell Tate
Glenn Nestel, Hager Bailly
Note comments,by those receiving information copies are welcome. I thought
that I would give you another chance after a more formal review by those listed
above and subsequent revision.
Information Copies to
All OWM Branch Chiefs
OWM Division Directors
Mike Cook, Director OWM
OW Quality Assurance Officers
Attachment
bcc. Lynn Riddick, Dyncorp, Kim Conmy, DynCorp
-------
Comments Received and Addressed
Most individuals that received the document for peer review did not respond.
Substantial review was received from Pat Bradley, Ben Lesser, and Glen Nestel.
Less comprehensive reviews were received from Mike Cook and Jim Wheeler.
Mike Cook indicated his belief that the document would be very useful to OWM,
Individual staff corrected the examples in Chapter 5. Ben Lesser and Pat Bradley
gave comments that were useful in improving the document and were for the most
part incorporated. Glen Nestel, gave the most thought provoking comments and
indicated the need to drop the term Quality Control. He stated that the term
quality control has a negative connotation because it is based upon the concept
that control is needed in the end of an activity because there will be mistakes.
This is in contrast with the more modern concept of Quality Management that
takes place throughout the entire development of the work products. Glen Nestel
also indicted the desirability of including TQM concepts in the draft in addition to
the considerable focus already present on quality and peer review tools. His
comments were incorporated. Copies of those reviews are included in a
supplemental file. The peer review comments were incorporated into this final
draft for pilot testing by the peer review leader.
Contract Management
The peer review leader served as the Task Manager for the development of the
Guide. Wendy Blake-Coleman was the Project Officer for the DynCorp contract.
My control of the project was predominately through having submitted chapers
and parts of chapters throughout the development of the draft document so that
they could be corrected and placed in a user friendly format.
Supplemental Project File
Additional quality and peer review information is contained in a supplemental
project file. This information includes a number of draft pieces of the report as
well as communication with the OW Project Officer. Also present in this
supplemental file are four peer reviewed copies of the guide and a copy of the
revised OWM pilot test copy.
[Propose to Add about hvo more example documentation files]
------- |