October 1, 2002
TABLE OF CONTENTS
INTRODUCTION SCOPE
AND APPLICABILITY QUALITY
MANAGEMENT PRINCIPLES 1996 SAFE
DRINKING WATER ACT AMENDMENTS FOR SAFETY AND HEALTH RISK ANALYSIS
PRINCIPLES BLS GUIDELINES FOR INFORMING USERS
OF INFORMATION QUALITY AND METHODOLOGY
INFORMATION QUALITY
RESPONSIBILITIES
INFORMATION CATEGORIES PRIVACY AND SECURITY
INFORMATION INFORMATION QUALITY ASSURANCE
PROCESS INFORMATION COMPLAINT AND
APPEAL PROCESS TRACKING AND
REPORTING INFORMATION COMPLAINTS AND APPEALS
APPENDIX I: INFORMATION
QUALITY GLOSSARY APPENDIX II: ADAPTING THE PRINCIPLES UNDER THE
1996 SAFE DRINKING WATER ACT AMENDMENTS FOR SAFETY AND HEALTH RISK
ANALYSES APPENDIX III: BLS GUIDELINES FOR INFORMING
USERS OF INFORMATION QUALITY AND METHODOLOGY APPENDIX
IV: SAMPLE INFORMATION QUALITY ASSURANCE TECHNIQUES AND
METHODS
INTRODUCTION
Background On December 21, 2000, Congress
passed Section 515 of the Treasury and General Government Appropriations Act
for Fiscal Year 2001(1) (the
Act). OMB issued final guidance(2) for implementing the Act, which required all
Federal agencies to:
- Issue information quality guidelines ensuring and maximizing
the quality, objectivity, utility, and integrity of information, including
statistical information, disseminated by the Department;
- Establish administrative mechanisms allowing affected persons
to seek and obtain correction of information maintained and disseminated by the
Department that does not comply with the OMB guidelines; and
- Report to the Director of OMB the number and nature of
complaints received regarding compliance with the OMB guidelines, including how
the complaints were resolved.
The OMB guidance directed Federal agencies to publish their draft
Information Quality guidelines on their websites by April 1, 2002. OMB further
clarified on March 4, 2002, that the website publishing date was extended to
May 1, 2002.
The Department of Labor (DOL) posted its draft guidelines to its
web site on May 1, 2002. On May 22, 2002, DOL posted an appendix addressing the
adaptation of the Safe Drinking Water Act to the development of risk
assessments. DOL published notice of the guidelines in the Federal Register on
May 1, 2002, and required that comments be submitted by May 31, 2002. DOL
extended the comment period to June 30, 2002, in response to public requests
and to OMB's decision to extend the submission date for agencies to provide
revised guidelines to OMB.
These guidelines represent changes made to the draft DOL
guidelines posted for public comment on May 1, 2002, and incorporate revisions
based upon feedback from OMB, public comments, and internal Departmental
review.
Purpose
The purpose of these guidelines is to establish Departmental guidance
for implementing an Information Quality program at DOL. This Information
Quality guidance is intended to enhance the quality of the information
disseminated by DOL.
SCOPE AND
APPLICABILITY
These guidelines are intended, within the
context of laws administered and enforced by DOL, to meet the information
quality objectives set forth in OMB's guidelines. They are intended to improve
the internal management of the Federal Government. They are not intended to
impose any binding requirements or obligations on DOL or the public or to
create any right or benefit, substantive or procedural, enforceable at law by a
party against the United States, its agencies, officers, or any person. They
are not intended to provide any right to judicial review.
These guidelines reflect this Department's commitment to
information quality as an important management objective that takes its place
alongside other Departmental objectives, such as ensuring the success of agency
missions, observing budget resource priorities and restraints, and providing
information to the public. Agencies should strive to assure that these goals
reinforce each other as much as is practicable. Where an agency believes that
they conflict, it should, consistent with its legal responsibilities, attempt
to reconcile them in a manner that the agency believes will best serve the
public interest and help the agency meet its statutory or program obligations.
Program efficiency must be a critical goal as DOL agencies carry out their
responsibilities under these guidelines. Thus, for example, it may not be in
the public interest for agencies to devote significant resources to correcting
information where the expenditure of such resources is not, in the agency's
view, cost effective in light of the significance of the data and the agency's
more pressing priorities and obligations.
The DOL's pre-dissemination reviews apply to information that DOL
first disseminates on or after October 1, 2002. Other aspects of these
guidelines, including the information correction process apply on or after
October 1, 2002, with respect to information that DOL disseminates on or after
October 1, 2002, regardless of when DOL first disseminated the information.
Information means any communication or representation of knowledge such as
facts or data, in any medium or form, including textual, numerical, graphic,
cartographic, narrative, or audiovisual forms. Dissemination includes agency
initiated or sponsored distribution of information to the public. It does not
include agency citation to or discussion of information that was prepared by
others and considered by the agency in the performance of its responsibilities,
unless an agency disseminates it in a manner that reasonably suggests that the
agency agrees with the information. Agency sponsored distribution of
information covers instances where an agency has directed a third party to
disseminate information or where the agency has the authority to review and
approve the information before release. By contrast, if an agency funds
research, but the researcher decides whether or not to disseminate the results,
the agency has not "sponsored" the dissemination, and the information is not
subject to these guidelines. In these instances, agencies should direct the
researcher to include an appropriate disclaimer in the publication. Similarly,
the guidelines would not cover publications of their research findings by
Departmental employees or Federal grantees or contractors when published in the
same manner as their academic colleagues. Again, the researchers should include
an appropriate disclaimer noting that the views are theirs and not necessarily
those of DOL.
These guidelines do not apply to the following:
- Information distribution limited to government employees or
agency contractors or grantees;
- Government information intended to
be limited to intra- or inter-agency use or sharing of information, such as
strategic plans, performance plans, program reports, operating plans, or
budgets;
- Responses to requests for Departmental records under the
Freedom of Information Act, the Privacy Act, the Federal Advisory Committee
Act, or other similar laws;
- Correspondence or other communications with individuals or
organizations;
- Press releases (except where the press release itself is the
primary source of the information);
- Congressional testimony;
- Archival records;
- Public filings;
- Dissemination of information through
subpoenas or adjudicative processes, such as those recognized under the
Administrative Procedure Act or established pursuant to regulation; provided,
however, that information originally disseminated through such vehicles could
subsequently become subject to these guidelines to the extent it is
re-disseminated more broadly through other vehicles;
- Information clearly represented as opinion and not an official
agency or Departmental representation;
- Policy guidance recommendations or statements or summaries of
agency policies, procedures, or programs;
- Statements of legal policy or interpretation, including briefs
filed with courts or administrative bodies; and
- Final agency decisions, settlements in litigation and
descriptions of these settlements, or determinations of legal force and effect,
such as wage determinations.
These guidelines apply to all agencies of DOL, except to the
extent that agencies have adopted tailored agency-specific guidelines.
For a glossary of numerous terms and their definitions used
throughout the rest of this document, please consult Appendix I. The
definitions are from the OMB guidance for the Act.
QUALITY
MANAGEMENT PRINCIPLES
Every agency should establish
information quality, as defined in OMB and DOL information quality guidelines,
as a performance goal. Quality includes the "utility," "objectivity," and
"integrity" of the information. The level of quality should be "appropriate to
the nature and timeliness of the information to be disseminated" and will be
affected by the nature of the underlying data. In considering utility, agencies
should evaluate the usefulness of particular information to those expected to
use it. The information also should be objective--"accurate, reliable, and
unbiased," and presented "in an accurate, clear, complete, and unbiased
manner." Agencies also should protect the integrity of information from
unauthorized access or revision
These objectives and guidelines are to be interpreted consistent
with DOL's statutory obligations. Where agencies are disseminating information
of a scientific, financial, or statistical nature, they should use sound
statistical and research methods to develop and analyze the data. Depending on
the type of information disseminated and consistent with statutory and
confidentiality restrictions, agencies should identify the sources of the
information and where appropriate, the supporting data, models, and error
sources.
Where agencies develop and disseminate "influential" scientific,
financial, or statistical information, they should provide a higher level of
transparency about data and methods. Unless prevented by confidentiality, legal
constraints, or other compelling interests, the level of transparency should be
such that qualified third parties could reproduce the information. In
identifying what kinds of information may be subject to reproducibility
standards, agencies should use commonly accepted scientific, financial, or
statistical standards. Agencies are encouraged to make arrangements that will
permit appropriate public access to the related original and supporting data
and analytical results. Regarding analytical results in situations where
agencies do not permit access to data or methods due to other compelling
interests, such as confidentiality protections, agencies should, unless
otherwise prohibited by law, generally disclose their data sources (at whatever
levels of generality are needed to preserve necessary confidentiality),
quantitative methods and assumptions that have been employed, and the types of
robustness checks used to assure the quality of results.
1996 SAFE DRINKING WATER ACT
AMENDMENTS FOR SAFETY AND HEALTH RISK ANALYSIS PRINCIPLES
With regard to analysis of risks to human health, safety, and the
environment maintained or disseminated by agencies, the Occupational Safety and
Health Administration (OSHA) and the Mine Safety and Health Administration
(MSHA), in performing risk analysis, are hereby adapting the standards
contained in the Safe Drinking Water Act, as set forth in Appendix II. DOL does
not anticipate that any other agencies will be performing such analysis of
risks for their programs. However, to deal with unforeseen contingencies, DOL
hereby adopts the Safe Drinking Water Act standards with respect to all
programs other than OSHA and MSHA. Should it be necessary in the future for
another DOL agency to perform such an analysis, DOL will consider, at that
time, whether it is appropriate to adapt the Safe Drinking Water Act
standards.
BLS GUIDELINES FOR INFORMING USERS
OF INFORMATION QUALITY AND METHODOLOGY
Appendix III contains the Bureau of Labor Statistics (BLS)
Guidelines for Informing Users of Information Quality and Methodology
supplement to the overall Departmental guidelines. These supplemental
guidelines reaffirm BLS commitment to both OMB and DOL information quality
guidelines. Moreover, as part of the Interagency Council on Statistical Policy,
BLS supports the Council's commitment to information quality.
INFORMATION QUALITY
RESPONSIBILITIES It is the responsibility of all
Departmental agencies to make information quality an important goal in every
phase of a product's development. The following responsibilities pertain to the
implementation of DOL's information quality guidelines.
Chief Information Officer (CIO)
- Maintain a leadership role in overseeing the implementation of
these guidelines and in providing guidance to the agencies on information
quality matters.
- Develop and submit to OMB the annual report on the number,
nature, and resolution of complaints.
- Coordinate, as appropriate, with other Federal organizations
on cross-agency information quality issues.
Agency Heads
- Apply, consistent with applicable statutes and regulations,
DOL's information quality policies, procedures, and guidance to
Department-sponsored information products that an agency has direct authority
to control.
- Ensure that, where Department-sponsored information does not
necessarily reflect the views of DOL, an appropriate disclaimer will be
included.
- Ensure that in its submissions to the OMB under the Paperwork
Reduction Act, the agency demonstrates how it is attempting to provide that
information will be collected, maintained, and used in a way consistent with
OMB and DOL information quality standards.
INFORMATION
CATEGORIES
Per OMB's guidance, information means any communication or
representation of knowledge such as facts or data, in any medium or form,
including textual, numerical, graphic, cartographic, narrative, or audiovisual
forms. This definition includes information that an agency disseminates from a
web page, but does not include the provision of hyperlinks to information that
others disseminate. This definition does not include opinions, where the
agency's presentation makes it clear that what is being offered is someone's
opinion rather than fact or the agency's views on information of the kind that
is subject to these guidelines.
DOL has identified two categories of information that are
disseminated to the public, with the level of quality control and review being
greater for influential information than for non-influential information.
Whether information is influential is to be determined on an item-by-item basis
rather than by aggregating multiple studies, documents, or other informational
items that may influence a single policy or decision.
Influential Definition:
This category contains scientific,
financial, or statistical information when agencies can reasonably determine
that dissemination will have or does have a clear and substantial impact on
important public policies or important private sector decisions.
To be influential, information should have a clear and substantial
impact. A clear impact is one that is determined by the agency to have a high
probability of occurring. A substantial impact is one that meets the levels of
significance described below.
In rulemaking, influential information is scientific, financial,
or statistical information that the agency believes will have a clear and
substantial impact on the resolution of one or more key issues in an
economically significant rulemaking, as that term is defined in section 3(f)(1)
of Executive Order 12866.
In non-rulemaking contexts, DOL should consider two factors in
relation to each other - breadth and intensity - in determining whether
information is influential. These factors need to be considered together.
Information that has a low cost or modest impact on a limited range of affected
parties is less likely to be influential than information that can have a very
costly or crucial impact on a broad range of parties. Of course even
information that has a low cost or modest impact on any one party can be
influential if it can impact a broad range of parties -- for example, an action
that could cost an individual employer only $20 could nevertheless be
influential if it impacts the vast number of employers in the United States,
since it would then have an aggregate effect in excess of $100 million. Within
that framework, in considering whether information has a high intensity impact,
agencies should use as a benchmark the $100 million figure used to determine
whether a rule is economically significant. It should be noted that the
definition of "influential" applies to information itself and not to the
decisions that the information may support. Even if a decision or action by an
agency is very important, a particular piece of information supporting it may
not be influential, for example, because it is cumulative to other information
or because it involves legal or policy issues.
Moreover, if it is merely arguable that an impact will occur, or
if it is a close judgment call, then the impact is probably not clear and
substantial. The "influential" designation is intended to be applied to
information only when clearly appropriate. Agencies should not designate
information products or types of information as influential on a regular or
routine basis. Nor should agencies place an "influential" label on the title
page or text of an information product.
Examples:
Principal economic indicators, such as Consumer Price Index, the
Employment Situation, and Producer Price Index; the Private Pension Bulletin;
and Unemployment Insurance Weekly Claims data.
Non-influential Definition:
All information disseminated to the
public that does not meet the criteria set forth in the influential information
definition.
Examples:
Fact sheets (e.g., Disaster Unemployment Assistance, Injury
Trends in Mining), OSHA Construction Resource Manual, technical information
issuances, annual reports, and studies (e.g., Pension and Health Benefits of
American Workers, Coverage Status of Workers under Employer-Provided Pension
Plans, and Study of 401(k) Plan Fees and Expenses, Study of Health Insurance
Coverage of the Unemployed).
PRIVACY AND
SECURITY INFORMATION AND PUBLIC
Regardless of the
category of information, all agencies will comply with the Privacy and Security
Statement posted on DOL's web site. DOL is strongly committed to maintaining
the privacy of information and the security of its computer systems. With
respect to the collection, use, and disclosure of information, DOL makes every
effort to ensure compliance with applicable Federal laws, including, but not
limited to, the Privacy Act of 1974, the Paperwork Reduction Act of 1995, the
Trade Secrets Act, and the Freedom of Information Act. DOL reaffirms its
commitment to keep the public appropriately informed.
As part of its efforts to ensure and maintain the integrity of the
information disseminated to the public, DOL's IT security policy and planning
framework is designed to protect information from unauthorized access or
revision and to ensure that the information is not compromised through
corruption or falsification.
INFORMATION QUALITY
ASSURANCE PROCESS
Departmental agencies
should use the information quality assurance process described below to
maximize the quality of information disseminated. Agencies should use
information quality assurance processes that are appropriate to the complexity
and importance of the product being developed. Agencies may use appropriate
pre-existing information quality assurance processes that are at least as
effective as those of DOL or OMB.
The quality assurance process should begin at the inception of the
product development process. At the initiation of the product development
process, agencies should consult existing Departmental and agency information
quality assurance guidelines. Agencies should determine the information
category of the product to be developed, the level of quality assurance needed,
and the appropriate techniques required to maximize and ensure information
quality.
There are numerous techniques and methods agencies can utilize to
ensure they consistently produce and disseminate quality information. Appendix
IV provides some sample techniques and methods derived from industry best
practices. Agencies should use the information quality assurance techniques and
methods that they determine are most appropriate for their information
products. If agencies choose to conduct a formal, independent, external peer
review of data and analytical results, the peer review should meet the
following general criteria: (a) peer reviewers should be selected primarily on
the basis of necessary technical expertise; (b) peer reviewers should be
expected to disclose to agencies prior technical or policy positions they may
have taken on the issues at hand; (c) peer reviewers should be expected to
disclose to agencies their sources of personal and institutional funding
(private or public sector); and (d) peer reviews should be conducted in an open
(made public) and rigorous manner.
Agencies should incorporate the selected quality assurance
techniques into the project development schedule. Throughout the product's
development, agencies should ensure that quality assurance decisions are
defensible and appropriate to the category of information involved. The product
may be subject to internal agency quality controls and any appropriate
Departmental reviews before being disseminated to the public. For example, if
an agency decides to post the information on the DOL web site, it should adhere
to DOL's and its own Public Web Site Review and Clearance Process. Agencies
should incorporate lessons learned into future product development activities
so as to improve DOL's overall quality management process.
INFORMATION
COMPLAINT AND APPEAL PROCESS
Because DOL is committed to
information dissemination programs based on high standards of quality, it
recognizes the value of public input. DOL therefore encourages the affected
public to suggest improvements in Departmental information quality practices
and to contact it when particular disseminated information may not meet the OMB
guidelines and the guidelines set forth above. DOL believes that in most cases,
informal contacts would be appropriate.
Sometimes agencies and affected persons may find it helpful to
resolve concerns about information in a more structured way and may choose to
follow a more formal process. DOL will make available to the public a list of
officials to whom complaints and appeals should be sent and where and how such
officials may be reached. Affected persons may submit such complaints and
appeals to the contact point in the DOL agency responsible for the information.
Each agency may designate one or more officials to review information
complaints and another official or officials who will be responsible for
appeals if the complainant is dissatisfied with the initial response to the
complaint. The agency should provide that the official conducting the second
level review is not the same official who responded to the initial request or
from the same office that prepared the information in question. In determining
the level of the person designated to respond to appeals, the agency may wish
to consider such factors as the qualifications of the person and the
significance of the information in question. An agency may, within its
discretion, wish to consider the designation of a panel to resolve appeals
involving influential information, when the agency believes that such an
appellate board is needed and is an efficient way to provide expertise or
perspective or otherwise to improve the resolution of the appeal. Designated
agency officials may consult with other agency or Departmental offices, as the
agency may deem appropriate to the resolution of the
complaint.
The purpose of the information complaint and appeal process is
to deal with information quality matters, not to resolve underlying substantive
policy or legal issues.
As is the case with other provisions of these guidelines, the
process is intended to improve the internal management of the Federal
Government. It is not intended to create any right or benefit, substantive or
procedural, enforceable at law by a party against the United States, its
agencies, officers, or any person. It is not intended to provide any right of
judicial review. Concerns regarding information in a rulemaking must, except as
provided below, be presented in the rulemaking in accordance with the
rulemaking's procedures.
Overview of Information Complaint
and Appeal Process Affected persons may indicate their
interest in following a more structured complaint and appeal process by
expressing that interest to the concerned agency. Complainants should:
- Identify themselves and indicate where and how they can be
reached;
- Identify, as specifically as possible, the information in
question;
- Indicate how they are affected by the information about which
they are complaining;
- Carefully describe the nature of the complaint, including an
explanation of why they believe the information does not comply with OMB,
Departmental, or agency-specific guidelines; and
- Describe the change requested and the reason why the agency
should make the change.
Failure to include this information may result in a complainant
not receiving a response to the complaint or greatly reducing the usefulness or
timeliness of any response. Complainants should be aware that they bear the
burden of establishing that they are affected persons and showing the need and
justification for the correction they are seeking, including why the
information being complained about does not comply with applicable
guidelines.
In deciding how to handle complaints, agencies should be
especially mindful of their legal obligations, program priorities, resource
constraints, and their duty to use resources efficiently. For example, agencies
have important responsibilities to issue rules and provide compliance guidance
to the public. Agencies must administer the complaint and appeal process
consistent with these obligations and their responsibilities to carry them out
in an expeditious manner.
Any structured process would not apply to an agency's archival
information or to public filings. Agencies may choose not to respond to
complaints about claimed defects that are frivolous or unlikely to have
substantial future impact.
Where procedures exist for dealing with information quality
issues, agencies may consider resolving complaints by referring them to these
procedures. For example, complaints about the quality of information in a
rulemaking are ordinarily to be submitted and handled in accordance with
rulemaking procedures. As agencies consider information quality issues within
the context of a rulemaking, they are reminded of their primary responsibility
to resolve these issues in a manner consistent with the Administrative
Procedure Act and the substantive statute pursuant to which the rule is being
issued. In unusual circumstances involving an information product related to a
rulemaking, agencies should consider an information complaint under these
information correction procedures. When the agency disseminates a study,
analysis, or other information prior to the final agency action or information
product, requests for correction will be considered prior to the final agency
action or information product in those cases if the agency determines that an
earlier response would not unduly delay issuance of the agency action or
information product and the complainant has shown a reasonable likelihood of
suffering actual harm from the agency's dissemination if the agency does not
resolve the complaint prior to the final agency action or information product.
In deciding what action may be appropriate in these unusual circumstances,
agencies should consider the factors previously discussed in these guidelines.
They also may consider: (1) the impact of the information on the complainant;
(2) the extent to which the complainant's concerns have been rendered moot as a
result of actions taken by the agency; (3) the mechanisms available under the
Administrative Procedure Act or other laws to resolve complainant's concerns;
and (4) the public interest to be served in pursuing further action on the
complaint.
Where an agency responds directly to a complaint, it should
respond in the manner that it deems most suitable, whether by letter,
telephone, email, or otherwise.
Agencies should try to respond to complaints and appeals within
sixty (60) days of their receipt, unless they deem a response within this time
period to be impracticable. If an agency believes that more time is required to
decide how to respond to a complaint or appeal, it should estimate the time
needed and notify the complainant within the 60-day period of the reasons for
the delay and the time that it estimates that a decision will be reached. Once
the agency had decided how to address the complaint, it should notify the
complainant.
If a complainant is dissatisfied with the initial response to the
complaint, he or she may submit an appeal to the designated contact point in
the agency responsible for the information.
A complainant may appeal within forty-five (45) days of the date
the agency notified the complainant how it would handle the complaint or one
hundred and five (105) days from the date on which an agency or agencies first
received the complaint, whichever is later. The appeal request should contain
the same contact and descriptive information that was provided in the original
complaint and the specific reasons why the initial agency response was not
satisfactory. Once an appeal decision has been rendered by the agency, it
should notify the complainant.
In processing initial complaints and appeal requests, DOL and its
agencies should be flexible and take into account, among other things, the
nature, significance, and volume of complaints, the agency's particular program
needs, and available review mechanisms.
TRACKING
AND REPORTING INFORMATION COMPLAINTS AND APPEALS
The
Office of the Chief Information Officer (OCIO) is responsible for reporting the
results of the Department's information quality efforts as required by OMB
guidance. OCIO anticipates using a system to centrally track and report
complaints and appeals.
In accordance with procedures to be specified by the CIO, DOL will
establish on its web site or agency web sites an information quality site to
keep the public informed about information quality on a timely basis. The
purpose of the information quality web site would be to inform the public about
the agency's information quality practices and procedures. The information
quality web site should include access to the agency's information quality
guidelines and an easy-to-understand explanation of the agency's procedures
regarding complaints (which will include an explanation of how a person may
file a complaint and, subsequently, an administrative appeal of the agency's
response to the complaint). The information quality web site also could contain
other types of information, such as a description of significant corrections
that the agency has made as a result of the information complaint and appeal
process. Each agency should determine the content of this information page
based on its mission, activities subject to the guidelines, and the expected
level of interest by members of the public.
APPENDIX I:
INFORMATION QUALITY GLOSSARY
OMB provides the
following definitions in its guidance for the Act.
1. "Quality" is an encompassing term comprising utility,
objectivity, and integrity. Therefore, the guidelines sometimes refer to these
four statutory terms, collectively, as "quality."
2. "Utility" refers to the usefulness of the
information for its intended users, including the public. In assessing the
usefulness of information that the agency disseminates to the public, the
agency needs to consider the uses of information not only from the perspective
of the agency but also from the perspective of the public. As a result, when
transparency of information is relevant for assessing the information's
usefulness from the public's perspective, the agency must take care to ensure
that transparency has been addressed in its review of the information.
3. "Objectivity" involves two distinct elements,
presentation and substance.
a. "Objectivity" includes whether disseminated information is
being presented in an accurate, clear, complete, and unbiased manner. This
involves whether the information is presented within a proper context.
Sometimes, in disseminating certain types of information to the public, other
information must also be disseminated in order to ensure an accurate, clear,
complete, and unbiased presentation. Also, the agency needs to identify the
sources of the disseminated information (to the extent possible, consistent
with confidentiality protections) and, in scientific, financial, or statistical
context, the supporting data and models, so that the public can assess for
itself whether there may be some reason to question the objectivity of the
sources. Where appropriate, supporting data should have full, accurate,
transparent documentation, and error sources affecting data quality should be
identified and disclosed to users.
b. In addition, "objectivity" involves a focus on ensuring
accurate, reliable, and unbiased information. In a scientific, financial, or
statistical context, the original and supporting data shall be generated, and
the analytical results shall be developed, using sound statistical and research
methods.
i. If data and analytic results have been subjected to formal,
independent, external peer review, the information may generally be presumed to
be of acceptable objectivity. However, this presumption is rebuttable based on
a persuasive showing by the petitioner in a particular instance. If
agency-sponsored peer review is employed to help satisfy the objectivity
standard, the review process employed shall meet the general criteria for
competent and credible peer review recommended by OMB-OIRA to the President's
Management Council (9/20/01)
(http://www.whitehouse.gov/omb/inforeg/oira_review-process.html), namely, "that
(a) peer reviewers be selected primarily on the basis of necessary technical
expertise, (b) peer reviewers be expected to disclose to agencies prior
technical/policy positions they may have taken on the issues at hand, (c) peer
reviewers be expected to disclose to agencies their sources of personal and
institutional funding (private or public sector), and (d) peer reviews be
conducted in an open and rigorous manner."
ii. If an agency is responsible for disseminating influential
scientific, financial, or statistical information, agency guidelines shall
include a high degree of transparency about data and methods to facilitate the
reproducibility of such information by qualified third
parties.
A. With regard to original and supporting data related
thereto, agency guidelines shall not require that all disseminated data be
subjected to a reproducibility requirement. Agencies may identify, in
consultation with the relevant scientific and technical communities, those
particular types of data that can practicably be subjected to a reproducibility
requirement, given ethical, feasibility, or confidentiality restraints. It is
understood that reproducibility of data is an indication of transparency about
research design and methods and thus a replication exercise (i.e., a new
experiment, test, or sample) shall not be required prior to each
dissemination.
B. With regard to analytic results related thereto, agency
guidelines shall generally require sufficient transparency about data and
methods that an independent reanalysis could be undertaken by a qualified
member of the public. These transparency standards apply to agency analysis of
data from a single study as well as to analyses that combine information from
multiple studies.
i Making the data and methods publicly available will
assist in determining whether analytic results are reproducible. However, the
objectivity standard does not override other compelling interests such as
privacy, trade secrets, intellectual property, and other confidentiality
protections.
ii In situations where public access to data and methods
will not occur due to other compelling interests, agencies shall apply
especially rigorous robustness checks to analytic results and document what
checks were undertaken. Agency guidelines shall, however, in all cases, require
a disclosure of the specific quantitative methods and assumptions that have
been employed. Each agency is authorized to define the type of robustness
checks, and the level of detail for documentation thereof, in ways appropriate
for it given the nature and multiplicity of issues for which the agency is
responsible.
C. With regard to analysis of risks to human health, safety,
and the environment maintained or disseminated by the agencies, agencies shall
either adopt or adapt the quality principles applied by Congress to risk
information used and disseminated pursuant to the Safe Drinking Water Act
Amendments of 1996 (42 U.S.C. 300g-1(b)(3)(A) and (B)). Agencies responsible
for dissemination of vital health and medical information shall interpret the
reproducibility and peer review standards in a manner appropriate to assuring
the timely flow of vital information from agencies to medical providers,
patients, health agencies, and the public. Information quality standards may be
waived temporarily by agencies under urgent situations (e.g., imminent threats
to public health or homeland security) in accordance with the latitude
specified in agency-specific
guidelines.
4. "Integrity" refers to the security of
information - protection of the information from unauthorized access or
revision, to ensure that the information is not compromised through corruption
or falsification.
5. "Information" means any communication or
representation of knowledge such as facts or data, in any medium or form,
including textual, numerical, graphic, cartographic, narrative, or audiovisual
forms. This definition includes information that an agency disseminates from a
web page, but does not include the provision of hyperlinks to information that
others disseminate. This definition does not include opinions, where the
agency's presentation makes it clear that what is being offered is someone's
opinion rather than fact or the agency's views.
6. "Government information" means information
created, collected, processed, disseminated, or disposed of by or for the
Federal Government.
7. "Information dissemination product" means any
book, paper, map, machine-readable material, audiovisual production, or other
documentary material, regardless of physical form or characteristic, an agency
disseminates to the public. This definition includes any electronic document,
CD-ROM, or web page.
8. "Dissemination" means agency-initiated or
sponsored distribution of information to the public (see 5 CFR 1320.3(d)
(definition of "Conduct or Sponsor"). Dissemination does not include
distribution limited to: government employees or agency contractors or
grantees; intra- or inter-agency use or sharing of government information; and
responses to requests for agency records under the Freedom of Information Act,
the Privacy Act, the Federal Advisory Committee Act, or other similar law. This
definition also does not include distribution limited to: correspondence with
individuals or persons; press releases; archival records; public filings;
subpoenas; or adjudicative processes.
9. "Influential" when used in the phrase
"influential scientific, financial, or statistical information" means that the
agency can reasonably determine that dissemination of the information will have
or does have a clear and substantial impact on important public policies or
important private sector decisions. Each agency is authorized to define
"influential" in ways appropriate for it given the nature and multiplicity of
issues for which the agency is responsible.
10. "Reproducibility" means that the information
is capable of being substantially reproduced, subject to an acceptable degree
of imprecision. For information judged to have more (less) important impacts,
the degree of imprecision that is tolerated is reduced (increased). If agencies
apply the reproducibility test to specific types of original or supporting
data, the associated guidelines shall provide relevant definitions of
reproducibility (e.g., standards for replication of laboratory data). With
respect to analytic results, "capable of being substantially reproduced" means
that independent analysis of the original or supporting data using identical
methods would generate similar analytic results, subject to an acceptable
degree of imprecision or error.
APPENDIX II:
ADAPTING THE PRINCIPLES UNDER THE 1996 SAFE DRINKING WATER ACT AMENDMENTS FOR
SAFETY AND HEALTH RISK ANALYSES
When disseminating influential information in the context of
analyses of safety, health, or environmental risks, the final OMB guidelines
instruct agencies to "
adopt or adapt the quality principles applied by
Congress to risk information used and disseminated pursuant to the Safe
Drinking Water Act Amendments of 1996 (42 U.S.C. 300g-1(b)(3)(A) & (B))."
According to the preamble to OMB's final guidelines (67 F.R. 375), these
principles reflect a "
basic standard of quality for the use of science in
agency decision making" and "
a basic quality standard for the
dissemination of public information about risks of adverse health effects."
Specifically, 42 U.S.C. 300g-1(b)(3)(A) states that "to the degree that an
agency action is based on science, the [EPA] Administrator shall use "(i) the
best available, peer-reviewed science and supporting studies conducted in
accordance with sound and objective scientific practices; and (ii) data
collected by accepted methods or best available methods (if the reliability of
the method and the nature of the decision justifies use of the data)." Under 42
U.S.C. 300g-1(b)(3)(B), "the Administrator shall ensure that the presentation
of information on public health effects is comprehensive, informative, and
understandable." Finally, in documents made available to the public to support
regulation, this section of the Safe Drinking Water Act Amendments requires
such documents to specify the following, to the extent practicable:
(i) each population addressed by any
estimate of public health effects;
(ii) the expected risk or central estimate of
risk for the specific populations;
(iii) each appropriate upper-bound or
lower-bound estimate of risk;
(iv) each significant uncertainty identified in
the process of the assessment of public health effects and studies that would
assist in resolving the uncertainty; and
(v) peer-reviewed studies known to the
Administrator that support, are directly relevant to, or fail to support any
estimate of public health effects and the methodology used to reconcile
inconsistencies in the scientific data.
Within the Department of Labor, analyses of safety and health
risks are performed primarily by the Occupational Safety Health Administration
(OSHA) and the Mine Safety and Health Administration (MSHA). Such analyses have
generally been done only in connection with promulgating safety and health
rules; as such, risk analyses disseminated by these agencies are subject to
statutory requirements governing the bases for regulatory decision making as
well as the public rulemaking process.
DOL is adapting the principles of the Safe Drinking Water Act
Amendments for both health and safety risk analyses. For health analyses, the
principles will be adapted as follows:
1. In taking agency actions that are based on the use of science
in the analysis of health risks, the agency shall use
a. the best available peer-reviewed science and supporting
studies conducted in accordance with sound and objective scientific practices;
and
b. data collected by accepted methods or best available
methods (if the reliability of the method and the nature of the decision
justifies use of the data), including;
i. exposure data such as that generated by
enforcement activity, contained in published literature, and submitted to the
rulemaking record; and ii. testimony and comment from experts familiar
with the underlying scientific information related to the risk analysis and
other relevant information in the rulemaking record.
2. In the
dissemination of public information about risks, the agency shall ensure that
the presentation of information about risk effects is comprehensive,
informative, and understandable, within the context of its intended purpose.
3. In a quantitative analysis of health risks made available to
the public, the agency shall specify, to the extent practicable:
a. each population addressed by any estimate
of public health effects; b. the expected risk or
central estimate of risk for the specific populations;
c. each appropriate upper-bound or lower-bound
estimate of risk; d. each significant uncertainty
identified in the assessment of public health effects and studies that would
assist in resolving the uncertainty; and e.
information, data, or studies, peer-reviewed where available,
known to the agency that support, are directly relevant to, or fail to support
any estimate of risk effects and a discussion that reconciles inconsistencies
in the data or information, and explains the rationale used by the agency to
rely on the data or information used for the risk analysis.
For safety risk analyses, the principles will be adapted as
follows:
1. In taking agency actions that are based on the use of science
in the analysis of safety risks, the agency shall use
a. the best available statistical data from surveys of
fatalities, injuries, and illnesses, and the best available peer-reviewed
science and supporting studies that describe the nature of the safety risks
being addressed;
b. data collected by accepted methods or best
available methods (if the reliability of the method and the nature of the
decision justifies use of the data), including;
i. incident reports compiled from an agency's information
collection or enforcement activities; ii. incident or accident investigation
reports provided by the public or private sectors; iii. relevant analyses
of such information or data, peer reviewed where available; and iv.
testimony of experts familiar with the causal nature of fatalities, injuries,
or illnesses being addressed in the safety risk analysis and other relevant
information in the rulemaking record.
2. In the dissemination of public information about safety
risks, the agency shall ensure that the presentation of information is
comprehensive, informative, and understandable, within the context of its
intended purpose.
3. In a quantitative risk analysis of safety risks made
available to the public, the agency will specify, to the extent
practicable: a. the agency's best estimate of the size of
the population at risk of such effects by industry sector; b. the agency's
best estimates of the total number and or rate of fatalities, injuries, or
illnesses that occur each year and that are relevant to the safety risks being
addressed; c. the possible range in the agency's best estimate of the
number or rate of fatalities, injuries, or illnesses, taking into account
possible uncertainties in the data underlying the estimate; d. data gaps
and other significant uncertainties identified in the assessment of risk
effects and the kind of data or information that would assist in reducing
uncertainty; and e. information, data, or studies, peer-reviewed if
available, known to the agency that support, are directly relevant to, or fail
to support any estimate of risk effects and a discussion that reconciles
inconsistencies in the data or information, and explains the rationale used by
the agency to rely on the data or information used for the risk analysis.
With regard to statutory requirements, Section 6(b)(5) of the OSH
Act of 1970 and Section 101(a)(6)(A) of the Federal Mine Safety and Health Act
of 1977 require the Secretary to set health standards, in part, "on the basis
of the best available evidence," and that development of standards be based
upon "research, demonstrations, experiments, and such other information as may
be appropriate." Section 6(b)(5) and Section 101(a)(6)(A) also state that
"
[i]n addition to the attainment of the highest degree of health and
safety protection for the employee, other considerations shall be the latest
available scientific data in the field..." Furthermore, Section 6(f) of the OSH
Act mandates that the Secretary's determinations be considered conclusive "if
supported by substantial evidence in the [rulemaking] record considered as a
whole."
Thus, the OSH Act and Mine Act reflect the basic principle
underlying the requirements of the Safe Drinking Water Act Amendments - that
agency actions be based on the best scientific information available at the
time of the agency action. OSHA's and MSHA's risk assessments disseminated in
past health rulemakings have relied on the kinds of scientific information
described in the Amendments, i.e., "peer-reviewed science and supporting
studies" as well as other data that the agency considers were collected by
"accepted methods or the best available methods." The agencies recognize that
peer review adds significant value to a scientific study. However, in
developing risk assessments to support rulemakings, the agencies also consider
all other information submitted to the record, including expert testimony,
written comments from the scientific community on data and other information
contained in the record, including risk analyses conducted by rulemaking
participants and submitted to the record. In those instances where agencies are
compelled to take actions to protect the public from serious risks absent the
availability of peer-reviewed scientific literature, DOL believes it consistent
with its adaptation of the SDWA principles, as well as applicable statutes, to
take such action based on data collected by "accepted methods or best available
methods" so long as the nature of the action justifies the use of the data.
Part of what can be considered the risk analysis in the context of
the Safe Drinking Water Act Amendments also appears in OSHA's and MSHA's
Economic Analyses for proposed and final health rules. The Economic Analysis
includes an analysis of worker exposures to the health hazard of interest,
estimates of the sizes of the exposed worker populations in affected industry
sectors, and an analysis of the numbers of exposure-related illnesses that
occur in those populations and the numbers of illnesses potentially avoided by
the new standard. In past rulemakings, OSHA and MSHA have found relatively few
peer-reviewed studies available from which the agencies could reliably
construct exposure profiles for all or most affected industry sectors.
Information and data typically relied upon by the agencies to conduct these
analyses include exposure data generated by enforcement activity, exposure data
submitted to the record by industry or labor organizations, industry studies
conducted by the National Institute for Occupational Safety and Health (NIOSH),
and data obtained by the agencies or their contractors during the conduct of
site visits to industrial facilities. In addition, OSHA has usually relied on
statistics published by the Bureau of Labor Statistics (BLS) or the U.S. Bureau
of the Census to develop estimates of the size of the population at risk.
Economic and cost data are normally not available from
peer-reviewed studies. Such data often comes from industry sources that require
confidentiality, suppliers of equipment and services, and industrial manuals.
Data on profits, sales, and other operations data come from census sources,
Standards and Poors, SRI International and similar sources, surveys,
extrapolations from site visits, and industry and government reports. Data
quality is met in these circumstances by clearly specifying sources, making
available non-confidential information, and providing the spreadsheets and
algorithms used by the agencies so persons can reproduce the analysis.
Analyses of safety risks conducted by OSHA and MSHA to support
safety standards are quite different from health risk analyses in terms of the
kinds of data and information generally available to the agencies. The goal of
a safety risk analysis is to describe the numbers, rates, and causal nature of
injuries related to the safety risks being addressed. OSHA and MSHA have
historically relied on injury and illness statistics from BLS, combined with
incident or accident reports from enforcement activities, incident or accident
reports submitted to the record from the private or public sectors, testimony
of experts who have experience dealing with the safety risks being addressed,
and information and data supplied by organizations that develop consensus
safety standards (such as the American National Standards Institute or the ASTM
International). Thus, DOL's adaptation of the Safe Drinking Water Act
principles for safety risk analysis reflects the use of injury and illness
statistics as a primary source of data, but also calls for the use of
peer-reviewed scientific data and supporting studies where they are
available.
In disseminating its health and safety risk analyses for proposed
and final rules, it has been OSHA's and MSHA's practice to state clearly its
reasons for using the kinds of information and data described above; this is
necessary to demonstrate that the agencies have relied on the "best available
evidence" in making its conclusions. Because of the requirements of rulemaking
procedures to consider all evidence and comment placed in the record by
interested parties, the Department intends to adapt the principles set forth in
the 1996 Safe Drinking Water Act Amendments to reflect that agencies must
consider their statute and case law and data and evidence contained in the
rulemaking docket, provided the agencies clearly state the reasons for relying
on particular data and evidence in the risk analysis. That is, in addition to
"peer-reviewed science and supporting studies" and "data collected by accepted
methods or best available methods," agencies may consider expert testimony,
public comment, and other data and information contained in the rulemaking
record, and may rely on such testimony and information in their risk analyses
provided that the agencies clearly communicate their rationale for selecting
such data and information and why it is consistent with statutory requirements
to use the best available information.
The principles outlined by the Safe Drinking Water Act Amendments
also contain a specification for reporting results of risk analyses, as
described above. For health risk analyses, OSHA and MSHA have historically
reported their "best estimate" of the risk to workers exposed to a health
hazard; this has typically been an estimate that the agencies refer to as a
"maximum likelihood" estimate derived from the statistical procedure of fitting
a mathematical exposure-response curve to dose-response data. The agencies also
typically have reported statistical upper limits of their estimates of risk.
The industry and exposure profiles presented in the Economic Analysis provide
estimates of the populations at risk, by affected industry sector. Finally,
during the course of rulemaking, OSHA and MSHA must consider and address data,
expert testimony, and public comment that deal with uncertainties in the risk
assessment and with conflicting scientific evidence. As part of demonstrating
that it has relied on the "best available evidence", the agency must also
clearly present its reasons for accepting certain studies or data and rejecting
others, and reconcile apparent discrepancies or conflicts in the available data
to the extent possible. These practices are consistent with the reporting
principles described by the Safe Drinking Water Act Amendments, as well as the
obligations of the OSH Act and the Mine Act.
These general principles also apply to OSHA's and MSHA's reporting
of results and conclusions from analyses of safety risks; that is, the agencies
make every effort to reliably estimate the sizes of the populations at risk and
the magnitude of the safety risk presented to workers, and to explain
uncertainties and apparent discrepancies in the available data. However, as
described above, the methods and underlying data relied on for safety risk
analyses are often different from that for health risk analyses; thus, DOL has
adapted the language of the Safe Drinking Water Act principles as applied to
the dissemination of information on safety risks to reflect the kinds of
results typically obtained from a safety risk analysis.
APPENDIX III: BLS
GUIDELINES FOR INFORMING USERS OF INFORMATION QUALITY AND
METHODOLOGY
The Office of Management and Budget (OMB) has issued
government-wide information quality guidelines in accordance with Section 515
of the Treasury and General Government Appropriations Act for Fiscal Year 2001.
The purpose of these guidelines is to ensure and maximize the quality, utility,
objectivity, and integrity of information disseminated by Federal agencies. The
guidelines direct each Federal agency to issue its own Section 515 guidelines.
As part of the Department of Labor, BLS follows DOL's information quality
guidelines, as well as the OMB Guidelines. Moreover, as part of the Interagency
Council on Statistical Policy, BLS supports the Council's commitment to
information quality. The following BLS Guidelines for Informing Users of
Information Quality and Methodology supplement the DOL guidelines that apply to
all Departmental agencies.
The Bureau of Labor Statistics is the principal fact-finding
agency for the Federal Government in the broad field of labor economics and
statistics. BLS is an independent national statistical agency within the
Department of Labor that collects, processes, analyzes, and disseminates
essential statistical data to the American public, business, and labor. BLS
also serves as a statistical resource to the Department of Labor.
BLS data must satisfy a number of criteria, including relevance to
current social and economic issues, timeliness in reflecting today's rapidly
changing economic conditions, accuracy, consistently high statistical quality,
and impartiality in both subject matter and presentation.
As a Federal statistical agency, BLS conducts work in an open
environment. Major changes in program design, scope, or methods are discussed
in advance with users and advisory committees and described in published
materials. Fair information practices are used, such as maintaining the
confidentiality of individual responses. Confidentiality of the information
that respondents furnish is assured by protecting the microdata, combining the
data reported, and issuing the findings in summary tables, analyses, and
reports. BLS values cooperation with data users and consults with a broad
spectrum of users of its data in order to make its products more useful. As
part of its customer pledge to the public, BLS promises to help users
understand the uses and limitation of the data.
BLS applies statistical information quality principles provided in
guidance from the Office of Management and Budget (OMB Statistical Policy
Directives, for example) as well as the National Research Council's Principles
and Practices for a Federal Statistical Agency. Moreover, all BLS information
products are subjected to a multi-stage review before they are disseminated to
the public.
A key component in ensuring information quality is integrity, or
the protection of data from corruption through unauthorized access. BLS data
integrity guidelines spell out procedures to protect the confidentiality of BLS
records, the process of data collection, and various security measures.
To inform users about information quality and methodology, BLS
provides descriptions of the methods and procedures used to develop and produce
its statistical products. These descriptions are prepared at various levels of
complexity and comprehensiveness to address the wide range of user needs.
Summary level technical notes are usually included with news releases. For most
programs, a periodical of record contains more comprehensive technical
material. In addition, Major Programs of the Bureau of Labor Statistics
provides a summary description of data availability, coverage, sources of data,
reference periods, major uses, and forms of publication. The BLS Handbook of
Methods covers most major programs and is updated every few
years.
A major purpose for providing users with information on
methodology is to assist them in determining whether the data adequately meet
their needs both in terms of closeness of concept and range of statistical
error.
BLS makes the information it disseminates and the methods used to
produce this information as transparent as possible, so they could, in
principle, be reproduced by qualified individuals. In practice, however, most
estimates included in the BLS information products are not directly
reproducible by the public because the underlying data used to produce them
contain confidential information about individual respondents. The
transparency, therefore, has the related goal of providing enough information
about methodology for the public to understand the information and to have
confidence in its preparation.
The level of documentation on methodology may differ among
statistical programs based on type of data (from households or establishments),
frequency of collection (monthly, annually, one-time), expected uses of the
information, budget, and how long the survey has been in existence. The type of
survey, census, or data collection process also may affect the existence of
generally accepted evaluation methods and data collection protocols may affect
the consistency of documentation.
Most of the information on information quality and methodology are
available in both print and electronic form to assist the broad range of users.
Current descriptions of specific BLS surveys and programs are available at
http://www.bls.gov/bls/descriptions.htm.
This page provides links to the relevant sections of the BLS Handbook of
Methods. For most programs, the Handbook provides a variety of information
that, as appropriate, may include a background summary, a description of the
concepts, the sources of data and collection methods, the sampling and
estimation procedures, and the uses and limitations of the statistics.
Additional related information may be available from a program's home page, and
links to these home pages are included with the descriptions.
As part of its commitment to information quality, BLS encourages
communication with its users. In addition to formal advisory councils from the
business, labor, and academic communities, BLS fosters discussions with the
public at large by making it easy to reach staff by a variety of formats,
including phone, mail, and email. A customer service guide is published
annually with the names, phone numbers, and email addresses of subject matter
specialists who can answer technical questions about the information BLS
issues. Every page on the BLS web site has a link to a subject matter contact,
a technical contact, and a general feedback contact. Every print publication
also contains contact information. For more information on how to contact BLS,
see its contact page.
Affected persons who believe that BLS has disseminated information
that does not meet its guidelines, or those of the DOL or OMB and who wish to
follow a formal complaint process, may send their complaint to the point of
contact that BLS is designating.
APPENDIX
IV: SAMPLE INFORMATION QUALITY ASSURANCE TECHNIQUES AND
METHODS
The table below provides some sample techniques and methods
derived from industry best practices. This is not intended to be an exhaustive
list. Agencies should select and apply techniques and methods depending on the
complexity, influence, and subject matter of each information product.
Techniques and
Methods |
Definition |
Applicability |
Peer Review |
An independent assessment of the
technical and scientific merit of research by individuals knowledgeable in the
particular subject of interest and with no unresolved conflict of
interest.
|
Peer review is an appropriate
technique for reviewing scientific studies and economic analyses. |
Certification |
Process of reviewing information prior to official release to ensure
that erroneous data are not released, or to identify data of marginal quality.
It is often conducted concurrently with an interpretative analysis of the data.
|
Certification is an appropriate technique for
statistical programs. |
Performance Measures |
Numerical indicators of the progress of the development of
information. |
Performance Measures should be used to help
management track the development of an information product and improve
information quality, but they generally should also be used in conjunction with
other, more rigorous quality assurance techniques. |
Check Lists |
A specific, step-based plan
designed to ensure all appropriate actions are taken.
|
Notes the steps in production
that can identify inconsistencies, mistakes, or weaknesses, and ensure
completeness. |
Bureau of Labor Statistics (BLS) Handbook of
Methods |
Publication explaining how the
BLS obtains and prepares the economic data it publishes.
|
Applicable to statistical data
collected by the BLS; and includes approaches and methodologies that could be
appropriate for other agencies conducting surveys. |
1 Public Law
106-554, App. C.
2 See Guidelines for Ensuring and Maximizing the Quality,
Objectivity, Utility, and Integrity of Information Disseminated by Federal
Agencies; Notice; Republication; Office of Management and Budget, Federal
Register Vol. 67, No. 36, pp. 452-460, February 22, 2002.
|