Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by the Department of Labor

INTRODUCTION
SCOPE AND APPLICABILITY
QUALITY MANAGEMENT PRINCIPLES
1996 SAFE DRINKING WATER ACT AMENDMENTS FOR SAFETY AND HEALTH RISK ANALYSIS PRINCIPLES
BLS GUIDELINES FOR INFORMING USERS OF INFORMATION QUALITY AND METHODOLOGY
INFORMATION QUALITY RESPONSIBILITIES
INFORMATION CATEGORIES
PRIVACY AND SECURITY INFORMATION
INFORMATION QUALITY ASSURANCE PROCESS
INFORMATION COMPLAINT AND APPEAL PROCESS
TRACKING AND REPORTING INFORMATION COMPLAINTS AND APPEALS
APPENDIX I: INFORMATION QUALITY GLOSSARY
APPENDIX II: ADAPTING THE PRINCIPLES UNDER THE 1996 SAFE DRINKING WATER ACT AMENDMENTS FOR SAFETY AND HEALTH RISK ANALYSES
APPENDIX III: BLS GUIDELINES FOR INFORMING USERS OF INFORMATION QUALITY AND METHODOLOGY
APPENDIX IV: SAMPLE INFORMATION QUALITY ASSURANCE TECHNIQUES AND METHODS

INTRODUCTION

Background
On December 21, 2000, Congress passed Section 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001⁽¹⁾ (the Act). OMB issued final guidance⁽²⁾ for implementing the Act, which required all Federal agencies to:

• Issue information quality guidelines ensuring and maximizing the quality, objectivity, utility, and integrity of information, including statistical information, disseminated by the Department;
• Establish administrative mechanisms allowing affected persons to seek and obtain correction of information maintained and disseminated by the Department that does not comply with the OMB guidelines; and
• Report to the Director of OMB the number and nature of complaints received regarding compliance with the OMB guidelines, including how the complaints were resolved.

The OMB guidance directed Federal agencies to publish their draft Information Quality guidelines on their websites by April 1, 2002. OMB further clarified on March 4, 2002, that the website publishing date was extended to May 1, 2002.

The Department of Labor (DOL) posted its draft guidelines to its web site on May 1, 2002. On May 22, 2002, DOL posted an appendix addressing the adaptation of the Safe Drinking Water Act to the development of risk assessments. DOL published notice of the guidelines in the Federal Register on May 1, 2002, and required that comments be submitted by May 31, 2002. DOL extended the comment period to June 30, 2002, in response to public requests and to OMB's decision to extend the submission date for agencies to provide revised guidelines to OMB.

These guidelines represent changes made to the draft DOL guidelines posted for public comment on May 1, 2002, and incorporate revisions based upon feedback from OMB, public comments, and internal Departmental review.

Purpose

The purpose of these guidelines is to establish Departmental guidance for implementing an Information Quality program at DOL. This Information Quality guidance is intended to enhance the quality of the information disseminated by DOL.

SCOPE AND APPLICABILITY

These guidelines are intended, within the context of laws administered and enforced by DOL, to meet the information quality objectives set forth in OMB's guidelines. They are intended to improve the internal management of the Federal Government. They are not intended to impose any binding requirements or obligations on DOL or the public or to create any right or benefit, substantive or procedural, enforceable at law by a party against the United States, its agencies, officers, or any person. They are not intended to provide any right to judicial review.

These guidelines reflect this Department's commitment to information quality as an important management objective that takes its place alongside other Departmental objectives, such as ensuring the success of agency missions, observing budget resource priorities and restraints, and providing information to the public. Agencies should strive to assure that these goals reinforce each other as much as is practicable. Where an agency believes that they conflict, it should, consistent with its legal responsibilities, attempt to reconcile them in a manner that the agency believes will best serve the public interest and help the agency meet its statutory or program obligations. Program efficiency must be a critical goal as DOL agencies carry out their responsibilities under these guidelines. Thus, for example, it may not be in the public interest for agencies to devote significant resources to correcting information where the expenditure of such resources is not, in the agency's view, cost effective in light of the significance of the data and the agency's more pressing priorities and obligations.

The DOL's pre-dissemination reviews apply to information that DOL first disseminates on or after October 1, 2002. Other aspects of these guidelines, including the information correction process apply on or after October 1, 2002, with respect to information that DOL disseminates on or after October 1, 2002, regardless of when DOL first disseminated the information. Information means any communication or representation of knowledge such as facts or data, in any medium or form, including textual, numerical, graphic, cartographic, narrative, or audiovisual forms. Dissemination includes agency initiated or sponsored distribution of information to the public. It does not include agency citation to or discussion of information that was prepared by others and considered by the agency in the performance of its responsibilities, unless an agency disseminates it in a manner that reasonably suggests that the agency agrees with the information. Agency sponsored distribution of information covers instances where an agency has directed a third party to disseminate information or where the agency has the authority to review and approve the information before release. By contrast, if an agency funds research, but the researcher decides whether or not to disseminate the results, the agency has not "sponsored" the dissemination, and the information is not subject to these guidelines. In these instances, agencies should direct the researcher to include an appropriate disclaimer in the publication. Similarly, the guidelines would not cover publications of their research findings by Departmental employees or Federal grantees or contractors when published in the same manner as their academic colleagues. Again, the researchers should include an appropriate disclaimer noting that the views are theirs and not necessarily those of DOL.

These guidelines do not apply to the following:

• Information distribution limited to government employees or agency contractors or grantees;
• Government information intended to be limited to intra- or inter-agency use or sharing of information, such as strategic plans, performance plans, program reports, operating plans, or budgets;
• Responses to requests for Departmental records under the Freedom of Information Act, the Privacy Act, the Federal Advisory Committee Act, or other similar laws;
• Correspondence or other communications with individuals or organizations;
• Press releases (except where the press release itself is the primary source of the information);
• Congressional testimony;
• Archival records;
• Public filings;
• Dissemination of information through subpoenas or adjudicative processes, such as those recognized under the Administrative Procedure Act or established pursuant to regulation; provided, however, that information originally disseminated through such vehicles could subsequently become subject to these guidelines to the extent it is re-disseminated more broadly through other vehicles;
• Information clearly represented as opinion and not an official agency or Departmental representation;
• Policy guidance recommendations or statements or summaries of agency policies, procedures, or programs;
• Statements of legal policy or interpretation, including briefs filed with courts or administrative bodies; and
• Final agency decisions, settlements in litigation and descriptions of these settlements, or determinations of legal force and effect, such as wage determinations.

These guidelines apply to all agencies of DOL, except to the extent that agencies have adopted tailored agency-specific guidelines.

For a glossary of numerous terms and their definitions used throughout the rest of this document, please consult Appendix I. The definitions are from the OMB guidance for the Act.

QUALITY MANAGEMENT PRINCIPLES

Every agency should establish information quality, as defined in OMB and DOL information quality guidelines, as a performance goal. Quality includes the "utility," "objectivity," and "integrity" of the information. The level of quality should be "appropriate to the nature and timeliness of the information to be disseminated" and will be affected by the nature of the underlying data. In considering utility, agencies should evaluate the usefulness of particular information to those expected to use it. The information also should be objective--"accurate, reliable, and unbiased," and presented "in an accurate, clear, complete, and unbiased manner." Agencies also should protect the integrity of information from unauthorized access or revision

These objectives and guidelines are to be interpreted consistent with DOL's statutory obligations. Where agencies are disseminating information of a scientific, financial, or statistical nature, they should use sound statistical and research methods to develop and analyze the data. Depending on the type of information disseminated and consistent with statutory and confidentiality restrictions, agencies should identify the sources of the information and where appropriate, the supporting data, models, and error sources.

Where agencies develop and disseminate "influential" scientific, financial, or statistical information, they should provide a higher level of transparency about data and methods. Unless prevented by confidentiality, legal constraints, or other compelling interests, the level of transparency should be such that qualified third parties could reproduce the information. In identifying what kinds of information may be subject to reproducibility standards, agencies should use commonly accepted scientific, financial, or statistical standards. Agencies are encouraged to make arrangements that will permit appropriate public access to the related original and supporting data and analytical results. Regarding analytical results in situations where agencies do not permit access to data or methods due to other compelling interests, such as confidentiality protections, agencies should, unless otherwise prohibited by law, generally disclose their data sources (at whatever levels of generality are needed to preserve necessary confidentiality), quantitative methods and assumptions that have been employed, and the types of robustness checks used to assure the quality of results.

1996 SAFE DRINKING WATER ACT AMENDMENTS FOR SAFETY AND HEALTH RISK ANALYSIS PRINCIPLES

With regard to analysis of risks to human health, safety, and the environment maintained or disseminated by agencies, the Occupational Safety and Health Administration (OSHA) and the Mine Safety and Health Administration (MSHA), in performing risk analysis, are hereby adapting the standards contained in the Safe Drinking Water Act, as set forth in Appendix II. DOL does not anticipate that any other agencies will be performing such analysis of risks for their programs. However, to deal with unforeseen contingencies, DOL hereby adopts the Safe Drinking Water Act standards with respect to all programs other than OSHA and MSHA. Should it be necessary in the future for another DOL agency to perform such an analysis, DOL will consider, at that time, whether it is appropriate to adapt the Safe Drinking Water Act standards.

BLS GUIDELINES FOR INFORMING USERS OF INFORMATION QUALITY AND METHODOLOGY

Appendix III contains the Bureau of Labor Statistics (BLS) Guidelines for Informing Users of Information Quality and Methodology supplement to the overall Departmental guidelines. These supplemental guidelines reaffirm BLS commitment to both OMB and DOL information quality guidelines. Moreover, as part of the Interagency Council on Statistical Policy, BLS supports the Council's commitment to information quality.

INFORMATION QUALITY RESPONSIBILITIES

It is the responsibility of all Departmental agencies to make information quality an important goal in every phase of a product's development. The following responsibilities pertain to the implementation of DOL's information quality guidelines.

Chief Information Officer (CIO)

• Maintain a leadership role in overseeing the implementation of these guidelines and in providing guidance to the agencies on information quality matters.
• Develop and submit to OMB the annual report on the number, nature, and resolution of complaints.
• Coordinate, as appropriate, with other Federal organizations on cross-agency information quality issues.
  

Agency Heads

• Apply, consistent with applicable statutes and regulations, DOL's information quality policies, procedures, and guidance to Department-sponsored information products that an agency has direct authority to control.
• Ensure that, where Department-sponsored information does not necessarily reflect the views of DOL, an appropriate disclaimer will be included.
• Ensure that in its submissions to the OMB under the Paperwork Reduction Act, the agency demonstrates how it is attempting to provide that information will be collected, maintained, and used in a way consistent with OMB and DOL information quality standards.

INFORMATION CATEGORIES

Per OMB's guidance, information means any communication or representation of knowledge such as facts or data, in any medium or form, including textual, numerical, graphic, cartographic, narrative, or audiovisual forms. This definition includes information that an agency disseminates from a web page, but does not include the provision of hyperlinks to information that others disseminate. This definition does not include opinions, where the agency's presentation makes it clear that what is being offered is someone's opinion rather than fact or the agency's views on information of the kind that is subject to these guidelines.

DOL has identified two categories of information that are disseminated to the public, with the level of quality control and review being greater for influential information than for non-influential information. Whether information is influential is to be determined on an item-by-item basis rather than by aggregating multiple studies, documents, or other informational items that may influence a single policy or decision.

Influential
Definition:

This category contains scientific, financial, or statistical information when agencies can reasonably determine that dissemination will have or does have a clear and substantial impact on important public policies or important private sector decisions.

To be influential, information should have a clear and substantial impact. A clear impact is one that is determined by the agency to have a high probability of occurring. A substantial impact is one that meets the levels of significance described below.

In rulemaking, influential information is scientific, financial, or statistical information that the agency believes will have a clear and substantial impact on the resolution of one or more key issues in an economically significant rulemaking, as that term is defined in section 3(f)(1) of Executive Order 12866.

In non-rulemaking contexts, DOL should consider two factors in relation to each other - breadth and intensity - in determining whether information is influential. These factors need to be considered together. Information that has a low cost or modest impact on a limited range of affected parties is less likely to be influential than information that can have a very costly or crucial impact on a broad range of parties. Of course even information that has a low cost or modest impact on any one party can be influential if it can impact a broad range of parties -- for example, an action that could cost an individual employer only $20 could nevertheless be influential if it impacts the vast number of employers in the United States, since it would then have an aggregate effect in excess of $100 million. Within that framework, in considering whether information has a high intensity impact, agencies should use as a benchmark the $100 million figure used to determine whether a rule is economically significant. It should be noted that the definition of "influential" applies to information itself and not to the decisions that the information may support. Even if a decision or action by an agency is very important, a particular piece of information supporting it may not be influential, for example, because it is cumulative to other information or because it involves legal or policy issues.

Moreover, if it is merely arguable that an impact will occur, or if it is a close judgment call, then the impact is probably not clear and substantial. The "influential" designation is intended to be applied to information only when clearly appropriate. Agencies should not designate information products or types of information as influential on a regular or routine basis. Nor should agencies place an "influential" label on the title page or text of an information product.

Examples:

Principal economic indicators, such as Consumer Price Index, the Employment Situation, and Producer Price Index; the Private Pension Bulletin; and Unemployment Insurance Weekly Claims data.

Non-influential
Definition:

All information disseminated to the public that does not meet the criteria set forth in the influential information definition.

Examples:

Fact sheets (e.g., Disaster Unemployment Assistance, Injury Trends in Mining), OSHA Construction Resource Manual, technical information issuances, annual reports, and studies (e.g., Pension and Health Benefits of American Workers, Coverage Status of Workers under Employer-Provided Pension Plans, and Study of 401(k) Plan Fees and Expenses, Study of Health Insurance Coverage of the Unemployed).

PRIVACY AND SECURITY INFORMATION AND PUBLIC

Regardless of the category of information, all agencies will comply with the Privacy and Security Statement posted on DOL's web site. DOL is strongly committed to maintaining the privacy of information and the security of its computer systems. With respect to the collection, use, and disclosure of information, DOL makes every effort to ensure compliance with applicable Federal laws, including, but not limited to, the Privacy Act of 1974, the Paperwork Reduction Act of 1995, the Trade Secrets Act, and the Freedom of Information Act. DOL reaffirms its commitment to keep the public appropriately informed.

As part of its efforts to ensure and maintain the integrity of the information disseminated to the public, DOL's IT security policy and planning framework is designed to protect information from unauthorized access or revision and to ensure that the information is not compromised through corruption or falsification.

INFORMATION QUALITY ASSURANCE PROCESS

Departmental agencies should use the information quality assurance process described below to maximize the quality of information disseminated. Agencies should use information quality assurance processes that are appropriate to the complexity and importance of the product being developed. Agencies may use appropriate pre-existing information quality assurance processes that are at least as effective as those of DOL or OMB.

The quality assurance process should begin at the inception of the product development process. At the initiation of the product development process, agencies should consult existing Departmental and agency information quality assurance guidelines. Agencies should determine the information category of the product to be developed, the level of quality assurance needed, and the appropriate techniques required to maximize and ensure information quality.

There are numerous techniques and methods agencies can utilize to ensure they consistently produce and disseminate quality information. Appendix IV provides some sample techniques and methods derived from industry best practices. Agencies should use the information quality assurance techniques and methods that they determine are most appropriate for their information products. If agencies choose to conduct a formal, independent, external peer review of data and analytical results, the peer review should meet the following general criteria: (a) peer reviewers should be selected primarily on the basis of necessary technical expertise; (b) peer reviewers should be expected to disclose to agencies prior technical or policy positions they may have taken on the issues at hand; (c) peer reviewers should be expected to disclose to agencies their sources of personal and institutional funding (private or public sector); and (d) peer reviews should be conducted in an open (made public) and rigorous manner.

Agencies should incorporate the selected quality assurance techniques into the project development schedule. Throughout the product's development, agencies should ensure that quality assurance decisions are defensible and appropriate to the category of information involved. The product may be subject to internal agency quality controls and any appropriate Departmental reviews before being disseminated to the public. For example, if an agency decides to post the information on the DOL web site, it should adhere to DOL's and its own Public Web Site Review and Clearance Process. Agencies should incorporate lessons learned into future product development activities so as to improve DOL's overall quality management process.

INFORMATION COMPLAINT AND APPEAL PROCESS

Because DOL is committed to information dissemination programs based on high standards of quality, it recognizes the value of public input. DOL therefore encourages the affected public to suggest improvements in Departmental information quality practices and to contact it when particular disseminated information may not meet the OMB guidelines and the guidelines set forth above. DOL believes that in most cases, informal contacts would be appropriate.

Sometimes agencies and affected persons may find it helpful to resolve concerns about information in a more structured way and may choose to follow a more formal process. DOL will make available to the public a list of officials to whom complaints and appeals should be sent and where and how such officials may be reached. Affected persons may submit such complaints and appeals to the contact point in the DOL agency responsible for the information. Each agency may designate one or more officials to review information complaints and another official or officials who will be responsible for appeals if the complainant is dissatisfied with the initial response to the complaint. The agency should provide that the official conducting the second level review is not the same official who responded to the initial request or from the same office that prepared the information in question. In determining the level of the person designated to respond to appeals, the agency may wish to consider such factors as the qualifications of the person and the significance of the information in question. An agency may, within its discretion, wish to consider the designation of a panel to resolve appeals involving influential information, when the agency believes that such an appellate board is needed and is an efficient way to provide expertise or perspective or otherwise to improve the resolution of the appeal. Designated agency officials may consult with other agency or Departmental offices, as the agency may deem appropriate to the resolution of the complaint.

The purpose of the information complaint and appeal process is to deal with information quality matters, not to resolve underlying substantive policy or legal issues.

As is the case with other provisions of these guidelines, the process is intended to improve the internal management of the Federal Government. It is not intended to create any right or benefit, substantive or procedural, enforceable at law by a party against the United States, its agencies, officers, or any person. It is not intended to provide any right of judicial review. Concerns regarding information in a rulemaking must, except as provided below, be presented in the rulemaking in accordance with the rulemaking's procedures.

Overview of Information Complaint and Appeal Process
Affected persons may indicate their interest in following a more structured complaint and appeal process by expressing that interest to the concerned agency. Complainants should:

• Identify themselves and indicate where and how they can be reached;
• Identify, as specifically as possible, the information in question;
• Indicate how they are affected by the information about which they are complaining;
• Carefully describe the nature of the complaint, including an explanation of why they believe the information does not comply with OMB, Departmental, or agency-specific guidelines; and
• Describe the change requested and the reason why the agency should make the change.

Failure to include this information may result in a complainant not receiving a response to the complaint or greatly reducing the usefulness or timeliness of any response. Complainants should be aware that they bear the burden of establishing that they are affected persons and showing the need and justification for the correction they are seeking, including why the information being complained about does not comply with applicable guidelines.

In deciding how to handle complaints, agencies should be especially mindful of their legal obligations, program priorities, resource constraints, and their duty to use resources efficiently. For example, agencies have important responsibilities to issue rules and provide compliance guidance to the public. Agencies must administer the complaint and appeal process consistent with these obligations and their responsibilities to carry them out in an expeditious manner.

Any structured process would not apply to an agency's archival information or to public filings. Agencies may choose not to respond to complaints about claimed defects that are frivolous or unlikely to have substantial future impact.

Where procedures exist for dealing with information quality issues, agencies may consider resolving complaints by referring them to these procedures. For example, complaints about the quality of information in a rulemaking are ordinarily to be submitted and handled in accordance with rulemaking procedures. As agencies consider information quality issues within the context of a rulemaking, they are reminded of their primary responsibility to resolve these issues in a manner consistent with the Administrative Procedure Act and the substantive statute pursuant to which the rule is being issued. In unusual circumstances involving an information product related to a rulemaking, agencies should consider an information complaint under these information correction procedures. When the agency disseminates a study, analysis, or other information prior to the final agency action or information product, requests for correction will be considered prior to the final agency action or information product in those cases if the agency determines that an earlier response would not unduly delay issuance of the agency action or information product and the complainant has shown a reasonable likelihood of suffering actual harm from the agency's dissemination if the agency does not resolve the complaint prior to the final agency action or information product. In deciding what action may be appropriate in these unusual circumstances, agencies should consider the factors previously discussed in these guidelines. They also may consider: (1) the impact of the information on the complainant; (2) the extent to which the complainant's concerns have been rendered moot as a result of actions taken by the agency; (3) the mechanisms available under the Administrative Procedure Act or other laws to resolve complainant's concerns; and (4) the public interest to be served in pursuing further action on the complaint.

Where an agency responds directly to a complaint, it should respond in the manner that it deems most suitable, whether by letter, telephone, email, or otherwise.

Agencies should try to respond to complaints and appeals within sixty (60) days of their receipt, unless they deem a response within this time period to be impracticable. If an agency believes that more time is required to decide how to respond to a complaint or appeal, it should estimate the time needed and notify the complainant within the 60-day period of the reasons for the delay and the time that it estimates that a decision will be reached. Once the agency had decided how to address the complaint, it should notify the complainant.

If a complainant is dissatisfied with the initial response to the complaint, he or she may submit an appeal to the designated contact point in the agency responsible for the information.

A complainant may appeal within forty-five (45) days of the date the agency notified the complainant how it would handle the complaint or one hundred and five (105) days from the date on which an agency or agencies first received the complaint, whichever is later. The appeal request should contain the same contact and descriptive information that was provided in the original complaint and the specific reasons why the initial agency response was not satisfactory. Once an appeal decision has been rendered by the agency, it should notify the complainant.

In processing initial complaints and appeal requests, DOL and its agencies should be flexible and take into account, among other things, the nature, significance, and volume of complaints, the agency's particular program needs, and available review mechanisms.

TRACKING AND REPORTING INFORMATION COMPLAINTS AND APPEALS

The Office of the Chief Information Officer (OCIO) is responsible for reporting the results of the Department's information quality efforts as required by OMB guidance. OCIO anticipates using a system to centrally track and report complaints and appeals.

In accordance with procedures to be specified by the CIO, DOL will establish on its web site or agency web sites an information quality site to keep the public informed about information quality on a timely basis. The purpose of the information quality web site would be to inform the public about the agency's information quality practices and procedures. The information quality web site should include access to the agency's information quality guidelines and an easy-to-understand explanation of the agency's procedures regarding complaints (which will include an explanation of how a person may file a complaint and, subsequently, an administrative appeal of the agency's response to the complaint). The information quality web site also could contain other types of information, such as a description of significant corrections that the agency has made as a result of the information complaint and appeal process. Each agency should determine the content of this information page based on its mission, activities subject to the guidelines, and the expected level of interest by members of the public.

APPENDIX I: INFORMATION QUALITY GLOSSARY

OMB provides the following definitions in its guidance for the Act.

1. "Quality" is an encompassing term comprising utility, objectivity, and integrity. Therefore, the guidelines sometimes refer to these four statutory terms, collectively, as "quality."

2. "Utility" refers to the usefulness of the information for its intended users, including the public. In assessing the usefulness of information that the agency disseminates to the public, the agency needs to consider the uses of information not only from the perspective of the agency but also from the perspective of the public. As a result, when transparency of information is relevant for assessing the information's usefulness from the public's perspective, the agency must take care to ensure that transparency has been addressed in its review of the information.

3. "Objectivity" involves two distinct elements, presentation and substance.

a. "Objectivity" includes whether disseminated information is being presented in an accurate, clear, complete, and unbiased manner. This involves whether the information is presented within a proper context. Sometimes, in disseminating certain types of information to the public, other information must also be disseminated in order to ensure an accurate, clear, complete, and unbiased presentation. Also, the agency needs to identify the sources of the disseminated information (to the extent possible, consistent with confidentiality protections) and, in scientific, financial, or statistical context, the supporting data and models, so that the public can assess for itself whether there may be some reason to question the objectivity of the sources. Where appropriate, supporting data should have full, accurate, transparent documentation, and error sources affecting data quality should be identified and disclosed to users.

b. In addition, "objectivity" involves a focus on ensuring accurate, reliable, and unbiased information. In a scientific, financial, or statistical context, the original and supporting data shall be generated, and the analytical results shall be developed, using sound statistical and research methods.

i. If data and analytic results have been subjected to formal, independent, external peer review, the information may generally be presumed to be of acceptable objectivity. However, this presumption is rebuttable based on a persuasive showing by the petitioner in a particular instance. If agency-sponsored peer review is employed to help satisfy the objectivity standard, the review process employed shall meet the general criteria for competent and credible peer review recommended by OMB-OIRA to the President's Management Council (9/20/01) (http://www.whitehouse.gov/omb/inforeg/oira_review-process.html), namely, "that (a) peer reviewers be selected primarily on the basis of necessary technical expertise, (b) peer reviewers be expected to disclose to agencies prior technical/policy positions they may have taken on the issues at hand, (c) peer reviewers be expected to disclose to agencies their sources of personal and institutional funding (private or public sector), and (d) peer reviews be conducted in an open and rigorous manner."

ii. If an agency is responsible for disseminating influential scientific, financial, or statistical information, agency guidelines shall include a high degree of transparency about data and methods to facilitate the reproducibility of such information by qualified third parties.

A. With regard to original and supporting data related thereto, agency guidelines shall not require that all disseminated data be subjected to a reproducibility requirement. Agencies may identify, in consultation with the relevant scientific and technical communities, those particular types of data that can practicably be subjected to a reproducibility requirement, given ethical, feasibility, or confidentiality restraints. It is understood that reproducibility of data is an indication of transparency about research design and methods and thus a replication exercise (i.e., a new experiment, test, or sample) shall not be required prior to each dissemination.

B. With regard to analytic results related thereto, agency guidelines shall generally require sufficient transparency about data and methods that an independent reanalysis could be undertaken by a qualified member of the public. These transparency standards apply to agency analysis of data from a single study as well as to analyses that combine information from multiple studies.

i Making the data and methods publicly available will assist in determining whether analytic results are reproducible. However, the objectivity standard does not override other compelling interests such as privacy, trade secrets, intellectual property, and other confidentiality protections.

ii In situations where public access to data and methods will not occur due to other compelling interests, agencies shall apply especially rigorous robustness checks to analytic results and document what checks were undertaken. Agency guidelines shall, however, in all cases, require a disclosure of the specific quantitative methods and assumptions that have been employed. Each agency is authorized to define the type of robustness checks, and the level of detail for documentation thereof, in ways appropriate for it given the nature and multiplicity of issues for which the agency is responsible.

C. With regard to analysis of risks to human health, safety, and the environment maintained or disseminated by the agencies, agencies shall either adopt or adapt the quality principles applied by Congress to risk information used and disseminated pursuant to the Safe Drinking Water Act Amendments of 1996 (42 U.S.C. 300g-1(b)(3)(A) and (B)). Agencies responsible for dissemination of vital health and medical information shall interpret the reproducibility and peer review standards in a manner appropriate to assuring the timely flow of vital information from agencies to medical providers, patients, health agencies, and the public. Information quality standards may be waived temporarily by agencies under urgent situations (e.g., imminent threats to public health or homeland security) in accordance with the latitude specified in agency-specific guidelines.

4. "Integrity" refers to the security of information - protection of the information from unauthorized access or revision, to ensure that the information is not compromised through corruption or falsification.

5. "Information" means any communication or representation of knowledge such as facts or data, in any medium or form, including textual, numerical, graphic, cartographic, narrative, or audiovisual forms. This definition includes information that an agency disseminates from a web page, but does not include the provision of hyperlinks to information that others disseminate. This definition does not include opinions, where the agency's presentation makes it clear that what is being offered is someone's opinion rather than fact or the agency's views.

6. "Government information" means information created, collected, processed, disseminated, or disposed of by or for the Federal Government.

7. "Information dissemination product" means any book, paper, map, machine-readable material, audiovisual production, or other documentary material, regardless of physical form or characteristic, an agency disseminates to the public. This definition includes any electronic document, CD-ROM, or web page.

8. "Dissemination" means agency-initiated or sponsored distribution of information to the public (see 5 CFR 1320.3(d) (definition of "Conduct or Sponsor"). Dissemination does not include distribution limited to: government employees or agency contractors or grantees; intra- or inter-agency use or sharing of government information; and responses to requests for agency records under the Freedom of Information Act, the Privacy Act, the Federal Advisory Committee Act, or other similar law. This definition also does not include distribution limited to: correspondence with individuals or persons; press releases; archival records; public filings; subpoenas; or adjudicative processes.

9. "Influential" when used in the phrase "influential scientific, financial, or statistical information" means that the agency can reasonably determine that dissemination of the information will have or does have a clear and substantial impact on important public policies or important private sector decisions. Each agency is authorized to define "influential" in ways appropriate for it given the nature and multiplicity of issues for which the agency is responsible.

10. "Reproducibility" means that the information is capable of being substantially reproduced, subject to an acceptable degree of imprecision. For information judged to have more (less) important impacts, the degree of imprecision that is tolerated is reduced (increased). If agencies apply the reproducibility test to specific types of original or supporting data, the associated guidelines shall provide relevant definitions of reproducibility (e.g., standards for replication of laboratory data). With respect to analytic results, "capable of being substantially reproduced" means that independent analysis of the original or supporting data using identical methods would generate similar analytic results, subject to an acceptable degree of imprecision or error.

APPENDIX II: ADAPTING THE PRINCIPLES UNDER THE 1996 SAFE DRINKING WATER ACT AMENDMENTS FOR SAFETY AND HEALTH RISK ANALYSES

When disseminating influential information in the context of analyses of safety, health, or environmental risks, the final OMB guidelines instruct agencies to "…adopt or adapt the quality principles applied by Congress to risk information used and disseminated pursuant to the Safe Drinking Water Act Amendments of 1996 (42 U.S.C. 300g-1(b)(3)(A) & (B))." According to the preamble to OMB's final guidelines (67 F.R. 375), these principles reflect a "…basic standard of quality for the use of science in agency decision making" and "…a basic quality standard for the dissemination of public information about risks of adverse health effects." Specifically, 42 U.S.C. 300g-1(b)(3)(A) states that "to the degree that an agency action is based on science, the [EPA] Administrator shall use "(i) the best available, peer-reviewed science and supporting studies conducted in accordance with sound and objective scientific practices; and (ii) data collected by accepted methods or best available methods (if the reliability of the method and the nature of the decision justifies use of the data)." Under 42 U.S.C. 300g-1(b)(3)(B), "the Administrator shall ensure that the presentation of information on public health effects is comprehensive, informative, and understandable." Finally, in documents made available to the public to support regulation, this section of the Safe Drinking Water Act Amendments requires such documents to specify the following, to the extent practicable:

(i) each population addressed by any estimate of public health effects;

(ii) the expected risk or central estimate of risk for the specific populations;

(iii) each appropriate upper-bound or lower-bound estimate of risk;

(iv) each significant uncertainty identified in the process of the assessment of public health effects and studies that would assist in resolving the uncertainty; and

(v) peer-reviewed studies known to the Administrator that support, are directly relevant to, or fail to support any estimate of public health effects and the methodology used to reconcile inconsistencies in the scientific data.

Within the Department of Labor, analyses of safety and health risks are performed primarily by the Occupational Safety Health Administration (OSHA) and the Mine Safety and Health Administration (MSHA). Such analyses have generally been done only in connection with promulgating safety and health rules; as such, risk analyses disseminated by these agencies are subject to statutory requirements governing the bases for regulatory decision making as well as the public rulemaking process.

DOL is adapting the principles of the Safe Drinking Water Act Amendments for both health and safety risk analyses. For health analyses, the principles will be adapted as follows:

1. In taking agency actions that are based on the use of science in the analysis of health risks, the agency shall use

a. the best available peer-reviewed science and supporting studies conducted in accordance with sound and objective scientific practices; and

b. data collected by accepted methods or best available methods (if the reliability of the method and the nature of the decision justifies use of the data), including;

i. exposure data such as that generated by enforcement activity, contained in published literature, and submitted to the rulemaking record; and
ii. testimony and comment from experts familiar with the underlying scientific information related to the risk analysis and other relevant information in the rulemaking record.

2. In the dissemination of public information about risks, the agency shall ensure that the presentation of information about risk effects is comprehensive, informative, and understandable, within the context of its intended purpose.

3. In a quantitative analysis of health risks made available to the public, the agency shall specify, to the extent practicable:

a. each population addressed by any estimate of public health effects;
b. the expected risk or central estimate of risk for the specific populations;
c. each appropriate upper-bound or lower-bound estimate of risk;
d. each significant uncertainty identified in the assessment of public health effects and studies that would assist in resolving the uncertainty; and
e. information, data, or studies, peer-reviewed where available, known to the agency that support, are directly relevant to, or fail to support any estimate of risk effects and a discussion that reconciles inconsistencies in the data or information, and explains the rationale used by the agency to rely on the data or information used for the risk analysis.

For safety risk analyses, the principles will be adapted as follows:

1. In taking agency actions that are based on the use of science in the analysis of safety risks, the agency shall use

a. the best available statistical data from surveys of fatalities, injuries, and illnesses, and the best available peer-reviewed science and supporting studies that describe the nature of the safety risks being addressed;

b. data collected by accepted methods or best available methods (if the reliability of the method and the nature of the decision justifies use of the data), including;

i. incident reports compiled from an agency's information collection or enforcement activities;
ii. incident or accident investigation reports provided by the public or private sectors;
iii. relevant analyses of such information or data, peer reviewed where available; and
iv. testimony of experts familiar with the causal nature of fatalities, injuries, or illnesses being addressed in the safety risk analysis and other relevant information in the rulemaking record.

2. In the dissemination of public information about safety risks, the agency shall ensure that the presentation of information is comprehensive, informative, and understandable, within the context of its intended purpose.

3. In a quantitative risk analysis of safety risks made available to the public, the agency will specify, to the extent practicable:

a. the agency's best estimate of the size of the population at risk of such effects by industry sector;
b. the agency's best estimates of the total number and or rate of fatalities, injuries, or illnesses that occur each year and that are relevant to the safety risks being addressed;
c. the possible range in the agency's best estimate of the number or rate of fatalities, injuries, or illnesses, taking into account possible uncertainties in the data underlying the estimate;
d. data gaps and other significant uncertainties identified in the assessment of risk effects and the kind of data or information that would assist in reducing uncertainty; and
e. information, data, or studies, peer-reviewed if available, known to the agency that support, are directly relevant to, or fail to support any estimate of risk effects and a discussion that reconciles inconsistencies in the data or information, and explains the rationale used by the agency to rely on the data or information used for the risk analysis.

With regard to statutory requirements, Section 6(b)(5) of the OSH Act of 1970 and Section 101(a)(6)(A) of the Federal Mine Safety and Health Act of 1977 require the Secretary to set health standards, in part, "on the basis of the best available evidence," and that development of standards be based upon "research, demonstrations, experiments, and such other information as may be appropriate." Section 6(b)(5) and Section 101(a)(6)(A) also state that "…[i]n addition to the attainment of the highest degree of health and safety protection for the employee, other considerations shall be the latest available scientific data in the field..." Furthermore, Section 6(f) of the OSH Act mandates that the Secretary's determinations be considered conclusive "if supported by substantial evidence in the [rulemaking] record considered as a whole."

Thus, the OSH Act and Mine Act reflect the basic principle underlying the requirements of the Safe Drinking Water Act Amendments - that agency actions be based on the best scientific information available at the time of the agency action. OSHA's and MSHA's risk assessments disseminated in past health rulemakings have relied on the kinds of scientific information described in the Amendments, i.e., "peer-reviewed science and supporting studies" as well as other data that the agency considers were collected by "accepted methods or the best available methods." The agencies recognize that peer review adds significant value to a scientific study. However, in developing risk assessments to support rulemakings, the agencies also consider all other information submitted to the record, including expert testimony, written comments from the scientific community on data and other information contained in the record, including risk analyses conducted by rulemaking participants and submitted to the record. In those instances where agencies are compelled to take actions to protect the public from serious risks absent the availability of peer-reviewed scientific literature, DOL believes it consistent with its adaptation of the SDWA principles, as well as applicable statutes, to take such action based on data collected by "accepted methods or best available methods" so long as the nature of the action justifies the use of the data.

Part of what can be considered the risk analysis in the context of the Safe Drinking Water Act Amendments also appears in OSHA's and MSHA's Economic Analyses for proposed and final health rules. The Economic Analysis includes an analysis of worker exposures to the health hazard of interest, estimates of the sizes of the exposed worker populations in affected industry sectors, and an analysis of the numbers of exposure-related illnesses that occur in those populations and the numbers of illnesses potentially avoided by the new standard. In past rulemakings, OSHA and MSHA have found relatively few peer-reviewed studies available from which the agencies could reliably construct exposure profiles for all or most affected industry sectors. Information and data typically relied upon by the agencies to conduct these analyses include exposure data generated by enforcement activity, exposure data submitted to the record by industry or labor organizations, industry studies conducted by the National Institute for Occupational Safety and Health (NIOSH), and data obtained by the agencies or their contractors during the conduct of site visits to industrial facilities. In addition, OSHA has usually relied on statistics published by the Bureau of Labor Statistics (BLS) or the U.S. Bureau of the Census to develop estimates of the size of the population at risk.

Economic and cost data are normally not available from peer-reviewed studies. Such data often comes from industry sources that require confidentiality, suppliers of equipment and services, and industrial manuals. Data on profits, sales, and other operations data come from census sources, Standards and Poors, SRI International and similar sources, surveys, extrapolations from site visits, and industry and government reports. Data quality is met in these circumstances by clearly specifying sources, making available non-confidential information, and providing the spreadsheets and algorithms used by the agencies so persons can reproduce the analysis.

Analyses of safety risks conducted by OSHA and MSHA to support safety standards are quite different from health risk analyses in terms of the kinds of data and information generally available to the agencies. The goal of a safety risk analysis is to describe the numbers, rates, and causal nature of injuries related to the safety risks being addressed. OSHA and MSHA have historically relied on injury and illness statistics from BLS, combined with incident or accident reports from enforcement activities, incident or accident reports submitted to the record from the private or public sectors, testimony of experts who have experience dealing with the safety risks being addressed, and information and data supplied by organizations that develop consensus safety standards (such as the American National Standards Institute or the ASTM International). Thus, DOL's adaptation of the Safe Drinking Water Act principles for safety risk analysis reflects the use of injury and illness statistics as a primary source of data, but also calls for the use of peer-reviewed scientific data and supporting studies where they are available.

In disseminating its health and safety risk analyses for proposed and final rules, it has been OSHA's and MSHA's practice to state clearly its reasons for using the kinds of information and data described above; this is necessary to demonstrate that the agencies have relied on the "best available evidence" in making its conclusions. Because of the requirements of rulemaking procedures to consider all evidence and comment placed in the record by interested parties, the Department intends to adapt the principles set forth in the 1996 Safe Drinking Water Act Amendments to reflect that agencies must consider their statute and case law and data and evidence contained in the rulemaking docket, provided the agencies clearly state the reasons for relying on particular data and evidence in the risk analysis. That is, in addition to "peer-reviewed science and supporting studies" and "data collected by accepted methods or best available methods," agencies may consider expert testimony, public comment, and other data and information contained in the rulemaking record, and may rely on such testimony and information in their risk analyses provided that the agencies clearly communicate their rationale for selecting such data and information and why it is consistent with statutory requirements to use the best available information.

The principles outlined by the Safe Drinking Water Act Amendments also contain a specification for reporting results of risk analyses, as described above. For health risk analyses, OSHA and MSHA have historically reported their "best estimate" of the risk to workers exposed to a health hazard; this has typically been an estimate that the agencies refer to as a "maximum likelihood" estimate derived from the statistical procedure of fitting a mathematical exposure-response curve to dose-response data. The agencies also typically have reported statistical upper limits of their estimates of risk. The industry and exposure profiles presented in the Economic Analysis provide estimates of the populations at risk, by affected industry sector. Finally, during the course of rulemaking, OSHA and MSHA must consider and address data, expert testimony, and public comment that deal with uncertainties in the risk assessment and with conflicting scientific evidence. As part of demonstrating that it has relied on the "best available evidence", the agency must also clearly present its reasons for accepting certain studies or data and rejecting others, and reconcile apparent discrepancies or conflicts in the available data to the extent possible. These practices are consistent with the reporting principles described by the Safe Drinking Water Act Amendments, as well as the obligations of the OSH Act and the Mine Act.

These general principles also apply to OSHA's and MSHA's reporting of results and conclusions from analyses of safety risks; that is, the agencies make every effort to reliably estimate the sizes of the populations at risk and the magnitude of the safety risk presented to workers, and to explain uncertainties and apparent discrepancies in the available data. However, as described above, the methods and underlying data relied on for safety risk analyses are often different from that for health risk analyses; thus, DOL has adapted the language of the Safe Drinking Water Act principles as applied to the dissemination of information on safety risks to reflect the kinds of results typically obtained from a safety risk analysis.

APPENDIX III: BLS GUIDELINES FOR INFORMING USERS OF INFORMATION QUALITY AND METHODOLOGY

The Office of Management and Budget (OMB) has issued government-wide information quality guidelines in accordance with Section 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001. The purpose of these guidelines is to ensure and maximize the quality, utility, objectivity, and integrity of information disseminated by Federal agencies. The guidelines direct each Federal agency to issue its own Section 515 guidelines. As part of the Department of Labor, BLS follows DOL's information quality guidelines, as well as the OMB Guidelines. Moreover, as part of the Interagency Council on Statistical Policy, BLS supports the Council's commitment to information quality. The following BLS Guidelines for Informing Users of Information Quality and Methodology supplement the DOL guidelines that apply to all Departmental agencies.

The Bureau of Labor Statistics is the principal fact-finding agency for the Federal Government in the broad field of labor economics and statistics. BLS is an independent national statistical agency within the Department of Labor that collects, processes, analyzes, and disseminates essential statistical data to the American public, business, and labor. BLS also serves as a statistical resource to the Department of Labor.

BLS data must satisfy a number of criteria, including relevance to current social and economic issues, timeliness in reflecting today's rapidly changing economic conditions, accuracy, consistently high statistical quality, and impartiality in both subject matter and presentation.

As a Federal statistical agency, BLS conducts work in an open environment. Major changes in program design, scope, or methods are discussed in advance with users and advisory committees and described in published materials. Fair information practices are used, such as maintaining the confidentiality of individual responses. Confidentiality of the information that respondents furnish is assured by protecting the microdata, combining the data reported, and issuing the findings in summary tables, analyses, and reports. BLS values cooperation with data users and consults with a broad spectrum of users of its data in order to make its products more useful. As part of its customer pledge to the public, BLS promises to help users understand the uses and limitation of the data.

BLS applies statistical information quality principles provided in guidance from the Office of Management and Budget (OMB Statistical Policy Directives, for example) as well as the National Research Council's Principles and Practices for a Federal Statistical Agency. Moreover, all BLS information products are subjected to a multi-stage review before they are disseminated to the public.

A key component in ensuring information quality is integrity, or the protection of data from corruption through unauthorized access. BLS data integrity guidelines spell out procedures to protect the confidentiality of BLS records, the process of data collection, and various security measures.

To inform users about information quality and methodology, BLS provides descriptions of the methods and procedures used to develop and produce its statistical products. These descriptions are prepared at various levels of complexity and comprehensiveness to address the wide range of user needs. Summary level technical notes are usually included with news releases. For most programs, a periodical of record contains more comprehensive technical material. In addition, Major Programs of the Bureau of Labor Statistics provides a summary description of data availability, coverage, sources of data, reference periods, major uses, and forms of publication. The BLS Handbook of Methods covers most major programs and is updated every few years.

A major purpose for providing users with information on methodology is to assist them in determining whether the data adequately meet their needs both in terms of closeness of concept and range of statistical error.

BLS makes the information it disseminates and the methods used to produce this information as transparent as possible, so they could, in principle, be reproduced by qualified individuals. In practice, however, most estimates included in the BLS information products are not directly reproducible by the public because the underlying data used to produce them contain confidential information about individual respondents. The transparency, therefore, has the related goal of providing enough information about methodology for the public to understand the information and to have confidence in its preparation.

The level of documentation on methodology may differ among statistical programs based on type of data (from households or establishments), frequency of collection (monthly, annually, one-time), expected uses of the information, budget, and how long the survey has been in existence. The type of survey, census, or data collection process also may affect the existence of generally accepted evaluation methods and data collection protocols may affect the consistency of documentation.

Most of the information on information quality and methodology are available in both print and electronic form to assist the broad range of users. Current descriptions of specific BLS surveys and programs are available at http://www.bls.gov/bls/descriptions.htm. This page provides links to the relevant sections of the BLS Handbook of Methods. For most programs, the Handbook provides a variety of information that, as appropriate, may include a background summary, a description of the concepts, the sources of data and collection methods, the sampling and estimation procedures, and the uses and limitations of the statistics. Additional related information may be available from a program's home page, and links to these home pages are included with the descriptions.

As part of its commitment to information quality, BLS encourages communication with its users. In addition to formal advisory councils from the business, labor, and academic communities, BLS fosters discussions with the public at large by making it easy to reach staff by a variety of formats, including phone, mail, and email. A customer service guide is published annually with the names, phone numbers, and email addresses of subject matter specialists who can answer technical questions about the information BLS issues. Every page on the BLS web site has a link to a subject matter contact, a technical contact, and a general feedback contact. Every print publication also contains contact information. For more information on how to contact BLS, see its contact page.

Affected persons who believe that BLS has disseminated information that does not meet its guidelines, or those of the DOL or OMB and who wish to follow a formal complaint process, may send their complaint to the point of contact that BLS is designating.

APPENDIX IV: SAMPLE INFORMATION QUALITY ASSURANCE TECHNIQUES AND METHODS

The table below provides some sample techniques and methods derived from industry best practices. This is not intended to be an exhaustive list. Agencies should select and apply techniques and methods depending on the complexity, influence, and subject matter of each information product.

¹ Public Law 106-554, App. C.

² See Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies; Notice; Republication; Office of Management and Budget, Federal Register Vol. 67, No. 36, pp. 452-460, February 22, 2002.