HHS Guidelines for Ensuring and Maximizing
the Quality, Objectivity, Utility, and Integrity
of Information Disseminated to the Public

Part I -- HHS Overview

Summary of HHS Agency Guidelines
OMB Guidelines
HHS Responsibilities

A. Summary

The U.S. Department of Health and Human Services (HHS) developed these guidelines to implement Office of Management and Budget (OMB) January 2002 requirement that all federal agencies issue guidelines for ensuring the quality of the information that they disseminate to the public. The Department of Health and Human Services is the United States government's principal agency for protecting the health of all Americans and providing essential human services, especially to those who are least able to help themselves. The Department includes more than 300 programs, covering a wide spectrum of activities.

In the course of carrying out program missions, agencies and staff offices within HHS disseminate a wide variety of information to the public, ranging from research and statistical reports to authoritative health and medical information. Many of these information dissemination activities and products rank among the highest quality scientific, statistical and programmatic information among federal agencies, and in many cases set the national and international standard for quality.

HHS is committed to disseminating information that meets the standards of quality set forth in OMB and in the guidelines discussed in this document. It is HHS's goal to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public. We strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. We are committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination. The guidelines that follow describe the quality assurance policies and practices that support information dissemination activities in HHS.

Following an overview of the OMB Guidelines and HHS guidelines, the quality assurance policies of each of the major operating agencies and staff offices of HHS are described. Each set of agency guidelines includes a description of a) the mission of the agency, b) the scope and applicability of the guidelines within the agency, c) the types of information that the agency disseminates, d) the dissemination methods employed by the agency, e) the policies, standards and practices that the agency employs to ensure the quality of the information it disseminates, and f) an administrative mechanism and contact points for each agency so that individuals may seek correction of any information that is believed not to meet the OMB, HHS, or agency-specific guidelines along with an administrative appeals process.

B. OMB Guidelines

On September 28, 2001, and as amended on February 22, 2002, OMB issued final Guidelines to implement section 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554). The statute directs OMB to "issue government wide guidelines that provide policy and procedural guidance to federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by federal agencies." By October 1, 2002, agencies must issue their own implementing guidelines. The guidelines only apply to information that is disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information.

In general, the OMB Guidelines require agencies to adopt a basic standard of quality as a performance goal and take appropriate steps to incorporate information quality criteria into agency information dissemination practices. Quality is to be ensured and established at levels appropriate to the nature and timeliness of the information to be disseminated, and specific standards may be adopted that are appropriate to the various categories of information that is disseminated. Agencies are to develop a process for reviewing the quality of information before it is disseminated. Further, information quality is to be treated as an integral step in every aspect of the information development process.

In issuing the Guidelines, OMB outlined several guiding principles. First, OMB designed the Guidelines to apply to a wide variety of government dissemination activities that may range in importance and scope. OMB also designed the Guidelines to be generic enough to fit all media, whether printed or electronic. OMB specifically sought to avoid the problems inherent in developing detailed, prescriptive, "one size fits all" guidelines that would artificially require all types of dissemination activities to be treated in the same manner. Second, OMB designed the Guidelines so that agencies will meet basic information quality standards. The Guidelines recognize that some government information may need to meet higher or more specific standards than others, depending on their purpose and scope. The more important the information, the higher the quality standards to which it might be held, for example, "influential scientific, financial or statistical information" described below. At the same time, OMB recognizes that information quality comes at a cost. Accordingly, agencies are encouraged to weigh the costs and benefits of higher information quality in the development of information, and the level of quality to which it will be held.

Third, OMB designed the Guidelines so that agencies can apply them in a common sense, workable manner. OMB expects agencies to use existing processes rather than create new and potentially duplicative or contradictory processes. Finally, OMB recognizes that the Guidelines cannot be implemented in the same way by all agencies. While the implementation may differ, the essence of the Guidelines will apply. The agencies must make their methods transparent by providing documentation, ensure quality by reviewing the underlying methods used, by consulting as needed with both experts and users, and by keeping users notified about corrections and revisions. These underlying principles apply equally well across the diversity of HHS agencies and information dissemination activities, and they have been adopted in the approach to the HHS Guidelines described below.

C. HHS Responsibilities

In accordance with the OMB Guidelines, agencies subject to the Paperwork Reduction Act (44USC 3502(1)) are required to:

Issue their own information quality guidelines ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by the agency not later than one year after the issuance of the OMB Guidelines;

Establish administrative mechanisms allowing affected persons to seek and obtain correction of information maintained and disseminated by the agency that does not comply with the guidelines. Agencies also are to specify appropriate time periods for agency decisions on whether and how to correct the information, and are to notify the affected persons of the action taken. If the person who requested the correction does not agree with the agency's decision (including the corrective action, if any), that person may file for reconsideration within the agency. The agency is to establish an administrative appeal process to review the initial decision, and specify appropriate time limits in which to resolve such requests for reconsideration.

Report periodically to the OMB Director on the number and nature of complaints received by the agency regarding the accuracy of information disseminated by the agency and how such complaints were resolved by the agency.

The HHS guidelines described in this plan incorporate the underlying principles that OMB used in designing their government-wide guidelines. First, the HHS guidelines apply to a wide range of government information dissemination activities across HHS and are generic enough to fit all types of media, including print, electronic, and other forms within HHS. Second, the HHS guidelines are intended to assure that all the information that is disseminated meets a basic level of quality and that more important information meets a more rigorous quality standard. Third, the HHS guidelines explicitly recognize the very different types of information that various HHS agencies disseminate depending on their missions, including the need for flexibility in implementation and avoidance of a "one size fits all" approach. Fourth, the statement of HHS information quality policies and procedures are issued in the form of guidelines and not a regulation.

HHS itself encompasses a broad and diverse range of health and human services programs which, while unified in their pursuit of broad goals, are themselves very diverse, encompassing the nation's largest health insurance plan, the nation's preeminent biomedical research agency, as well as most of the nation's federal capacity for public health protection and preparedness and income assistance to needy families. Accordingly, the HHS approach to implementation of the OMB Guidelines is designed to allow HHS agencies and offices to use existing agency quality assurance mechanisms, and apply the guidelines in a flexible manner that recognizes the mission of the agency, the wide range of data that is disseminated and the frequent reliance on third party sources.

D. Framework for HHS Guidelines

Purpose
These Guidelines describe the policies and procedures that HHS agencies employ to ensure the quality of the information they disseminate and the administrative complaint mechanisms that HHS agencies make available to the public. The Guidelines provide policy and procedural guidance to HHS staff and are intended to inform the public about agency quality assurance policies and procedures.

HHS views the guidelines as an evolving document and process. HHS will continually review the performance of the guidelines in the context of agency statutes and missions and will make revisions to the guidelines as necessary.
Definitions
1. "Quality" is an encompassing term comprising utility, objectivity, and Integrity. Therefore, the Guidelines sometimes refer to these four statutory terms, collectively, as "quality."
2. "Utility" refers to the usefulness of the information to its intended users, including the public. In assessing the usefulness of information that the agency disseminates to the public, the agency needs to consider the uses of the information not only from the perspective of the agency but also from the perspective of the public. As a result, when transparency of information is relevant for assessing the information's usefulness from the public's perspective, the agency must take care to ensure that transparency has been addressed in its review of the information.
3. "Objectivity" involves two distinct elements, presentation and substance.
  "Objectivity" includes whether disseminated information is being presented in an accurate, clear, complete, and unbiased manner. This involves whether the information is presented within a proper context. Sometimes, in disseminating certain types of information to the public, other information must also be disseminated in order to ensure an accurate, clear, complete, and unbiased presentation. Also, the agency needs to identify the sources of the disseminated information (to the extent possible, consistent with confidentiality protections) and, in a scientific, financial, or statistical context, the supporting data and models, so that the public can assess for itself whether there may be some reason to question the objectivity of the sources. Where appropriate, data should have full, accurate, transparent documentation, and error sources affecting data quality should be identified and disclosed to users.
  
  In addition, "objectivity" involves a focus on ensuring accurate, reliable, and unbiased information. In a scientific, financial or statistical context, the original and supporting data shall be generated, and the analytic results shall be developed, using sound statistical and research methods.
  1. If data and analytic results have been subjected to formal, independent, external peer review, the information may generally be presumed to be of acceptable objectivity. However, this presumption is rebuttable based on a persuasive showing by the petitioner in a particular instance. If agency-sponsored peer review is employed to help satisfy the objectivity standard, the review process employed shall meet the general criteria for competent and credible peer review recommended by OMB-OIRA to the President's Management Council (9/20/01) (www.whitehouse.gov/omb/inforeg/oira_review-process.html), namely, that (a) peer reviewers be selected primarily on the basis of necessary technical expertise, (b) peer reviewers be expected to disclose to agencies prior technical/policy positions they may have taken on the issues at hand, (c) peer reviewers be expected to disclose agencies their sources of personal and institutional funding (private or public sector), and (d) peer reviews be conducted in an open and rigorous manner."
  2. If an agency is responsible for disseminating influential scientific, financial, or statistical information, agency guidelines shall include a high degree of transparency about the data and methods to facilitate the reproducibility of such information by qualified third parties.
    Original and supporting data must be subject to commonly accepted scientific, financial, and statistical standards related thereto. However, agency guidelines shall not require that all disseminated data be subjected to a reproducibility requirement. Agencies may identify, in consultation with the relevant scientific and technical communities, those particular types of data that can practicably be subjected to a reproducibility requirement, given ethical, feasibility, or confidentiality constraints. It is understood that reproducibility of data is an indication of transparency about research design and methods and thus a replication exercise (i.e., a new experiment, test, or sample) shall not be required prior to each dissemination.
    
    With regard to analytic results, agency guidelines shall generally require sufficient transparency about data and methods that an independent reanalysis could be undertaken by a qualified member of the public. These transparency standards apply to agency analysis of data from a single study as well as to analyses that combine information from multiple studies.
    
    Making the data and methods publicly available will assist in determining whether analytic results are reproducible. However, the objectivity standard does not override other compelling interests such as privacy, trade secrets, intellectual property, and other confidentiality protections.
    
    In situations where public access to data and methods will not occur due to other compelling interests, agencies shall apply especially rigorous robustness checks to analytic results and document what checks were undertaken. However, agency guidelines, in all cases, shall require a disclosure of the specific data sources that have been used and the specific quantitative methods and assumptions that have been employed. Each agency is authorized to define the type of robustness checks, and level of detail for documentation thereof, in ways appropriate for it given the nature and multiplicity of issues for which the agency is responsible.
    
    With regard to analysis of risks to human health, safety and the environment maintained or disseminated by the agencies, agencies shall either adopt or adapt the quality principles applied by Congress to risk information used and disseminated pursuant to the Safe Drinking Water Act Amendments of 1996 (42 U.S.C. 300g-1(b)(3)(A) & (B)). Agencies responsible for dissemination of vital health and medical information shall interpret the reproducibility and peer-review standards in a manner appropriate to assuring the timely flow of vital information from agencies to medical providers, patients, health agencies, and the public. Information quality standards may be waived temporarily by agencies under urgent situations (e.g., imminent threats to public health or homeland security) in accordance with the latitude specified in agency-specific guidelines.
4. "Integrity" refers to the security of information -- protection of the information from unauthorized access or revision, to ensure that the information is not compromised through corruption or falsification.
5. "Information" means any communication or representation of knowledge such as facts or data, in any medium or form, including textual, numerical, graphic, cartographic, narrative, or audiovisual forms. This definition includes information that an agency disseminates from a web page, but does not include the provision of hyperlinks to information that others disseminate. This definition does not include opinions, where the agency's presentation makes it clear that what is being offered is someone's opinion rather than fact or the agency's views.
6. "Government information" means information created, collected, processed, disseminated, or disposed of by or for the Federal Government.
7. "Information dissemination product" means any book, paper, map, machine-readable material, audiovisual production, or other documentary material, regardless of physical form or characteristic, an agency disseminates to the public. This definition includes any electronic document, CD-ROM, or web page.
8. "Dissemination" means agency initiated or sponsored distribution of information to the public (see 5 C.F.R. 1320.3(d) (definition of "Conduct or Sponsor"). Dissemination does not include distribution intended to be limited to government employees or agency contractors or grantees; intra- or inter-agency use or sharing of government information; and responses to request for agency records under the Freedom of Information Act, the Privacy Act, the Federal Advisory Committee Act or other similar law. This definition also does not include distribution limited to correspondence with individuals or persons, archival records, public filings, subpoenas or adjudicative processes.
9. "Influential," when used in the phrase "influential scientific, financial, or statistical information," means that the agency can reasonably determine that dissemination of the information will have or does have a clear and substantial impact on important public policies or important private sector decisions. Each agency is authorized to define "influential" in ways appropriate for it given the nature and multiplicity of issues for which the agency is responsible.
10. "Reproducibility" means that the information is capable of being substantially reproduced, subject to an acceptable degree of imprecision. For information judged to have more (less) important impacts, the degree of imprecision that is tolerated is reduced (increased). If agencies apply the reproducibility test to specific types of original or supporting data, the associated guidelines shall provide relevant definitions of reproducibility (e.g., standards for replication of laboratory data). With respect to analytic results, "capable of being substantially reproduced" means that independent analysis of the original or supporting data using identical methods would generate similar analytic results, subject to an acceptable degree of imprecision or error.
Scope and Applicability of the Guidelines
The HHS guidelines described in this implementation plan apply to substantive information dissemination activities that are initiated or sponsored by HHS agencies. The pre-dissemination review described in the guidelines only applies to information that is disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information. The guidelines do not apply to the large proportion of extramural scientific research activity supported by HHS whose dissemination is the sole responsibility of the academic researcher rather than HHS. The guidelines do apply to the dissemination of information by federal intramural researchers if the dissemination is agency-initiated or sponsored. Otherwise, appropriate disclaimers are to be included in the report or speech to distinguish the research from agency views and positions. The guidelines do not apply to distribution of information limited to correspondence with individuals or persons, press releases that support the announcement or give public notice of information that the agency disseminates elsewhere, archival material, public filings, subpoenas or adjudicative processes. Nor do they apply to opinions, when the agency's presentation makes clear that what is disseminated is someone's opinion rather than fact or agency views.
Overview of HHS Information Dissemination and Quality Assurance
1. The HHS Mission
  The Department of Health and Human Services is the United States government's principal agency for protecting the health of all Americans and providing essential human services, especially to those who are least able to help themselves. The Department includes more than 300 programs, covering a wide spectrum of activities. Program responsibilities include:
  - Ensuring a safe and healthy America
  - Conducting and supporting medical and social science research
  - Preventing outbreaks of infectious diseases
  - Assuring the safety of foods and drugs
  - Administering the Medicare and Medicaid programs
  - Providing financial assistance for low income families
  - Improving maternal and child health
  - Administering the Head Start program
  - Preventing child abuse and domestic violence
  - Providing assistance to States for substance abuse treatment and prevention
  - Providing services for older Americans
  - Assuring comprehensive health services for Native Americans
2. Categories of Information Disseminated
  The development and dissemination of timely and high quality data and information is a critical component of the missions of many HHS programs, as well as those of HHS partners in the health and human services communities. As a result, HHS plays a major role in information dissemination, as a producer and user of high quality data and information, as a collaborator with partners in the health and human services communities, and as a national leader in health and human services information policy.
  
  In carrying out their diverse statutes and missions, HHS agencies develop and support the dissemination of the following types of substantive information:
  - Scientific research studies (the results of biomedical, behavioral, services research and social science research)
  - Statistical and analytic studies (the results of surveys, statistical systems as well as analytical and modeling studies and public use data files
  - Programmatic, administrative and regulatory information, including program evaluations
  - Authoritative health, medical and human services information aimed at consumers and health and human services professionals.
  - Public health surveillance, epidemiology and risk assessment studies and information
3. Types of Dissemination Mechanisms
  HHS agencies disseminate information through a wide range of methods and print and electronic media. These include publication in peer reviewed literature, published reports, periodicals, newsletters, brochures, clearinghouses, websites, CD-ROM and other electronic media.
4. Overview of Quality Assurance Policies and Practices in HHS
  Depending upon their specific statutes and missions and the nature of the information they disseminate to the public, HHS agencies currently use a variety of quality assurance methods and procedures. These methods and procedures are designed to maximize the quality of HHS information, including the objectivity, utility, and integrity.
  In addition, HHS is subject to a number of statutory requirements that protect the sensitive information it gathers and maintains on individuals. Among these are:
  - Health Insurance Portability and Accountability Act of 1996
  - Privacy Act of 1974
  - Computer Security Act of 1987
  - Office of Management and Budget (OMB) Circulars A-123, A-127, and A-130
  - Government Information Security Reform Act
  - Federal Managers' Financial Integrity Act (FMFIA) of 1982
5. HHS Information Quality Goals
  The development of data and information within HHS is generally undertaken within the context of two overarching goals:
  - attention to information quality as a total and continuing process, and
  - commitment to making data and information supported with public funds available to the public, consistent with confidentiality concerns and resource availability.
  Further, when HHS agencies prepare a Paperwork Reduction Act (PRA) clearance submission, they strive to engage in a data development effort that will result in information that will be collected maintained, and used in a way that is consistent with OMB, HHS, and agency-specific information quality guidelines. As a general policy, HHS views data and information quality as a continuing process that begins at the inception of the information development process with project conceptualization and carries through all phases of data planning, design and execution, including information dissemination activities. Further, HHS agencies that support or sponsor research and statistical activities are encouraged to not only describe the methods and data sources in a clear and transparent manner in the reports they release, but also to make the data used for the report available to the public through public use data files, restricted access research files, research data centers, data archives and other mechanisms consistent with confidentiality, legal and proprietary restrictions.
  
  In addition, HHS agencies review the quality (including the objectivity, utility, and integrity) of information before it is disseminated and treat information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination. The quality assurance processes that are used to ensure the quality of specific categories of information are described below.
6. Influential Scientific, Financial and Statistical Information
  
  From time to time, HHS agencies disseminate influential scientific, financial and statistical information as defined in the OMB Guidelines. In such instances, care is taken to ensure that the information is substantially reproducible and replicable. This goal is accomplished by using reliable data sources and sound analytical techniques, and by employing a high degree of transparency about the data, sources, methods, measures, assumptions and limitations used to develop the information in order to facilitate reproducibility by qualified third parties. In the case of original or supporting data, most major epidemiological and statistical activities sponsored by HHS agencies have well developed public use data dissemination programs that make much of the data available to the public in standardized, de-identified micro-data files.
  
  Because of confidentiality, ethical and feasibility constraints and legal obligations to third parties supplying the information, there may be instances where original or supporting data may not be available to the public, but HHS agencies typically will work with qualified third parties to facilitate understanding, and transparency in data sources and methods will be emphasized in the report or in reference documents. In the case of analytical studies, HHS agencies will make provisions for sufficient transparency about data and methods so that an independent reanalysis could be undertaken by a qualified member of the public.
7. Health, Safety and Environmental Information
  Several HHS agencies have science-based missions and use such information in decision making. These agencies adapt the quality standards discussed in the Safe Drinking Water Act Amendments of 1996 (SDWA) (42 U.S.C. 300g-1(b)(3)(A) and (B)). The adaptation involves the commitment of these agencies to use: a.) the best available science and supporting studies conducted in accordance with sound and objective scientific practices, including peer reviewed studies when available and b.) data collected by accepted methods (if reliability of the method and the nature of the decision justify use of the data.) Additional detail relating to the standards employed by individual agencies is described in Part II, the individual agency section of these guidelines.
8. Urgent Public Health and Safety Information
  Several HHS agencies are responsible for dissemination of authoritative health, medical and safety information on a real time basis in order to protect the health of the public against urgent and emerging threats. Accordingly, nothing in these guidelines relating to reproducibility or peer review shall be construed to limit or delay the timely flow of vital information from agencies to medical providers, patients, health agencies, and the public. HHS reserves the right to waive information quality standards temporarily for agencies addressing urgent situations (e.g., imminent threats to public health or homeland security) in accordance with the latitude described in both the OMB and agency specific guidelines.

E. Overview of HHS Agency Complaint Procedures

The OMB Guidelines require all agencies to establish administrative mechanisms allowing affected persons to seek and obtain correction of information disseminated by the agency that does not comply with OMB, HHS or agency-specific guidelines. Agencies also are to specify appropriate time periods for agency decisions on whether and how to correct the information, and are to notify the affected persons of the action taken. If the person who requested the correction does not agree with the agency's decision (including the corrective action, if any), that person may file for reconsideration within the agency. The agency is to establish an administrative appeal process to review the initial decision, and specify appropriate time limits in which to resolve such requests for reconsideration. The agency need not respond substantively to information not covered by the guidelines or to frivolous or repetitive requests for correction.

Requests for correction of information quality pursuant to Section 515 are to be directed to the respective agency. A common format for complaints has been developed across HHS. The approach is described below. To accompany the actual implementation of the Guidelines in October 2002, HHS has created a department-wide website describing in user friendly terms the procedures and contact persons for submitting requests for corrections. The web site can be accessed at www.hhs.gov/infoquality.

Responsibility of the Complainant

In general, to seek an information quality request for correction of information disseminated by any HHS agency, individuals should follow the procedures described below. Requests for correction that are specific and provide evidence to support the need for correction will enable the agency to provide a satisfactory response. Complainants should be aware that they bear the "burden of proof" with respect to the necessity for correction as well as with respect to the type of correction they seek.

A complaint or request for correction of information must be in written hard copy or electronic form, be sent to the agency designated address by mail or electronic-mail (e-mail); and state that an information quality request for correction is being submitted. In terms of content, the complaint letter must contain:

a detailed description of the specific material that needs to be corrected including where the material is located, i.e. the publication title, date, and publication number, if any, or the website and web page address (url), or the speech title, presenter, date and place of delivery;
the specific reasons for believing the information does not comply with OMB, HHS, or agency-specific guidelines and is in error and supporting documentation, if any;
the specific recommendations for correcting the information;
a description of how the person submitting the complaint is affected by the information error; and
the name, mailing address, telephone number, e-mail address, and organizational affiliation, if any, of the individual making the complaint.

Responsibility of the Agency

Based on a review of the information provided, the agency will determine whether a correction is warranted and if, so what action to take. The agency will respond to the requestor by letter or e-mail. The agency's response will explain the findings of the review and the actions that the agency will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information, the magnitude of the correction and the resource requirements for the correction. The response will describe how the complainant may request reconsideration. The agency will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

Appeals for Reconsideration

If the individual submitting the complaint does not agree with the agency's decision (including the corrective action), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the agency's decision. The appeal shall state the reasons why the agency response is insufficient or inadequate. Complainants shall attach a copy of their original request and the agency response to it, clearly mark the appeal with the words, "Information Quality Appeal," and send the appeal to the specific agency appeals address.

The agency official who handles the original complaint will not have responsibility for resolving the appeal. The agency will respond to all requests for appeals within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

Rulemakings and Other Public Comment Procedures

Existing public comment procedures for rule-makings and other formal agency actions already provide well established procedural safeguards that allow affected persons to raise information quality issues on a timely basis. Accordingly, agencies will use these existing procedures to respond to information quality complaints that arise in this process.

In cases where the agency disseminates a study, analysis, or other information prior to the final agency action or information product, requests for correction will be considered prior to the final agency action or information product in those cases where in the agency's judgment issuing an earlier response would not unduly delay issuance of the agency action or information product and the complainant has shown a reasonable likelihood of suffering actual harm from the agency's dissemination if the agency does not resolve the complaint prior to the final agency action or information product.

Last revised: November 13, 2003

The White House | FirstGov