OMB Quality Guidelines
Background
Section 515 of the Treasury and General Government Appropriations Act
for Fiscal Year 2001 (Public Law 106-554), directed the U.S. Office
of Management and Budget (OMB) to issue government-wide guidelines that
"provide policy and procedural guidance to Federal agencies for
ensuring and maximizing the quality, objectivity, utility, and integrity
of information (including statistical information) disseminated by Federal
agencies." Information, as defined by OMB, includes any communication
or representation of knowledge, such as facts or data, in any medium
or form, including textual, numerical, graphic, cartographic, narrative
or audiovisual forms. Dissemination refers to any agency initiated or
sponsored distribution of information to the public (OMB, Guidelines
for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity
of Information Disseminated by Federal Agencies, February 22, 2002,
67 FR 8452-8460).
NCES provides the public with a wide variety of information about the
condition of American education. Information quality is important to
NCES because educators, researchers, policymakers, and the public use
NCES products for a variety of purposes. Thus it is important that information
products that NCES disseminates are accurate and reliable. Most of the
information products are available both as printed and electronic documents.
They are announced on the NCES website (nces.ed.gov), and most electronic
versions can be accessed and downloaded directly from the website.
Purpose and Scope
NCES guidelines have been identified as Standards for the last 15 years,
thus we will retain that label. The purpose of these Standards is to
describe NCES policy and procedures for reviewing and substantiating
the quality of information before it is disseminated. These Standards
are consistent with those issued by OMB and the Department of Education.
These Standards represent a performance goal for NCES and are intended
to improve the quality of the information NCES shares with the public.
In addition to the NCES Standards, the Department of Education and
OMB have more general Information Quality Guidelines that apply to NCES.
What is more, NCES will follow the request for corrections and appeal
process described in the Department Information Quality Guidelines.
www.ed.gov/policy/gen/guid/infoqualguide.html
The Standards are applicable to any information that NCES disseminates
after October 1, 2002. In addition, some previously released information
products continue to be used for decision-making or are relied upon
by the Department of Education and the public as official, authoritative,
government data; these data are, in effect, constantly being re-disseminated
and thus are subject to these Standards and to the Department and OMB
Information Quality Guidelines. Previously released information products
that do not meet these criteria are considered archived information
and thus are not subject to the Guidelines.
In addition to archived reports, these Standards do not cover all other
information held or disseminated by NCES. The Department of Education
Information Quality Guidelines include a list of excluded items, although
that list also applies to NCES, the items that are particularly relevant
to NCES are included here. For example, the guidelines generally do
not cover: internal information such as employee records; internal procedural,
operational, or policy manuals prepared for the management and operations
of the Department of Education (and NCES) that are not primarily intended
for public dissemination; information collected or developed by NCES
that is not disseminated to the public, including documents intended
only for inter-agency or intra-agency communications; opinions that
are clearly identified as such, and that do not represent facts or NCES
views; correspondence with individuals; comments received from the public
in response to Federal Register notices, electronic links to
information on other websites; and research findings published by NCES
data cooperatives or grantees, unless NCES represents or uses the information
as the official position of the Department, or in support of the official
position of the Department, or has authority to review and approve the
information before release.
For information covered by Information Quality Guidelines, the NCES
Standards provide a basic standard of quality that can be defined based
on the three elements of quality as defined by OMB: utility, objectivity,
and integrity. These elements are intended to ensure that information
disseminated by the NCES is useful, accurate, reliable, unbiased, and
secure.
Framework
Utility refers to the usefulness of the information
to its intended users. The usefulness of information disseminated by
NCES should be considered from the perspective of NCES, educators, education
researchers, policymakers, and the public. Utility is achieved by staying
informed of information needs and developing new products and services
where appropriate.
NCES wants to ensure that information it disseminates meets the needs
of the intended users. NCES relies upon internal reviews and analyses,
along with feedback from advisory committees, educators, education researchers,
policymakers, and the public to ensure that information disseminated
by NCES meets the needs of intended users. In addition, all information
products should be grammatically correct and clearly written in plain
English. The target audience should be clearly identified, and the product
should be understandable to that audience.
Consistent with OMB guidance, the goal is to maximize the usefulness
of information and minimize the cost to the government and the public.
When disseminating its information products, NCES will utilize all feasible
and available dissemination channels so that the public, education researchers,
and policymakers can locate NCES information in an equitable and timely
fashion.
The information disseminated by NCES includes administrative and statistical
data. NCES collects and disseminates administrative data from universe
collections of elementary and secondary and postsecondary institutions.
These universe collections are based on reports aggregated from records
from schools, school districts, and states. NCES also collects and disseminates
data from a number of sample survey data collections that are designed
to fill the information needs for statistical data. NCES supports both
ongoing sample survey data collections and special purpose surveys that
are designed to fill data gaps or information needs that are identified
through internal review, legislative mandates, or input from data users
outside the Department. All statistical reports and related products
are reviewed to ensure their usefulness to the intended users. Where
appropriate, contact information is available on each publication to
facilitate feedback and questions by users.
The specific NCES standards that contribute directly to the utility
and the dissemination of information include those on the
Initial Planning of Surveys(1-1),
Publication and Product Planning (1-2), and
the Release
and Dissemination of Reports and Data Products (7-3).
Objectivity refers to whether information is accurate,
reliable, unbiased, and is presented in an accurate, clear, and unbiased
manner. It involves both the content of the information and the presentation
of the information. This includes complete, accurate, and easily understood
documentation of the source of the information, with a description of
the sources of any errors that may affect the quality of the data, when
appropriate. Objectivity is achieved by using reliable information sources
and appropriate techniques to prepare information products.
NCES strives to present information to the public in an accurate, clear,
complete, and unbiased manner. Prior to dissemination to the public,
all products are reviewed for objectivity using sound statistical methods
and the principles of transparency and reproducibility, as delineated
in the OMB Information Quality Guidelines. In addition, all products
undergo editorial and technical peer review to assist NCES in meeting
this goal.
NCES is committed to the principles for objectivity in administrative
and statistical data that are outlined in the Department of Education's
Guidelines. To that end, we have specific standards that relate to each
of the Department's principles:
- In formulating a data collection plan goals of the study should
be clearly described ¾ Initial Planning of Surveys (1-1),
Design of Surveys (2-1),
Developing a Request for Proposal (RFP) for Surveys
(2-3).
- The subjects to be studied and the data to be collected should be
clearly defined, using broadly understood concepts and definitions
¾ Initial
Planning of Surveys (1-1), Codes and Abbreviations
(1-4), Defining Race and Ethnicity Data (1-5),
Design of Surveys (2-1), Developing
a Request for Proposal (RFP) for Surveys (2-3),
Maintaining Data Series (2-5).
- The data collection techniques should be well thought out, clearly
articulated, and designed to use state of the art methodologies in
the data collection ¾
Initial Planning of Surveys (1-1), Design of Surveys
(2-1), Survey Response Rate Parameters (2-2), Developing a Request
for Proposal (RFP) for Surveys (2-3), Pretesting Survey Systems (2-4),
Educational Testing (2-6), Coverage for Frames and Samples (3-1),
Achieving Acceptable Response Rates (3-2), Monitoring and Documenting
Survey Contracts (3-3).
- In designing the work, every effort should be made to minimize the
amount of time required for survey participants ¾
Achieving Acceptable
Response Rates (3-2).
- The source of data should be reliable. In the case of sample survey
data, the sample should be drawn from a complete list of items to
be tested or evaluated, the appropriate respondents must be identified,
correctly sampled, and queried with survey instruments that have been
properly developed and tested ¾
Initial Planning of Surveys (1-1), Design
of Surveys (2-1), Pretesting Survey Systems (2-4),
Coverage for Frames and Samples (3-1).
- Response rates should be monitored during data collection. When
necessary, appropriate steps should be taken to ensure the respondents
are a representative sample ¾
Computation of Response Rates (1-3),
Survey Response Rate Parameters (2-2),
Achieving Acceptable Response Rates
(3-2), Monitoring and Documenting Survey Contracts (3-3),
Nonresponse Bias Analysis (4-4).
- Care should be taken to ensure the confidentiality of personally
identifiable data, as required by law, during data collection, processing,
and analysis of the resulting data ¾
Maintaining Confidentiality (4-2).
- Upon completion of the work, the data should be processed in a manner
sufficient to ensure that the data are cleaned and edited to help
ensure that the data are accurate and reliable ¾
Initial Planning of Surveys (1-1),
Design of Surveys (2-1), Monitoring and Documenting
Survey Contracts (3-3), Editing and Imputation of Item Nonresponse (4-1),
Evaluation of Surveys (4-3).
- he data collection should be properly documented and stored, and
the documentation should include an evaluation of the quality of the
data with a description of any limitations of the data ¾
Monitoring and Documenting Survey Contracts (3-3),
Documenting a Survey System
(3-4), Machine Readable Products (7-1).
- Data should be capable of being reproduced or replicated based on
information included in the documentation including, for example:
a) The source(s) of the information;
b) The date the information was current;
c) Any known limitations on the information;
d) The reason why the information is provided;
e) Descriptions of any statistical techniques or mathematical operations
applied to source data; and
f) Identification of other sources of potentially corroborating
or conflicting information.
The relevant standards include ¾
Monitoring and Documenting Survey
Contracts (3-3), Documenting a Survey System (3-4),
Machine Readable
Products (7-1), Survey Documentation in Reports (7-2).
- If secondary analysis of data is employed, the source should be
acknowledged, the reliability of the data should be confirmed and
documented, and any shortcomings or explicit errors should be acknowledged
(e.g., the representativeness of the data, measurement error, data
preparation error, processing error, sampling errors, and nonresponse
errors) ¾
Survey Documentation in Reports (7-2).
- The analysis should be selected and implemented to ensure that the
data are correctly analyzed using modern statistical techniques suitable
for hypothesis testing. Techniques may vary from simple tabulations
and descriptive analysis to multivariate analysis of complex interrelationships.
Care should be taken to ensure that the techniques are appropriate
for the data and the questions under inquiry ¾
Statistical Analysis,
Inference, and Comparisons (5-1), Variance Estimation (5-2),
Rounding (5-3), Tabular and Graphic Presentations of Data (5-4).
- Reports should also include the reason the information is provided,
its potential uses, and cautions as to inappropriate extractions or
conclusions, and the identification of other sources of corroborating
or conflicting information ¾
Survey Documentation in Reports (7-2).
- Descriptions of the data and all analytical work should be reported
in sufficient detail to ensure that the findings could be reproduced
using the same data and methods of analysis; this includes the preservation
of the data set used to produce the work ¾
Monitoring and Documenting
Survey Contracts (3-3), Documenting a Survey System (3-4),
Evaluation of Surveys (4-3), Machine Readable Products (7-1),
Survey Documentation in Reports (7-2).
- All reports, data, and documentation should undergo editorial and
technical review to ensure accuracy and clarity prior to dissemination.
Qualified technical staff and peers outside the Department should
do the technical review ¾
Review of Reports and Data Products (6-1).
- To ensure the utility of the work, all work must be conducted and
released in a timely manner ¾
Publication and Product Planning (1-2),
Release and Dissemination of Reports and Data Products (7-3).
- There should be established procedures to correct any identified
errors. These procedures may include the publication of errata sheets,
revised publications, or Web postings ¾
Review of Reports and Data Products
(6-1), Release and Dissemination of Reports and Data Products (7-3).
Integrity refers to the security or protection of
information from unauthorized access or revision. Integrity ensures
that the information is not compromised through corruption or falsification.
NCES has in place appropriate security provisions for the protection
of confidential information that is contained in all identified systems
of records. In accordance with statutory and administrative provisions
governing the protection of information, NCES protects administrative
records and sample survey data that include personally identifiable
information, especially survey data that are collected under a pledge
of confidentiality. Applicable provisions governing the protection of
information include the following:
- Privacy Act;
- Computer Security Act of 1987;
- Freedom of Information Act;
- OMB Circulars A-123, A-127, and A-130;
- Federal Policy for the Protection of Human Subjects;
- Government Information Security Reform Act; and
- Education Sciences Reform Act of 2002
- E-Government Act of 2002
The relevant standard is Maintaining Confidentiality (4-2).
Influential Information
The OMB guidelines for implementing section 515 recognize that some
government information needs to meet higher quality standards than a
basic standard of quality. The level of effort required to ensure the
quality of information is tied to the uses of the information. Information
that is defined as "influential" requires a higher level of
effort to ensure its' quality and reproducibility. Scientific, financial,
and statistical information is considered influential if the Department
can reasonably determine that the information is likely to have a clear
and substantial impact on important public policies or private sector
decisions if disseminated.
Influential information must be accompanied by supporting documentation
that allows an external user to clearly understand the steps involved
in producing the information and, to be able to reproduce the information.
Any influential original data files must describe the design, collection,
and processing of the data in sufficient detail that an interested third
party could understand the specifics of the original data and, if necessary,
independently replicate the data collection. In the case of influential
analytic results, the mathematical and statistical processes used to
produce the report must be described in sufficient detail to allow an
independent analyst to substantially reproduce the findings using the
original data and identical methods.
When full public access to NCES data and methods is not possible due
to other compelling interests, NCES will apply especially rigorous robustness
checks to analytic results and will document the checks that were undertaken.
In those cases where protecting the confidentiality of individually
identifiable data precludes the full release of a data file, persons
seeking access to such data and methods are required to follow applicable
NCES requirements and procedures for seeking such access. In all cases,
the interest in transparency of the agency's data shall not override
other compelling interests such as privacy, intellectual property, and
other confidentiality protections (16 CFR 4.9-4.11 and OMB Guidelines,
par V.b.3.ii.B.j.).
Inasmuch as it is not always possible to predict in advance all of
the uses of the information included in NCES data collections, all information
collected and disseminated by NCES is held to the standards of quality,
reproducibility, and documentation that are required for influential
information.
CHART A. NCES ORGANIZATION