|
Chapter 7
Evaluating Information Quality
Ensuring
data quality requires regular assessments of all aspects of data collection and
processing and the implementation of corrective actions as appropriate. This can be accomplished by incorporating
data quality checks within routine data collection processes and information
product releases (Section 7.1), independently reviewing data products and data
collection systems for standards compliance (Section 7.2), and targeting
evaluations to diagnose and resolve serious data problems (Section 7.3).
7.1 Continuing Activities
Standard 7.1: BTS information products and the processes
that BTS uses to create them must routinely include an evaluation component.
Key Terms: information product
Guideline 7.1.1: Quality Assurance
All BTS information products and the
processes that BTS uses to create them must routinely include:
- Process
checks throughout data collection (Guideline 3.3.1), data processing
(Section 4.5), and information dissemination (Section 6.7),
- Pre-dissemination
review of information products (Guideline 6.1.3), and
- Measurement
of performance (Guideline 4.5.1) and of information quality (Guideline
5.2.2).
Guideline 7.1.2: Periodic Quality and Performance Self-Assessment
Product and project managers should
periodically conduct a self-assessment of the quality and performance of their
products and processes.
- Assess
quality and performance on an annual basis for products and processes that
occur at least once a year. Assess less
frequently for products and processes that occur less than once a year.
- The
assessment should highlight significant events that occurred during the
past period, any events anticipated to occur during the next period, and identify
strengths, weaknesses, and improvement opportunities.
- Submit
a summary of the self-assessment findings to the Director or the Director’s
designated manager.
- Use assessment
results to update internal and public documentation (Section 6.8) and to
improve data quality.
Related Information
Bureau of Transportation Statistics.
2005. BTS Statistical Standards Manual, Chapters 3-6. Washington, DC.
Available at http://www.bts.gov/programs/statistical_policy_and_research/bts_statistical_standards_manual/index.html as of July
29, 2005.
U.S. Department of Transportation. 2002. The Department of Transportation Information Dissemination Quality Guidelines, Chapter 6 (Evaluating Information Quality). Washington, DC. Available at http://dms.dot.gov/ombfinal092502.pdf as of January 19, 2005.
Approval Date: September 23, 2005
7.2 Data Quality Reviews
Standard 7.2: Independent statistical reviews are required of
all BTS data collection systems (including those handling external-source data)
that BTS uses to produce information products.
Key Terms: external source,
information product
Guideline 7.2.1: Independent Review Team
An independent data quality review team
should include:
- The
Director or the Director’s designee responsible for statistical methods
and standards, who should establish the team,
- At
least one person knowledgeable about BTS statistical standards but not
involved in the data collection process, and
- At
least one person familiar with the data collection process.
Guideline 7.2.2: Review Areas
The independent data quality review should focus
on compliance with statistical standards (BTS 2005, OMB 2005) and with design
specifications (Section 2.4). The review
should include:
- The
most recent self-assessment report (Section 7.1),
- The
data collection design specifications (Section 2.4),
- An
historical review of problems with identified by staff in collecting the
data, the primary data users in applying the data to their needs, and data
providers in reporting the data (Chapters 2 and 3),
- If
sampling is used, a review of the sample design and the sample selection
and maintenance processes (Sections 2.2 and 3.2),
- A
review of data processing problems, such as problems in converting raw
data files to databases, problems with lack of editing or edit resolution,
and problems with missing data (Chapter 4),
- Review
of the procedures for dissemination of data through various media, and of
the source and accuracy information provided to users (Chapter 6),
- Verification
that users can independently reproduce estimates, including sampling error
estimates where applicable, contained in the information products coming
out of the system (Sections 5.3 and 6.8), and
- Verification
that documentation is accurate, complete (Sections 2.4, 3.4, 4.6, 5.3, and
6.8), and current.
Guideline
7.2.3: Review Outputs
Outputs from the data quality review should
include:
- A
report to the Director on review findings,
- A
reply from the office responsible for the data collection system, and
- A
quality improvement plan, prepared by the office responsible for the data
collection system.
Guideline 7.2.4: Follow-up Review
A follow-up review should verify that the
improvements have been implemented.
Related Information
Bureau of Transportation Statistics. 2005. BTS Statistical Standards Manual. Washington, DC.
Available at http://www.bts.gov/programs/statistical_policy_and_research/bts_statistical_standards_manual/index.html as of July 29, 2005.
Office of Management and Budget (OMB). 2005. Standards for Statistical Surveys (Proposed). Washington, DC. July 14.
U.S. Department of Transportation. 2002. The Department of Transportation Information Dissemination Quality Guidelines, Chapter 6 (Evaluating Information Quality). Washington, DC. Available at http://dms.dot.gov/ombfinal092502.pdf as of January 19, 2005.
Approval Date: September 23, 2005
7.3 Data Evaluation Projects
Standard 7.3: BTS should undertake a data evaluation project if analysis of
the data reveals that key data elements fail to meet data quality requirements.
Key Terms: primary data user,
secondary data user
Guideline 7.3.1: Evaluation Project Teams
An evaluation project team should report to
the Director or to a designee with authority to allocate resources in support
of the team’s mission. The team members should
include:
- A
team leader who is not involved in the data collection process,
- Personnel
selected for their expertise but not involved in the data collection
process, possibly including non-BTS staff, and
- Personnel
who are directly involved in the data collection process.
Guideline 7.3.2: Evaluation Plan
The project team should plan the evaluation
as a type of data analysis (Guideline 5.1.2) that targets specific problems or
issues in BTS data products. Solicit
input from the following sources, with the greatest weight given to the primary
users:
- Primary
users for whom BTS designs information products, and normally include analysts
in DOT, Congress, and BTS,
- Secondary
users, including commercial interests and the general public,
- BTS data
collection experts who can identify additional process quality issues, and
- Sponsored
independent expert reviews.
Guideline 7.3.3: Conduct of an Evaluation Study
The major tasks in an evaluation study are:
- Specifying
the processes contributing to the observed data quality problem,
- Identifying
the root problems,
- Ascertaining
solutions to the problems, and
- Drafting
an improvement plan to address the problems identified. In some cases, the recommendations could
lead to a redesign of the data collection system (Chapter 2).
Guideline 7.3.4: Implementation of the Study Recommendations
The project team should report the evaluation
results and recommendations to the Director, or to the Director’s designee
(7.3.1). Upon concurrence, the office
conducting the data collection is responsible for the implementation of the
recommendations. A follow-up data
evaluation should verify that the recommendations have been implemented and
that the data meet quality requirements.
Related Information
U.S. Department of Transportation. 2002. The Department of Transportation Information Dissemination Quality Guidelines, Chapter 6 (Evaluating Information Quality). Washington, DC. Available at http://dms.dot.gov/ombfinal092502.pdf as of January 19, 2005.
Approval Date: September 23, 2005
|
|