Chapter 7
Evaluating Information Quality

Ensuring data quality requires regular assessments of all aspects of data collection and processing and the implementation of corrective actions as appropriate.  This can be accomplished by incorporating data quality checks within routine data collection processes and information product releases (Section 7.1), independently reviewing data products and data collection systems for standards compliance (Section 7.2), and targeting evaluations to diagnose and resolve serious data problems (Section 7.3).

7.1 Continuing Activities

Standard 7.1:  BTS information products and the processes that BTS uses to create them must routinely include an evaluation component.

Key Terms:  information product

Guideline 7.1.1:  Quality Assurance

All BTS information products and the processes that BTS uses to create them must routinely include:

• Process checks throughout data collection (Guideline 3.3.1), data processing (Section 4.5), and information dissemination (Section 6.7),
• Pre-dissemination review of information products (Guideline 6.1.3), and
• Measurement of performance (Guideline 4.5.1) and of information quality (Guideline 5.2.2).

Guideline 7.1.2:  Periodic Quality and Performance Self-Assessment

Product and project managers should periodically conduct a self-assessment of the quality and performance of their products and processes.

• Assess quality and performance on an annual basis for products and processes that occur at least once a year.  Assess less frequently for products and processes that occur less than once a year.
• The assessment should highlight significant events that occurred during the past period, any events anticipated to occur during the next period, and identify strengths, weaknesses, and improvement opportunities.
• Submit a summary of the self-assessment findings to the Director or the Director’s designated manager.
• Use assessment results to update internal and public documentation (Section 6.8) and to improve data quality.

Related Information

Bureau of Transportation Statistics. 2005.  BTS Statistical Standards Manual, Chapters 3-6.  Washington, DC.  Available at http://www.bts.gov/programs/statistical_policy_and_research/bts_statistical_standards_manual/index.html as of July 29, 2005.

U.S. Department of Transportation.  2002.  The Department of Transportation Information Dissemination Quality Guidelines, Chapter 6 (Evaluating Information Quality).  Washington, DC.  Available at http://dms.dot.gov/ombfinal092502.pdf as of January 19, 2005.

Approval Date:   September 23, 2005

7.2 Data Quality Reviews

Standard 7.2:  Independent statistical reviews are required of all BTS data collection systems (including those handling external-source data) that BTS uses to produce information products.

Key Terms:  external source, information product

Guideline 7.2.1:  Independent Review Team

An independent data quality review team should include:

• The Director or the Director’s designee responsible for statistical methods and standards, who should establish the team,
• At least one person knowledgeable about BTS statistical standards but not involved in the data collection process, and
• At least one person familiar with the data collection process.

Guideline 7.2.2:  Review Areas

The independent data quality review should focus on compliance with statistical standards (BTS 2005, OMB 2005) and with design specifications (Section 2.4).  The review should include:

• The most recent self-assessment report (Section 7.1),
• The data collection design specifications (Section 2.4),
• An historical review of problems with identified by staff in collecting the data, the primary data users in applying the data to their needs, and data providers in reporting the data (Chapters 2 and 3), 
• If sampling is used, a review of the sample design and the sample selection and maintenance processes (Sections 2.2 and 3.2),
• A review of data processing problems, such as problems in converting raw data files to databases, problems with lack of editing or edit resolution, and problems with missing data (Chapter 4),
• Review of the procedures for dissemination of data through various media, and of the source and accuracy information provided to users (Chapter 6),
• Verification that users can independently reproduce estimates, including sampling error estimates where applicable, contained in the information products coming out of the system (Sections 5.3 and 6.8), and
• Verification that documentation is accurate, complete (Sections 2.4, 3.4, 4.6, 5.3, and 6.8), and current.

Guideline 7.2.3:  Review Outputs

Outputs from the data quality review should include:

• A report to the Director on review findings,
• A reply from the office responsible for the data collection system, and
• A quality improvement plan, prepared by the office responsible for the data collection system.

Guideline 7.2.4:  Follow-up Review

A follow-up review should verify that the improvements have been implemented.

Related Information

Bureau of Transportation Statistics.  2005.  BTS Statistical Standards Manual.  Washington, DC. Available at http://www.bts.gov/programs/statistical_policy_and_research/bts_statistical_standards_manual/index.html as of July 29, 2005.

Office of Management and Budget (OMB).  2005.  Standards for Statistical Surveys (Proposed).  Washington, DC.  July 14.

U.S. Department of Transportation.  2002.  The Department of Transportation Information Dissemination Quality Guidelines, Chapter 6 (Evaluating Information Quality).  Washington, DC.  Available at http://dms.dot.gov/ombfinal092502.pdf as of January 19, 2005.

Approval Date:   September 23, 2005

7.3 Data Evaluation Projects

Standard 7.3:  BTS should undertake a data evaluation project if analysis of the data reveals that key data elements fail to meet data quality requirements.

Key Terms:  primary data user, secondary data user

Guideline 7.3.1:  Evaluation Project Teams

An evaluation project team should report to the Director or to a designee with authority to allocate resources in support of the team’s mission.  The team members should include:

• A team leader who is not involved in the data collection process,
• Personnel selected for their expertise but not involved in the data collection process, possibly including non-BTS staff, and
• Personnel who are directly involved in the data collection process. 

Guideline 7.3.2:  Evaluation Plan

The project team should plan the evaluation as a type of data analysis (Guideline 5.1.2) that targets specific problems or issues in BTS data products.  Solicit input from the following sources, with the greatest weight given to the primary users:

• Primary users for whom BTS designs information products, and normally include analysts in DOT, Congress, and BTS, 
• Secondary users, including commercial interests and the general public,

• BTS data collection experts who can identify additional process quality issues, and   
• Sponsored independent expert reviews.

Guideline 7.3.3:  Conduct of an Evaluation Study

The major tasks in an evaluation study are:

• Specifying the processes contributing to the observed data quality problem,
• Identifying the root problems,
• Ascertaining solutions to the problems, and
• Drafting an improvement plan to address the problems identified.  In some cases, the recommendations could lead to a redesign of the data collection system (Chapter 2).

Guideline 7.3.4:  Implementation of the Study Recommendations

The project team should report the evaluation results and recommendations to the Director, or to the Director’s designee (7.3.1).  Upon concurrence, the office conducting the data collection is responsible for the implementation of the recommendations.  A follow-up data evaluation should verify that the recommendations have been implemented and that the data meet quality requirements.

Related Information

U.S. Department of Transportation.  2002.  The Department of Transportation Information Dissemination Quality Guidelines, Chapter 6 (Evaluating Information Quality).  Washington, DC.  Available at http://dms.dot.gov/ombfinal092502.pdf as of January 19, 2005.

Approval Date:   September 23, 2005