Slide 1: FMCSA Data Quality Commercial Vehicle Safety Alliance Annual Conference Hartford, CT April 24, 2006 Slide 2: What Programs are Available to States to Improve Data Quality? Betsy Benkowski Sr. Transportation Specialist Analysis Division Slide 3: FMCSA Data Quality Programs Parts of the FMCSA Data Quality Improvement Program 1) DataQs (highlighted box) 2) Ongoing Evaluation of State Safety Data Quality 3) Monthly Monitoring of State Data Quality 4) UMTRI Off-site Analysis of State Crash Data 5) On-site State Data Quality Reviews 6) Training - Analysis and Improvement of State PARs 7) Safety Data Improvement Funding Slide 4: What is the DataQs Program? It is: A single location for data challenge entry and response A web-based interface to view the status of data challenges and supporting documentation through-out the challenge process A support tool that assists FMCSA and States in generating responses It is not: A change in either the State or FMCSA data correction guidelines A system that allows or requires FMCSA to intervene in the State data correction process Slide 5: DataQs Challenge-Type Breakdown Total Challenges 12,332 55% Total Inspection, 40% Total Crash Slide 6: FMCSA Data Quality Programs 1) DataQs 2) Ongoing Evaluation of State Safety Data Quality (highlighted box) 3) Monthly Monitoring of State Data Quality 4) UMTRI Off-site Analysis of State Crash Data 5) On-site State Data Quality Reviews 6) Training - Analysis and Improvement of State PARs 7) Safety Data Improvement Funding Slide 7: Ongoing Evaluation of State Safety Data Quality Objective: To evaluate the completeness, timeliness, accuracy and consistency of the State-reported crash and roadside inspection data in the Motor Carrier Management Information System (MCMIS). Slide 8: Evaluation Methodology 5 Performance Measures, 1 Overriding Indicator --> Overall State Ratings (Good, Fair, Poor) --> Quarterly Results Slide 9: Generate Quarterly Results Slide 10: How the Evaluation Results Improved Online 24 Month Comparison of Overall State Ratings Good: March 2004 24 States March 2006 31 States Fair: March 2004 13 States March 2006 10 States Poor: March 2004 14 States March 2006 10 States Slide 11: FMCSA Data Quality Programs 1) DataQs 2) Ongoing Evaluation of State Safety Data Quality 3) Monthly Monitoring of State Data Quality (highlighted box) 4) UMTRI Off-site Analysis of State Crash Data 5) On-site State Data Quality Reviews 6) Training - Analysis and Improvement of State PARs 7) Safety Data Improvement Funding Slide 12: What is Monthly Monitoring of State Data Quality? A program that assists FMCSA and States to: Monitor State-reported CMV crash and inspection data quality on a monthly basis Measure progress towards the quarterly State data quality rating Analyze current and historical MCMIS and DataQs management reports Provide current status and historical trends of CMV data quality Slide 13: What Analysis is Conducted in the Monthly Monitoring Process? Review of MCMIS Online Reports Review 8 reports Crash & inspection timeliness, accuracy, and duplicate analysis Identify Discrepancies Review DataQs Management Reports Indication of persistent problems in data collection or correction Review data challenges entered and open over certain time limits Slide 14: How are Monthly Monitoring Results Presented? Results provide historic view of data quality State Crash Timeliness Historical Report Slide 15: FMCSA Data Quality Programs 1) DataQs 2) Ongoing Evaluation of State Safety Data Quality 3) Monthly Monitoring of State Data Quality 4) UMTRI Off-site Analysis of State Crash Data (highlighted box) 5) On-site State Data Quality Reviews 6) Training - Analysis and Improvement of State PARs 7) Safety Data Improvement Funding Slide 16: 4. What is the UMTRI Analysis of State Crash Data? Provides an analysis of the State’s data reporting to the FMCSA MCMIS file Similar evaluation method for each State State police accident report (PAR) file is evaluated for all reportable crashes Comparison of the PAR case file to the MCMIS Crash file Slide 17: UMTRI Analysis of State Crash Data Outcomes Determine the nature and extent of underreporting Determine patterns of MCMIS crash file under- or over-reporting Determine sources of under- or over-reporting List of Reports California, Florida, Illinois, Michigan, Missouri, New Mexico, New Jersey, North Carolina, Ohio Slide 18: FMCSA Data Quality Programs 1) DataQs 2) Ongoing Evaluation of State Safety Data Quality 3) Monthly Monitoring of State Data Quality 4) UMTRI Off-site Analysis of State Crash Data 5) On-site State Data Quality Reviews (highlighted box) 6) Training - Analysis and Improvement of State PARs 7) Safety Data Improvement Funding Slide 19: What is the FMCSA State Data Quality Review Program? FMCSA team goes on-site to a State to: Review a State’s crash and roadside inspection collection and reporting processes Works with the States to identify areas for improving the data quality, and establishing methods for monitoring the effectiveness of the improvements Slide 20: What is the status of the FMCSA State Data Quality Review Program? Conducted Pilot in New Hampshire – June 2005 Conducted data quality review in North Carolina, January 2006 and New Jersey, March 2006 Conducting 1 State Data Quality Review every Quarter Slide 21: FMCSA Data Quality Programs 1) DataQs 2) Ongoing Evaluation of State Safety Data Quality 3) Monthly Monitoring of State Data Quality 4) UMTRI Off-site Analysis of State Crash Data 5) On-site State Data Quality Reviews 6) Training - Analysis and Improvement of State PARs (highlighted box) 7) Safety Data Improvement Funding Slide 22: CVM Crash Data Collection Training Lessons include: Identifying the appropriate Commercial Motor Carrier Determining Reportable Crashes Identifying Vehicle Configuration and Cargo Body Type Determining Sequence of Events Recording Hazardous Materials Recording proper CDL 12 States Trained: CA, MN, OH, NE, KS, MI, NH, WI, LA, IA, RI, AR Slide 23: Analysis of State PARs, displays an example of a colored indicator chart showing "Comparison of State Crash Reports with SAFENET Selection Criteria and Key Data Elements" Slide 24: Analysis of State PARs, displays an example of a colored indicator chart showing "Crash Report Ratings by Evaluation Factor" Slide 25: FMCSA Data Quality Programs 1) DataQs 2) Ongoing Evaluation of State Safety Data Quality 3) Monthly Monitoring of State Data Quality 4) UMTRI Off-site Analysis of State Crash Data 5) On-site State Data Quality Reviews 6) Training - Analysis and Improvement of State PARs 7) Safety Data Improvement Funding (highlighted box) Slide 26: What is Safety Data Improvement Funding? SaDIP – State Safety Data Improvement Program Provides funding via grants and cooperative agreements to State agencies to help them improve the collection and reporting of eligible large truck and bus safety data Slide 27: 7. Safety Data Improvement Funding Projects Initiated by States Hire staff to manage data quality improvement programs Revise outdated crash report forms Hire staff to code and enter crash data Train law enforcement officers in collecting crash data Develop software to transfer data from the State repository to SAFETYNET Purchase equipment to support data quality activities Purchase software for field data collection and data transfer Slide 28: Safety Data Improvement Funding From 2002-2005 almost $21 million awarded in grant and cooperative funding Slide 29: A&I Data Quality Module Presents data quality analysis and reports Located in A&I Online website http://ai.fmcsa.dot.gov Provides quarterly evaluations of State-reported data in MCMIS Provides an electronic means for filing concerns about Federal and State data Access tools for developing State strategic plans (e.g., CVSPs, State safety plan, etc.) Slide 30: A&I Data Quality Module - graphic displays website home page of module Slide 31: Safety Data Improvement Funding Contact Betsy Benkowski (202) 366-5387 Betsy.Benkowski@fmcsa.dot.gov http://dataqs.fmcsa.dot.gov