Slide 1 Federal Motor Carrier Safety Administration What Programs are Available to the States to Improve Data Quality? August 2005 Slide 2 FMCSA Programs Monthly Monitoring of State Data Quality + Ongoing Evaluation of State-reported Motor Carrier Crash and Inspection Data + UMTRI Off-site Analysis of State Crash Data + On-site State Data Quality Reviews + DataQs + Training (FMCSA, State & Local Officials) + Safety Data Improvement Funding + Analysis and Improvement of State PARs = FMCSA Data Quality Improvement Programs Slide 3 What is DataQs? A web-based system for filing data quality concerns about Federal and State data Slide 4 What is DataQs? It is: A single location for data challenge entry and response A web-based interface to view the status of data challenges and supporting documentation throughout the challenge process A support tool that assists FMCSA and States in generating responses It is not: A change in either the State or FMCSA data correction guidelines A system that allows or requires FMCSA to intervene in the State data correction process Slide 5 What Does DataQs Do? Allows users to submit data quality challenges related to motor carrier safety data Routes the challenges to the appropriate organization for resolution based on the type of challenge Slide 6 What Does DataQs Do? Allows all involved in a data quality challenge to view the challenge and its status at any time Stores all information related to the challenge in a centralized place, e.g., correspondence between the user that submitted the challenge and the organization that is assigned the challenge, documentation pertaining to the challenge, status of the challenge, etc. Slide 7 Who Can Use DataQs? Motor Carriers and the General Public can use DataQs to submit data quality challenges related to motor carrier safety data FMCSA HQ, Field, and State personnel use DataQs to enter, respond to, track and manage the challenges Slide 8 DataQs Activity Graphic showing Website hits as of 7/27/05. Slide 9 DataQs Challenge Type Breakdown Carrier Information (USDOT#, Op. Auth., Insurance) 2% Enforcement (Compliance Rev., Safety Audit, Enf. Action) 1% Inspections 53% Crashes 41% Other 3% Total Challenges 3,721 Slide 10 DataQs Inspection Challenge Type Breakdown Missing from Carrier's Report 3% Duplicate 2% Assigned to Wrong Carrier 62% Incorrect Data (Other) 9% Incorrect Data (Violation) 24% Total Challenges 3,721 Slide 11 DataQs Crash Challenge Type Breakdown Missing from Carrier's Report 3% Not Preventable 7% Not Reportable (No fatality, injury, or towaway) 41% Duplicate 7% Incorrect Data 14% Assigned to Wrong Carrier 28% Total Challenges 2,828 Slide 12 What Makes For A Successful Challenge? Challenge Type Crash Data – Assigned to Wrong Carrier, Duplicate, Not Reportable Inspection Data – Assigned to Wrong Carrier Providing Supporting Documents Crash challenges improves from 65% to 86% Inspection challenges: 68% to 80% Carrier information: 35% to 87% State Differences Some States have very high rate of corrective actions taken compared to others (TN, WY, NM, FL, OH, VA) Slide 13 What Makes For An Unsuccessful Challenge? Challenge Type Crash – Not Preventable Inspection – Violation Data (Citation vs. Conviction) Inspection – Duplicate Enforcement Data (Compliance Review, Safety Audits, Enforcement Actions) State Differences Some States have a lower rate of corrective actions taken compared to others (KS, WA, KY, CT, SC, TX, WI, OH) Slide 14 FMCSA Programs Monthly Monitoring of State Data Quality (highlighted) + Ongoing Evaluation of State-reported Motor Carrier Crash and Inspection Data + UMTRI Off-site Analysis of State Crash Data + On-site State Data Quality Reviews + DataQs + Training (FMCSA, State & Local Officials) + Safety Data Improvement Funding + Analysis and Improvement of State PARs = FMCSA Data Quality Improvement Programs Slide 15 What is Monthly Monitoring of State Data Quality? A program that assists FMCSA and States to: Monitor State-reported CMV crash and inspection data quality on a monthly basis Measure progress towards the quarterly State data quality rating Analyze current and historical MCMIS and DataQs management reports Provide current status and historical trends of CMV data quality Slide 16 What Data & Resources are used for Monthly Monitoring? State-reported data Fatal and Non-fatal Commercial Motor Vehicle Crash data Roadside Inspection data Resources Motor Carrier Management Information System (MCMIS) Crash and Inspection files National Highway Traffic Safety (NHTSA) Fatality Analysis Reporting System (FARS) Crash files MCMIS online reports DataQs management reports Slide 17 What Analysis is Conducted in the Monthly Monitoring Process? State Safety Data Quality (SSDQ) Analysis Compares the quality of State-reported crash and inspection data to FMCSA standards Uses the same evaluation methodology used by FMCSA to rate States’ data quality on a quarterly basis Consists of 5 measures and one overriding indicator Results in an overall rating of “Good”, “Fair”, or “Poor” Slide 18 What Analysis is Conducted in the Monthly Monitoring Process? Review of MCMIS Online Reports Review 8 reports Crash & Inspection Timeliness, Accuracy, and Duplicate analysis Identify Discrepancies Review DataQs Management Reports Indication of persistent problems in data collection or correction Review data challenges entered and open over certain time limits Slide 19 How are Monthly Monitoring Results Presented? Two types of reports are prepared Headquarters Progress Report State Progress Report Reports will be posted online Results available on A&I Online’s Data Quality Module (http://ai.fmcsa.dot.gov) Site will be protected with UAS password (UAS password available through FMCSA Technical Support) Slide 20 Results provide historic view of data quality State Crash Timeliness Historical Report Slide 21 Slide 14 FMCSA Programs Monthly Monitoring of State Data Quality + Ongoing Evaluation of State-reported Motor Carrier Crash and Inspection Data + UMTRI Off-site Analysis of State Crash Data (highlighted) + On-site State Data Quality Reviews + DataQs + Training (FMCSA, State & Local Officials) + Safety Data Improvement Funding + Analysis and Improvement of State PARs = FMCSA Data Quality Improvement Programs Slide 22 UMTRI Analysis of State Crash Data What is the Program? Provides an analysis of the State’s data reporting to the FMCSA Motor Carrier Management Information System Crash file Similar evaluation method for each State State police accident report (PAR) file is evaluated for all reportable crashes Comparison of the PAR case file to the MCMIS Crash file Slide 23 UMTRI Analysis of State Crash Data Outcomes Determine the nature and extent of underreporting Determine patterns of MCMIS Crash File under- or over-reporting Determine sources of under- or over-reporting List of Reports California, Florida, Illinois, Michigan, Missouri, New Mexico, North Carolina, Ohio Slide 24 FMCSA Programs Monthly Monitoring of State Data Quality + Ongoing Evaluation of State-reported Motor Carrier Crash and Inspection Data + UMTRI Off-site Analysis of State Crash Data (highlighted) + On-site State Data Quality Reviews (highlighted) + DataQs + Training (FMCSA, State & Local Officials) + Safety Data Improvement Funding + Analysis and Improvement of State PARs = FMCSA Data Quality Improvement Programs Slide 25 What is the FMCSA State Data Quality Review Program? FMCSA team goes on-site to a State to: Review a State’s crash and roadside inspection collection and reporting processes Works with the States to identify areas for improving the data quality, and establishing methods for monitoring the effectiveness of the improvements. Slide 26 What does the FMCSA State Data Quality Review Program Involve? 1. State selection/coordination for the on-site review 2. Pre-analysis of State crash and inspection data 3. An on-site visit consisting of interviews/observations of State data collection process 4. Prepare recommendations/strategies for improving State’s data quality 5. Follow-up and monitoring effectiveness of improvements Slide 27 Who does the FMCSA State Data Quality Review Team Consist of? Data Quality Review Team – FMCSA HQ, Volpe Center staff, NISR, and Data Nexus Crash data expertise Roadside inspection data expertise System/Technical expertise Collection and Process expertise Slide 28 What is the status of the FMCSA State Data Quality Review Program? Conducted Pilot in State of NH – (June ‘05) In FY ’06 - plan 1 State Data Quality Review every quarter Slide 29 FMCSA Programs Monthly Monitoring of State Data Quality + Ongoing Evaluation of State-reported Motor Carrier Crash and Inspection Data + UMTRI Off-site Analysis of State Crash Data (highlighted) + On-site State Data Quality Reviews (highlighted) + DataQs + Training (FMCSA, State & Local Officials) + Safety Data Improvement Funding (highlighted) + Analysis and Improvement of State PARs = FMCSA Data Quality Improvement Programs Slide 30 Safety Data Improvement Funding What is the Program? CVARS – Commercial Vehicle Analysis Reporting System (grants and cooperative agreements) Provides funding via grants and cooperative agreements to State agencies to help them improve the collection and reporting of eligible large truck and bus safety data Slide 31 Safety Data Improvement Funding Assistance Federal funding available up to 100% No State match is required Existing projects are also eligible for assistance Expert staff available to help States identify data quality challenges and design workable solutions Slide 32 Safety Data Improvement Funding Projects Initiated by States Hire staff to manage data quality improvement programs Revise outdated crash report forms Hire staff to code and enter crash data Train law enforcement officers in collecting crash data Develop software to transfer data from the State repository to SAFETYNET Purchase equipment to support data quality activities Purchase software for field data collection and data transfer Slide 33 Safety Data Improvement Funding Grants and cooperative agreements awarded in 2002 - 2005 totaled nearly $ 18,400,000 Slide 34 Safety Data Improvement Funding Contact Betsy Benkowski FMCSA (202) 366-5387 Betsy.Benkowski@fmcsa.dot.gov Slide 35 A&I Data Quality Module Program Presents data quality analysis and reports Located in A&I Online website (http://ai.fmcsa.dot.gov) Provides quarterly evaluations of State-reported data in MCMIS Provides an electronic means for filing concerns about Federal and State data Access tools for developing State strategic plans (e.g., CVSPs, State safety plan, etc.) Slide 36 A&I Data Quality Module (Shown is a graphic of the home page of the A&I Data Quality Module.)