SafeStat Effectiveness Study Update March 2004 Prepared for: Federal Motor Carrier Safety Administration 400 Seventh Street SW Washington, D.C. 20590 Prepared by: John A. Volpe National Transportation Systems Center Motor Carrier Safety Assessment Division, DTS-47 Kendall Square Cambridge, MA 02142 Background In 1998, an effectiveness study was devised to confirm that motor carriers that SafeStat identifies are indeed high safety risk carriers. Safety risk at any given time is defined as the likelihood of having crashes in the near future. By examining the SafeStat post-identification crash experience of identified carriers, this study essentially tested SafeStat's crash rate prediction capability and represents the "bottom-line" assessment of its performance. Beyond confirming SafeStat's effectiveness, the results of this study were used to refine SafeStat to further emphasize the components of the system that are the most closely related to high future crash rates. Since 1998, there have been changes made to the SafeStat algorithm, and in the quality of data used by SafeStat, as well as changes to FMCSA motor carrier safety program policy and in the motor carrier industry. To better determine the current effectiveness of SafeStat in identifying the high safety risk carriers, the effectiveness study has been updated using the current version of the SafeStat algorithm (version 8.5) and recent motor carrier data (available as of March 2003). The report summarizes the results of this updated study. Description of the Effectiveness Study The effectiveness study was accomplished by: (1) performing a simulated SafeStat carrier identification using historical data; (2) observing the crash involvement over the immediate 18 months after SafeStat was run for both the carriers identified by SafeStat as having poor safety status and other carriers not so identified by SafeStat, but which had sufficient data to be identified; and (3) comparing the post-identification crash rates of both groups of carriers. If SafeStat is effective in identifying unsafe carriers (i.e., carriers having a high risk of being involved in future crashes), then the carriers identified as having a poor safety status would be expected to have higher post-selection crash rates than the carriers that were not identified by SafeStat. The greater the post-selection crash rate for the identified carriers relative to those carriers not identified, the more effective SafeStat would be in identifying unsafe motor carriers. Rather than use the most recent SafeStat results and having to wait for a period of time to collect post-identification crash data, the analysis was performed using historical data. The study was conducted by simulating a carrier identification by SafeStat on an earlier date (March 24, 2001) and then observing the carriers' crash involvement that occurred over the next 18 months (from March 2001 to September 2002). This procedure simulated carrier identification by the current version of SafeStat as if it had been run as of March 24, 2001 using safety events that occurred prior to that date, and allowed enough time for sufficient subsequent crash reporting to accurately measure the post-identification crash rates. The study was performed using data available as of March 2003. A relatively small number of carriers (150 out of 118,907) was excluded from the study due to erroneous data. Figure 1. Effectiveness Analysis Timeline From this simulation run of SafeStat, carriers that had sufficient data to be scored were placed into the following groups based on their overall SafeStat results in order to compare the "post-selection crash performance": 1. Carriers identified as "at-risk" (Category A and B Carriers), 2. Other carriers identified as having a poor safety status according to SafeStat (Category C carriers), and 3. Carriers with sufficient data but not identified by SafeStat as having a poor safety status The post-identification crash rate of each group was calculated as the number of reported crashes per 1000 power units (PUs). The number of PUs is defined as the total number of trucks, tractors, hazardous-material tank trucks, motor coaches, and school buses that are owned or leased by a motor carrier. The carrier PU information was based on census data that reside in the centralized FMCSA national database, the Motor Carrier Management Information System (MCMIS). The crash data were based upon the crashes reported by the states (according to the National Governors' Association (NGA) standard) that occurred during the post-selection period (March 2001 to September 2002). These data also reside in the MCMIS. Each reported crash was weighted based on the severity and timing of the crash. The severity weighting scheme placed emphasis on crashes with greater consequences, while the time weighting placed emphasis on crashes that occurred soon after the SafeStat identification run. Severity weights were assigned as follows: a weight of 0.5 for property damage only, a weight of 1.0 for crashes involving injuries/fatalities or hazardous material release, and a weight of 1.5 for crashes involving injuries/fatalities and hazardous material release. Time weights were assigned to each crash as follows: a weight of 1.5 for crashes that occurred within the first six months of 18 month post-selection time period, a weight of 1.0 for crashes that occurred 7 to 12 months into the post-identification time period, and a weight of 0.5 for crashes that occurred in the last 6 months of the time period. Each crash had its severity weight multiplied by its time weight to obtain an overall weight. In each carrier group, the numbers of weighted crashes were summed and divided by the number of total PUs to provide a weighted crash rate for the group. The following section discusses the results for each carrier group. Results Overall Effectiveness of SafeStat The post-selection crash rates for the SafeStat identified and not identified carrier groups were examined both in terms of their overall SafeStat Scores and in terms of the four Safety Evaluation Areas (SEAs) ? Accident, Driver, Vehicle, and Safety Management ? that determine the overall SafeStat Scores. The rates are shown in Table 1. Table 1. Post-Selection Crash Rates Carrier Group Number of Carriers Weighted Crash Rate* % Higher than Not Identified Carriers All Identified 5,952 42.5 73% At-Risk (Category A&B) 3,595 52.0 112% Other Identified (Category C) 2,357 29.4 20% Not Identified 112,805 24.6 - * Number of weighted crashes per 1,000 power units from 3/2001 to 9/2002. These results confirm that SafeStat did identify carriers with a higher crash risk. The group of all carriers that SafeStat identified as poor performers had a 73% higher crash rate than carriers that were not identified. The carriers designated as "at-risk" by SafeStat had a much higher crash rate (112% greater) than the carriers that were not identified. A majority of these "at-risk" carriers were identified in part because they had previous problems with respect to their crash rates (i.e., they had deficient Accident SEA values). The SafeStat-identified carriers in the "other identified" group (Category C carriers), which did not have high Accident SEA values but were in the worst 25th percentile in two of the other SEAs, posed a 20% greater crash risk than the carriers that were not identified. This result shows that SafeStat has the proactive capability to identify carriers that are likely to be involved in crashes even though they previously did not have exceptionally high crash rates. Effectiveness of Individual SEAs Further testing was done to determine the effectiveness of the principal components of SafeStat. This was accomplished by placing carriers into groups based on their performance results for each particular SEA (i.e., Accident, Driver, Vehicle, or Safety Management (SM)). The results for carriers with high individual SEA values compared to those with lower SEA values are as follows (Carriers with high SEA values were in the worst 25th percentile and were designated as the worst performers in that particular evaluation area. Conversely, carriers with no high SEA values were not in the worst 25th percentile, and therefore, were not among the poorest performers in that SEA.): Table 2. Crash Rates of Carriers with and without High SEAs Safety Evaluation Area Number of Carriers Weighted Crash Rate* % Greater than Carriers without the High SEA High Accident SEA 3,838 63.5 169% No High Accident SEA 114,919 23.6 - High Driver SEA 12,391 39.2 63% No High Driver SEA 106,366 24.1 - High Vehicle SEA 18,745 27.0 6% No High Vehicle SEA 100,012 25.4 - High SM SEA 4,448 38.3 53% No High SM SEA 114,309 25.0 - * Number of weighted crashes per 1,000 power units from 3/2001 to 9/2002. Accident SEA - The results confirm what may seem intuitively to be obvious: carriers with high crash rates in the past are likely to continue to have high crash rates in the future. In other words, past crash rate performance is a good indicator of future crash rate performance. The effectiveness study shows a 169% greater post-selection crash rate for carriers with poor Accident SEAs compared to carriers that were not identified as having poor Accident SEAs. Comparing SEAs, the Accident SEA is by far the most effective SEA for identifying high-risk carriers. Driver SEA - The Driver SEA (with a 63% higher crash rate for carriers with poor Driver SEAs) is the next most effective SEA. These results from the study are especially impressive because the criteria for the Driver SEA are based on violations and are independent of crash history. Vehicle SEA - Carriers with poor Vehicle SEAs did have a slightly higher crash rate (6%) than carriers without poor Vehicle SEAs. This result is much lower than the Driver SEA, thus implying that there is a stronger relationship between driver violations and crash risk than vehicle violations. Safety Management SEA - The Safety Management SEA is also effective in identifying carriers with high crash rates. Indicators in this SEA are based on safety regulation compliance supporting the association of safety regulations with crash risk. Carriers with high Safety Management SEAs had a 53% higher post-identification crash rate than carriers that did not have high Safety Management SEA values. Effectiveness of Individual Indicators Further testing was done to determine the effectiveness of indicators of SafeStat. The indicators are used to derive the SEA values. The follow table shows the results for indicators for the carriers in the worst 25th percentile. Table 3. Crash Rates of Carriers with High Indicators (>=75) Indicator >= 75 Number of Carriers Weighted Crash Rate* % Greater than Carrier Population Accident Involvement Indicator (AII) 3,811 70.0 174% Recordable Accident Indicator (RAI) 489 35.5 39% Driver Inspection Indicator (DII) 10,727 37.9 48% Driver Review Indicator (DRI) 1,573 46.4 82% Moving Violation Indicator (MVI) 5,468 63.3 148% Vehicle Inspection Indicator (VII) 18,736 27.0 6% Vehicle Review Indicator (VRI) 23 27.6 8% Enforcement History Indicator (EHI) 1,789 39.5 54% Safety Mgmt. Review Indicator (SMRI) 3,016 35.4 39% Hazardous Material Review Indicator (HMRI) 374 35.0 37% Carrier Population 118,757 25.6 - * Number of weighted crashes per 1,000 power units from 3/2001 to 9/2002. The Accident Involvement Indicator (AII), based on state-reported crashes, is clearly the indicator with the strongest link to future crash risk. The newest indicator, the Moving Violation Indicator (MVI), based on moving violations recorded during roadside inspections, is very close to the AII in it effectiveness with identifying high risk carriers. The Vehicle Inspection Indicator (VII) had nominal value in identifying high risk carriers, but it identified the largest portion of carriers (18,736). Comparison with the 1998 Effectiveness Study When comparing results of the original effectiveness study to the new results, it is clear that the original results showed that SafeStat was identifying carriers with relatively higher crash rates. The original study showed the At-Risk Group and Other Identified Group with future crash rates that were 169% and 41% higher, respectively, than the group of carriers not identified (compared to 112% and 20% for the same groups in the new study – see Table 1). There is, however, a major change in the environment from the original study conducted 1998 to this new study. The pool of carriers used in the original study was operating in a "pre-SafeStat" environment. During the timeframe of the new study, the carriers identified by SafeStat as potential high crash risk carriers were subject to targeting by federal and state safety programs, such as compliance reviews, road inspections, the PRISM program, and SafeStat On-line. The main purpose of these programs is to prevent future crashes from occurring. The effectiveness of these SafeStat-influenced safety programs in reducing crashes does to some extent mute the effectiveness of SafeStat to identify high crash risk carriers as measured in this study. The effectiveness of the SafeStat-influenced programs is significant. According to the a longitudinal study conducted over a similar timeframe as the new effectiveness study, Category A and B carriers reduced their crash rate by 45% over a period of 18 months after they were identified in SafeStat. Repercussions of these programs on SafeStat-targeted carriers can lead to companies leaving the industry. The results of the new study showed that 21% of at-risk carriers (and 14% of the other identified carriers) were no longer actively operating interstate commercial motor vehicles after the 18 months from the March 2001 SafeStat identification. These numbers are significantly higher than the attrition rate of 5% of the carriers not identified by SafeStat. Conclusion Despite these complications, the new effectiveness study is still showing that SafeStat does work. The individual parts of SafeStat and SafeStat as a whole do identify carriers that are likely to have significantly higher crash rates than carriers not identified. Volpe National Transportation Systems Center, Measuring the FMCSA's Safety Objectives from Year 2000 to 2002, July 2003, pg 6-3. (http://ai.volpe.dot.gov/CarrierResearchResults/CarrierResearchResults.asp?file=PDFs/StrategicPlan.pdf) 1