dFMCSA-RI-05-036
| Prepared for: Federal Motor Carrier Safety Administration Office of Information Management Analysis Division, MC-RIA 400 Seventh Street, S.W. Washington, DC 20590 |
Prepared by: John A. Volpe National Transportation Systems Center Office of System and Economic Assessment Motor Carrier Safety Assessment Division, DTS-47 Kendall Square Cambridge, MA 02142 |
This report documents the methodology and results from a model, called the Compliance Review (CR) Effectiveness Model, that measures the effectiveness of one of the key safety programs of the Federal Motor Carrier Safety Administration (FMCSA), the compliance review. The model was developed for the FMCSA by the Research and Innovative Technology Administration’s (RITA) John A. Volpe National Transportation Systems Center (the Volpe Center) in Cambridge, MA. This work is part of an effort to assess the effectiveness of the FMCSA’s principal safety programs. The work also addresses the requirements of the Government Performance and Results Act (GPRA) of 1993, which obligates federal agencies to measure the results of their programs as part of the budget cycle process.
The CR Effectiveness Model is one of two models that provide a baseline of the effectiveness of FMCSA safety programs through the use of standard safety performance measures. This baseline allows the FMCSA to judge the relative performance of its programs on a periodic basis by reflecting the changes in benefits resulting from each program. The results of these analyses are also intended to provide a basis for FMCSA resource allocation and budgeting decisions that will more closely optimize the effectiveness and efficiency of its motor carrier safety programs.
In addition to the CR Effectiveness Model, another model, called the Intervention Model, has been developed to measure the effectiveness of and estimate benefits resulting from roadside inspection and traffic enforcement activities. These two models have been developed to estimate the benefits of these FMCSA safety programs in terms of crashes avoided, lives saved, and injuries avoided.
Dale Sienicki, Chief of the Analysis Division in the Office of Information Management, manages the project for the FMCSA. The Volpe Center project manager is Donald Wright, Chief of the Motor Carrier Safety Assessment Division in the Office of System and Economic Assessment. The analysis was performed by Jon Ohman with assistance from Julie Nixon and Kevin Gay, all of the Volpe Center. Technical support was provided by Leon Parkin of Chenega Advanced Solutions & Engineering (CASE), LLC, under contract to the Volpe Center. Scott Valentine of the FMCSA’s Analysis Division deserves special thanks for his assistance in obtaining data that were used in the model.
Section
EXECUTIVE SUMMARY
1. INTRODUCTION
1.1. Background
1.2. Project Objective
1.3. Project Scope
2. COMPLIANCE REVIEW EFFECTIVENESS MODEL
2.1. Compliance Reviews
2.2. Methodology of the Model
2.3. Results of Implementation of Model for Carriers with Compliance Reviews in 2002
2.4. Extended Benefits of Compliance Reviews
3. ADDITIONAL ANALYSIS
3.1. Overview
3.2. Carrier Size
3.3. State of Domicile of Carrier
APPENDIX A. CALCULATION OF PRE-CR AND POST-CR AVERAGE CRASH RATES FOR CONTROL GROUP
APPENDIX B. ALLOCATION OF CHANGE IN AVERAGE CRASH RATE OF CONTROL GROUP TO CHANGES IN CRASH RATE AND CRASH REPORTING
LIST OF ILLUSTRATIONS
FIGURES
Figure
2-1. Compliance Review Effectiveness Model
2-2. Timeline for a Carrier with a Compliance Review on January 21, 2002
TABLES
Table
ES-1. Results of Implementation of Compliance Review Effectiveness Model for Carriers with Compliance Reviews in 2002
ES-2. Benefits from Compliance Reviews Conducted in 2002 – Estimated for 2002-2003 and Projected for 2003-2006
2-1. Results of Implementation of Compliance Review Effectiveness Model for Carriers Compliance Reviews in 2002
2-2. Benefits from Compliance Reviews Conducted in 2002 – Estimated for 2002-2003 and Projected for 2003-2006
3-1. Results of Implementation of Model by Carrier Size
3-2. Results of Implementation of Model by State of Domicile of Carrier
This report documents the methodology and results from a model that measures the effectiveness of one of the key safety programs of the Federal Motor Carrier Safety Administration (FMCSA), the compliance review (CR) program. The research was conducted by the Research and Innovative Technology Administration’s (RITA) John A. Volpe National Transportation Systems Center (the Volpe Center) in Cambridge, MA under a project plan agreement with the FMCSA. The work on the FMCSA Safety Program Effectiveness Measurement Project addresses the requirements of the Government Performance and Results Act (GPRA) of 1993, which obligates federal agencies to measure the results of their programs as part of the budget cycle process.
This report describes the methodology of the Compliance Review Effectiveness Model and presents the results of the implementation of the model for carriers receiving CRs in 2002. The benefits of the compliance review program are calculated in terms of crashes avoided, lives saved, and injuries avoided.
The on-site compliance review is perhaps the single greatest resource-consuming activity of the FMCSA. Thousands of CRs are conducted each year. In the year 2002, federal and state enforcement personnel conducted over 12,000 CRs on individual motor carriers. It is intended that through education, heightened safety regulation awareness, and the enforcement effects of the CR, carriers will improve the safety of their commercial vehicle operations, and, ultimately, reduce the number and severity of crashes in which they are involved.
The CR Effectiveness Model was developed to determine the effectiveness of the CR program. The model shows the direct impact of compliance reviews on carrier safety, but not the "deterrent" effects (i.e., the "threat" of having a CR for carriers that did not actually have a CR). The model is based on the individual and cumulative "before and after" changes in the safety performance of carriers that received CRs. The model compares a motor carrier’s crash rate following an on-site compliance review to its crash rate in the 12 months prior to that review. The model uses (1) crash data reported by the states and (2) power unit data reported by carriers or obtained during CRs, to calculate both before and after CR crash rates.
The CR Effectiveness Model succeeds the Compliance Review (CR) Impact Assessment Model, which was used to produce estimates of benefits for carriers with CRs in 1998, 1999, 2000, and 2001. [1] The CR Impact Assessment Model had been criticized for its shortcomings such as (1) the lack of a control group to adjust the results for underlying factors in the general carrier population, and (2) the inability to estimate the benefits of CRs beyond the calendar year following the CRs. The CR Effectiveness Model has features that address these shortcomings. These features are among the several differences between the two models:
The estimates produced by the CR Effectiveness Model establish new benchmarks and are not directly comparable to the estimates produced by the CR Impact Assessment Model.
The CR Effectiveness Model was implemented for carriers with CRs in 2002 to estimate the number of crashes (and associated fatalities and injuries) avoided in 2002-2003. The results for the first year following the review (2002-2003) are shown in Table ES-1. It is likely that these benefits continue beyond the first year following the review based on a study being conducted at the Volpe Center in which a continued lower crash rate was observed for carriers receiving reviews in prior years. It appears from this study that there was a permanent reduction in the average crash rate of carriers that receive CRs lasts for several years after receiving a CR. See below under Extended Benefits of Compliance Reviews.
| Model Implementation for Motor Carriers with CRs in: | 2002 |
|---|---|
| Compliance reviews conducted | 12,139 |
Motor carriers that received compliance reviews and:
|
9,172 |
| Estimated percentage reduction in average crash rate due to compliance reviews | 12.6 |
| Model Results (i.e., Benefits) Estimated for: | 2002-2003 |
| Crashes avoided | 1,426 |
| Fatal crashes avoided | 53 |
| Injury crashes avoided | 677 |
| Towaway crashes avoided | 696 |
| Lives saved | 62 |
| Injuries avoided | 1,087 |
The current methodology of the CR Effectiveness Model only estimates the benefits that occur in the 12 months following the CRs in a given year. Based on the results of research being conducted at the Volpe Center, it appears that the reduction in the average crash rate of carriers continues for several years after the CRs.
Two groups of carriers have been studied: (1) carriers that received CRs in 1999, and (2) carriers that received CRs in 2000. The average crash rate for each group was tracked over time using state-reported crash data and power unit data reported by carriers or obtained during CRs, which are the same data used in the CR Effectiveness Model.
In each group, the average crash rate declined in the two-year period following the CRs (i.e., the post-CR period) and remained at that new, lower level in the succeeding two-year periods. A methodology was developed to closely replicate these results and to project results for future years. This methodology involves projecting (1) the reduction in the pre-CR average crash rate and (2) the number of post-CR power units in each year. The projected number of crashes avoided is then calculated for each year using the same formula used in the CR Effectiveness Model.
Preliminary projections of the numbers of crashes avoided by carriers that received CRs in 2002 have been made for future periods (i.e., 2003-2004, 2004-2005, and 2005-2006). These projections should be regarded only as preliminary indicators of the extended benefits of compliance reviews rather than as official estimates. The research on this topic is still in progress. As more years of data become available for analysis and verification, this methodology will be refined to produce more accurate projections.
Table ES-2 shows the number of crashes avoided in 2002-2003 by carriers that received CRs in 2002 that was estimated by the CR Effectiveness Model. Table ES-2 also shows the projected numbers of crashes avoided in the periods 2003-2004, 2004-2005, and 2005-2006, as well as the projected percentage reduction in the pre-CR average crash rate and the projected post-CR power unit total for each of the three periods.
| Benefit Time Period | ||||
|---|---|---|---|---|
| 2002-2003 (Estimated) | 2003-2004 (Projected) | 2004-2005 (Projected) | 2005-2006 (Projected) | |
| Pre-CR Average Crash Rate* | 4.032 | 4.032 | 4.032 | 4.032 |
| Change in Average Crash Rate of CR Group due to CRs | –12.6% | –12.1% | –13.3% | –13.9% |
| Actual/Projected Post-CR Power Units | 280,685 | 277,878 | 275,099 | 272,348 |
| Estimated/Projected Crashes Avoided | 1,426 | 1,356 | 1,475 | 1,526 |
* – Crashes per 100 power units
To further measure the effectiveness of the compliance review program, the results of the implementation of the model were broken out by carrier size (i.e., number of power units) and by the state of domicile of the carrier.
The breakout of the results of the model implementation by carrier size showed that the carriers with 20 or fewer power units had the largest reduction in the average crash rate in the 12 months following their CRs.
The results of the implementation of the model were also broken out by state. Two states each had over 100 crashes avoided in 2002-2003 as a result of CRs conducted in 2002, while ten other states each had 50 or more crashes avoided.
1.1. BACKGROUND
During the 1980s, Congress passed several acts intended to strengthen motor carrier safety regulations. This led to the implementation of safety-oriented programs both at the federal and state levels. The Surface Transportation Assistance Act of 1982 established the Motor Carrier Safety Assistance Program, a grants‑in‑aid program to states to conduct roadside inspection and traffic enforcement programs aimed at commercial motor vehicles. The 1984 Motor Carrier Safety Act directed the U.S. Department of Transportation (U.S. DOT) to establish safety fitness standards for carriers. The U.S. DOT, in conjunction with the states, implemented the Motor Carrier Safety Assistance Program (MCSAP) to fund the roadside inspection and traffic enforcement programs and the safety fitness determination process and rating system (based on on‑site safety audits called compliance reviews).
It is expected that a major benefit of these programs has been and will continue to be an improved level of safety in the operation of commercial motor vehicles. Previously, however, there was no means to measure the benefits and effectiveness of these programs. The Safety Program Effectiveness Measurement Project was established to identify major functions and operations (programs) associated with the FMCSA mission and to develop results-oriented performance measures for those functions and operations, as called for in the Government Performance and Results Act (GPRA) of 1993.
1.2. PROJECT OBJECTIVE
Program evaluation should be viewed as a continuous management process that encourages the organization to reflect periodically upon how it is implementing its programs. Program effectiveness should be reassessed in light of the mission, available resources, changing requirements, political climate, technological change, public demands, and costs. Periodic review of the results of the evaluations will ensure that the activities are working, i.e., that they are delivering what was promised. This report is intended to satisfy the desire of the FMCSA to verify the effectiveness of one of its motor carrier safety programs, the compliance review program. The immediate objective of this effort is to measure how much of an impact the safety program activities have on avoiding crashes involving motor carriers and reducing resulting injuries and fatalities.
One of the main objectives of the Safety Program Effectiveness Measurement Project is to provide a baseline of the effectiveness of the selected programs through the use of standard safety performance measures. This baseline allows the FMCSA to judge the relative performance of its programs on a periodic basis by reflecting the benefits resulting from each program. The results of these analyses are intended to provide a basis for FMCSA resource allocation and budgeting decisions that will more closely optimize the effectiveness and efficiency of its motor carrier safety programs.
1.3. PROJECT SCOPE
The scope of this overall effort is limited to the major identifiable operational FMCSA programs and their effectiveness in reducing crashes and avoiding injuries and fatalities. Currently the Safety Program Effectiveness Measurement Project includes the compliance review, roadside inspection, and traffic enforcement activities and programs performed and supported by the FMCSA. Two models have been developed to estimate the benefits of these programs: the Compliance Review Effectiveness Model and the Intervention Model (for roadside inspections and traffic enforcements). The benefits of these programs are calculated in terms of crashes avoided, lives saved, and injuries avoided.
An objective of the project is to continue to improve these models and update the results on a recurring basis. The models will serve the program-specific requirement to measure program effectiveness as well as the broader function of supporting annual budget requirements and helping to determine the best resource allocation among program elements.
This report describes the methodology of the Compliance Review Effectiveness Model and presents the final results of the implementation of the model for carriers receiving compliance reviews (CRs) in 2002, including estimates of crashes avoided by carrier size and state of domicile.
This model succeeds the Compliance Review Impact Assessment Model, which was used to produce estimates of benefits for carriers with CRs in 1998, 1999, 2000, and 2001. [2] The CR Impact Assessment Model had been criticized for its shortcomings such as (1) the lack of a control group to adjust the results for underlying factors in the general carrier population, and (2) the inability to estimate the benefits of CRs beyond the calendar year following the CRs. The CR Effectiveness Model has features that address these shortcomings. These features are among the several differences between the two models:
The estimates produced by the CR Effectiveness Model will establish new benchmarks, which may differ from the level of the estimates produced by the CR Impact Assessment Model.
2.1. COMPLIANCE REVIEWS
The on-site compliance review (CR) is perhaps the single greatest resource-consuming activity of the FMCSA. Thousands of CRs are conducted each year. In the year 2002, federal and state enforcement personnel conducted over 12,000 CRs on individual motor carriers. In addition to actually conducting CRs, the FMCSA invests in: extensive analysis of the requirements of the Federal Motor Carrier Safety Regulations (FMCSR), enhancements to the design of the CR to better assess safety performance and compliance with the FMCSR, continued safety investigator training, enhancements to prioritization methodologies such as SafeStat [3] to determine who should receive CRs, and enhancements to information systems to report and store the results of the CRs that are conducted.
When performing CRs, FMCSA and state safety investigators spend many hours examining the safety records of individual motor carriers to assess their compliance and safety performance. The investigators also discuss their findings with the carriers’ safety managers to improve understanding of their safety programs. After a review is completed, the carrier is assigned a safety rating (i.e., satisfactory, conditional, or unsatisfactory). If serious violations are discovered, an enforcement case is initiated and a fine may be imposed. The CR results are also incorporated, with other safety data (i.e., crashes, roadside inspection results, moving violations, and closed enforcement cases), into SafeStat to reassess the carrier’s safety status. It is intended that through education, heightened safety regulation awareness, and the enforcement effects of the CR, carriers will improve the safety of their commercial vehicle operations, and, ultimately, reduce the number and severity of crashes in which they are involved.
2.2. METHODOLOGY OF THE MODEL
The CR Effectiveness Model was developed to determine the effectiveness of the CR program. The model shows the direct impact of compliance reviews on carriers that received CRs, but not the "deterrent" effects (i.e., the "threat" of having a CR) on all carriers. In addition, the model only estimates the benefits that occur in the 12 months following a CR. The model is based on the individual and cumulative "before and after" changes in the safety performance of carriers that received CRs. The model compares a motor carrier’s crash rate in the 12 months after an on-site compliance review to its crash rate in the 12 months prior to that review. The model uses (1) crash data reported by the states and (2) power unit data reported by carriers or obtained during compliance reviews, to calculate both crash rates. The data are stored in the FMCSA’s Motor Carrier Management Information System (MCMIS).
2.3. RESULTS OF IMPLEMENTATION OF MODEL FOR CARRIERS WITH COMPLIANCE REVIEWS IN 2002
A diagram of the CR Effectiveness Model, as implemented for carriers with CRs in 2002, is shown in Figure 2-1. The model estimates the number of crashes (and associated fatalities and injuries) avoided in the 12 months following the CRs. Thus, the benefits from the CRs conducted in 2002 occurred in both 2002 and 2003.
| d |
Figure 2-1. Compliance Review Effectiveness Model
A step-by-step description of the implementation procedure follows. The step numbers (shown in parentheses) correspond to the numbers in parentheses in the diagram.
(1) Identify carriers with one or more compliance reviews (CRs) in 2002 that were still active 12 months after their CRs.
There were 9,172 carriers that received CRs in 2002, were still active 12 months after their CRs (i.e., throughout their post-CR periods), and met the following conditions:
(2) Calculate the pre-CR average crash rate.
The 9,172 carriers that received CRs in 2002 and were still active 12 months after their CRs had a pre-CR average crash rate of 4.032 crashes per 100 power units. This average was obtained by dividing the total number of carriers’ state-reported crashes in the 12 months before their 2002 CRs by the total number of carriers’ power units and then multiplying by 100. The power unit data came from the MCMIS Census File. The data were obtained from compliance reviews and updated Form MCS-150 information submitted by carriers. In the rate calculation for each carrier, the power unit data were taken from the SafeStat run for the month following the carrier’s CR. That way, the power unit data used in the rate calculation would reflect the power unit data collected during the CR.
(3) Calculate the post-CR average crash rate.
The 9,172 carriers that received CRs in 2002 and were still active 12 months after their CRs had a post-CR average crash rate of 3.712 crashes per 100 power units. This average was obtained by dividing the total number of carriers’ state-reported crashes in the 12 months after their 2002 CRs by the total number of carriers’ power units and then multiplying by 100. The power unit data came from the MCMIS Census File. In the rate calculation for each carrier, the power unit data were taken from the SafeStat run one year after the run used to calculate the carrier’s pre-CR crash rate.
For example, if a carrier had a CR on January 21, 2002, then power unit data from the February 2002 SafeStat run would have been used to calculate its pre-CR crash rate, and power unit data from the February 2003 SafeStat run would have been used to calculate its post-CR crash rate. The carrier’s pre-CR period (i.e., the 12 months prior to the CR) would have been January 21, 2001 to January 20, 2002, while its post-CR period (i.e., the 12 months after the CR) would have been January 22, 2002 to January 21, 2003. This information is shown in the timeline in Figure 2-2.
Figure 2-2. Timeline for a Carrier with a Compliance Review on January 21, 2002
(4) Calculate the reduction in the average crash rate.
(4a) Calculate the reduction using the data for the carriers with CRs in 2002.
The percentage change in the average crash rate of carriers with CRs in 2002 was calculated as follows:
Percentage Change in Average Crash Rate of Carriers with CRs in 2002
Post-CR Average Crash Rate – Pre-CR Average Crash Rate
= —————————————————————————— X 100
Pre-CR Average Crash Rate
3.712 – 4.032
= ——————— X 100
4.032
= –7.94% (i.e., a decrease of 7.94 percent)
(4b) Adjust the reduction for underlying factors in the general carrier population.
The change in the average crash rate of the carriers that received CRs (i.e., the CR group) was not adjusted for underlying factors occurring in the general carrier population. For example, if the average crash rate of all carriers had decreased during the same period in which the CR group’s average crash rate decreased, then the reduction in the CR group’s average crash rate calculated in Step 4a would have been exaggerated. That is, not all of the reduction would have been the result of the CRs. Conversely, if the average crash rate of the general carrier population had increased during this period, then the reduction in the CR group’s average crash rate calculated in Step 4a would have been less than the actual crash rate reduction due to the CRs.
Another factor that must be considered in the analysis of carriers that received CRs in 2002 is improved crash reporting. Over the past several years, the FMCSA has made a concerted effort to improve the timeliness and completeness of crash reporting by the states. As a result, crashes are being reported earlier and more completely. This improved crash reporting will tend to increase the post-CR average crash rate and produce a smaller crash rate reduction in the CR group’s average crash rate than actually occurred.
To eliminate the effects of outside factors, a control group of carriers was selected. This control group consisted of all carriers that did not receive CRs in 2002. Any change in the average crash rate of the control group must have been due to factors affecting the entire carrier population, e.g., improved crash reporting. Thus, the change in the average crash rate of the control group was calculated and then subtracted from the change in the average crash rate of the carriers that received CRs in 2002. The difference resulting from this calculation represents the change in the average crash rate of the carriers that received CRs in 2002 that was solely the result of the CRs.
To be eligible for the control group, a carrier had to meet the following conditions:
There were 396,478 carriers that met these criteria.
The change in the average crash rate of the control (i.e., non-CR) group was calculated as follows:
Percent Change Post-CR Average Crash Rate – Pre-CR Average Crash Rate
in Average = ————————————————————————— X 100
Crash Rate Pre-CR Average Crash Rate
The pre-CR crash rate is the average crash rate for the entire pre-CR period, i.e., 2001-2002, while the post-CR crash rate is the average crash rate for the entire post-CR period, i.e., 2002-2003. The pre-CR and post-CR average crash rates were calculated as follows: [4]
Pre-CR Crashes in 2001 + Crashes in 2002
Average = ———————————————————————————
Crash Rate Power Units at the end of 2001 + Power Units at the end of 2002
Post-CR Crashes in 2002 + Crashes in 2003
Average = ———————————————————————————
Crash Rate Power Units at the end of 2002 + Power Units at the end of 2003
The 396,478 carriers in the control group had a pre-CR average crash rate of 1.656 crashes per 100 power units and a post-CR average crash rate of 1.733 crashes per 100 power units.
The percentage change in the average crash rate of the control group was calculated as follows:
Percentage Change in Average Crash Rate of Control Group
Post-CR Average Crash Rate – Pre-CR Average Crash Rate
= ——————————————————————————— X 100
Pre-CR Average Crash Rate
= ———————— X 100
1.656
= +4.65% (i.e., an increase of 4.65 percent)
This increase in the average crash rate of the control group (and therefore, the general carrier population) is the sum of the effects of the improved crash reporting and any change in the average crash rate of the general carrier population. To determine how much of the increase was due to each factor, a separate series of calculations was performed. The calculations showed that there was a 2.90 percent increase in the average crash rate of the general carrier population. Therefore, the 4.65 percent increase in the average crash rate of the control group (and therefore, the general carrier population) was the sum of a 2.90 percent increase due to an increase in the crash rate of the general carrier population and a 1.75 increase due to improved crash reporting. These calculations are shown in Appendix B.
Therefore, the adjusted change in the average crash rate due to the CRs conducted in 2002 was:
Adjusted Change in Crash Rate due to CRs Conducted in 2002
= Percentage Change in Average Crash Rate of Carriers with CRs in 2002
–
Percentage Change in Average Crash Rate of Control Group due to Increased Crash
Reporting
= (–7.94) – (4.65)
= –12.6% (i.e., a decrease of 12.6 percent)
(5) Calculate the number of post-CR power units, i.e., the number of power units 12 months after the CRs in 2002.
The 9,172 carriers that received CRs in 2002 and were still active 12 months after their CRs had a total of 280,685 power units 12 months after their CRs. This number was used to calculate the post-CR average crash rate in Step 3.
(6) Estimate the number of crashes avoided in 2002-2003 as a result of the CRs conducted in 2002.
The estimated number of crashes avoided in 2002-2003 by the 9,172 carriers that received CRs in 2002 and were still active 12 months after their CRs was calculated as follows:
Crashes Avoided in 2002-2003 by Carriers with CRs in 2002
= Pre-CR Average Crash Rate X Crash Rate Reduction (%) X Post-CR Power Units
= 4.032 crashes per 100 power units X 12.6% X 280,685 power units
= 1,426 crashes
Next, estimates were made of the number of crashes avoided in 2002-2003 by the carriers receiving CRs in 2002 by severity, i.e., fatal, injury, and towaway. [5] State-reported crash data in the MCMIS were used to compute these proportions. For crashes involving large trucks or motorcoaches (i.e., cross-country or intercity buses) in 2002-2003, the period in which the benefits of the CRs conducted in 2002 would occur, 3.7 percent were fatal crashes, 47.5 percent were injury crashes, and 48.8 percent were in towaway crashes.
Applying these proportions to the estimate of 1,426 crashes avoided produced the following results:
Fatal crashes = 1,426 X 3.7% = 53
Injury crashes = 1,426 X 47.5% = 677
Towaway crashes = 1,426 X 48.8% = 696
(7) Calculate the average numbers of fatalities and injuries per crash in 2002-2003.
The average number of fatalities per fatal crash was calculated from data from the Fatality Analysis Reporting System (FARS), which is maintained by the National Highway Traffic Safety Administration (NHTSA). The period in which the benefits of the CRs conducted in 2002 occurred is 2002-2003. For crashes in 2002-2003 involving large trucks or motorcoaches (i.e., cross-country or intercity buses), the ratio was 1.17 fatalities per fatal crash.
The number of injuries per crash involves fatal as well as injury crashes, since fatal crashes can also result in injuries. State-reported crash data from the MCMIS Crash File were used to compute the average numbers of injuries in fatal and injury crashes. For 2002-2003 large truck and motorcoach crashes, the averages were as follows:
(8) Calculate the benefits (i.e., lives saved and injuries avoided) that occurred in 2002-2003.
The estimated number of lives saved in the crashes avoided in 2002-2003 by the carriers with CRs in 2002 was calculated as follows:
Number of lives saved in fatal crashes avoided in 2002-2003 by carriers with CRs in
2002
= Number of fatal crashes avoided X Average number of fatalities per fatal crash
= 53 X 1.17
= 62 lives saved
The estimated number of injuries avoided in the crashes avoided in 2002-2003 by the carriers with CRs in 2002 was calculated as follows:
Number of injuries avoided in crashes avoided in 2002-2003 by carriers with CRs in
2002
= Number of fatal crashes avoided X Average number of injuries per fatal crash
+
Number of injury crashes avoided X Average number of injuries per injury crash
= 53 X 1.09 + 677 X 1.52
= 1,087 injuries avoided
Table 2-1 summarizes the estimated benefits that occurred in 2002-2003 as a result of the CRs conducted in 2002 on the 9,172 carriers that were still active 12 months after their CRs and met the additional criteria listed in the table.
| Model Implementation for Motor Carriers with CRs in: | 2002 |
|---|---|
| Compliance reviews conducted | 12,139 |
Motor carriers that received compliance reviews and:
|
9,172 |
| Estimated percentage reduction in average crash rate due to compliance reviews | 12.6 |
| Model Results (i.e., Benefits) Estimated for: | 2002-2003 |
| Crashes avoided | 1,426 |
| Fatal crashes avoided | 53 |
| Injury crashes avoided | 677 |
| Towaway crashes avoided | 696 |
| Lives saved | 62 |
| Injuries avoided | 1,087 |
2.4. EXTENDED BENEFITS OF COMPLIANCE REVIEWS
The current methodology of the CR Effectiveness Model only estimates the benefits that occur in the 12 months following the CRs in a given year. Based on the results of research being conducted at the Volpe Center, it appears that the reduction in the average crash rate of carriers that receive CRs lasts for several years after the CRs.
Two groups of carriers have been studied: (1) carriers that received CRs in 1999, and (2) carriers that received CRs in 2000. The average crash rate for each group was tracked over time using state-reported crash data and power unit data reported by carriers or obtained during CRs, which are the same data used in the CR Effectiveness Model.
In each group, the average crash rate declined in the two-year period following the CRs (i.e., the post-CR period) and remained at that new, lower level in the succeeding two-year periods. A methodology was developed to closely replicate these results and to project results for future years. This methodology involves projecting (1) the reduction in the pre-CR average crash rate and (2) the number of post-CR power units in each year. The projected number of crashes avoided is then calculated for each year using the same formula used in the CR Effectiveness Model.
Table 2-2 shows the number of crashes avoided in 2002-2003 by carriers that received CRs in 2002 that was estimated by the CR Effectiveness Model. Table 2-2 also shows the projected numbers of crashes avoided in the periods 2003-2004, 2004-2005, and 2005-2006, as well as the projected percentage reduction in the pre-CR average crash rate and the projected post-CR power unit total for each of the three periods.
| Benefit Time Period | ||||
|---|---|---|---|---|
| 2002-2003 (Estimated) |
2003-2004 (Projected) |
2004-2005 (Projected) |
2005-2006 (Projected) |
|
| Pre-CR Average Crash Rate* | 4.032 | 4.032 | 4.032 | 4.032 |
| Change in Average Crash Rate of CR Group due to CRs | –12.6% | –12.1% | –13.3% | –13.9% |
| Actual/Projected Post-CR Power Units | 280,685 | 277,878 | 275,099 | 272,348 |
| Estimated/Projected Crashes Avoided | 1,426 | 1,356 | 1,475 | 1,526 |
* – Crashes per 100 power units
The projections in Table 2-2 should be regarded as preliminary indicators of the extended benefits of compliance reviews rather than as official estimates. The research on this topic is still in progress. As more years of data become available for analysis and verification, this methodology will be refined to produce more accurate projections.
3. ADDITIONAL ANALYSIS
3.1. OVERVIEW
The results of the implementation of the model were broken out by carrier size (i.e., number of power units) and by the state of domicile of the carrier.
The results of these analyses revealed the types of carriers that will most likely respond positively to CRs. By focusing on carriers that are likely to respond positively to CRs, the effectiveness of the compliance review program may be improved. Alternative treatment approaches may be suggested for carriers that are at risk, but will most likely not respond positively to CRs.
The changes in the average crash rates of (1) the carriers that received CRs in 2002 and (2) the control group (i.e., the carriers that did not receive CRs in 2002) were calculated individually for each power unit group and state.
The sums of the estimates of crashes avoided by power group and state did not equal the estimate of 1,426 crashes avoided that was obtained in Section 2.3. Therefore, the estimates were prorated to sum to this number. The estimated numbers of crashes avoided and the percent changes in the average crash rates shown in Tables 3-1 and 3-2 reflect this prorating procedure.
3.2. CARRIER SIZE
The results of the implementation of the model were broken out by carrier size as measured by the number of power units at the time of the CR, i.e., the number of pre-CR power units.
Table 3-1 shows the results of the implementation of the model for the four power unit groups:
Table 3-1 shows, for each power unit group, the number of carriers in the group that received CRs in 2002, the pre-CR average crash rate, and the adjusted percent change in the average crash rate after receiving the CRs. Table 3-1 also shows, for each power unit group, the estimated number of crashes avoided as a result of the CRs. (Note: A number in parentheses indicates an increase in the number of crashes.)
The smaller carriers, those with 20 or fewer power units, had the greatest reduction in the average crash rate as well as the largest number of estimated crashes avoided as a result of the program. For carriers with 1-5 power units, the post-CR average crash rate showed a decrease of 49.8 percent from the pre-CR average crash rate, resulting in 513 crashes avoided. For carriers with 6-20 power units, the crash rate decrease was 30.2 percent, resulting in 674 crashes avoided.
| Number of Pre-CR Power Units | Number of Carriers with CRs in 2002 | Pre-CR Average Crash Rate* | Percent Change in Average Crash Rate | Estimated Number of Crashes Avoided in 2002-2003 |
|---|---|---|---|---|
| 1 - 5 | 3,497 | 9.202 | -49.8 | 513 |
| 6 - 20 | 3,503 | 5.666 | -30.2 | 674 |
| 21-100 | 1,810 | 4.526 | -9.1 | 307 |
| >101 | 362 | 3.064 | +1.4 | (68) |
| All Carriers | 9,172 | 4.032 | -12.6 | 1,426 |
* – Crashes per 100 power units
For carriers with 21-100 power units, the post-CR average crash rate showed a decrease of 9.1 percent, resulting in 307 crashes avoided. Carriers with 101 or more power units had an increase of 1.4 percent in their average crash rate and an increase of 68 crashes.
The results of this analysis are consistent with the results of analyses of data from the previous model, the Compliance Review Impact Assessment Model. [6] The major difference in the results is that in the previous analyses, the average crash rate of the carriers with 101 or more power units decreased slightly, while in the current analysis the rate increased slightly.
3.3. STATE OF DOMICILE OF CARRIER
The results of the implementation of the model were broken out by the carrier’s state of domicile. Table 3-2 shows the results of the implementation for the 50 states and the District of Columbia. Since there were not enough Canadian or Mexican carriers receiving CRs in 2002 to summarize at the province/state level, these results were summarized at the national level.
Table 3-2 shows, for each state (or country), the number of carriers that received CRs in 2002, the pre-CR average crash rate, and the adjusted percent change in the average crash rate after receiving the CRs. Table 3-2 also shows, for each state (or country), the estimated number of crashes avoided as a result of the CRs. (Note: A number in parentheses indicates an increase in the number of crashes.)
| State/Country of Domicile | Number of Carriers with CRs in 2002 | Pre-CR Average Crash Rate* | Percent Change in Average Crash Rate | Estimated Number of Crashes Avoided in 2002-2003 |
|---|---|---|---|---|
| Alabama | 118 | 8.244 | –22.8 | 74 |
| Alaska | 19 | 0.000 | ------ | 0 |
| Arizona | 99 | 4.631 | –11.4 | 8 |
| Arkansas | 176 | 5.876 | +1.2 | (5) |
| California | 305 | 3.317 | –2.1 | 4 |
| Colorado | 191 | 3.405 | –8.6 | 12 |
| Connecticut | 117 | 2.744 | –47.2 | 41 |
| District of Columbia | 8 | 1.515 | –36.5 | 1 |
| Delaware | 41 | 5.219 | –3.0 | 1 |
| Florida | 169 | 6.218 | –25.4 | 78 |
| Georgia | 323 | 5.627 | +2.7 | (9) |
| Hawaii | 33 | 2.104 | –44.6 | 7 |
| Idaho | 114 | 5.817 | –15.0 | 18 |
| Illinois | 392 | 3.943 | –27.7 | 125 |
| Indiana | 286 | 2.499 | –9.5 | 61 |
| Iowa | 142 | 2.141 | –22.9 | 44 |
| Kansas | 230 | 5.351 | –10.6 | 23 |
| Kentucky | 196 | 5.395 | –19.3 | 43 |
| Louisiana | 127 | 2.884 | +5.7 | (6) |
| Maine | 59 | 3.659 | –66.1 | 24 |
| Maryland | 146 | 1.665 | +27.8 | (32) |
| Massachusetts | 128 | 3.503 | –29.6 | 26 |
| Michigan | 179 | 4.675 | –15.4 | 63 |
| Minnesota | 286 | 3.986 | –10.8 | 27 |
| Mississippi | 198 | 5.083 | –15.4 | 46 |
| Missouri | 496 | 2.933 | –10.9 | 72 |
| Montana | 36 | 5.475 | –48.1 | 18 |
| North Carolina | 198 | 5.886 | –20.9 | 56 |
| North Dakota | 84 | 4.283 | –63.4 | 50 |
| Nebraska | 139 | 5.458 | –28.9 | 42 |
| Nevada | 85 | 3.638 | –55.6 | 19 |
| New Hampshire | 44 | 2.236 | –42.6 | 5 |
| New Jersey | 188 | 3.982 | –31.7 | 90 |
| New Mexico | 89 | 3.015 | +11.3 | (7) |
| New York | 155 | 4.695 | –15.6 | 29 |
| Ohio | 569 | 3.447 | –2.3 | 21 |
| Oklahoma | 167 | 6.038 | –27.8 | 97 |
| Oregon | 165 | 3.831 | –10.6 | 6 |
| Pennsylvania | 290 | 2.636 | –26.0 | 80 |
| Rhode Island | 54 | 3.147 | –20.7 | 4 |
| South Carolina | 113 | 9.195 | –32.3 | 43 |
| South Dakota | 79 | 5.794 | +0.9 | (1) |
| Tennessee | 201 | 5.067 | +16.8 | (76) |
| Texas | 624 | 4.934 | –13.3 | 142 |
| Utah | 138 | 3.480 | +1.6 | (2) |
| Virginia | 138 | 3.796 | +3.6 | (11) |
| Vermont | 25 | 2.691 | –28.1 | 2 |
| Washington | 239 | 3.695 | +23.7 | (31) |
| West Virginia | 96 | 4.046 | –45.4 | 22 |
| Wisconsin | 319 | 4.296 | –9.4 | 42 |
| Wyoming | 119 | 3.722 | –50.0 | 24 |
| Canada | 169 | 3.863 | –9.8 | 15 |
| Mexico | 15 | 0.662 | –95.7 | 1 |
| Total | 9,172 | 4.032 | –12.6 | 1,426 |
* – Crashes per 100 power units
Table 3-2 shows that two states, Texas (142) and Illinois (125), each had more than 100 crashes avoided in 2002-2003 due to CRs performed in 2002. Ten other states had 50 or more crashes avoided. Ten states showed increases in the number of crashes in 2002-2003 for carriers that received CRs in 2002.
There are several factors that affect the state estimates of crashes avoided. The equation that is used to calculate the number of crashes avoided consists of three factors: the pre-CR average crash rate, the percentage reduction in the average crash rate due to the CRs, and the number of post-CR power units. The states with the largest numbers of crashes avoided are usually among the states with the highest numbers of post-CR power units, which is a function of the number of carriers receiving CRs. The more carriers in a state that receive reviews, the greater the number of post-CR power units that results, which increases the potential for a large number of crashes to be avoided. For example, Texas had a reduction in its average crash rate of only 13.3 percent, but had 142 crashes avoided because it had 624 carriers with CRs in 2002. On the other hand, Maine had a reduction in its average crash rate of 66.1 percent, but had only 24 crashes avoided because it had only 59 carriers with CRs in 2002.
Another factor that influenced the state results was the proportion of the carriers with zero crashes in the pre-CR period in each state that received CRs in 2002. Of the total of 9,172 carriers that received reviews in 2002, 5,390, or 58.8 percent, had pre-CR crash rates of zero. Thus, the crash rates of these carriers could either stay the same or increase, but not decrease. If a state had an especially high percentage of these carriers, it would make it difficult for that state’s average crash rate to decrease significantly. For example, the state of Washington had an increase in its average crash rate of 23.7 percent. One reason for this increase is that 73.2 percent of the carriers in Washington that received CRs in 2002 had pre-CR crash rates of zero.
In addition, the relatively low number of carriers in each state that received CRs in 2002 makes the state results subject to the influence of a few large carriers, i.e., carriers with large numbers of power units. As shown in Table 3-1, there were 362 carriers with 101 or more power units that received CRs in 2002. While these carriers made up 3.9 percent of the 9,172 carriers being analyzed, they accounted for 56.0 percent of the total post-CR power units. Thus, the data from a single large carrier could greatly affect an individual state’s results.
APPENDIX A
CALCULATION OF PRE-CR AND POST-CR
AVERAGE CRASH RATES FOR CONTROL GROUP
The pre-CR and post-CR average crash rates for the control (i.e., non-CR) group are actually the weighted averages of the average crash rates of the individual years, as shown by the following derivation.
The weighted average of the crash rates of two individual years is calculated by the equation:
Weighted Average Crash Rate
= ------------
where Rn = the average crash rate for year n, and
Wn = the weight for year n.
Rn, the average crash rate for year n, is defined as:
Cn
= ---------
Pn
where Cn = the number of crashes in year n, and
Pn = the number of power units at the end of year n
In this case, Wn, the weight for year n, is defined as Pn, the number of power units at the end of year n
Therefore, the weighted average of the crash rates for years 1 and 2
= ---------
= ---------
Therefore, the weighted average of the average crash rates for the control group for 2001 and 2002
Crashes in 2001 + Crashes in 2002
= ————————————————————————————
Power Units at the end of 2001 + Power Units at the end of 2002
= Pre-CR Average Crash Rate for the Control Group.
Also, the weighted average of the average crash rates for the control group for 2002 and 2003
Crashes in 2002 + Crashes in 2003
= ————————————————————————————
Power Units at the end of 2002 + Power Units at the end of 2003
= Post-CR Average Crash Rate for the Control Group.
APPENDIX B
ALLOCATION OF CHANGE IN AVERAGE CRASH RATE OF CONTROL GROUP
TO CHANGES IN CRASH RATE AND CRASH REPORTING
The 396,478 carriers in the control group had a pre-CR average crash rate of 1.656 crashes per 100 power units and a post-CR average crash rate of 1.733 crashes per 100 power units.
The percentage change in the average crash rate of the control group was calculated as follows:
Percentage Change in Average Crash Rate of Control Group
Post-CR Average Crash Rate – Pre-CR Average Crash Rate
= ——————————————————————————— X 100
Pre-CR Average Crash Rate
1.733 – 1.656
= ———————— X 100
1.656
= +4.65% (i.e., an increase of 4.65 percent)
This increase in the average crash rate of the control group, and therefore, the general carrier population, is the sum of the effects of changes in crash reporting and a change in the average crash rate of the general carrier population. To determine how much of the increase was due to each factor, the change in the average crash rate of the general carrier population was calculated.
To verify if the crash rate actually increased during the period in which the benefits from the CRs conducted in 2002 would have occurred (i.e., 2002-2003), data independent of the state-reported crash data used in the CR Effectiveness Model were used to calculate the large truck crash rates for the periods 2001-2002 and 2002-2003. The percentage change in the two crash rates was then calculated.
These crash rates were calculated using large truck crash data from the Fatality Analysis Reporting System (FARS) and the General Estimates System (GES), which are maintained by the National Highway Traffic Safety Administration (NHTSA). Counts of fatal crashes were obtained from the FARS, which contains data on a census of fatal crashes. Counts of injury crashes and property-damage-only crashes were obtained from the GES, which produces crash estimates from a national probability sample of all police-reported crashes. Crashes are included in the sample whether or not they are reported by the states to the FMCSA.
The NHTSA crash classification system differs from the National Governors’ Association (NGA) standard used by the states to report crashes to the FMCSA. In both systems, a fatal crash is defined as a crash resulting in at least one fatality, although the NHTSA rule specifically requires that at least one death occur within 30 days of the crash. For non-fatal crashes, the differences are much greater.
The NGA categories of non-fatal crashes are injury and towaway:
The NHTSA categories of non-fatal crashes are injury and property-damage-only:
The NHTSA non-fatal crash categories include many more crashes of lower severity than do the NGA non-fatal crash categories. Since it is the change in crash rates that is being measured, rather than the crash rates themselves, using the FARS and GES data should provide a reasonable indication of the change in the NGA crash rate calculated using the FMCSA’s MCMIS data.
Power unit data were obtained from the Federal Highway Administration (FHWA). The FHWA collects truck registration data from the 50 states and the District of Columbia. The data obtained were the numbers of large trucks registered in the U.S. in 2001, 2002, and 2003.
The change in the average crash rate of the general carrier population, as measured by the FARS and GES data, was calculated as follows:
Percent Change
in Average
Crash Rate= Post-CR Average Crash Rate – Pre-CR Average Crash Rate
——————————————————————————— X 100
Pre-CR Average Crash Rate
The pre-CR crash rate is the average crash rate for the entire pre-CR period, i.e., 2001-2002, while the post-CR crash rate is the average crash rate for the entire post-CR period, i.e., 2002-2003. The pre-CR and post-CR average crash rates were calculated as follows:
Pre-CR
Average
Crash Rate= Crashes in 2001 + Crashes in 2002
——————————————————————————— X 100
Large Trucks Reg. in 2001 + Large Trucks Reg. in 2002
Post-CR
Average
Crash Rate= Crashes in 2002 + Crashes in 2003
——————————————————————————— X 100
Large Trucks Reg. in 2002 + Large Trucks Reg. in 2003
The general carrier population had a pre-CR average crash rate of 5.233 crashes per 100 power units and a post-CR average crash rate of 5.385 crashes per 100 power units.
The percentage change in the average crash rate of the general carrier population was calculated as follows:
Percentage Change in Average Crash Rate of General Carrier Population
Post-CR Average Crash Rate – Pre-CR Average Crash Rate
= ——————————————————————————— X 100
Pre-CR Average Crash Rate
5.385 – 5.233
= ———————— X 100
5.233
= +2.90% (i.e., an increase of 2.90 percent)
Thus, the large truck crash rate increased 2.90 percent from 2001-2002 to 2002-2003.
Therefore, the increase in the crash rate of the control group caused by changes in crash reporting was:
Percentage Change in Average Crash Rate of Control Group (from state-reported data)
– Percentage Change in Average Crash Rate of General Carrier Population (from FARS and GES data)
= 4.65% – 2.90%
= 1.75%
Therefore, the 4.65 percent increase in the average crash rate of the control group, and therefore, the general carrier population, was the sum of a 2.90 percent increase due to an increase in the crash rate of the general carrier population and a 1.75 increase due to changes in crash reporting.
[1] A report documenting these results can be found at http://ai.fmcsa.dot.gov/CarrierResearchResults/PDFs/CRIAFinalReport.pdf
[2] A report documenting these results can be found at http://ai.fmcsa.dot.gov/CarrierResearchResults/PDFs/CRIAFinalReport.pdf
[3] SafeStat (Safety Status Measurement System) is an automated, data-driven analysis system that is designed to incorporate on-road safety performance information and enforcement history with on-site compliance review information in order to measure the relative safety fitness of interstate motor carriers. A thorough description of SafeStat methodology can be found in: John A. Volpe National Transportation Systems Center, Motor Carrier Safety Assessment Division, DTS-47, SafeStat, Motor Carrier Safety Status Measurement System, Methodology: Version 8.6, January 2004. This document is available at http://ai.fmcsa.dot.gov/SafeStat/safestat.asp?file=method.pdf.
[4] The pre-CR average crash rate is actually the weighted average of the average crash rates for 2001 and 2002. The post-CR average crash rate is actually the weighted average of the average crash rates for 2002 and 2003. A detailed derivation of these formulas can be found in Appendix A.
[5] A fatal crash results in at least one fatality. An injury crash results in no fatalities, but bodily injury to at least one person who, as a result of the injury, immediately receives medical treatment away from the scene of the crash. A towaway crash results in no fatalities or injuries requiring transport for immediate medical attention, but in one or more motor vehicles incurring disabling damage as a result of the crash, requiring the vehicle(s) to be transported away from the scene by a tow truck or other motor vehicle.
[6] A report documenting these results can be found at http://ai.fmcsa.dot.gov/CarrierResearchResults/PDFs/CRIAFinalReport.pdf
