FMCSA-RI-02-005

This report documents the methodology and results from an improved model to measure the effectiveness of one of the key safety programs of the Federal Motor Carrier Safety Administration (FMCSA). The research was conducted by the Research and Special Programs Administration’s (RSPA) John A. Volpe National Transportation Systems Center (the Volpe Center) in Cambridge, MA under a project plan agreement with the FMCSA. The work on FMCSA Program Performance Measures addresses the requirements of the Government Performance and Results Act (GPRA) of 1993, which obligates federal agencies to measure the effectiveness of their programs as part of the budget cycle process.

Work on FMCSA Program Performance Measures was initiated during FY 93. In December 1994, a report titled "Office of Motor Carriers Safety Program - Performance Measurement" was prepared. That report provided a comprehensive breakdown of Office of Motor Carriers (OMC) safety programs and activities and described about a dozen potential evaluation models. (Note: The OMC later became the FMCSA.) Based on the OMC’s review, the Volpe Center revised the report and recommended four evaluation models to assess the key OMC programs: roadside inspections conducted by participating states under the Motor Carrier Safety Assistance Program (MCSAP), on-site compliance reviews conducted by the OMC field offices and the states, commercial vehicle traffic enforcement also performed by the states under the MCSAP, and a comprehensive assessment of combined effects. Two initial evaluation models covering the roadside inspection program and the compliance review program were described in detail in a December 1998 report titled "OMC Safety Program Performance Measures." A review panel was convened to evaluate these models and made recommendations for improvement. The Volpe Center incorporated these recommendations together with other Volpe Center defined improvements into two "second-generation" models that measure the effectiveness of these two programs. This report describes the implementation of the second-generation Compliance Review Impact Assessment Model covering the compliance review program.

At the FMCSA, the project is managed by Dale Sienicki of the Office of Data Analysis and Information Systems, Analysis Division. The Volpe Center project manager is Donald Wright of the Economic Analysis Division in the Office of System and Economic Assessment. The analysis was performed at the Volpe Center by Jon Ohman, with assistance from David Madsen of the Volpe Center; Leon Parkin of EG&G Services, under contact to the Volpe Center; and Dr. Thomas M. Corsi of the Supply Chain Management Center, Robert H. Smith School of Business, University of Maryland, College Park, Maryland.

This report documents the methodology and results from an improved model to measure the effectiveness of one of the key safety programs of the Federal Motor Carrier Safety Administration (FMCSA), the compliance review (CR) program. The research was conducted by the Research and Special Programs Administration’s (RSPA) John A. Volpe National Transportation Systems Center (the Volpe Center) in Cambridge, MA under a project plan agreement with the FMCSA. The work on FMCSA Program Performance Measures addresses the requirements of the Government Performance and Results Act (GPRA) of 1993, which obligates federal agencies to measure the effectiveness of their programs as part of the budget cycle process.

This report describes the methodology and the implementation of the initial Compliance Review Impact Assessment Model, the limitations of that model, efforts made to address and remedy the limitations, and, finally, the description of the second-generation Compliance Review Impact Assessment Model. The results from the implementations of the improved second-generation model using CRs performed in 1998 and 1999 are presented. Finally, future planned work involving refinements to the model and the analyses of CR effectiveness on different risk classes of carriers is described.

Perhaps the single greatest resource-consuming activity of the FMCSA is the compliance review (CR). Thousands of CRs are conducted each year. In the most recent year, 2000, over 12,000 CRs were conducted on individual motor carriers by federal and state enforcement personnel. It is intended that through education, heightened safety regulation awareness, and enforcement effects of the CR, carriers will improve the safety of their commercial vehicle operations, and ultimately, reduce their crash rates.

The initial CR Impact Assessment Model was developed to determine the effectiveness of the CR program. This analytic model shows the direct impact of compliance reviews on carrier safety, but not the "deterrent" effects (i.e., the "threat" of having a CR). The model is based on the individual and cumulative "before and after" changes in the safety performance of carriers that received CRs. The model compares a motor carrier’s crash rate in a time period after an on-site compliance review to its crash rate prior to that review. To make this comparison, the model uses crash and mileage data collected during compliance reviews.

As part of the CR procedure, investigators are required to obtain the number of recordable crashes (crashes involving fatalities, injuries, or "towaways," in which an involved vehicle cannot leave the crash scene due to damage) in which the carrier was involved over the past 12 months as well as the number of vehicle miles traveled (VMT) by the carrier’s fleet over the same 12 months. Therefore, crash rates (in the form of the number of recordable crashes per million VMT) for all carriers having received CRs can be calculated.

Since the initial CR Impact Assessment Model determines the change in crash rates from before to after CRs, it requires not only pre-CR crash rates but also crash rates after the CRs. Due to data availability limitations, the initial model is based on crash reduction rates for carriers that received two CRs one to two years apart. The earlier (or initial) CR provides the pre-CR crash rate data and the subsequent (or follow-up) CR provides the post-initial CR crash rate data.

The initial CR Impact Assessment Model has one major limitation. It only uses data from carriers with two or more compliance reviews (CRs) that are one to two years apart to determine the change between pre and post-CR crash rates. This crash rate change, however, is extrapolated to all carriers receiving CRs by the model to derive total program benefits.

The model requires an assumption that the carriers that receive single CRs experience the same reduction in crash rate as carriers that receive multiple CRs during that same period. A carrier can get follow-up reviews for a variety of reasons. One reason for a subsequent CR is that the carrier’s safety status did not improve sufficiently after the first CR to avoid being targeted for a second CR. This reason for a subsequent review may potentially make the subset of carriers with multiple CRs different from the rest of the carriers receiving single CRs. Therefore, it would be preferable to base the estimate of crash reduction on a representative sample of all carriers receiving reviews rather than just the subgroup subjected to subsequent reviews.

The solution to this limitation was to conduct the Compliance Review Follow-Up Survey to obtain post-CR crash rates for all reviewed carriers not just those carriers with subsequent CRs. The survey results represent all carriers receiving CRs and can be used to calibrate the crash rate reductions used by the model.

The second-generation CR Impact Assessment Model addresses the limitation of the initial model by using data from the Compliance Review Follow-Up Survey to calculate post-CR crash rates, instead of data from subsequent CRs. The survey provides estimates of the change in the average crash rate, the change in vehicle miles traveled (VMT), and the decrease in VMT due to carrier attrition.

The second-generation CR Impact Assessment Model was implemented for 1999. There were 6,055 carriers that received CRs in 1998. These carriers had a total of 13,844 million vehicle miles traveled (VMT) and an average crash rate of .823 crashes per million VMT. The model, using results from the 2000 Compliance Review Follow-Up Survey, produced the following estimates:

The second-generation Compliance Review Impact Assessment Model was implemented for 2000, to produce an estimate of the number of crashes (and associated fatalities and injuries) avoided in 2000 as a result of the compliance reviews conducted in 1999.

The second-generation Compliance Review Impact Assessment Model uses the following estimates produced by the Compliance Review Follow-Up Survey:

• Change in VMT, and

• Decrease in VMT due to carrier attrition.

No follow-up survey of carriers that received CRs in 1999 was performed in 2001. Therefore, the above parameters in the model had to be estimated. For this implementation, the estimates obtained from the 2000 Compliance Review Follow-Up Survey were extrapolated one year.

There were 8,877 carriers that received CRs in 1999. These carriers had a total of 17,409 million vehicle miles traveled (VMT) and an average crash rate of .804 crashes per million VMT. The model, using extrapolated results from the 2000 Compliance Review Follow-Up Survey, produced the following estimates:

There are some caveats, however, for the above estimates.

Implementing the second-generation Compliance Review Impact Assessment Model using extrapolated results from the previous year’s Compliance Review Follow-Up Survey may not be valid if conditions that would affect the results of the survey change.

The second-generation Compliance Review Impact Assessment Model uses survey estimates of the change in the average crash rate, the change in VMT, and the decrease in VMT due to carrier attrition.

The change in the average crash rate depends on the population of carriers receiving compliance reviews (CRs). If the number of carriers receiving CRs increases sharply from one year to the next, then the additional carriers that receive reviews will be further down on the SafeStat CR prioritization list than the carriers already receiving reviews. These carriers will have better safety statuses, and, probably, lower crash rates, than the carriers already receiving reviews, i.e., those carriers higher up on the list. Note that as the number of carriers that received CRs increased from 6,055 in 1998 to 8,877 in 1999, the average crash rate from those CRs decreased from .823 crashes per million miles in 1998 to .804 crashes per million miles in 1999.

Since these carriers have lower crash rates than the other carriers, their average improvement (i.e., decrease) in the crash rate may not be as large as the average improvement shown by the carriers higher up on the prioritization list. Therefore, as the number of carriers receiving CRs increases, the resulting reduction in the average crash rate will tend to decrease.

Vehicle miles traveled (VMT) is largely a function of economic conditions. If the economy is strong, then active carriers are likely to increase their VMT from year to year. Also, carrier attrition is likely to be low, since fewer carriers will go out of business when the economy is strong than when it is weak.

Using the previous year’s survey to estimate the change in VMT and VMT lost due to carrier attrition assumes that economic conditions are similar in both years. If economic conditions change from one year to the next, then these estimates will be less reliable than those from a new survey would be. Economic conditions changed from 1999 to 2000. Since the economy slowed in 2000, the carriers that received CRs in 1999 probably did not increase their VMT from 1999 to 2000 as much as the carriers that received CRs in 1998 increased their VMT from 1998 to 1999. Carrier attrition was also probably greater from 1999 to 2000 than it was from 1998 to 1999. Therefore, the decrease in VMT due to carrier attrition was probably greater than the extrapolated estimate, and the increase in total VMT was probably less than the extrapolated estimate of 8.8 percent.

Since the model implementation probably overestimated both the degree of crash reduction from 1999 to 2000 and the number of VMT in 2000 for carriers receiving CRs in 1999, the estimate of 1,500 crashes avoided in 2000 is likely too high. It would probably be more accurate to consider this figure to be an upper bound of the actual number of crashes avoided instead of a point estimate.

Since there was another sizable increase in the number of carriers receiving CRs in 2000 (to over 11,000), and since economic conditions (and thus VMT) are subject to significant change, it is recommended that a new follow-up survey (of the carriers receiving CRs in 2000) be conducted in 2002, instead of continuing to extrapolate the results of the 2000 survey.

The survey will be conducted on a sample of the carriers receiving CRs, rather than on all such carriers as in the 2000 survey. The smaller sample size will allow more resources to be devoted to data quality issues. This survey will also be designed based on the results of research to be conducted on the results of the 2000 survey. This research is designed to further refine the CR Impact Assessment Model and to increase the effectiveness of the CR program.

One issue to be studied is whether there is a relationship between crash rate reduction following a CR and carrier size. If the two items are related, then the CR Impact Assessment Model will be refined to account for the size of the carriers receiving CRs.

To increase the effectiveness of the CR program, two analyses will be performed to determine the types of carriers that will most likely respond positively to CRs.

In one analysis, the results of the implementation of the model will be broken out by carrier safety status, i.e., the carrier’s SafeStat category before receiving the initial CR. The results will be examined to see if carriers in the higher risk categories, A and B, that are targeted for CRs reduce their crash rates more than carriers in the lower risk categories, C-G, or vice versa.

Another analysis will involve carriers that received more than one compliance review. The results of the model implementation will be analyzed to determine where the greatest crash reduction occurs: after the first CR, the second CR, etc. This analysis will determine if there are diminishing returns from performing additional CRs on the same carriers.

Since the early 1980s, Congress has passed several acts that strengthened federal motor carrier safety regulations and led to Federal Motor Carrier Safety Administration (FMCSA) programs to enforce them. The Surface Transportation Assistance Act of 1982 established the Motor Carrier Safety Assistance Program, a grants-in-aid program to states to conduct roadside inspection and enforcement programs aimed at commercial motor vehicles. The 1984 Motor Carrier Safety Act directed the Department of Transportation (DOT) to establish safety fitness standards for carriers. In response to this legislation, the DOT, in conjunction with the states, implemented the Motor Carrier Safety Assistance Program (MCSAP) to establish and fund the roadside inspection and enforcement program and the Safety Fitness Determination Process (SFDP) and rating system based on on-site safety audits (called compliance reviews).

It is expected that a major benefit of these programs has been and will continue to be an improved level of safety in the operation of commercial motor vehicles. Previously, however, there was no means to measure the benefits and effectiveness of these programs. This project was established to identify major functions and operations (programs) associated with the FMCSA mission and to develop results-oriented performance measures for those functions and operations, as called for in the Government Performance and Results Act (GPRA) of 1993.

Program evaluation should be viewed as a continuous management process that encourages the organization to reflect periodically upon how it is implementing its programs. Program effectiveness should be reassessed in light of the mission, available resources, changing requirements, political climate, technological change, public demands, and costs. Periodic review of the results of the evaluations will ensure that the activities are working, i.e., that they are delivering what was promised. This report is intended to satisfy the desire of the FMCSA to verify the effectiveness of one of its motor carrier safety programs, the compliance review program. The immediate objective of this effort is to measure how much of an impact the safety program activities have on avoiding crashes involving interstate motor carriers and reducing resulting injuries and fatalities.

One of the long-term objectives is to provide a baseline of the effectiveness of the selected programs through the use of standard safety performance measures that can be compared to future safety performance. This baseline will allow the FMCSA to judge the relative performance of its programs on a periodic basis by reflecting the benefits resulting from changes in each program. This capability will provide the FMCSA with a powerful analytical tool that can estimate the effects of changes within an activity and the effects of changes in resources between program activities. The results of these analyses will provide a basis for FMCSA resource allocation and budgeting decisions that will more closely optimize the effectiveness and efficiency of its motor carrier safety programs.

The scope of this overall effort is limited to the major identifiable programs and their effectiveness in reducing crashes and avoiding injuries and fatalities. It is hypothesized that the FMCSA safety program elements exert a positive influence causing changes in driver behavior and carrier operations ultimately leading to improvements in the level of motor carrier safety. It is recognized, however, that motor carriers are also affected by the highway environment and factors other than the influences of the FMCSA safety program elements that may intervene, impact, or influence motor carrier safety. No attempt is made here to account for these other exogenous influences on motor carrier safety performance, crash rates, and their associated consequences, i.e., fatalities and injuries.

The Program Performance Measures project includes the roadside inspection, compliance review, and traffic enforcement activities and programs performed and supported by the FMCSA. This report is concerned with the compliance review program and describes the CR Impact Assessment Model. An improved Roadside Inspection and Traffic Enforcement program performance measurement model, called the Intervention Model, has also been developed and will be described in a subsequent companion report. An objective of the project is to continue to improve these safety program measures and models and run them on a recurring basis. The models will serve the program specific requirement to measure program effectiveness as well as the broader function of supporting annual budget requirements and helping to determine the best resource allocation among program elements.

This report covers the background as well as the current work. It describes the methodology and the implementation of the initial Compliance Review Impact Assessment Model, the reviews and limitations of that model, efforts made to address and remedy the limitations, and, finally, the description of the second-generation Compliance Review Impact Assessment Model. The results from the implementations of the model using CRs performed in 1998 and 1999 are derived and presented. Finally, future planned work involving refinements to the model and the analyses of CR effectiveness on different risk classes of carriers is described.

The on-site compliance review (CR) is perhaps the single greatest resource-consuming activity of the FMCSA. Thousands of CRs are conducted each year. In the year 2000, over 12,000 CRs were conducted on individual motor carriers by federal and state enforcement personnel. In addition to actually conducting CRs, the FMCSA invests in extensive analysis of the requirements of the Federal Motor Carrier Safety Regulations (FMCSRs), design of the CR to assess safety performance and compliance with the FMCSRs, safety investigator training, prioritization methodologies such as SafeStat to determine who should receive CRs, and information systems to report and store the results of the CRs that are conducted. When performing CRs, FMCSA and state safety investigators spend many hours examining the safety records of individual motor carriers to assess their compliance and safety performance. The investigators also discuss their findings with the carriers’ safety managers to improve understanding of their safety programs. After the reviews are completed, the results are incorporated with other safety data in SafeStat to reassess carrier safety status and are used to assign overall safety ratings (i.e., satisfactory, conditional, unsatisfactory). In the instances where serious violations are discovered, enforcement cases are initiated and fines may be imposed. It is intended that through education, heightened safety regulation awareness, and the enforcement effects of the CR, carriers will improve the safety of their commercial vehicle operations, and, ultimately, reduce their crash rates.

The initial CR Impact Assessment Model was developed to determine the effectiveness of the CR program. This analytic model shows the direct impact of compliance reviews on carrier safety, but not the "deterrent" effects (i.e., the "threat" of having a CR). The model is based on the individual and cumulative "before and after" changes in the safety performance of carriers that received CRs. The model compares a motor carrier’s crash rate in a time period after an on-site compliance review to its crash rate prior to that review. To make this comparison, the model uses crash and mileage data collected during compliance reviews. The results of compliance reviews are stored in the FMCSA’s Motor Carrier Management Information System (MCMIS), which is the source of the data used by the model.

As part of the CR procedure, investigators are required to obtain the number of recordable crashes (crashes involving fatalities, injuries, or "towaways," in which an involved vehicle cannot leave the crash scene due to damage) in which the carrier was involved over the past 12 months as well as the number of vehicle miles traveled (VMT) by the carrier’s fleet over the same 12 months. Therefore, crash rates (in the form of the number of recordable crashes per million VMT) for all carriers having received CRs can be calculated.

Since the initial CR Impact Assessment Model determines the change in crash rates from before to after CRs, it requires not only pre-CR crash rates but also crash rates after the CRs. Due to data availability limitations, the initial model is based on crash rates for carriers that received two CRs one to two years apart. The earlier (or initial) CR provides the pre-CR crash rate data and the subsequent (or follow-up) CR provides the post-initial CR crash rate data. As shown in a prior report, the average crash rate for these carriers improved (i.e., decreased) after the initial CRs from .750 crashes per million VMT to .661 crashes per million VMT, a decrease of 12 percent. This reduction in the average crash rate was then used not just for carriers with two CRs, but for all carriers with one or more CRs in a base year, to calculate the total CR program benefits.

The initial CR Impact Assessment Model has one major limitation. It only uses data from carriers with two or more compliance reviews (CRs) that are one to two years apart to determine the change between pre and post-CR crash rates. This crash rate change, however, is extrapolated to all carriers receiving CRs by the model to derive total program benefits.

The model requires an assumption that the carriers that receive single CRs experience the same reduction in crash rate as carriers that receive multiple CRs during that same period. A carrier can get follow-up reviews for a variety of reasons. For example, the carrier needed an enforcement follow-up, a complaint was filed against the carrier, or the carrier itself requested a new CR to improve its rating. Another reason for a subsequent CR is that the carrier’s safety status did not improve sufficiently after the first CR to avoid being targeted for a second CR. Some of these reasons for a subsequent review may potentially make the subset of carriers with multiple CRs different from the rest of the carriers receiving single CRs. Therefore, it would be preferable to base the estimate of crash reduction on a representative sample of all carriers receiving reviews rather than just the subgroup subjected to subsequent reviews.

The solution to this limitation was to conduct the Compliance Review Follow-Up Survey to obtain post-CR crash rates for all reviewed carriers not just those carriers with subsequent CRs. The survey results represent all carriers receiving CRs and can be used to calibrate the crash rate reductions used by the model. The Compliance Review Follow-Up Survey was conducted in 2000 on all carriers reviewed during 1998, and is described in Appendix A. The second-generation model, which uses the results of the survey, is described in Section 3.

A diagram of the second-generation Compliance Review Impact Assessment Model is shown in Figure 3-1. This model uses data from the Compliance Review Follow-Up Survey to calculate post-CR crash rates, instead of data from subsequent CRs, which were used in the initial model. Therefore, the model can use data from all carriers receiving CRs in a given year, not just those carriers receiving two CRs one to two years apart.

The second-generation Compliance Review Impact Assessment Model was implemented for 1999 in the following steps, which correspond to the numbered steps in Figure 3-1. This implementation produced an estimate of the number of crashes (and associated fatalities and injuries) avoided in 1999 as a result of the compliance reviews conducted in 1998.

There were 6,055 carriers that met this criterion.

The 6,055 carriers with CRs in 1998 had a pre-CR average crash rate of .823 crashes per million vehicle miles traveled (VMT). This average was obtained from the carriers’ 1998 CR data by multiplying the total number of the carriers’ crashes (11,389) by 1 million and then dividing by the carriers’ total VMT (13,844 million). This aggregate rate is equivalent to averaging each carrier’s crash rate weighted by its VMT.

The carriers with survey data had the following pre-CR and post-CR average crash rates:

• Post-CR: .747 crashes per million VMT

The percentage change in the average crash rate was -10.3 percent (i.e., a reduction of 10.3 percent), calculated as follows:

Percentage Change in Average Crash Rate

Post-CR Average Crash Rate - Pre-CR Average Crash Rate

= --------------------------------------------------------------------------- X 100

Pre-CR Crash Rate

.747 - .833

= ----------------- X 100

.833

= -10.3%

The 1999 VMT by the 6,055 carriers with CRs in 1998 was calculated as follows:

1999 VMT

= (1998 VMT – AVMT) X (1 + C)

where

AVMT = Decrease in carrier VMT from 1998 to 1999 due to carrier attrition, and

C = Percentage change in VMT from 1998 to 1999.

The 6,055 carriers with CRs in 1998 had a total of 13,844 million vehicle miles traveled in 1998.

The Compliance Review Follow-Up Survey found the decrease in carrier VMT from 1998 to 1999 due to carrier attrition, i.e., the 1998 CR VMT of reviewed carriers that ceased operations before or during 1999, to be 1,076 million miles. (Details of this calculation can be found in Appendix A.)

The carriers in the survey reported an 8.8 percent increase in VMT from 1998 to 1999.

Therefore, the estimated total VMT for 1999 was:

1999 VMT

= (13,844 – 1,076) million miles X (1 + .088)

= 12,768 million miles X 1.088

= 13,892 million miles

The estimated number of crashes avoided in 1999 by the 6,055 carriers with CRs in 1998 was calculated as follows:

Crashes avoided in 1999

= Pre-CR Average Crash Rate X Crash Rate Reduction X 1999 VMT

= .823 crashes per million miles X 10.3% X 13,892 million miles

= 1,178 crashes,

rounded to 1,200 crashes

The estimate of the number of crashes avoided was rounded to the nearest 100 crashes, due to the limited precision of the estimates produced by the second-generation model.

Next, estimates were made of the number of crashes avoided in 1999 by severity, i.e., fatal, injury, and towaway. The most recent estimates of the proportions of crashes by severity are found in the "Truck and Bus Crash Factbook 1995." This report was published in 1997 by the University of Michigan Transportation Research Institute" under contract to the Federal Highway Administration’s Office of Motor Carriers, which later became the FMCSA.

According to the report, of the trucks involved in crashes on U.S. roads in 1995, 3.6 percent were involved in fatal crashes, 40.0 percent were involved in injury crashes, and 56.4 percent were involved in towaway crashes.

Applying these proportions to the total of 1,200 crashes avoided produced the following results:

Injury crashes = 1,200 X 40.0% = 480

Towaway crashes = 1,200 X 56.4% = 677

The average number of fatalities per fatal crash was calculated from data from the Fatality Analysis Reporting System (FARS), which is maintained by the National Highway Traffic Safety Administration (NHTSA). For 1999 crashes involving large trucks or intercity buses, the ratio was 1.19 fatalities per fatal crash.

The number of injuries per crash involves fatal as well as injury crashes, since fatal crashes can also result in injuries. State-reported crash data in the MCMIS were used to compute the average numbers of injuries in fatal and injury crashes. For 1999 large truck and bus crashes, the averages were as follows:

• Injury crashes: 1.60 injuries per crash

The estimated number of lives saved in the crashes avoided in 1999 was calculated as follows:

Number of lives saved in fatal crashes in 1999

= Number of fatal crashes avoided X Average number of fatalities per fatal crash

= 43 X 1.19

= 51 lives saved

The estimated number of injuries avoided was calculated as follows:

Number of injuries avoided in 1999

= Number of fatal crashes avoided X Average number of injuries per fatal crash +

Number of injury crashes avoided X Average number of injuries per injury crash

= 43 X 1.26 + 480 X 1.60

= 822 injuries avoided

In summary, the implementation of the second-generation CR Impact Assessment Model for 1999 produced the following estimates:

• Number of crashes avoided in 1999: 1,200

The second-generation Compliance Review Impact Assessment Model was implemented for 2000 in the following steps, which correspond to the numbered steps in Figure 3-1. This implementation produced an estimate of the number of crashes (and associated fatalities and injuries) avoided in 2000 as a result of the compliance reviews conducted in 1999.

The second-generation Compliance Review Impact Assessment Model uses the following estimates produced by the Compliance Review Follow-Up Survey:

• Change in VMT, and

• Decrease in VMT due to carrier attrition.

No follow-up survey of carriers that received CRs in 1999 was performed in 2001. Therefore, the above parameters in the model had to be estimated. For this implementation, the estimates obtained from the 2000 Compliance Review Follow-Up Survey were extrapolated one year.

There were 8,877 carriers that met this criterion.

The 8,877 carriers with CRs in 1999 had an average crash rate of .804 crashes per million vehicle miles traveled (VMT). This average was obtained from the carriers’ 1998 CR data by multiplying the total number of the carriers’ crashes (13,995) by 1 million and then dividing by the carriers’ total VMT (17,409 million). This aggregate rate is equivalent to averaging each carrier’s crash rate weighted by its VMT.

Since no survey was performed in 2001, there was no post-CR average crash rate. The 10.3 percent crash rate reduction obtained in the 2000 survey was used again for this implementation of the model.

The estimated 2000 VMT by the 8,877 carriers with CRs in 1999 was calculated as follows:

2000 VMT

= (1999 VMT – AVMT) X (1 + C)

where

AVMT = Decrease in carrier VMT from 1999 to 2000 due to carrier attrition, and

C = Percentage change in VMT from 1999 to 2000.

The 8,877 carriers with CRs in 1999 had a total of 17,409 million vehicle miles traveled in 1999.

The decrease in carrier VMT from 1999 to 2000 due to carrier attrition, i.e., the 1999 CR VMT of reviewed carriers that ceased operations before or during 2000, was estimated to be 1,286 million miles. (Details of this calculation can be found in Appendix B.)

Since no survey was performed in 2001, the 8.8 percent increase in carrier VMT from 1998 to 1999 found in the 2000 Compliance Review Follow-Up Survey was used as an estimate of the percentage change in carrier VMT from 1999 to 2000.

Therefore, the estimated total VMT for 2000 was:

2000 VMT

= (17,409 – 1,286) million miles X (1 + .088)

= 16,123 million miles X 1.088

= 17,542 million miles

The estimated number of crashes avoided in 2000 by the 8,877 carriers with CRs in 1999 was calculated as follows:

Crashes avoided in 2000

= Pre-CR Average Crash Rate X Crash Rate Reduction X 2000 VMT

= .804 crashes per million miles X 10.3% X 17,542 million miles

= 1,453 crashes,

rounded to 1,500 crashes

The estimate of the number of crashes avoided was rounded to the nearest 100 crashes, due to the limited precision of the estimates produced by the second-generation model.

Next, estimates were made of the number of crashes avoided in 2000 by severity, i.e., fatal, injury, and towaway. The most recent estimates of the proportions of crashes by severity are found in the "Truck and Bus Crash Factbook 1995." This report was published in 1997 by the University of Michigan Transportation Research Institute" under contract to the Federal Highway Administration’s Office of Motor Carriers, which later became the FMCSA.

According to the report, of the trucks involved in crashes on U.S. roads in 1995, 3.6 percent were involved in fatal crashes, 40.0 percent were involved in injury crashes, and 56.4 percent were involved in towaway crashes.

Applying these proportions to the total of 1,500 crashes avoided produced the following results:

Injury crashes = 1,500 X 40.0% = 600

Towaway crashes = 1,500 X 56.4% = 846

The average number of fatalities per fatal crash is calculated from data from the Fatality Analysis Reporting System (FARS), which is maintained by the National Highway Traffic Safety Administration (NHTSA). The latest year for which estimates are available is 1999. For 1999 crashes involving large trucks or intercity buses, the ratio was 1.19 fatalities per fatal crash.

The number of injuries per crash involves fatal as well as injury crashes, since fatal crashes can also result in injuries. State-reported crash data in the MCMIS were used to compute the average numbers of injuries in fatal and injury crashes. For 1999 large truck and bus crashes, the averages were as follows:

The estimated number of lives saved in the crashes avoided in 2000 was calculated as follows:

Number of lives saved in fatal crashes in 2000

= Number of fatal crashes avoided X Average number of fatalities per fatal crash

= 54 X 1.19

= 64 lives saved

The estimated number of injuries avoided was calculated as follows:

Number of injuries avoided in 2000

= Number of fatal crashes avoided X Average number of injuries per fatal crash +

Number of injury crashes avoided X Average number of injuries per injury crash

= 54 X 1.26 + 600 X 1.60

= 1,028 injuries avoided

In summary, the implementation of the second-generation CR Impact Assessment Model for 2000 produced the following estimates:

Section 3.3 describes the implementation of second-generation Compliance Review Impact Assessment Model using findings from the previous year’s Compliance Review Follow-Up Survey. Extrapolating the results of the previous year’s survey to the current year may not be valid if conditions that would affect the results of the survey change.

The second-generation Compliance Review Impact Assessment Model uses the following estimates produced by the Compliance Review Follow-Up Survey:

• Change in VMT, and

• Decrease in VMT due to carrier attrition.

The change in the average crash rate depends on the population of carriers receiving compliance reviews (CRs). If the number of carriers receiving CRs increases sharply from one year to the next, then the additional carriers that receive reviews will be further down on the SafeStat CR prioritization list than the carriers already receiving reviews. These additional carriers will have better safety statuses, and, probably, lower crash rates, than the carriers already receiving reviews, i.e., those carriers higher up on the list.

Since these carriers will probably have lower crash rates than the other carriers, their average improvement (i.e., decrease) in the crash rate may not be as large as the average improvement shown by the carriers higher up on the prioritization list. Therefore, as the number of carriers receiving CRs increases, the resulting reduction in the average crash rate will tend to decrease. In other words, increasing the number of carriers receiving CRs by X percent will probably result in a reduction in the average crash rate of less than X percent.

Vehicle miles traveled (VMT) is a function of economic conditions. If the economy is strong, then active carriers are likely to increase their VMT from year to year. Also, carrier attrition is likely to be low, since fewer carriers will go out of business when the economy is strong than when it is weak.

Using the previous year’s survey to estimate the change in VMT and VMT lost due to carrier attrition assumes that economic conditions are similar in both years. If economic conditions change from one year to the next, then these estimates will be less reliable than those from a new survey would be.

In the case of carriers receiving CRs in 1999, both of these factors apply. The number of carriers receiving CRs went from 6,055 in 1998 to 8,877 in 1999, an increase of 46.6 percent. The average crash rate for these CRs, however, decreased from .823 crashes per million miles in 1998 to .804 crashes per million miles in 1999. Thus, the 10.3 percent decrease in the average crash rate that the Compliance Review Follow-Up Survey found for carriers receiving CRs in 1998 might be too high an estimate for the carriers receiving CRs in 1999. Since there was another sizable increase in the number of carriers receiving CRs in 2000 (to over 11,000), it would be advisable to conduct a new follow-up survey of those carriers in 2002, instead of extrapolating the results of the follow-up survey conducted in 2000 one more year.

Economic conditions also changed from 1999 to 2000. Since the economy slowed in 2000, the carriers that received CRs in 1999 probably did not increase their VMT from 1999 to 2000 as much as the carriers that received CRs in 1998 increased their VMT from 1998 to 1999. Carrier attrition was also probably greater from 1999 to 2000 than it was from 1998 to 1999. Therefore, the decrease in VMT due to carrier attrition was probably greater than the extrapolated estimate (See Appendix B.), and the increase in total VMT was probably less than the extrapolated estimate of 8.8 percent. The current uncertain economic outlook is another argument for conducting a follow-up survey of the carriers receiving CRs in 2000 rather than extrapolating the old survey estimates again.

Since the model implementation probably overestimated both the degree of crash reduction from 1999 to 2000 and the number of VMT in 2000 for carriers receiving CRs in 1999, the estimate of 1,500 crashes avoided in 2000 is likely too high. It would probably be more accurate to consider this figure to be an upper bound of the actual number of crashes avoided instead of a point estimate.

Additional work on the Compliance Review Impact Assessment Model is planned. This work is designed to 1) further refine the Compliance Review Impact Assessment Model, and 2) increase the effectiveness of the compliance review program.

Initial analysis has suggested that there is a relationship between crash rate reduction following a CR and carrier size. If the two items are related, i.e., if the CR has different levels of effectiveness based on carrier size, then the Compliance Review Impact Assessment Model will be refined to account for the size of the carriers receiving CRs.

Certain carriers may respond better to compliance reviews (i.e., lower their crash rates more) than other carriers do. The survey results and model will be used to determine which carriers do or do not improve after receiving CRs and the extent of the improvement of those that do improve. For instance, the results of the implementation of the model will be broken out by carrier safety status, i.e., the carrier’s SafeStat category before receiving the initial CR. In this case, the results will be studied to see if carriers in the higher risk categories, A and B, that are targeted for CRs reduce their crash rates more than carriers in the lower risk categories, C-G, or vice versa. Carriers in the higher risk categories currently receive priority for CRs. They are often deficient in the SafeStat Safety Evaluation Areas (SEAs) reflecting safety performance (e.g., crashes), while carriers in the lower risk categories often have more safety compliance deficiencies (which may lead to safety performance problems if not addressed).

For carriers that received more than one compliance review, the results of the model implementation will also be broken out by the number of CRs the carrier received. The results will be analyzed to determine where the greatest crash rate reduction occurs for carriers with multiple CRs: after the first CR, the second CR, etc. The analysis will determine if there are diminishing returns from performing additional CRs on the same carriers.

The results of this analysis will reveal the types of carriers that will most likely respond positively to CRs. Alternative treatment approaches may be suggested for carriers that are at risk, but will most likely not respond positively to CRs. By focusing on carriers that are likely to respond positively to CRs, the effectiveness of the compliance review program may be improved.

Based on the discussion in Section 3.4, it is recommended that a new follow-up survey be conducted in 2002, instead of extrapolating the results of the 2000 survey one more year. The results of this survey will produce estimates that reflect the two factors discussed in Section 4.3:

Both of these factors tend to lower the number of crashes avoided by carriers that receive CRs. A new survey will calibrate the model, i.e., provide current estimates of:

• Change in VMT, and

• Decrease in VMT due to carrier attrition.

The survey will be designed based on several factors, including the results of the research described in Sections 4.2 and 4.3. To address the problem of the overreporting of crash data (See Appendix A.), the questionnaire will be redesigned and new survey procedures will be instituted. The survey will be conducted on a sample of the carriers receiving CRs, rather that on all such carriers as in the 2000 survey, to allow more time and effort to be devoted to data quality control.

The purpose of the Compliance Review (CR) Follow-Up Survey was to measure the post-compliance review crash rate change for all carriers receiving CRs, not just those carriers that also received subsequent CRs. The survey was conducted to calibrate the second-generation CR Impact Assessment Model, which is described in Section 3.

There were 6,055 motor carriers had received CRs in 1998. In late March and early April 2000, survey packages were mailed to the 5,623 of these carriers that

• had overall safety ratings, and

• had United States mailing addresses.

Each survey package consisted of a cover letter, a questionnaire, and a postage-paid return envelope. The questionnaire asked for:

• the total vehicle miles traveled (VMT) in 1999.

Survey packages were not mailed to the following groups of carriers:

In late May, a reminder package was mailed to the carriers that had not responded to the initial mailing. This package consisted of the same items as the original survey package, except for a reminder letter that was substituted for the original cover letter.

Finally, a nonresponse follow-up was conducted by telephone on a sample of carriers not responding to the survey.

Table A-1 shows the response to the Compliance Review Follow-Up Survey.

As shown in Table A-1, 4,397 questionnaires with usable data were received. Another 540 carriers did not return usable questionnaires, but received CRs between July 1999 and June 2000. These data were considered to be comparable to the calendar year 1999 data that were obtained from the questionnaires, and were used in the analysis. Thus, usable data were obtained from 4,937 (86.6 percent) of the of the 5,701 carriers in the survey.

Another 267 carriers were excluded from the analysis. Of these carriers, 230 were out of the motor carrier business. Some of these 230 companies were confirmed to be out of business, while others could not be located and were presumed to be out of business. Still other companies were still in business, but no longer operating as motor carriers.

Another 29 carriers were excluded from the analysis because of inconsistent or questionable data, while 5 carriers said that the requested data were unavailable, e.g., leasing companies. Finally, 3 companies selected for the nonresponse follow-up refused to provide data.

Of the 5,701 carriers in the survey, 5,204 (91.3 percent) were accounted for.

Table A-2 shows the results of the survey. As shown in row 1, questionnaire data only were obtained from 3,385 carriers, while recent (July 1999 to June 2000) CR data only were obtained from 540 carriers. Both questionnaire and recent CR data were obtained from 1,012 carriers.

The overall results show an increase from the pre-CR to the post-CR crash rate, regardless of whether questionnaire or recent CR data were used for the 1,012 carriers with both sources of data. The results exhibit a stark difference between the data obtained from the questionnaires and the data obtained from the recent CRs. The 3,385 carriers with only questionnaire data showed a 14.4 percent increase in the average crash rate, while the 540 carries with only recent CR data showed a 21.3 percent decrease. The probable cause of this disparity is that many carriers included all 1999 insurance incidents in their questionnaire counts, not just recordable crashes.

* - Crashes per million miles

This argument for this probable cause is based on several facts. Some carriers noted on their questionnaires that they had included nonrecordable crashes in their data. Other carriers enclosed copies of their accident registers with their questionnaires. An examination of these registers revealed the presence of nonrecordable crashes. Still other carriers stated in telephone calls that they obtained their crash counts from their insurance claims.

The final justification of the argument that carriers included nonrecordable crashes on their questionnaires comes from an examination of the data from the 1,012 carriers with both questionnaire and recent CR data. Table A-3 shows both the questionnaire and recent CR data used to calculate the average post-CR crash rates for this group of carriers. While the VMT data reported by the carriers on the questionnaires were comparable (within 2.1 percent) with the VMT data collected by the safety investigators (SIs) during the CRs, the carriers reported 23.5 percent more crashes than did the SIs during the CRs.

The reasons for this disparity lie with the persons doing the crash reporting. Safety investigators collected the crash data on site during compliance reviews. Thus, they could screen out nonrecordable crashes listed on the carriers’ accident registers. Carriers, however, often used insurance claims to determine crash counts, thus overstating the actual number of recordable crashes. To obtain an estimate of this overreporting, the post-CR crash rates obtained for the 1,012 carriers using the SI-reported CR data and the carrier-reported questionnaire data were used. The crash rates are expressed in crashes per million miles.

Post-CR Crash Rate using SI-Reported CR Data .803

------------------------------------------------------------------------------ = ------- = .827

Post-CR Crash Rate using Carrier-Reported Questionnaire Data .971

Therefore, the overreporting adjustment factor was .827 or 82.7 percent. This factor was applied to the post-CR crash rate for the 3,385 carriers with questionnaire data only to obtain an adjusted crash rate.

Adjusted Post-CR Crash Rate for Carriers with Questionnaire Data Only

= Post-CR Crash Rate for Carriers with Questionnaire Data Only X Adjustment Factor

= .872 crashes per million miles X 82.7%

= .721 crashes per million miles

This adjusted crash rate was used to recalculate the results shown in Table A-2. The results of these recalculations are shown in Table A-4, which contains only four of the six data columns found in Table A-2. Since the CR data had been determined to be more reliable than the questionnaire data, Table A-4 shows only the CR data results for the 1,012 carriers with both types of data.

The 3,385 carriers with only questionnaire data now exhibit a 5.4 percent decrease in the average crash rate instead of a 14.4 percent increase.

The overall average post-CR crash rate was recalculated using the adjusted crash rate for the 3,385 carriers with questionnaire data only. The overall rate is an average of the three post-CR crash rates (.721, .819, and .803 crashes per million miles) weighted by the total post-CR VMT for each group of carriers. The recalculated rate of .747 crashes per million miles represents an overall crash rate reduction of 10.3 percent.

A.5. SURVEY RESULTS – CARRIER ATTRITION

Implementation of the second-generation model requires an estimate of carrier attrition during the year under examination, i.e., the year whose crash count is affected by the compliance reviews performed the year before. In this case, to estimate the number of crashes avoided in 1999, one needs to estimate the decrease in VMT from 1998 to 1999 due to carrier attrition. That is, one needs to estimate the 1998 CR VMT of the carriers that became inactive before the end of 1999.

There are two categories of attrition carriers:

Prior to the survey, it was determined from the MCMIS Census File that 354 carriers that had received CRs in 1998 were no longer active. These pre-survey attrition carriers’ total 1998 CR VMT was 721.5 million miles.

During the survey, 230 carriers were found to be inactive (i.e., out of business, unable to be located, etc.). These respondent in-survey attrition carriers’ total 1998 CR VMT was 329.3 million miles, compared to 12,176 million miles for all 5,204 carriers that were accounted for in the survey, i.e., all respondent carriers. Using VMT as the basis for measuring attrition, the respondent in-survey attrition rate was calculated as follows:

Respondent In-Survey Attrition Rate

VMT of Respondent In-Survey Attrition Carriers

= ---------------------------------------------------------------

VMT of All Respondent Carriers

329.3 million miles

= -------------------------------

12,176 million miles

= .027, or 2.7 percent

Estimates of attrition must also be made for carriers not accounted for in the survey. There were 497 carriers that did not respond to the survey. Of these carriers, 444 were mailed questionnaires, but did not return them. The remaining 53 carriers came from the 69 Canadian carriers that were not mailed questionnaires for logistical reasons, and the 9 carriers whose 1998 CR information was not in the MCMIS at the start of the survey. Of the 69 Canadian carriers, 23 had recent CR data, leaving 46 nonrespondents. Of the 9 other carriers, 2 had recent CR data, leaving 7 nonrespondents.

The 497 nonrespondent carriers’ total 1998 CR VMT was 946.0 million miles. Applying the respondent in-survey attrition rate of 2.7 percent to this total produced the following estimate of nonrespondent in-survey attrition mileage:

Nonrespondent In-Survey Attrition Mileage

= Respondent In-Survey Attrition Rate X Nonrespondent Carrier 1998 CR VMT

= 2.7% X 946.0 million miles

= 25.5 million miles

Therefore, the estimated total attrition mileage for 1999 was:

Total Attrition Mileage

= Pre-Survey Attrition Mileage + In-Survey Attrition Mileage

= Pre-Survey Attrition Mileage +

= 721.5 million miles + (329.3 + 25.5) million miles

= (721.5 + 354.8) million miles

= 1,076 million miles

This section describes the estimation of the VMT lost to carrier attrition from 1999 to 2000. This quantity is needed in the implementation of the second-generation Compliance Review Impact Assessment Model for 2000, i.e., the calculation of the number of crashes (and associated fatalities and injuries) avoided in 2000 as a result of CRs performed in 1999. This implementation is described in Section 3.3.

The decrease in carrier VMT from 1999 to 2000 due to carrier attrition, i.e., the 1999 CR VMT of carriers that ceased operations before or during 2000, consists of two components:

As of March 2001, when a survey would have begun, 675 of the 8,877 had become inactive. These 675 pre-survey attrition carriers had a total 1999 CR VMT of 839 million miles.

Since no survey was conducted in 2001, the in-survey attrition mileage was estimated, using the results of the 2000 Compliance Review Follow-Up Survey. In that survey, the VMT of the carriers that were found to be inactive during the survey amounted to 2.7 percent of the VMT of all the carriers accounted for in the survey, i.e., all respondent carriers.

If a survey of carriers that received CRs in 1999 had been conducted in 2001, it would have included 8,202 carriers with total 1999 CR VMT of 16,570 million miles. These numbers were obtained by subtracting the known inactive carriers and their associated VMT from all carriers, i.e.,

Number of Carriers in Survey

= Number of All Carriers – Number of Known Inactive Carriers

= (8,877 – 675) carriers

= 8,202 carriers

VMT of Carriers in Survey

= VMT of All Carriers – VMT of Known Inactive Carriers

= (17,409 – 839) million miles

= 16,570 million miles

Applying the 2.7 percent in-survey attrition rate obtained in the 2000 survey produced the following estimate:

In-Survey Attrition Mileage

= In-Survey Attrition Rate X VMT of Carriers in Survey

= 2.7% X 16,570 million miles

= 447 million miles

Thus, the estimated total attrition mileage for 2000 was:

Total Attrition Mileage

= Pre-Survey Attrition Mileage + In-Survey Attrition Mileage

= (839 + 447) million miles

= 1,286 million miles