Performance measures show if intended outcomes are occurring and assess any trends. Program evaluation uses analytic techniques to assess the extent to which our programs are contributing to those outcomes and trends. As required by the Government Performance and Results Act of 1993, the Department's FY 2006—2011 Strategic Plan includes an updated list of new program evaluations planned for those fiscal years. This appendix provides a summary of DOT's program evaluation efforts and a report on program evaluations scheduled for completion in FY 2006. In addition, updates of FY 2005 evaluations that were not completed when last year’s PAR went to press are also included to maintain continuity across fiscal years.
TYPES OF PROGRAM EVALUATIONS
Program evaluation is an assessment, through objective measurement and systematic analysis, of the manner and extent to which programs achieve intended outcomes. Evaluations are of the following types:
PROGRAM EVALUATION MANAGEMENT
DOT staff, contractors, academic institutions, the Office of the Inspector General (OIG), or the Government Accountability Office (GAO) may conduct program evaluations. Program evaluation efforts are designed to ensure that the finished evaluations are useful regardless of who conducts the evaluation or the methodology used.
The programs selected for evaluations are vetted through the Department’s strategic planning process. Each modal administration nominates programs that are then reviewed by a strategic planning executive committee to ensure two things: 1) adequate breadth of program evaluations across modal administrations; and 2) alignment to the strategic goals developed through the planning process. The OIG and the GAO continue their own program evaluations independent of this schedule, as deemed appropriate.
FY 2006 Program Evaluation Summaries
A summary of DOT program evaluations scheduled for completion in FY 2006 follows.
EVALUATION OF NATIONAL TRAFFIC SAFETY ENFORCEMENT MOBILIZATIONS
Safety belt use is the most effective countermeasure available to passenger vehicle occupants to prevent fatalities and injuries in highway motor vehicle traffic crashes. This being the case, NHTSA has encouraged States to aggressively enforce laws mandating the use of safety belts in motor vehicles. Data indicates that safety belt use has increased over the preceding six year period as shown below:
As part of the enforcement effort, the Click It Or Ticket (CIOT) campaign was established to promote safety belt use. The CIOT program includes: the use of high visibility enforcement mobilizations by local law enforcement officials, paid advertisements focusing on safety belt enforcement, measurement of motorists’ awareness of safety belt campaigns, and measurement of the change in the rate of seat belt use.
Related Strategic Goal: Safety
The purpose of this outcome evaluation is to determine the effectiveness of various CIOT program components in increasing the overall safety belt usage rate. Data collection is ongoing to determine the amount of funds spent for enforcement actions and advertising. In addition, the results of State reported surveys of safety belt use, motorist knowledge/attitude surveys administered at Driver Licensing offices, and a national telephone survey conducted pre/post yearly mobilization efforts are being analyzed to track progress.
Both the FY 2004 and FY 2005 evaluation results will be published by the end of 2006 and the FY 2006 results will be published by December 2007.
EVALUATION OF FRA’s RESEARCH, DEVELOPMENT, AND DEMONSTRATION PROGRAM
The Federal Railroad Administration (FRA) Research, Development and Demonstration (RD&D) Program conducts work that directly supports the agency's regulatory safety mission. The purpose of the program is to support FRA's efforts to sustain the safety and efficiency of the rail system.
Related Strategic Goal: Safety
Based on the recommendations of a 2004 program assessment, FRA is in the process of completing a strategic framework for managing the program and its component research projects. This will involve developing multi-year RD&D program plans that contain detailed schedule and budget information as well as clear explanations of how projects support Department-wide goals. After the completion of the framework and program plans in FY 2007, annual reviews of the program will commence.
RAILROAD SAFETY BOARD EVALUATION
The Railroad Safety Board reviews and approves, or denies, waiver petitions, block-signal applications, and requests for special approval submitted by railroads and other parties subject to the applicable regulations. A third party review of the Railroad Safety Board was initiated in order to improve the effectiveness and efficiency of the Board. The evaluation was performed by private contractor, Zeta-Tech Associates, Inc.
Related Strategic Goal: Safety, Organizational Excellence
The evaluation objectives included the following:
In performing the process evaluation, Zeta-Tech interviewed FRA staff, Safety Board Members, and railroad representatives. They also reviewed documentation outlining the waiver process and attended multiple Railroad Safety Board meetings. Based on the recommendations provided by Zeta-Tech, FRA has taken the actions indicated in the list below.
FMCSA PERFORMANCE MEASUREMENT AND LINKAGES PROGRAM EVALUATION
The Performance Measurement and Linkages Program evaluation focused on an assessment of the underlying mission, goals, objectives and strategies of Federal Motor Carrier Safety Administration (FMCSA), and development of strategic plans, integrated performance budgets and operational plans based on sound performance measurement.
Related Strategic Goal: Organizational Excellence
This process evaluation focused on the current quantitative measures being used by the Agency in defining accomplishments, and recommended ways to more closely align the planning and budgeting processes. It also provided analysis into the validity of Agency performance measures and related measurement tools.
The findings identified the following recommendations:
FMCSA has already begun implementing the evaluation recommendations. Upcoming actions include the following:
In addition to the findings related to current strategies and measures, the evaluation also provided recommendations for improvement of the Agency’s internet-based Accomplishments Tracking System and identified issues to be considered in long-term authorization planning.
EVALUATION OF THE COMPLIANCE REVIEW IMPACT ASSESSMENT MODEL
A Compliance Review (CR) is an onsite examination of a motor carrier’s operations to determine the carrier’s safety fitness. FMCSA, in cooperation with the Volpe National Transportation Systems Center, has developed an analytic model to measure the effectiveness of the Compliance Review in terms of crashes avoided, injuries avoided, and lives saved. This tool provides FMCSA management with the information it needs to address the requirements of the Government Performance and Results Act (GPRA) of 1993, which obligates Federal agencies to measure the effectiveness of their programs as part of the budget cycle process. It also provides FMCSA and State safety program managers with a quantitative basis for optimizing the allocation of field safety resources. This analytic tool is known as the CR Effectiveness Model.
Related Strategic Goal: Safety
The CR Effectiveness Model shows the direct impact of compliance reviews on motor carrier safety, but not the "deterrent" effects (i.e., the effect on a carrier's behavior due to the potential of having a CR). The model is based entirely on "before and after" changes in the safety performance of motor carriers that received CRs. The model compares a motor carrier's crash rate in the 12-month period after a CR to its crash rate in the 12-month period prior to that review. To make this comparison, the model uses: (1) crash data reported by the states, and (2) power unit data reported by carriers or obtained during CRs.
This impact evaluation focused on CRs conducted in 2002 and 2003 to identify the extent to which the model could be used and to identify the associated benefits. In 2002, 12,139 compliance reviews were conducted. The analytical model was able to assess the impact of 9,172 of these reviews (some compliance reviews are removed from the model because the motor carrier receiving the compliance review was not active 12 months after the CR, had zero power units, or had crash and power unit data that did not pass edit checks designed to screen out erroneous data). Based on this assessment, it is estimated that during the period from 2002 to 2003, 1,426 crashes were avoided, 1,087 injuries were avoided, and 62 lives were saved as result of performing compliance reviews in 2002.
In 2003, 11,086 compliance reviews were conducted. The analytical model was able to assess the impact of 8,587 of these reviews. Based on this assessment, it is estimated that during the period from 2003 to 2004, 2,276 crashes were avoided, 1,651 injuries were avoided, and 90 lives were saved as result of performing compliance reviews in 2003.
EVALUATION OF THE ROADSIDE INSPECTION/ TRAFFIC ENFORCEMENT ANALYTICAL MODEL
FMCSA, in cooperation with the Volpe National Transportation Systems Center, has developed an analytic model to measure the effectiveness of roadside inspections and traffic enforcements in terms of crashes avoided, injuries avoided, and lives saved. This model provides FMCSA management with information to address the requirements of the Government Performance and Results Act of 1993 (GPRA), which obligates Federal agencies to measure the effectiveness of their programs as part of the budget cycle process. It also provides FMCSA and State safety program managers with a quantitative basis for optimizing the allocation of safety resources in the field. This analytic model is known as the Intervention Model.
Related Strategic Goal: Safety
The Intervention Model is based on the premise that the two programs— Roadside Inspection and Traffic Enforcement—directly and indirectly contribute to a reduction in crashes. The model includes two components that are used for measuring these different effects, the direct effects model component and indirect effects model component. Direct effects are based on the assumption that vehicle and/or driver defects discovered and then corrected at the roadside reduce the probability that these vehicles/drivers will be involved in subsequent crashes. In order to measure the direct effects of the intervention, the model assigns crash risk probabilities to each of the violations found at the roadside. The model then calculates direct-effect-prevented crashes according to the number and type of violations detected and corrected during the intervention.
Indirect effects are the by-products of the carriers' increased awareness of FMCSA programs and the consequences that the programs could impose if steps are not taken to ensure and/or maintain higher levels of safety. In order to measure indirect effects, which are essentially changes in behavior involving driver preparation, practices and vehicle maintenance, the model calculates motor carrier responses to exposure to the programs, and the resulting reduction in potentially crash-causing violations.
Most recently, the model was implemented to measure program effectiveness during the 2004 activity year using March 25, 2005, data extracted from the Motor Carrier Management Information System (MCMIS). The number of inspections as well as the model results are shown below for 2004 and the previous two years.
| Calendar Year | 2002 | 2003 | 2004 |
|---|---|---|---|
| Roadside Inspections | 2,255,921 | 2,215,762 | 2,211,875 |
| Traffic Enforcements | 762,561 | 791,157 | 803,032 |
| Total Interventions | 3,018,482 | 3,006,919 | 3,014,907 |
| Calendar Year | 2002 | 2003 | 2004 |
|---|---|---|---|
| Roadside Inspections | 16,387 | 17,151 | 18,673 |
| Traffic Enforcements | 12,716 | 13,062 | 13,615 |
| Total Interventions | 781 | 722 | 722 |
COMMERCIAL DRIVERS LICENSE PROGRAM EVALUATION
The objective of the Commercial Drivers License (CDL) program evaluation is to assess how well the CDL program is meeting its implicit goal of improving highway safety, which includes preventing unqualified drivers from obtaining a CDL and ensuring that commercial drivers are disqualified from driving when appropriate. Phase I of the evaluation used existing data to assess the program’s effectiveness and was completed in July 2005. Phase II of the evaluation is gathering additional data and is scheduled for completion in 2007.
Related Strategic Goal: Safety
Findings from the Phase I impact evaluation include:
As a result of these findings, FMCSA has taken significant steps to improve oversight of the CDL program, including implementation of more thorough and more frequent reviews of State CDL programs. The final portion of the review (Phase II) is accumulating diverse stakeholder feedback through surveys and focus groups to help determine why the States are not fully successful, with the goal of recommending additional improvements to the CDL program.
EVALUATION OF DOT’S RESEARCH AND DEVELOPEMENT STRATEGIC PLANNING PROCESS
Congress directed the Government Accountability Office (GAO) to assess Research and Innovative Technology Administration’s (RITA’s) coordination of DOT’s research and development activities and to evaluate how RITA is resolving concerns about the process raised by its predecessor organization—the Research and Special Programs Administration (RSPA).
Related Strategic Goal: Organizational Excellence
The purpose of the GAO evaluation was to: (1) determine how RITA’s responsibilities for overseeing DOT’s Research, Development, and Technology (RD&T) activities differ from RSPA’s; (2) identify RITA’s practices for coordinating, facilitating, and reviewing RD&T activities; and (3) evaluate the progress RITA has made in implementing previous GAO recommendations made to RSPA in 2003.
GAO collected information through legislative histories, document reviews, and interviews to compare RITA with RSPA with respect to mission, organizational structure, oversight of RD&T activities, budgetary resources, and strategic goals.
The findings from the GAO evaluation were as follows:
GAO made several recommendations to the Secretary of Transportation to enhance RITA’s ability to manage and ensure the effectiveness of RD&T activities. GAO urged RITA to develop and incorporate into its annual budget process and upcoming RD&T strategic plan:
EVALUATION OF THE NATION’S HIGHWAYS, BRIDGES AND TRANSIT (CONDITION AND PERFORMANCE REPORT)
The Conditions and Performance (C&P) Report provides Congress and other decision makers an appraisal of highway, bridge and transit physical conditions, operational performance, financing mechanisms, and future investment requirements. The C&P Report consolidates conditions, performance, and finance data provided by States, local governments, and transit operators to provide a national summary.
Related Strategic Goals: Safety, Mobility, Environmental Stewardship, Global Connectivity, Security, and Organizational Excellence
Executive Order 12893, Principles for Federal Infrastructure Investments (January 1994), directs each Executive Department and Agency with infrastructure responsibilities to base investments on systematic analysis of expected benefits and costs, including both quantitative and qualitative measures. The highway investment requirements in the C&P Report are developed in part from the Highway Economic Requirements System, which quantifies user, agency and societal costs for various types and combinations of improvements including travel time, vehicle operating, safety, capital, maintenance, and emissions costs. The National Bridge Investment Analysis System uses engineering and benefit-cost analysis to evaluate bridge investment requirements. Transit investment analysis is based on the Transit Economic Requirements Model (TERM), which consolidates engineering and cost/benefit analysis. TERM identifies the investments needed to replace and rehabilitate existing assets, improve operating performance, and expand transit systems to address the growth in travel demand and evaluate these needs to select future investments. The 2006 version of the C&P Report was completed in May 2006 and will be released upon review and approval by the Office of Management and Budget.
MANAGING THE PHYSICAL SECURITY OF FAA FACILITIES EVALUATION
The FAA operates systems and facilities, including Air Traffic Control Centers, Terminal Radar Approach Control facilities, air traffic control towers, and supporting facilities that collectively make up the National Airspace System (NAS). The President, through the publication of Homeland Security Presidential Directive-7, dated December 17, 2003, designated the NAS as part of the Nation’s critical infrastructure because of commercial aviation’s role in fostering and sustaining the national economy and ensuring the safety and mobility of air travelers.
Related Strategic Goal: Security
The FAA has established physical security requirements to ensure the safety and security of the NAS, FAA personnel, and assets. These security requirements are implemented through the FAA’s Facility Security Management Program. All staffed NAS facilities are periodically assessed and inspected for program compliance. Security shortfalls or “findings” are aggressively tracked until corrected. Once all required security measures are implemented at a facility, the facility receives security “accreditation.” The FAA has completed assessments at all of its staffed facilities and continues to work toward completing accreditation.
The DOT OIG conducted an independent evaluation to assess the adequacy of physical security at FAA facilities. The OIG concluded that the FAA has continued to improve its security measures since the September 11th attacks and has taken steps to strengthen its physical security environment. Nevertheless, physical security weaknesses at FAA facilities were identified and the OIG made recommendations for improvement that the FAA is actively pursuing.
The OIG's report is protected as sensitive security information and is not released to the public.
FY 2005 program evaluation updates
For those evaluations that were scheduled for completion in FY 2005 and did not have results available for publication in the FY 2005 Performance and Accountability Report, an update is provided below.
EVALUATION OF AUTOMOBILE SIDE IMPACT PROTECTION
Side impacts rank second only to frontal impacts as a cause of occupant fatalities in cars, light trucks and vans. Federal Motor Vehicle Safety Standard 214 sets minimum performance requirements in side impacts. The requirements were phased into passenger cars during model years 1994 to 1997 and extended to light trucks and vans in model year 1999.
Related Strategic Goal: Safety
The results of this outcome evaluation will not be published by the end of FY 2006 as planned. This evaluation is subject to new Office of Management and Budget (OMB) peer review requirements. Although the review is nearing completion, the added level of review will delay publication until sometime in FY 2007.
LARGE TRUCK CRASH CAUSATION STUDY
The Government Accountability Office and the Department of Transportation’s Inspector General stated in separate reports in 1999 that the lack of large truck crash causation data hampers FMCSA program effectiveness. In addition, the Motor Carrier Safety Improvement Act of 1999 authorized funding for a study of the causes of commercial vehicle crashes. In response, in cooperation with the National Highway Traffic Safety Administration (NHTSA), FMCSA initiated the Large Truck Crash Causation Study (LTCCS); the first-ever National study of the causes of crashes involving large trucks.
Related Strategic Goal: Safety
Nationally representative data on the primary and secondary causes of serious large truck crashes were collected by teams of trained investigators from NHTSA’s National Automotive Sampling System and FMCSA-funded State truck inspectors. The goal of the LTCCS was to determine the reasons for, and factors contributing to, serious large truck crashes, so FMCSA can take the results of this process evaluation and implement the most effective countermeasures to reduce crash occurrence and severity.
The LTCCS collected data on crashes in 24 sites in 17 States from 2000 through 2003. All the crash data were collected and encoded into a database. In March 2006, FMCSA completed a report to Congress on the LTCCS results. Information and findings from the evaluation included:
FMCSA is analyzing the results of this evaluation and will use the findings to improve the Agency’s safety programs.
BUS CRASH CAUSATION STUDY
In 2004, FMCSA initiated the Bus Crash Causation Study (BCCS) as a complementary and follow-on effort to the Large Truck Crash Causation Study (LTCCS). The goal of the BCCS is to determine the reasons for, and factors contributing to, serious bus crashes, so FMCSA can take the results of this process evaluation and implement the most effective countermeasures to reduce crash occurrence and severity.
Related Strategic Goal: Safety
The BCCS uses the same data collection methodology as followed in the LTCCS. Data collection was initiated in 2004 and will continue through 2006, with a release of initial data and findings to Congress and the public scheduled for FY 2007. By August 2006, investigations of 32 bus crashes had been conducted.