Performance measures show if intended outcomes are occurring and assess any trends. Program evaluation uses analytic techniques to assess the extent to which our programs are contributing to those outcomes and trends. As required by the Government Performance and Results Act of 1993, the Department's FY 2006 - 2011 Strategic Plan includes an updated list of new program evaluations planned for those fiscal years. This appendix provides a summary of DOT's program evaluation efforts and a report on program evaluations scheduled for completion in FY 2007.
Program evaluation is an assessment, through objective measurement and systematic analysis, of the manner and extent to which programs achieve intended outcomes. Evaluations are of the following types:
DOT staff, contractors, academic institutions, the Office of the Inspector General (OIG), or the Government Accountability Office (GAO) may conduct program evaluations. Program evaluation efforts are designed to ensure that the finished evaluations are useful regardless of who conducts the evaluation or the methodology used.
The programs selected for evaluations are vetted through the Department's strategic planning process. Each Operating Administration nominates programs that are then reviewed by a strategic planning executive committee to ensure two things: 1) adequate breadth of program evaluations across modal administrations; and 2) alignment to the strategic objectives developed through the planning process. The OIG and the GAO continue their own program evaluations independent of this schedule, as deemed appropriate.
A summary of DOT program evaluations scheduled for completion in FY 2007 follows.
Operator error has long been a significant factor in pipeline incidents and accidents. After three major incidents in the 1980s, the National Transportation Safety Board recommended development of qualification requirements for pipeline personnel carrying out safety duties. Congress directed regulatory action in subsequent legislation, and PHMSA finalized a rule in 1999.
The purpose of this outcome evaluation was to determine the benefits and impacts of the personnel qualification regulations issued in 1999. Information was collected through program operations and incident reports and two public meetings to explore progress in implementing the program.
The evaluation found several deficiencies in operator programs during the initial phase of implementation, but also a number of solutions emerging from: the pipeline industry itself, professional standards organizations, and State/federal inspectors and program managers. PHMSA and State pipeline safety agencies completed the initial inspection of Operator Qualification (OQ) programs for all federally-regulated operators and most State-regulated operators. The results of this initial inspection led to development of a national consensus standard to improve operators' programs and to the development of a model plan to assist small operators in complying with the 1999 rule. After the new standard for OQ programs was finalized and implemented, operators reported improved operational ability and safety, improved operating and maintenance procedures, and increased awareness of requirements. While evaluators observed some reduction in the number of incidents and accidents attributed to operator error, it is too early to attribute performance trends solely to the new operator qualification program standard.
This evaluation was completed in December 2006, and a report was provided to Congress in January 2007. The report recommended continuing inspections of OQ programs, monitoring the safety performance of operators and trends in accidents/incidents, clarifying the OQ regulations, and consideration of regulatory changes. As a result, PHMSA has expanded the Frequently Asked Questions published on the agency's Web site, participated with State partners in the first American Society of Mechanical Engineers' committee meeting to review the new standard and begun efforts toward developing new standards focused on new construction.
This evaluation will identify current and future business and technology performance gaps that inhibit PHMSA's ability to efficiently, effectively, and reliably execute mission activities.
The primary goal of this process evaluation is to better align the Information Technology (IT) portfolio with PHMSA's business model in order to provide the greatest value while reducing redundancies and costs and creating a more efficient IT services delivery program. We have conducted a significant amount of pre-program review activities in preparation for the IT evaluation. These activities include the development of a draft PHMSA Strategic Plan, identification of the PHMSA IT portfolio, mapping of IT systems to PHMSA lines of business, and the identification of related costs. These pre-program activities have already led to the discovery of redundant IT systems and reinforced our assumption that the current solutions are not meeting business expectations and/or not providing maximum business value.
On September 12, 2007, PHMSA awarded a contract to a third party vendor to assist in completing the evaluation effort. The evaluation will use the following approach:
The expected completion date for this evaluation is in early 2008. Results from the evaluation will be reported in next year's PAR.
In 1997, the White House Commission on Aviation Safety and Security issued a challenge to the FAA and the aviation industry - to reduce the air carrier fatal accident rate by 80 percent in ten years. This challenge became known as the Safer Skies program.
The purpose of this program evaluation is to determine how successful the Safer Skies program, which FAA began in 1998, was in increasing aviation safety. This outcome evaluation looked at the three main areas of emphasis for Safer Skies - Commercial Aviation, General Aviation and Cabin Safety. These areas focused on improved data and analysis, as well as improved human factors in operations and maintenance.
The results of this outcome evaluation, for each of the evaluation's focus areas, were as follows:
Commercial Aviation - In reviewing Commercial Air Carrier Fatal Accidents, the ten-year target called for an 80 percent reduction in the commercial air carrier fatal accident rate. Although we did not achieve the target set ten years ago, by the end of FY 2007, FAA has achieved a rate of 0.022 fatal accidents per 100,000 departures - a 57 percent drop. While we did not make this ambitious target, this is a significant reduction.
General Aviation Fatal Accidents - A review of the data from the three years prior to the development of Safer Skies (1994-1996) shows an average of 418 fatal general aviation accidents per year. From 2004 to 2006, the data indicate an average of 353 fatal general aviation accidents per year - approximately a 16 percent decrease.
Cabin Safety Commercial Fatalities - The three years prior to the development of Safer Skies (1994-1996) saw an average of 269 deaths per year, with 45 average deaths per fatal accident. A review of the FYs 2005 - 2007 fatality data indicates an average of 29 deaths per year, with 11 average deaths per fatal accident. This is a reduction of 89 percent and 76 percent, respectively.
The transformation of the aviation industry in both complexity and size has dramatically changed. This should also be reflected in how FAA conducts surveillance and measures progress. To date, “diagnostic surveillance,” looking at the causes of accidents after the fact, has been effective. However, this method of surveillance will result in an unacceptable level of fatal accidents as traffic doubles or triples by 2025. The results of the evaluation yielded two recommendations for FAA - modify the commercial fatal accident rate to address fatalities and develop a system safety approach.
The Aviation Safety organization developed a new commercial air carrier fatal accident rate performance measure (Fatalities per 100 million enplanements) and is in the process of developing a Safety Management System policy for FAA in FY 2008.
The Maritime Administration planned to have an independent auditor conduct an impact assessment of the Maritime Security Program in FY 2007. However, funding was not available for this project, and as a result, a decision was made to defer the evaluation. The Maritime Administration plans to complete this evaluation in FY 2008. Results from this evaluation will be reported in the FY 2008 DOT Performance and Accountability Report.
NHTSA has encouraged States to aggressively enforce laws affecting the safety of motorists on the Nation's highways. These laws include mandating the use of seat belts in motor vehicles and discouraging drivers from operating a motor vehicle while impaired. As part of this enforcement effort, the “Click It Or Ticket” (CIOT) campaign was established to promote seat belt use and the “Drunk Driving: Over the Limit; Under Arrest” campaign was established to discourage impaired driving.
Bergen County police officer Jeff Roberts, right, hands a seat belt safety pamphlet to a driver during a “Click It or Ticket” checkpoint stop near the entrance to the George Washington Bridge in Fort Lee, N.J., Monday, May 21, 2007. The checkpoint was one of three set up in New Jersey Monday to raise awareness of seat belt safety and give out $46 tickets to those motorists not wearing seat belts. (AP Photo/Mike Derer)
The purpose of this outcome evaluation is to evaluate high visibility national enforcement efforts and the resulting impact on driver's behaviors. The evaluation considered the use of paid advertisements focusing on seat belt enforcement, measured motorists' awareness of seat belt campaigns, and ultimately measured the change in seat belt use rate and the reduction in alcohol related fatalities.
NHTSA is in the process of conducting an evaluation of the 2006 national mobilization effort and is preparing a separate report on the May national seat belt mobilization and the Fall impaired driving crackdown. They will be completed and published in the Fall of 2008.
A Compliance Review (CR) is an onsite examination of a motor carrier's operations to determine the carrier's safety fitness. FMCSA, in cooperation with the Volpe National Transportation Systems Center, has developed an analytic model to measure the effectiveness of the CR in terms of crashes avoided, injuries avoided and lives saved. This tool provides FMCSA management with the information it needs to address the requirements of the Government Performance and Results Act (GPRA) of 1993, which obligates Federal agencies to measure the effectiveness of their programs as part of the budget cycle process. It also provides FMCSA and State safety program managers with a quantitative basis for optimizing the allocation of field safety resources. This analytic tool is known as the CR Effectiveness Model.
The CR Effectiveness Model shows the direct impact of compliance reviews on motor carrier safety, but not the “deterrent” effects (i.e., the effect on a carrier's behavior due to the potential of having a CR). The model is based entirely on “before and after” changes in the safety performance of motor carriers that received CRs. The model compares a motor carrier's crash rate in the 12-month period after a CR, to its crash rate in the 12-month period prior to that review. To make this comparison, the model uses: (1) crash data reported by the States, and (2) power unit data reported by carriers or obtained during CRs.
This impact evaluation focused on CRs conducted in 2004 to identify the extent to which the model could be used to identify the associated benefits. In 2004, 10,671 CRs were conducted. The analytical model was able to assess the impact of 8,042 of these reviews (some compliance reviews were removed from the model because the motor carrier receiving the CR was not active 12 months after the CR, had zero power units, or had crash and power unit data that did not pass edit checks designed to screen out erroneous data). Based on this assessment, it is estimated that during the period from 2004 to 2005, 2,720 crashes were avoided, 1,889 injuries were avoided and 107 lives were saved as result of performing compliance reviews in 2004.
FMCSA and its State partners conduct roadside inspections and traffic enforcements of large trucks and buses to ensure that the vehicles and drivers are operating safely within the Federal Motor Carrier Safety Regulations. FMCSA, in cooperation with the Volpe National Transportation Systems Center, has developed an analytic model to measure the effectiveness of roadside inspections and traffic enforcements in terms of crashes avoided, injuries avoided and lives saved. This tool provides FMCSA management with the information it needs to address the requirements of the Government Performance and Results Act (GPRA) of 1993, which obligates Federal agencies to measure the effectiveness of their programs as part of the budget cycle process. It also provides FMCSA and State safety program managers with a quantitative basis for optimizing the allocation of field safety resources. This analytic tool is known as the Intervention Model.
The Intervention Model is based on the premise that the two programs-Roadside Inspection and Traffic Enforcement-directly and indirectly contribute to a reduction in crashes. The model includes two components that are used for measuring these different effects, the direct effect model component and the indirect effect model component. Direct effects are based on the assumption that vehicle and/or driver defects discovered and then corrected at the roadside reduce the probability that these vehicles/drivers will be involved in subsequent crashes. In order to measure the direct effects of the intervention, the model assigns crash risk probabilities to each of the violations found at the roadside. The model then calculates direct-effect-prevented crashes according to the number and type of violations detected and corrected during the intervention.
Indirect effects are the by-products of the carriers' increased awareness of FMCSA programs and the consequences that the programs could impose if steps are not taken to ensure and/or maintain higher levels of safety. In order to measure indirect effects, which are essentially changes in behavior involving driver preparation, practices and vehicle maintenance, the model calculates motor carrier responses to exposure to the programs, and the resulting reduction in potentially crash-causing violations.
Most recently, the model was implemented to measure program effectiveness during the 2005 activity year using March 31, 2006, data extracted from the Motor Carrier Management Information System (MCMIS). The number of inspections and the model results are shown below for 2005.
| Number of Inspections | ||
|---|---|---|
| Calendar Year | 2004 | 2005 |
| Roadside Inspections | 2,211,875 | 2,194,567 |
| Traffic Enforcement | 803,032 | 827,719 |
| Total Interventions | 3,014,907 | 3,022,286 |
| 2005 Traffic Enforcement Activity Level Results | |||
|---|---|---|---|
| Crashes Avoided | Injuries Avoided | Lives Saved | |
| Traffic Enforcement Activity | 3,416 | 2,369 | 127 |
| Roadside Inspection Activity | 3,216 | 2,230 | 120 |
| Combined Activity | 2,583 | 1,791 | 96 |
| Total | 9,215 | 6,390 | 343 |
The FRA Research, Development and Demonstration (RD&D) Program directly supports DOT's regulatory safety mission. The purpose of the program is to facilitate FRA's efforts to enhance the safety and efficiency of the Nation's rail system. In FY 1998, Congress directed FRA to expand its collaborative efforts with the Transportation Research Board (TRB) and to review its Research and Development (RD&D) program. The purpose of the TRB review was to validate how FRA's RD&D program activities supported the agency's safety mission and to identify opportunities to improve overall program performance. TRB continues to provide FRA with an annual review of its RD&D program, which helps inform agency leadership on the progress of its RD&D activities.
Federal Railroad Administrator Joseph Boardman speaks during a news conference at the CSX rail yard in Atlanta, Thursday, March 2, 2006. Boardman said, human factors are the leading cause of train accidents and the agency is researching railroad worker fatigue and is working to analyze close calls, incidents that nearly cause train accidents. (AP Photo/John Bazemore)
The TRB conducted peer reviews of FRA's research, development, and demonstration programs, and in doing so, focused on three major issues: safety, capacity, and efficiency. The review encompassed both freight and passenger rail activities and was conducted during a three-day workshop, which included interviews of FRA staff, presentations from FRA program managers, discussions and debate by industry and subject matter experts, and various other data gathering methods.
TRB's overall findings were positive. The Board concluded that ongoing research being pursued by FRA should continue and not be given a lower priority solely because of proposed new research. The committee did conclude that there are a number of additional research areas that should be explored, including:
TRB also made four recommendations designed to improve FRA's management of the RD&D program. FRA is in the process of developing a response to these recommendations, which would include an action plan to address the recommendations where appropriate.
The Safety Assurance and Compliance Program (SACP) was initiated in the mid-1990s to identify and resolve systemic safety issues on large multi-regional railroad systems through a “macro” approach (i.e., examining the railroad system, all at one time). SACP is credited with resolving hundreds of safety issues, many of which were very complex. Over time, however, SACP grew in several different directions and was criticized, internally and externally, for a variety of perceived shortcomings.
On July 29, 2005, the Associate Administrator for Safety issued a directive implementing a new element of the FRA safety program, Railroad System Oversight (RSO), to replace SACP. The SACP had served as a critical part of Railroad Safety Enforcement, so this directive reflected a significant modification of FRA's safety program.
Fundamental changes in the safety oversight process for large railroad systems were implemented as a result of this initiative. In all, these changes were designed to improve communication, efficiency, use of FRA resources, and to focus better on those issues and concerns of greatest importance to the FRA safety program.
This outcome evaluation of RSO, one year after its implementation, will assess the effectiveness of the program and determine if the anticipated operational improvements (e.g., improved resource utilization, better organizational communication) were achieved.
The evaluation will include the following activities:
We are currently conducting an evaluation of the Rail Safety Oversight group. The evaluation will be completed in November 2007 and evaluation results will be reported in the DOT FY 2008 Performance and Accountability Report.
This undated photo provided by the Insurance Institute for Highway Safety shows a side impact test on a 2006 Ford Five Hundred vehicle with optional side airbags. (AP Photo/Insurance Institute for Highway Safety)
Side impacts rank second only to frontal impacts as a cause of occupant fatalities in cars, light trucks, and vans. In 2003, over 9,000 fatalities, approximately 29 percent of all occupant fatalities in cars and light trucks occurred in crashes initiated by a side impact. Since the 1970's, NHTSA, the manufacturers, and others in the safety community have worked hard to reduce the fatality risk in side impacts, especially of the most vulnerable occupant, the “nearside” occupant: the driver in a left-side impact and the right-front passenger in a right-side impact.
In recent years, four tangible improvements in side impact protection have been implemented and were evaluated in this outcome evaluation:
Effect of side air bags for nearside occupants:
Torso air bags plus head-protection air bags reduce the fatality risk of nearside front seat occupants in single- and multi-vehicle crashes by 24 percent. Torso bags alone reduce the fatality risk by an estimated 12 percent. Through 2005, there were few vehicles equipped with head-protection air bags only (no torso bags), not enough for a separate statistical analysis. However, the results suggest that torso and head-protection air bags are both effective in nearside impacts and make approximately equal contributions to fatality reduction. Overall torso and head-protection air bags could have saved an estimated 1791 lives in calendar year 2003 if every passenger car, light truck, and van had been so equipped. In September 2007, NHTSA announced new upgraded side impact safety requirements for all passenger vehicles, which is expected to save over 300 lives and prevent nearly 400 serious injuries every year. The new standard requires auto manufacturers - for the first time ever - to provide head protection in side-impact crashes, as well as enhance other protections for passengers involved in such crashes.
Combined effect of improved structure, padding and side airbags:
Side impact protection could have saved an estimated 2,934 lives (nearside and far side occupants) in calendar year 2003 if every car on the road had been equipped with side air bags (head and torso), improved side structure and padding, and every light truck and van had been equipped with side air bags. The associated reduction in fatality risk for the combined effect is 42 percent in two-door passenger cars, 30 percent in four-door cars, and 15 percent in light trucks and vans. The complete NHTSA report can be found at http://dmses.dot.gov/docimages/p89/460665.pdf.
Safety Status Measurement System (SafeStat) is an automated analysis system developed for the Federal Motor Carrier Safety Administration (FMCSA). The system combines current and historical safety performance data to measure the relative safety fitness of interstate commercial motor carriers. SafeStat enables FMCSA to quantify and monitor the safety status of motor carriers and guides the deployment of resources to focus on carriers posing the greatest safety risk.
GAO and the DOT Office of Inspector General (OIG) conducted process evaluations of SafeStat and made several recommendations to FMCSA to improve the system. The reports issued by GAO and OIG were largely positive and indicate that FMCSA does a good job in identifying carriers that pose high crash risks thereby ensuring the thoroughness and consistency of compliance reviews (CRs).
GAO's assessments found that FMCSA could more effectively address fatalities due to crashes involving a commercial motor vehicle if it better targeted CRs to those carriers that pose the greatest crash risks. OIG's assessment found that although improvements have been made to the data relied upon in SafeStat; problems still exist with the reporting of crash data.
In June 2007, GAO published an audit report titled, “Motor Carrier Safety: A Statistical Approach Will Better Identify Commercial Carriers That Pose High Crash Risks Than Does the Current Federal Approach” (Report No. GAO-07-585), and recommended that FMCSA apply a negative binomial regression model to enhance the current SafeStat methodology. FMCSA believes that the approach looks promising but is concerned that using the binomial regression model may result in less emphasis on safety regulatory areas. FMCSA agreed to consider amending current policies to place increased CR priority on certain motor carriers that are identified by SafeStat as deficient in the area of prior crashes.
In August 2007, GAO published audit findings in a report titled, “Motor Carrier Safety: Federal Safety Agency Identifies Many High-Risk Carriers but Does Not Assess Maximum Fines as Often as Required by Law” (Report No. GAO-07-584), and recommended that FMCSA improve its targeting of carriers that posed high crash risks by prioritizing CRs for carriers with very poor scores (such as the worst five percent) in the accident safety evaluation area, so that these carriers will be selected for CRs regardless of their scores in the other areas. GAO recommended this approach should FMCSA decide not to implement the negative binomial regression model, as recommended in the report issued in June 2007. FMCSA agreed to implement GAO's recommended approach. In the longer term, FMCSA, under the CSA 2010 initiative, is moving towards new methods for better targeting its safety compliance resources which promises further improvement in identifying high-risk motor carriers.
In June 2007, OIG issued correspondence to Representative Thomas Petri regarding the quality of the underlying data used by SafeStat. The review, conducted at Representative Petri's request, found that, although improvements have been made, problems still exist with the reporting of crash data to FMCSA. While States are reporting more commercial motor vehicle crashes to FMCSA, OIG found anomalies that caused OIG to question the completeness of the non-fatal crash reporting. OIG recommended that FMCSA implement a new, more reliable estimate that would allow evaluation of non-fatal crash reporting, both nationally and state-by-state, before the Department makes all SafeStat scores available to the public. FMCSA acknowledged the need to develop a new, more reliable estimate and has begun work to implement the estimate.
The primary goal of the FMCSA is to ensure the safe operation of interstate motor carriers and hazardous materials shippers. A program evaluation was conducted to assess the safety effectiveness, program and process efficiency, and cost effectiveness of alternative inspection regimes for FMCSA's Motor Carrier Compliance Review (CR) Program. The alternative inspection regimes study evaluated four intervention tactics: educational initiatives, warning letters, off-site investigations, and focused on-site investigations. These four interventions are among the broad array of progressive interventions being considered by the Comprehensive Safety Analysis (CSA) 2010 program.
Currently the Agency's CR program can only inspect a small percentage of the motor carrier industry. One of CSA 2010's goals is to increase the Agency's influence over the safety behavior of motor carriers (measured in terms of education, visits, an array of disciplinary treatments, and warnings). Each of the four interventions were analyzed using quantitative data (when available) and qualitative data. The warning letter intervention showed the greatest promise of safety improvement. The educational initiative, which used the new entrant program as a proxy, helped support the belief that education has a positive impact on safety improvement. FMCSA obtained data from other agencies to evaluate the impact of educational initiatives and discovered that those that employed this tactic were convinced that education improved safety, though they lacked clear performance data to support their claims. Other agencies were also assessed to support the effectiveness of employing off-site investigations (evaluating data gathered using sophisticated information systems which record/monitor safety behavior without visiting the motor carrier site); the qualitative data also supported using this tactic. Finally, qualitative data indicated that focused reviews provided safety improvements and cost savings. As a result of our investigation of alternative inspection regimes, the study recommended that the CSA 2010 team proceed with the current model which includes educational initiatives, warning letters, off-site investigations, and focused reviews.
This evaluation was conducted by the Federal Transit Administration to comply with the requirements of public law 49 U.S.C. 5309(c)(6), to assess the costs, benefits, and efficiencies of public-private partnerships (PPP) for fixed guideway transit capital projects.
The scope of the evaluation includes the results of a comprehensive review of available literature on PPPs and the results of large-scale transit projects developed as PPPs within the continental U.S. since the year 2000. The evaluation considered performance factors that determine a sponsor's satisfaction with a PPP, including facets such as total project costs, project delivery timeframe, overall cost-effectiveness and service delivery.
The methodology used included a literature review of PPPs in surface transportation, specific transit experience with PPPs both in the United States and other developed countries, as well as profiles and interviews with transit project sponsors using PPPs in the United States since 2000.
The results of the evaluation were that many transit agencies are considering PPPs to leverage public resources, lower costs, improve services, and transfer risks associated with fixed guideway development, financing, operations, and maintenance. They are also considering PPPs for capital replacement, expansion and program management. Transit PPPs are likely to attract growing private sector interest including transit-oriented development, joint development, and multimodal development. These additional PPP approaches can help transit agencies increase revenues, reduce project costs, or both by tapping into the resources of economic development on a proactive basis. Multimodal PPPs enable transit agencies to combine their resources with resources from other modes that have joint needs for related infrastructure improvements, such as highways, railroads, or airports.
The evaluation found that in certain instances, the increased involvement of the private sector may not prevent a project from experiencing cost or schedule overruns or quality problems. While the involvement of the private sector in a transit capital project can help improve the cost-effectiveness and timeliness of project delivery and provide other benefits in terms of risk transfer and access to financial markets, it is not a guarantee of successful delivery. Greater involvement by the private sector does not make a transit project of dubious feasibility automatically become feasible.
Recommendations resulting from this evaluation include: