Railroad-Highway Grade Crossing Handbook - 8 Evaluation of Projects and Programs

VIII

Evaluation of Projects and Programs

An integral part of any highway-rail grade crossing improvement program is the evaluation of individual projects and the overall program. The Federal-Aid Policy Guide (FAPG) specifies that each state's highway safety improvement program should include an evaluation of the program. This evaluation component is to include a determination of the effects the improvements have in reducing collisions, collision potential, and collision severity. This process should include:

•    the cost of and the safety benefits derived from the various means and methods used to mitigate or eliminate hazards;

•    a record of collision experience before and after the implementation of a highway safety improvement project; and

•    a comparison of collision numbers, rates, and severity observed after the implementation of a highway safety improvement project with the collision numbers, rates, and severity expected if the improvement had not been made.

In addition, the evaluation program is to include an annual evaluation and report of the state's overall safety improvement program and the state's progress in implementing the individual federal programs, such as the Section 203 crossing program.

Evaluation is an assessment of the value of an activity as measured by its success or failure in achieving a predetermined set of goals or objectives. The ultimate goal of evaluation is to improve the agency's ability to make future decisions regarding the improvement program. These decisions can be aided by conducting formal effectiveness and administrative evaluations of ongoing and completed improvement projects and programs.

In the Highway Safety Evaluation: Procedural Guide, two types of evaluation are addressed:

effectiveness evaluation and administrative evaluation. These two types will be discussed in this chapter only in sufficient detail for the user to be aware of the need and the procedures. However, the reader should refer to the procedural guide for more details. Also, the following references provide more useful information on safety evaluation procedures:

•    Lunenfeld, H. Evaluation of Traffic Operations, Safety and Positive Guidance Projects. Washington, DC: Federal Highway Administration (FHWA), Report No. FHWA-l0-80-1, October 1980.

•    Tarrants, W.E. and C.H. Veigel. The Evaluation of Highway Traffic Safety Programs. Washington, DC: National Highway Traffic Safety Administration, Report DOT-HS-80525, February 1978.

•    Council, F.M., et al. Accident Research Manual. Washington, DC: FHWA, Report FHWA/RD-80/016, February 1980.

•    Berg, W.D. Experimental Design for Evaluating the Safety Benefits of Railroad Advance Warning Signs. Washington, DC: FHWA, Report FHWA-RD-7978, April 1979.

A. Project Evaluation

Highway-rail grade crossing improvements that have as their objective the enhancement of safety should be evaluated as to their effectiveness. This can be done for individual projects and should be done for the overall improvement program. An effectiveness evaluation for safety purposes is the statistical and economic assessment of the extent to which a project or program achieves its ultimate safety goal of reducing the number and/or severity of collisions. It also can be expanded to include an assessment of the intermediate effects related to safety enhancement. The latter type of evaluation becomes particularly relevant for crossings because the low number of collisions occurring at a crossing may preclude any meaningful collision-based evaluation of individual crossings or a small number of them.

The procedural guide lists seven functions that should be followed in conducting an effectiveness evaluation:

•    Develop an evaluation plan.

•    Collect and reduce data.

•    Compare measures of effectiveness.

•    Perform statistical tests.

•    Perform economic analyses.

•    Prepare evaluation documents.

•    Develop and update a database.

The essential elements of the principal functions are described below.

The evaluation plan addresses issues such as the selection of projects for evaluation, project purposes, evaluation objectives and measures of effectiveness (MOEs), experimental plans, and data requirements.

Although it would be desirable to evaluate all improvement projects, manpower and fiscal capabilities do not always permit this. Consequently, when selecting projects for evaluation, the following factors should be considered:

•    Improvement types that are questionable as to their effectiveness.

•    Projects that have sufficient data necessary for statistical analysis.

•    Projects that are directly related to collision reduction.

If the number of collisions occurring before the improvement is too few to allow a significant reduction of collisions to occur, the project may be evaluated along with other similar projects. This is frequently the situation with crossings because they experience very few collisions. If projects are aggregated for evaluation, it is essential that:

•     Countermeasures for each be identical.

•     Types of locations be similar.

•     Project purposes be similar.

The experimental plan selected should be consistent with the nature of the project and the completeness and availability of data. The most common experimental plans for evaluating safety improvement projects are a before-and-after study with control sites and a before-and-after study.

The most desirable MOE for crossing safety improvements would be the reduction of collision frequency or severity. However, because a long period of time may be required to amass an adequate sample size, especially for individual projects, evaluations can be made based on other measures such as:

•    Traffic performance—speed, stopping behavior, and conflicts; or

•    Driver behavior—looking, compliance, and awareness.

The evaluation plan describes the types and amounts of data necessary for the evaluation. Data for the “before” situation could be obtained from the engineering study (see Chapter III) used to assist in determining the crossing problem and appropriate improvement. Additional data, if not available from historical records, will have to be collected before the improvement is made. If the MOE involves collision data, several years of data would be required. Traffic and driver behavior data can be collected four to six weeks after project implementation.

The effect of the project(s) on the selected MOE must be determined. Computations are made to determine the expected value of the MOE if the project(s) had not been implemented, and the difference between the expected MOE and the actual observed value of the MOE. This difference should then be tested to determine if it is statistically significant.

An important objective of an effectiveness evaluation is to obtain a complete picture of how well the completed project is performing from a safety standpoint. Economic analysis provides another perspective. From such analysis, an assessment of cost and collision reduction effects, in combination, may be made. This aspect of an evaluation is very important because it is possible to have a very effective project that is cost-prohibitive in terms of future use under similar circumstances.

There are many economic analysis techniques. The two most commonly used for evaluating completed highway safety improvement projects are the benefit-cost and cost-effectiveness methods.

An effectiveness database is an accumulation of project results that are directly usable as input to future project selection. The database:

•    Contains pertinent information on the collision-reducing capabilities of countermeasures and/ or projects.

•    Must be continually updated with new effectiveness evaluation information.

•    Should only contain evaluation results from reliable and properly conducted evaluations.

With such a database, collision reduction factors can be established and refined over time. These factors in turn can be used in determining the most cost-effective improvements.

B. Program Evaluation

The preceding section outlined the process for conducting evaluations of one or more improvement projects. This evaluation process can and should be applied to the entire crossing improvement program or components of it. The entire program would consist of activities including physical improvements to the crossing, changes in railroad or highway traffic operations, and changes in law enforcement and driver education.

Throughout the program, it may be useful for the policy-maker to identify whether certain specific program subsets are effective. These program subsets could include types of improvements such as:

•    Installation of flashing lights.

•    Relocation of crossing.

•    Illumination.

•    Sight distance improvements.

•    Combinations of two or more types.

The steps and procedures in conducting the program (or subset of the program) effectiveness evaluation are essentially the same as for projects. FHWA's Highway Safety Evaluation: Procedural Guide should be referred to for details.

C. Administrative Evaluation

This evaluation is the assessment of the scheduling, design, construction, and operational review activities undertaken during the implementation of the crossing improvement program. It evaluates these activities in terms of actual resource expenditures, planned versus actual resource expenditures, and productivity.

In the FHWA procedural guide, eight steps are recommended for administrative evaluation:

•    Select evaluation subjects.

•    Review project (program) details.

•    Identify administrative issues.

•    Obtain available data sources.

•    Prepare administrative data summary tables.

•    Evaluate administrative issues.

•    Prepare and distribute the evaluation report.

•    Develop and update database.

D. References

Berg, William D. Experimental Design for Evaluating the Safety Benefits of Railroad Advance Warning Signs. Washington, DC: Federal Highway Administration (FHWA), Report FHWA-RD79-78, April 1979.

Council, F.M., et al. Accident Research Manual. Washington, DC: FHWA, Report FHWA/RD-80/016, February 1980.

Federal-Aid Policy Guide. Washington, DC: FHWA, updated periodically.

Goodell-Grivas, Inc. Highway Safety Evaluation: Procedural Guide. Washington, DC: FHWA, FHWA-TS-81-219, October 1980.

Lunenfeld, H. Evaluation of Traffic Operations, Safety and Positive Guidance Projects. Washington, DC: FHWA, Report No. FHWA-10-80-1, October 1980.

Tarrants, W.E. and C.H. Veigel. The Evaluation of Highway Traffic Safety Administration. Report DOT-HS80-525, February 1978.