Program Evaluation

Program Description

Program evaluation is the systematic assessment of the strengths and weaknesses of a program with respect to implementation, impact, and long-term viability. The goal of evaluation is to provide documentation for accountability, direction for policy making, and guidance for designing programs that are feasible, useful, and effective. Evaluation begins by identifying important stakeholders: their interests in given programs, information needs, and success criteria. Logic models are designed to clarify a program’s implementation and theory of change. Performance metrics are then developed and methodologies created to generate the data and conduct analyses needed to meet those information needs. Good evaluation methodology recognizes that programs must be assessed relative to the criteria used to justify their funding, but that intermediate outcomes are also important, as are impacts that may evolve over a program’s life cycle. It also obtains and analyzes relevant information, reports results in a timely manner, and promotes appropriate and effective use of the findings. Sound program evaluation meets standards of utility, feasibility, propriety, and accuracy. The FRA is working closely with the greater evaluation community to build an internal evaluation capability consisting of human capital, a critical mass of exerts, formal and informal evaluation processes, tools, and methodologies. The goal is to improve its own evaluation, and to serve as a model for evaluation in the Department of Transportation. Some specific objectives of this program area include:

• on-going needs assessments for human factors research in the rail industry;
• continuing clarification and development of program performance goals based on industry needs;
• implementation and on-going refinement of a human factors project selection model;
• the development of processes, procedures and methodologies to improve the utilization, impact, and effectiveness of human factors research in the rail industry;
• the development of program evaluation plans and logic models to support  implementation of significant or large scale industry demonstrations;
• development of key indicators and program performance metrics to be used when assessing program outcomes and impacts; and evaluation capacity building, both internal and external to FRA. Click the link to see a list of program evaluation standards .

Current Projects

• Program Evaluation: Implementation and Impact Assessment Tools

Completed Projects

• 2008 Transportation Research Board Workshop #133: Increasing Utilization, Impact, and Effectiveness of Research and Development: Stakeholder Involvement Strategies in Evaluation
  (links to materials pending)

• Panel Session at 2007 American Evaluation Association Conference: Effectiveness and Impact of Federal Safety Risk Reduction Programs: Evaluation Experience and Lessons Learned from Three Government Agencies Efforts to Improve Industry Safety
  (links to materials pending)

• 2007 Transportation Research Board Human Factors in Transportation Workshop #141: Proactive Risk Management Safety Approaches for Managing Human-Factors-Caused Accidents in the Railroad Industry: Alternatives to Compliance
  (links to materials pending)

• 2005 Transportation Research Board Human Factors Workshop #133: Evaluating and Assessing Safety Education Program Impacts: A Benefit to All Modes
  (links to materials pending)

• 2004 Transportation Research Board Human Factors Workshop #128: Program Evaluation
  (links to materials pending)

• 2004 Safety Rules Revisions Impact on Safety Culture, Incident Rates and Liability Claims in the U.S. Railroad Industry: A Summary of Lessons Learned

• 2003 Safety Culture: TRB Human Factors in Transportation Workshop 105