Program evaluation is one of the three major elements of the Government Performance and Results Act (GPRA). The statute calls for agencies to use program evaluations to assess the manner and extent to which Federal programs achieve intended objectives. The statute further calls for an agency's Performance Plan to include a summary of the findings of program evaluations completed in the fiscal year covered in the report. Finally, GPRA calls for a schedule for future program evaluations to be presented in Strategic Plans.
In response to these mandates, DOT reports its completed program evaluations annually in “The Department of Transportation Performance and Accountability Reports.”[8] To the extent that the results of completed program evaluations, Program Assessment Rating Tool (PART) reviews, and the reports of the DOT Office of the Inspector General and the General Accountability Office illuminate how we might achieve results more effectively or address future conditions, we considered these in writing the strategies presented in this Strategic Plan.
Table 7 below presents the schedule for future DOT program evaluations. These evaluations represent a cross-section of DOT programs that must be well-managed in full support of budget-performance integration. DOT defines the methodologies used in Table 7 program evaluations as follows.
Impact Evaluations use empirical data to compare measurable program outcomes with what would have happened in the absence of the program. These represent the highest standard of program evaluations and are often the most difficult and expensive to construct and interpret.
Outcome Evaluations assess the extent to which programs achieve their outcome-oriented objectives. Outcome evaluations use quantitative methods to assess program effectiveness.
Process Evaluations assess the extent to which a program is operating as intended. While a true process evaluation will use objective measurement and analysis, it does not address the causal links between intervention and outcome.
Cost-Benefit and Cost-Effectiveness Analyses compare a program's outputs or outcomes with the costs to produce them. This type of analysis conforms to program evaluation when applied systematically to existing programs and when measurable outputs and outcomes are monetized.
Agency |
Strategic Goals[9] |
O E |
Methodology |
Title |
FY Completed |
||||
---|---|---|---|---|---|---|---|---|---|
RITA |
X |
Process - NRC Review and GAO Study |
RD&T Strategic Plan |
2006 |
|||||
FMCSA |
X |
Outcome - conducted by an independent auditor |
Commercial Driver's License Program |
2006 |
|||||
FMCSA |
X |
X |
X |
X |
X |
Process - conducted by an independent auditor |
Performance Linkages |
2006 |
|
FAA |
X |
Outcome |
Facility Security Program |
2006 |
|||||
PHMSA |
X |
X |
Outcome |
Retrospective assessment of benefits and impacts of the pipeline safety operator qualification regulations. |
2007 |
||||
PHMSA |
X |
X |
X |
Process - conducted by an independent auditor |
Assess the current information technology (IT) program to identify overlapping and redundant IT investments, systems and services. |
2007 |
|||
FAA |
X |
X |
Outcome |
Safer Skies |
2007 |
||||
MARAD |
X |
Impact assessment - conducted by an independent auditor |
Maritime Security Program |
2007 |
|||||
FAA |
X |
Outcome |
Operational Error Program |
2008 |
|||||
NHTSA |
X |
Outcome - conducted by an independent auditor |
Evaluation of National Mobilizations |
2006 2007 2008 |
|||||
FMCSA |
X |
Outcome - conducted by an independent auditor |
Compliance Review (CR) Effectiveness |
Annually 2006 through 2011 |
|||||
FMCSA |
X |
Outcome - conducted by an independent auditor |
Roadside Inspection and Traffic Enforcement Effectiveness |
Annually 2006 through 2011 |
|||||
FRA |
X |
X |
X |
X |
Process conducted by an independent auditor |
Review of FRA's Research, Development and Demonstration Programs |
Annually 2006 through 2011 |
||
FTA/FHWA |
X |
X |
X |
X |
Cost Effectiveness |
Infrastructure Investment Needs Report (Conditions and Performance Report) |
Biennially 2006 through 2011 |
||
FRA |
X |
X |
Outcome - conducted by an independent auditor |
Railroad Safety Enforcement |
2006 2007 |
||||
NHTSA |
X |
Outcome |
Side Impact Protection and Side Air Bags |
2007 |
|||||
FMCSA |
X |
Process - conducted by an independent auditor |
SafeStat Program |
2007 |
|||||
FMCSA |
X |
Process - conducted by an independent auditor |
Alternative Inspection Regimes |
2007 |
|||||
FTA |
X |
X |
Cost Benefit |
Public/Private Partnership Program |
2007 |
||||
PHMSA |
X |
X |
X |
Process |
Evaluation of outreach and training programs |
2008 |
|||
FMCSA |
X |
X |
X |
Process/cost effectiveness - conducted by an independent auditor |
State Division Effectiveness |
2008 |
|||
FMCSA |
X |
X |
Process/cost effectiveness - conducted by an independent auditor |
Motor Carrier Safety Assistance Program (MCSAP) |
2008 |
||||
FHWA/FTA |
X |
Cost effectiveness |
Infrastructure Investment Needs Report |
2008 |
|||||
FAA |
X |
Process |
Aircraft Delay Reduction Program |
2008 |
|||||
FMCSA |
X |
Process – conducted by an independent auditor |
Quality Assurance Review – Grants Management |
2008 |
|||||
FTA |
X |
Outcome |
Job Access and Reverse Commute Formula Grant Program |
2008 |
|||||
MARAD |
X |
Impact assessment - conducted by an independent auditor |
Maritime Education Programs |
2008 |
|||||
PHMSA |
X |
X |
Outcome Process |
Evaluation of the pipeline safety State grants program |
2009 |
||||
PHMSA |
X |
Process |
Evaluation of processing pipeline and hazmat enforcement cases |
2009 |
|||||
PHMSA |
X |
Process |
Evaluation of readiness among emergency responders |
2009 |
|||||
RITA |
X |
Process - conducted by an independent auditor |
UTC Program |
2009 |
|||||
FAA |
X |
Outcome |
Strategic Sourcing |
2009 |
|||||
FMCSA |
X |
Outcome – conducted by an independent auditor |
New Entrant Safety Audits |
2009 |
|||||
FMCSA |
X |
Cost effectiveness – conducted by an independent auditor |
Information Management Program |
2009 |
|||||
FTA |
X |
Outcome |
Contracted Para-transit Pilot Program |
2009 |
|||||
MARAD |
X |
Process – conducted by an independent auditor |
Ship Disposal |
2009 |
|||||
FMCSA |
X |
Outcome - conducted by an independent auditor |
Motor Coach Operations |
2010 |
|||||
RITA |
X |
X |
Cost Benefit |
BTS – Airline Program Web Filing |
2010 |
||||
FMCSA |
X |
Process |
Quality Assurance Review – Enforcement Activities |
2010 |
|||||
FHWA |
X |
X |
X |
X |
X |
Outcome |
Strategic Highway Research Program II—Report on Implementation of Results |
2010 |
|
FHWA |
X |
X |
X |
X |
X |
Process |
Strategic Highway Research Program II—Programmatic Evaluation |
2010 |
|
FAA |
X |
Process |
Streamlined Environmental Impact Statement Process |
2010 |
|||||
MARAD |
X |
Outcome - conducted by an independent auditor |
Cargo Preference – Food Aid Cargoes |
2010 |
|||||
PHMSA |
X |
X |
Impact |
Effectiveness of the pipeline safety integrity management program |
2011 |
||||
FAA |
X |
Outcome |
Runway Safety Program |
2011 |