![]() ![]() |
![]() ![]() |
Planning |
![]() |
Evaluation |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
Guidance for the Preparation of TCSP Evaluation Plans - June 2001Table of Contents
List of FiguresList of Tables
1. Importance of EvaluationThe purpose of the Transportation and Community and System Preservation (TCSP) Pilot Program is to fund innovative projects that will increase the knowledge of the costs and benefits of different approaches to integrating transportation investments and strategies, community preservation, land development patterns, and environmental quality. Planning and implementation projects may be undertaken at the neighborhood, local, metropolitan, State, and regional levels by States, local governments, and metropolitan planning organizations (MPOs) working in cooperation with non-traditional partners. The TCSP is a pilot program explicitly designed to encourage innovative strategies and techniques, the results of which can then be used by other public and private organizations throughout the country. While TCSP funding is not sufficient to implement projects on a nationwide basis, all organizations nonetheless will benefit by being able to easily tap into the experience of others to learn what might be applicable for their own situations and how these new transportation strategies and techniques can be most effectively implemented. To accomplish this learning and the desired resultant transfer of experience, the evaluation of individual projects is a key component of the TCSP program. Evaluation of projects which are new or experimental in character will indicate the success of various activities at achieving the desired transportation, community, and system preservation objectives. The lessons learned from this process will be used in evaluating the overall TCSP program and will help develop more effective TCSP projects in the future. As a result, the TCSP program will provide an important nationwide learning experience. In keeping with the TCSP program's emphasis on learning and evaluation, grant applicants should bear in mind the following points:
In presenting this guidance for the preparation of evaluation plans, it is recognized that, "one size does not fit all." This guidance, therefore, provides ideas for evaluation plans rather than a mandated approach. [Top of Section, Table of Contents, Top of Document] 2. Grantee Roles and ResponsibilitesEvaluation PlanTCSP grantees are responsible for conducting a systematic evaluation of their TCSP project. Each grant application should include an evaluation plan which describes how the grantee proposes to evaluate the project. This will assist in demonstrating the applicant's commitment to the evaluation component. The strength of the evaluation component, including identification of resources required, will be an important factor in the selection of final grant awards. FHWA will use the results from individual evaluations, in conjunction with other overall program evaluation criteria and methods, in assessing the overall effectiveness of the TCSP program. As results and lessons learned from individual TCSP grant awards become available and the overall program can be assessed, the FHWA will coordinate and disseminate results, tools, and information developed through the program. In the evaluation plan submitted, grantees should identify program goals and objectives, performance measures, measurement techniques, potential data sources, and schedule milestones. Proposals should identify existing sources of information which will be utilized (either qualitative or quantitative), and should also identify any new data collection efforts which may be required or useful for evaluating the effectiveness of the program. The evaluation plan also should contain clear roles, responsibilities, commitments by participants, and a budget estimate. The resources required for evaluation activities should be included in the overall grant budget proposed for the project. As a component of the TCSP program evaluation, a grant workshop is planned for the Spring of 1999, at which grantees will share experiences and initial results from their projects. Budgets for grant applications should include travel for the key investigator to this workshop, as well as a second such conference, as part of the evaluation component. Assistance with EvaluationThe remainder of this document provides guidance relating to the development of an evaluation plan. The purpose of this guidance is to provide ideas rather than a mandated approach, and agencies should not be discouraged from applying for TCSP program funding simply because they lack expertise in particular evaluation methods. It is more important that grant applicants commit to undertaking a systematic evaluation, including the designation of project resources, than they demonstrate proficiency in any particular evaluation method. Grant applicants not already having the desired level of in-house evaluation expertise may want to consider working in cooperation with another agency or a university. [Top of Section, Table of Contents, Top of Document] General Approach to EvaluationThis section provides an overview of how to develop an evaluation plan, with more detail on how to structure and conduct the evaluation described in Section 4.0. In particular, Section 4.0 identifies specific techniques that may be used, issues to consider, and key questions to ask in evaluating a TCSP project. Steps in Developing an Evaluation PlanGrant applicants are encouraged to take the following steps in developing an approach to project evaluation, as illustrated in Figure 1: Figure 1. Steps in Developing an Evaluation Plan
Table 1. Examples of TCSP Project Goals and Objectives
Once potential performance measures, data sources, and evaluation methods have been identified, an overall evaluation plan should be developed for collecting and analyzing the required information. This includes identifying the individual work tasks required to carry out the evaluation and establishment of the associated budget and timeline for these tasks. What Should Be Evaluated?A TCSP evaluation should focus on identifying both the magnitude and the distribution of the costs and benefits of a project, and on those aspects of the planning and implementation process that will be useful to other organizations in deciding whether or not to implement similar strategies. Thus, evaluations can focus on three different aspects of a TCSP project: process, products, and outcomes. Appropriate goals and objectives, performance measures, and evaluation methods will differ for each, as will the timeframe over which the evaluation is conducted.
These three aspects of a project are interrelated and important to the evaluation of a TCSP project. Outcome goals are of ultimate interest to society, but achievement of process and product goals can indicate the likelihood of success at achieving the desired outcomes. Process and product goals are also desirable for their own sake. For example, an open and participatory process is important for ensuring that all viewpoints and potential impacts are considered. The involvement of non-traditional partners will help to identify strategies that encourage private sector development patterns that are consistent with the goals of the TCSP program. Examining the linkages among process, product, and outcome also can be useful. For example, desirable outcomes can be facilitated by the relationships developed during a planning process. Conversely, difficulties encountered during implementation may be traceable to the unintentional omission of an important factor during the planning stage. Finally, evaluation of all aspects of a project serves as an important learning tool, helping to identify both successful and unsuccessful approaches to a problem. Evaluation ReportsAn initial evaluation plan is to be included by an applicant as part of the application for a TCSP grant. This initial plan then may be refined in negotiating the terms of a grant awarded to the applicant. While the evaluation plan is expected to cover the basic approach proposed for evaluating a TCSP planning or implementation grant, the details of an evaluation plan, such as the statistical basis for a stratified sampling plan, may not be fully developed until after a project is actually underway. The evaluation activities associated with a TCSP grant should result in one or more reports. The purpose of these reports is to provide the information needed by other organizations throughout the country to decide whether similar projects would be beneficial within their jurisdictions, and how they should go about planning or implementing this particular kind of action. The initial evaluation report should document the process by which the TCSP grant project was developed or implemented, as well as the final product of the grant. This report can be produced shortly after completion of the project. Initial information on the results or outcomes of the project may also be available soon after completion and can be documented in this initial evaluation report. It is possible, however, that the full impacts of a project will not occur immediately, and that additional documentation of project outcomes will be appropriate in the future as data on longer-term impacts become available. The proposed approach to reporting should be explained in the evaluation plan portion of the grant application. [Top of Section, Table of Contents, Top of Document] 4. Detailed Evaluation GuidanceThis section provides more detailed guidance on evaluating the process, product, and outcomes of TCSP projects. For process and product evaluations, key questions for obtaining information as background to the evaluation are identified. For outcome evaluations, specific techniques and issues to consider in either estimating or measuring the impacts of the TCSP project are identified. For all three types of evaluations, examples of goals and objectives, performance measures, and evaluation methods relevant to TCSP projects are provided. It is important to identify in each case a baseline from which a change is being determined. For a process evaluation, this can be simply a comparison of the new or TCSP planning process with the existing or traditional approach. For a product evaluation, this can include an assessment of how the final project differs from what was initially proposed. Baseline considerations in estimating project outcomes include issues of time scale and differentiating project impacts from parallel changes in other significant factors, as discussed in Section 4.3. Process EvaluationEvaluation of the process by which the TCSP plan or project was produced or implemented can serve a number of useful functions. Process evaluation can identify reasons for success or failure of the plan or project as well as specific strategies and tactics which were most effective. Evaluation of specific aspects of the process, such as who participated and their respective roles, also can help indicate how likely the product is to achieve success. For example, extensive participation of a variety of affected parties or groups may mean that the project is more likely to be successful, since potential obstacles and stumbling blocks can be resolved. A number of techniques can be used to gather information for evaluating the process, including:
Questions that can be asked as a basis for evaluating the process include:
Documenting answers to the above questions can determine the degree to which the process met its defined goals and objectives. Some process-related goals and objectives for the TCSP program, as well as associated performance measures, are shown in Table 2. Local agencies may also hold other goals and objectives for activities carried out under the TCSP program. Documenting the answers to these questions also will help in identifying circumstances or actions that influenced the level of success of the final product. Table 2. TCSP Process Evaluation
Evaluation of improved linkages to metropolitan or statewide planning process, as encouraged by TEA-21, is of particular importance, although this may not be relevant to all TCSP grants. As applicable, grantees might evaluate their ability to improve connections through the funded project with the broad metropolitan or statewide transportation planning processes at the center of TEA-21. Linkages to the planning process can be flexible, and could be demonstrated, for example, by:
With respect to the public involvement process for transportation planning in particular, federal guidelines suggest the following desirable outcomes of public involvement (Reference: A Guide to Metropolitan Transportation Planning Under ISTEA: How the Pieces Fit Together. U.S. Department of Transportation, Washington, D.C., 1995.):
Product EvaluationProduct evaluation focuses on what was produced by the planning or implementation activity. A description of the project as it was actually produced or implemented can serve as an interim step in identifying the likely outcomes or impacts of the project. Some general questions that can be asked about the product include:
Evaluation of the product of a TCSP activity will differ significantly depending on whether the activity is a planning or implementation grant. In the case of an implementation activity, product evaluation can focus on describing what was actually built, or what service was developed, and why it is significant. In the case of a planning activity, product evaluation will focus on the content of the plan, agreement, etc. (e.g., what will be achieved if the plan is implemented or the agreement carried out); adoption of the plan; and on provisions to ensure successful implementation of the plan or agreement. While development of the plan or project consistent with the original scope of work, timeline, and budget may be a criterion, this should not limit flexibility in making mid-course modifications to a project. As planning and implementation progresses, it is possible that changes to the project may be incorporated that result in an improved product compared to the original proposal. Table 3 shows examples of goals and objectives and performance measures for evaluating the product of a planning grant. Table 4 shows examples of goals and objectives and performance measures for evaluating the product of an implementation grant. Table 3. TCSP Product Evaluation: Planning Grant
Table 4. TCSP Product Evaluation: Implementation Grant
Outcome EvaluationOutcome evaluation focuses on determining the effectiveness of the project at achieving particular transportation, community, and system preservation objectives, such as reductions in emissions or preservation of open space. Measuring the outcomes of a project is, in many ways, the most difficult aspect of evaluation. Numerous factors must be considered, such as distinguishing the impacts of the program from other concurrent changes and identifying the time scale over which impacts occur. Measurement of outcomes, however, is ultimately of critical importance to determining whether a project is worthwhile. Therefore, grant applicants are encouraged to give careful thought to how the impacts of the proposed programs can be directly assessed. Applicants are encouraged to seek agreement with both traditional and non-traditional partners regarding the specific set of outcome measures to be evaluated. This section provides guidance regarding issues to consider in evaluating the outcomes of projects funded through TCSP. Section 5.0 provides additional references on how to design an evaluation program and implement specific evaluation methods. Approaches to Measuring OutcomesThree general approaches may be taken to measuring the outcomes of a project:
As appropriate, grant applicants should identify a balanced set of techniques that allow evaluation of the economic, environmental, mobility, and social equity effects of strategies or investments. In developing proposals, grant applicants are encouraged to predict - at least from a qualitative standpoint - the potential impact of the proposed project on each of the outcome performance measures which have been identified. Proponents are also encouraged to predict impacts on a quantitative basis, using available modeling or sketch planning tools, although in many cases appropriate tools may not exist or may not be readily usable. The applicant also should develop a plan for measuring the impacts of the project once it has been implemented as part of the evaluation plan in the applicant's proposal. Ideally, this plan will include data collection and/or analysis which is capable of quantifying the impacts of the project on identified performance measures. It also may include, in some situations, development or refinement of analytical models to predict the impacts of the project. In many cases, however, it is likely that accurate quantitative measurements or forecasts will either be difficult to obtain or will not be relevant to the type of project being implemented. In this case, qualitative assessments should be performed in order to gauge the magnitude and nature of project impacts. General Measurement IssuesImportant issues to consider in designing an evaluation plan - whether quantitative or qualitative - include:
Collection of before-and-after data on both the affected population and control groups can be a particularly effective means of isolating the effects of a program, as illustrated in Figure 2. Figure 2. Use of Control Group in Before-and-After Data Collection ![]() Data collection plans, where possible, should account for seasonal fluctuations in the variables being measured, in addition to identifying longer-term trends. For example, many areas experience higher levels of pedestrian and bicycle activity in summer than in winter.
Available Evaluation Methods and Data SourcesTable 5 illustrates examples of outcome-related goals and objectives of the overall TCSP program, along with associated performance measures and methods for evaluating these measures. These performance measures are provided as examples and may not be relevant to all projects or measurable in all situations. Grant applicants are encouraged to define their own short list of meaningful, performance measures, as well as those goals and objectives which may be important locally. Applicants are further encouraged to identify the most appropriate and feasible evaluation methods for developing these performance measures. Table 5. TCSP Outcome Evaluation
Table 6 identifies potential existing data sources that can be used for project evaluation. Table 7 identifies methods for collecting new data as well as applications for each method. Table 6. Potential Existing Data Sources for Evaluation
Table 7. Methods for Collecting New Data
[Top of Section, Table of Contents, List of Tables, Top of Document] 5. Evaluation ReferencesThe following documents provide additional guidance on designing and implementing a data collection and evaluation plan. References also are provided on qualitative analysis methods and on the design of planning processes. In addition to addressing generic evaluation issues and methods, many of these documents describe evaluations of specific transportation programs. Transportation-Related Data Collection, Evaluation, and Experimental DesignInstitute of Transportation Engineers. Manual of Transportation Engineering Studies. H. Douglas Robertson, ed. Prentice Hall: Englewood Cliffs, NJ (1994). This reference manual discusses data collection methods for traffic volumes and speeds, public transportation, pedestrian activity, goods movement, environmental impacts, and other transportation data. The manual also discusses general methodological issues including experimental design, survey design, and statistical analysis methods. Available through the Institute of Transportation Engineers bookstore at 525 School Street, S.W., Suite 410, Washington, D.C. 20024-2797; Phone: 202/554-8050; Fax: 202/863-5486; Internet: http://www.ite.org. Cambridge Systematics, Inc. and Barton Aschman Associates. Travel Survey Manual. Prepared for the U.S. Department of Transportation and U.S. Environmental Protection Agency, Publication No. FHWA-PL-96-029 (Manual) and FHWA-PL-96-030 (Appendices), (1996). This guidance manual discusses the design, implementation, and uses of various types of surveys used in transportation planning, including household travel surveys, vehicle intercept and external station surveys, transit onboard surveys, commercial vehicles surveys, workplace and establishment surveys, visitor surveys, parking surveys, and stated response surveys. The manual can be ordered from the U.S. Department of Transportation at: TASC Subsequent Distribution Office, Ardmore East Business Center, 3341 Q 75th Avenue, Landover, MD 20785; Fax: 301/386-5394; e-mail: SDS.Info@OST.DOT.GOV. Refer to complete title, Travel Survey Manual and Appendices, and publication numbers FHWA-PL-96-029 and FHWA-PL-96-030 when ordering. Richardson, Anthony, E.S. Ampt, and A.H. Meyburg. Survey Methods for Transport Planning. Wiley-Interscience Publications: New York, NY (1995). This book discusses elements in designing and implementing various types of surveys used in transportation planning. Specific elements include selection of survey method, sampling procedures, survey instrument design, survey administration, and data processing and analysis. Cambridge Systematics, Inc., Economic Impact Analysis of Transit Investments: Guidebook for Practitioners. Transit Cooperative Research Program: Report 35, National Academy Press, Washington, D.C. (1998). This reports presents 12 evaluation methods for use in evaluating the economic impacts of transit projects. The report describes uses of each method, advantages and disadvantages, data sources, examples, and provides guidance for selecting methods. Many of the methods and issues discussed are generically relevant to the evaluation of all types of transportation-related projects, as well as to the evaluation of impacts other than economic impacts. Transit Cooperative Research Program (TCRP) reports can be ordered through the Internet at http://www.nas.edu/trb/index.html or by writing: Transportation Research Board, National Research Council, 2101 Constitution Avenue, NW, Washington, D.C. 20418. Casey, Robert F. and John Collura. Advanced Public Transportation Systems: Evaluation Guidelines. Prepared by the U.S. Department of Transportation, Volpe National Transportation Systems Center, for the Federal Transit Administration, Publication Nos. FTA-MA-26-0007-94-2 and DOT-VNTSC-FTA-93-9 (January 1994). This report provides guidelines for evaluating Advanced Public Transportation Systems, including; identification of performance measures; techniques for collection, deriving, and analyzing data; issues in experimental design; survey methods and execution; and statistical methods. Much of the guidance is relevant to the evaluation of transportation programs in general. The report is available through the National Technical Information Service, Springfield, VA, and on the Internet at http://ntl.bts.gov/DOCS/ate.html. Qualitative Assessment TechniquesKrueger, Richard A. Focus Groups: A Practical Guide for Applied Research. Sage Publications: Thousand Oaks, CA (1992). Mishler, Elliot G. Research Interviews: Context and Narrative. Harvard University Press: Cambridge, MA (1986). Yin, Robert. Case Study Research: Design and Methods. Sage Publications: Beverly Hills, CA (1992). Planning ProcessesUnited States Department of Transportation. A Guide to Metropolitan Transportation Planning Under ISTEA: How the Pieces Fit Together. Publication No. FHWA-PD-95-031 (1995). Available on the Internet at: http://ntl.bts.gov/DOCS/424MTP.html United States Department of Transportation. Statewide Transportation Planning Under ISTEA: A New Framework for Decision-Making. Publication No. FHWA-PD-96-026 (1996). United States Department of Transportation. Metropolitan Transportation Planning Under ISTEA: The Shape of Things to Come (1997). United States Department of Transportation, Volpe National Transportation Systems Center, Enhanced Planning Reviews of 14 Metropolitan Areas, prepared for FTA and FHWA, 1991-1997. Available on the Internet at: http://www.fta.dot.gov Innes, Judith. Planning Through Consensus Building. Journal of the American Planning Association (Autumn 1996). Ozawa, Connie. Recasting Science: Consensual Procedures in Public Policy-Making. Westview Press (1991). Susskind, Lawrence, and J. Cruikshank. Breaking the Impasse: Consensual Approaches to Resolving Public Disputes. Basic Books: New York, NY (1987). [Top of Section, Table of Contents, Top of Document] |