SAFETEA-LU Intelligent Transportation Systems (ITS) Reporting and Evaluation Guidelines

I. Introduction

A. Background

The enactment of the Safe, Accountable, Flexible, Efficient Transportation Equity Act: A Legacy for Users (SAFETEA-LU) [Pub. L. 109-59, 119 Stat. 1809 (2005)] on August 10, 2005 represents the largest surface transportation investment in our Nation’s history. The two landmark bills that brought surface transportation into the 21 st century – the Intermodal Surface Transportation Efficiency Act of 1991 (ISTEA) and the Transportation Equity Act for the 21 st Century (TEA-21) – shaped the highway program to meet the Nation’s changing transportation needs. SAFETEA-LU builds on this foundation, supplying the funds and refining the programmatic framework for investments needed to maintain and grow our vital transportation infrastructure.

SAFETEA-LU addresses the many challenges facing our transportation system today – challenges such as improving safety, reducing traffic congestion, improving efficiency in freight movement, increasing intermodal connectivity, and protecting the environment – as well as laying the groundwork for addressing future challenges. SAFETEA-LU promotes more efficient and effective Federal surface transportation programs by focusing on transportation issues of national significance, while giving State and local transportation decision makers more flexibility for solving transportation problems in their communities.

The effective implementation of the ITS activities specified in Subtitle C of SAFETEA-LU will rely heavily on active and rigorous evaluations of those activities. Recognizing the evolving nature of the ITS Program, Congress included a mandate in SAFETEA-LU which requires the Secretary of Transportation (the Secretary) to issue guidelines and requirements for the reporting and evaluation of operational tests and deployment projects carried out under Subtitle C, Intelligent Transportation System Research. A prerequisite for the continuing support of decision makers addressing policy and investment issues will be a clear understanding of ITS effectiveness. At the Federal level, the Government Performance and Results Act [Pub. L. 103-62, 107 Stat. 285 (1993)] has established a formal process for program and budget planning. ITS benefits information will be needed for assessing the efficacy of Federal investment in ITS. At the project level, agencies and partners in public-private project initiatives require the means to monitor costs and system effectiveness on a continuing basis to support management of systems and operations.

SAFETEA-LU marks an evolution from TEA-21 where references to ITS were found primarily in Title V (Research). ITS is now referenced in several titles of SAFETEA-LU, however the Reporting and Evaluation Guidelines addressed in this document apply only to operational tests and deployment projects funded under the provisions of Title V, Subtitle C.

B. Legislative Requirements

The Congress has recognized the critical role of ITS evaluation in section 5305(i)(1)(A) of subtitle C of SAFETEA-LU that prescribes:

Subparagraph (A):
“IN GENERAL. -- The Secretary shall issue guidelines and requirements for the reporting and evaluation of operational tests and deployment projects carried out under this subtitle.”

Subparagraph (B):
“OBJECTIVITY AND INDEPENDENCE. -- The guidelines and requirements issued under Subparagraph (A) shall include provisions to ensure the objectivity and independence of the reporting entity so as to avoid any real or apparent conflict of interest or potential influence on the outcome by parties to any such test or deployment project or by any other formal evaluation carried out under this subtitle.”

Subparagraph (C):
“FUNDING. -- The guidelines and requirements issued under Subparagraph (A) shall establish reporting funding levels based on the size and scope of each test or project to ensure adequate reporting of the results of the test or project.”

Under the period governed by SAFETEA-LU, the ITS Program is conducting only field operational tests. Deployments, though specified by the legislation, will not be further addressed. The definition of field operational test is included under “Project Categorization” in Section II.A.

C. Objective

The objective of this document is to fulfill the “guidelines and reporting requirements” imposed on the Secretary by SAFETEA-LU, Title V, Subtitle C, Section 5305(i)(1)(A), (B), and (C) as referenced above.

D. Purposes of Reporting and Evaluation Guidelines

In order to satisfy the applicable Title V, Subtitle C criteria imposed by SAFETEA-LU, guidelines must be issued which accomplish the following purposes:

• Define the categories of projects carried out under Subtitle C, Intelligent Transportation System Research, to which the guidelines and reporting requirements are applicable.
• Define the provisions to ensure the objectivity and independence of the reporting entity so as to avoid any real or apparent conflict of interest or potential influence in the outcome by parties to any such field operational test, or by any other formal evaluation carried out under Subtitle C.
• Define the reporting funding levels based on the size and scope of each field operational test so as to ensure adequate reporting of the results of the test.
• Provide a general description of the procedures and requirements that project partnerships should expect to conduct when carrying out project evaluations.
• Provide access through Web site address and Web links to detailed evaluation procedures including examples of specific evaluation plans and reports. Guidelines in this document do not address detailed evaluation procedures, but they are linked to resources providing extensive detail and examples.

II. Terms of Reference

The establishment of evaluation guidelines for projects funded with Federal ITS funding requires identification of project types.

A. Project Categorization

Operational Tests:

In general, SAFETEA-LU, Section 5306 (a) specifies that “the Secretary shall carry out a comprehensive program of intelligent transportation system research, development, and operational tests of intelligent vehicles and intelligent infrastructure systems and other similar activities that are necessary to carry out this subtitle” (i.e., subtitle C). The legislation specifies that evaluations are to be done for “operational tests” and “deployment projects” carried out under this Subtitle (Title V, Subtitle C).

Therefore, projects subject to the provisions of Section 5305 (i) “Reporting” are:

• “Operational tests and deployment projects whose objectives fulfill the intent of Section 5306 (a), and
• Are carried out under this “subtitle” (i.e., Subtitle C.)

Under the period governed by SAFETEA-LU, the ITS Program conducts only field operational tests. Therefore, the following definition applies: A field operational test deploys one or more technologies in an operational setting under real-world conditions to test the hypotheses of the research question. It is designed to assess whether or not, or the extent to which, the technology application(s) achieve(s) the research objectives. The field operational test documents, and to the extent feasible, quantifies the test outcomes through evaluation.

These Guidelines and the provisions of Subtitle C do not apply to field operational tests, deployment projects and other research activities funded under other subtitles of SAFETEA-LU or the Federal Aid Program. Field operational tests or deployment projects funded under TEA-21 will conform to the TEA-21 Evaluation Guidelines (accessible at http://www.its.dot.gov/evaluation/eguide_tea21.htm).

B. Reporting Funding Levels

The purpose of this section is to identify two categories of evaluations for field operational tests based on allocated funding levels. In order to fulfill the mandate specified in Section 5305 (i) (1) (C ) of Title V, Subtitle C that these guidelines “shall establish reporting funding levels based on the size and scope of each test or project that ensure adequate reporting of the results of the test or project,” the following procedures prescribe the category of evaluation to be conducted based on allocated funding. The costs associated with producing a final report are addressed in II.C.3:

1. Self-Evaluations (sometimes referred to as Local Evaluations): Projects valued up to, but not exceeding $1 million. These projects will submit reports documenting the findings of self-evaluations (see II.C.1).
2. National Evaluations: Projects valued above $1 million. These projects will submit final evaluation reports documenting the results and findings of formal, in-depth independent evaluations (see II.C.2).
3. Project value includes total implementation costs and operational costs that are relevant to testing the defined research question, and to determine the quantitative or qualitative outcomes over the duration of the field operational test (this equates to the cost of the field operational test value). It includes Federal and matching funds and may span several fiscal years. Costs will not include initial research to develop the technology or application.

C. Evaluation Categorization & Field Operational Test Funding Allocations

For the purposes of these guidelines, there are two categories of project evaluations.

1. Self-Evaluations – Projects funded by title V, Subtitle C that are valued up to, but not to exceed $1 million will conduct self-evaluations. In the interests of sound management practices, the project partnerships managing projects that meet this funding level will conduct locally executed evaluations funded from project resources. These self-evaluations, also identified as local evaluations, should strive to follow the evaluation procedures outlined in the ITS Evaluation Resource Guide.
2. National Evaluations –National evaluations are sponsored and funded by the ITS Program. These are formal, in-depth evaluations of ITS field operational tests carried out under Title V, Subtitle C. These evaluations may be funded, partially or entirely, with Subtitle C resources allocated by the ITS Joint Program Office. The total budget for evaluation will be determined by the project partnership, in coordination with ITS JPO staff or the appropriate USDOT modal designee, and should be established in the context of evaluation cost experience encountered during the period in which ITS field operational tests have taken place. Factors discussed below should be considered.

The basic elements of a field evaluation (as distinguished from a technical evaluation conducted under controlled laboratory or test site environment conditions) whose objective is to determine the impacts of technology deployment on the transportation system are discussed in general terms in the ITS Evaluation Resource Guide. A brief examination of the essential components of a field evaluation serves to highlight its features. Major tasks include, but are not limited to:

• Development of an Evaluation Strategy that establishes the foundation for, and the overall approach to, the evaluation.
• Development of an Evaluation Plan that refines the evaluation approach established in the Evaluation Strategy, and that provides focus through the formulation of outcome-based hypotheses, and the identification of tests to be conducted.
• Development of Detailed Test Plans to guide the conduct of each test specified in the Evaluation Plan.
• Pre-and post-deployment data collection and analysis culminating in the drafting and review/coordination of a Final Evaluation Report by all stakeholders.

3. Funding Levels for Supporting Evaluations - Experience with evaluations over the course of the transportation acts preceding SAFETEA-LU (ISTEA and TEA-21), has demonstrated that evaluation activities conducted by qualified evaluation organizations require a commitment of resources on the order of 10% of total (Federal and matching) project funding. The final evaluation report is considered an integral component of the evaluation, and its cost is included in the amount allocated for evaluation. Therefore, it is desirable that projects should allocate 10% of total project value for evaluation.

4. Project Partnerships and Evaluation Oversight - Project evaluations will be conducted under the auspices of the project partnership. It is understood that the ITS JPO program manager who recommended the commitment of Subtitle C funds for a project (for the purposes of these Guidelines, a field operational test) is a member of the project partnership. National evaluations will be carried out in accordance with arrangements negotiated among the project partners. Depending on these arrangements, the evaluation may be closely monitored by the appropriate ITS JPO Program manager and/or a modal designee, or the oversight process may be delegated to the designated evaluation point of contact at the project partnership. The evaluation reporting procedures, however, must adhere to the outlines discussed in the ITS Evaluation Resource Guide.

III. Selection of Evaluating Organizations

A. General

Title V, Subtitle C, Section 5305 (i) (1) (B) requires that these guidelines “shall include provisions to ensure the objectivity and independence of the reporting entity so as to avoid any real or apparent conflict of interest or potential influence on the outcome by parties to any such test or deployment project or by any other formal evaluation carried out under this subtitle.”

B. Provisions for National Evaluations

The required provisions for the selection of organizations to conduct national evaluations are outlined as follows:

1. Organizations (entities) demonstrating appropriate technical qualifications are eligible for selection to perform a national evaluation of ITS field operational tests.
2. As an integral part of submitting documentation demonstrating organizational qualifications to perform a national evaluation, interested entities are required to include a statement certifying commitment to the conduct of a completely objective and independent evaluation. This statement must include a commitment ensuring that any subcontracting organizations will be required to adhere to the standard of objectivity and independence established by the evaluating entity.

• In the interests of avoiding not only real organizational conflicts of interest, but also the appearance of such conflicts, or potential influence on the outcome of evaluations, entities interested in pursuing evaluation opportunities must disclose the existence of current, or previous business relationships with the project partnership conducting the ITS JPO-sponsored project.
• Disclosure must include a description of the nature of previous business relationships, if applicable, and their duration. If there is a current business arrangement in effect in an area of activity unrelated to the ITS project, it must be described in sufficient detail to facilitate comprehensive understanding by all project partners. The appropriate ITS JPO Program manager will review the disclosure statement, and render a judgment. ITS JPO Program manager will communicate with the existing, or prospective contracting officer’s technical representative if appropriate, and the Office of Chief Counsel, to ensure that the provisions of these Guidelines are coordinated with procurement actions.
• The result of the assessment of previous and current business relationships should support a statement that there are no organizational conflicts of interest at the time of proposal review.
• The review of the project should also reveal no potential conflicts of interest arising from subsequent evaluation activities for all members of the team being proposed for the conduct of the evaluation task.
• As a part of the certification, the project partners will document their expectation that, in the event that an evaluation entity would be required to assess any technologies in which it has a financial or corporate interest, however remote, that entity will voluntarily recuse itself from that aspect of the evaluation. A financial or corporate interest is further defined to include the case where an entity interested in pursuing an evaluation role :
  • is the corporate parent, subsidiary or maintains a financial interest in the supplier of the technology that will be evaluated, or
  • is a supplier of a technology that will be evaluated, or
  • is a supplier of a technology that is in competition with another technology to be evaluated, or
  • if that entity holds an existing or pending patent for a technology that is similar to the one to be evaluated, or
  • if that entity is in the process of merging with, or being acquired by another entity that is a supplier, or
  • if it is a patent holder of the technology to be evaluated, then the entity must disclose the relevant conditions.
• Decisions relating to the need for the interested entity to recuse itself from evaluation activities will be made by the appropriate ITS JPO Program manager under advice from the Office of Chief Counsel.
• Any subcontractor involved in conducting the field operational test (for example, implementing technology) should be excluded from participating in the evaluation of that field operational test.
• A copy of the documentation demonstrating compliance with the provisions of Subtitle C, section 5305 (i) (1) (B) will be forwarded to the ITS JPO member of the project partnership, i.e., the appropriate JPO Program manager.

C. Provisions for self-evaluations

The guidelines for the conduct of self-evaluations include the following:

1. The evaluations should be conducted under the auspices of the project partners.
2. The project partners should form an evaluation team from project resources, and appoint an evaluation point-of-contact (the evaluation manager) who will manage the evaluation.
3. The evaluation team conducting a project self-evaluation is expected to use resources made available by project partners and Web-based tools identified in this document.
4. The process employed for forming evaluation organizations to conduct self-evaluations should attempt, to the extent possible, to adhere to the principles of objectivity and independence cited in Title V, Subtitle C, Section 5305 (i) (1) (B) of SAFETEA-LU.

IV. Administrative Guidelines

A. Report Submission Requirements

The operational tests of intelligent transportation systems infrastructure and deployment projects carried out under Title V, Subtitle C are sponsored by the USDOT Research and Innovative Technology Administration’s ITS JPO.

All projects funded by Subtitle C, whether independently evaluated by an evaluation entity or self-evaluated using project resources, are expected to deliver a final evaluation report. In the interests of facilitating final report delivery and posting on the Electronic Document Library, there are several options for submission.

The appropriate ITS JPO Program manager will determine if the final evaluation report should be posted on the JPO Electronic Document Library (EDL). If the program manager elects to have the reporting entity submit the report to the EDL directly, the following procedures apply:

1. Submit final evaluation reports and executive summary via email to itspubs@dot.gov (with CC to the appropriate ITS JPO Program manager). Self-evaluations must be submitted to itspubs@dot.gov with a CC to the ITS JPO Program Assessment Manager. The correct e-mail address will be provided on request by calling the ITS JPO Reception Desk @ (202) 366-9536.
2. Also send 3 copies in electronic format (e.g., CDs, diskettes, etc.) via package delivery service to the Publications Distribution Manager (for statutory transmittal to the National Technical Information Service (NTIS)):

Publications Distribution Manager
ITS Joint Program Office – HOIT
U.S. Department of Transportation
1200 New Jersey Avenue, SE,
Washington, DC 20590

3. Accompanying the report and evaluation must be an EDL profile sheet and NTIS form 1700.7, which can be downloaded from the EDL front page accessible at: (www.its.dot.gov/itsweb/welcome.htm)
4. Electronic submission must be compliant with section 508 of the Americans with Disabilities Act (the document must be accessible via keyboard). Acceptable formats are HTML and MSWord. Mac formats are unacceptable.

If the program manager elects to have the report submitted to him/her for final review prior to posting on the EDL, the same requirements apply, but the documents should be addressed to:

Intelligent Transportation Systems
Joint Program Office (HOIT)
Attn: __________________ Program Manager
U.S. Department of Transportation
1200 New Jersey Avenue, SE, HOIT
Washington, DC 20590

B. References

In lieu of incorporating detailed procedural guidance for the conduct of evaluations in this document, an ITS Evaluation Resource Guide has been developed. This comprehensive resource for supporting evaluation planning is accessible at http://www.its.dot.gov/evaluation/eguide_resource.htm.