Chapter 10
Evaluation and Assessment of RD&T

To ensure the effectiveness of RD&T, the Department continually assesses its research programs using three primary mechanisms: (1) systematic application of the Administration’s R&D Investment Criteria and Program Assessment Rating Tool (PART); (2) internal annual reviews of operating administration RD&T programs; and (3) external RD&T coordination and review. This evaluation strategy is shown in Table 10-1.

R&D Investment Criteria and Part

To guide the planning and management of research across the Federal Government, the White House Office of Management and Budget (OMB) has established three broad investment criteria for RD&T: relevance, quality, and performance. Each of the criteria has both prospective and retrospective elements:

Relevance. Programs must have complete plans, with clear goals and priorities; must articulate their potential public benefits; and must be relevant to national and customer needs. Agencies must assess relevance periodically through both prospective and retrospective independent review.

Quality. Programs must use clearly stated, defensible methods for awarding funding; those allocating funds through means other than a competitive, merit-based process must justify funding methods and document how quality is maintained.

Performance. Programs must maintain long-term objectives, with annual measures and targets, and define appropriate outputs, outcomes, schedules, and decision points. Agencies should express program results in terms of public benefits and document retrospective performance annually.

The R&D Investment Criteria incorporate established best practices for research evaluation as identified by the National Academy of Sciences, Government Accountability Office, and others. OMB uses the PART to assess how well agencies are implementing the criteria. Essentially a questionnaire, the PART measures all aspects of program planning, budgeting, and management. It has four sections: Program Purpose and Design. Assesses whether a program’s purpose and design are clear and sound.

Strategic Planning. Assesses whether the agency has established valid long-term and annual goals for the program.

Program Management. Assesses agency management of the program, including financial oversight and improvement efforts.

Program Results and Accountability. Assesses whether a program is meeting its annual and long-term goals and how effective the program is based on independent evaluations.

On the basis of the PART, OMB rates programs as Effective, Moderately Effective, Adequate, Ineffective, or Results Not Demonstrated. To date, OMB has used the PART to assess RD&T programs in FAA, FHWA, FRA, NHTSA, and PHMSA. Each program has been rated Moderately Effective or better and has implemented all PART recommendations. The PART results for FTA’s research program will be released in February 2007.

Annual Internal Program Review

Within the Department, the primary mechanism for ensuring implementation of the R&D Investment Criteria and PART is the annual review of modal RD&T programs.

Through the RD&T Planning Council and RD&T Planning Team, RITA works with the operating administrations to conduct annual reviews of RD&T programs to assess how well they are applying the criteria, particularly whether programs are evaluated according to established best practices. Members of a Program Review Working Group represent each of the Department’s research-performing administrations and the Office of the Secretary.

The objectives of these internal reviews are to:

• Continuously improve transportation research management and performance.
• Identify, share, and learn best practices.
• Identify opportunities for leveraging RD&T resources and for crossmodal research initiatives.
• Prevent unnecessary research duplication.

RITA reports recommendations and conclusions to the RD&T Planning Council at the end of each year’s cycle of reviews. DOT’s Program Review Working Group assessed all operating administration RD&T programs in FY 2005 and again in FY 2006. The working group found no research duplication and identified a number of areas for crossmodal collaboration, including geospatial technologies, freight capacity, security, alternative energy technologies, and advanced materials and sensors.

External Coordination and Review

A critical element of the Department’s RD&T evaluation strategy is regular consultation and engagement with stakeholders. Such efforts avoid duplication, uphold the technical quality of DOT research, and ensure that RD&T programs are wise public investments that address critical needs.

Within the operating administrations, stakeholder input is essential for establishing RD&T priorities, programmatic activities, and performance metrics. Of particular importance is the conduct of regular external program evaluations. Table 10-2 summarizes the various review mechanisms employed, the results of the most recent RD&T reviews, and program evaluations in FY 2006. The following paragraphs provide details about these and other RD&T planning activities.

FAA

A primary method by which FAA ensures RD&T effectiveness is its Research, Engineering, and Development Advisory Committee. Established by Congress in 1989, this committee reports to the FAA Administrator on RD&T issues and provides a link between agency research and similar efforts in industry, academia, and government. The committee considers aviation research needs in six key areas: air traffic services, airport technology, aircraft safety, aviation information security, human factors, and the environment. It meets twice a year with FAA senior managers and annually reviews FAA’s RD&T budget. Committee members represent corporations, universities, associations, consumers, and other agencies.

Another body, the Commercial Space Transportation Advisory Committee, advises on RD&T in commercial space transportation safety. This committee comprises senior executives from the U.S. commercial space transportation industry, including entrepreneurial firms as well as large aerospace companies; space-related State government officials; academia; and representatives from space advocacy organizations.

FHWA

Within FHWA, the TRB Research and Technology Coordinating Committee reviews RD&T, advises on research activities, and provides policy-level recommendations on program direction and management. The committee consists of 18 members from the States, academia, and private sector. It provides high-level, overarching advice on the composition and management of FHWA’s research and technology (R&T) program in the form of letter reports to the Administrator and more in-depth TRB Special Reports. In particular, the committee examined national highway research and technology needs and roles in TRB Special Report 261, which urged FHWA to focus on long-term, fundamental research; address gaps and emerging issues; conduct policy research; and engage in technology transfer.

Another mechanism for evaluating FHWA research is the Lab Assessment Program, which provides independent evaluation of the research performed by FHWA laboratories. The assessment process is designed to enhance lab quality and performance by providing feedback to managers and staff on their work. It provides a credible, professional, and objective assessment that further improves stakeholder confidence in the value of the work performed and the outcomes produced. The lab assessment is a form of expert and peer review by technical and scientific experts whose knowledge and expertise enable them to make reliable and unbiased judgments regarding the conduct of the reviewed research. The assessment process is continuous, with each lab assessed once every four years. Assessment criteria are based on the Administration’s R&D Investment Criteria.

Finally, FHWA has broad interaction with R&T stakeholder groups, including the Association of American State Highway and Transportation Officials, and engages customers throughout the entire R&T process. For stakeholder input to the Intelligent Transportation Systems (ITS) Program, FHWA had previously relied on ITS America as a Utilized Federal Advisory Committee. The Department has determined that this program has matured to the point where a formal DOT Advisory Committee is needed and is in the process of establishing such a body.

FMCSA

FMCSA obtains input on its R&T program from various stakeholders, including the National Transportation Safety Board, safety advocacy groups, the national enforcement community, the motor carrier industry, commercial driver groups, truck manufacturers, the driver training community, sleep researchers, insurance representatives, truck manufacturers, and the motor coach industry. The agency holds annual stakeholder forums to solicit recommendations for R&T projects and improved program planning. This input is reflected in a recently completed five-year strategic plan and in annual budget submissions.

FRA

FRA research is guided by inputs from both internal and external stakeholders, including FRA’s Office of Safety, the Association of American Railroads Railway Technology Working Committee and Tank Car Committee, the American Public Transportation Association Research Needs Conference, and the Highway–Railroad Grade Crossing Research Needs Conference.

Of particular importance is the TRB Committee for Review of the FRA Research, Development, and Demonstration Programs. Initially established by Congress, this committee conducts an annual assessment of FRA’s Railroad R&D program. Reviews address FRA’s research management structure and approach; current direction and allocation of funds devoted to various program areas; and whether there is an appropriate balance of Federal, State, and private-sector input and cost sharing. The committee includes representatives from States, railroads, labor unions, universities, financial institutions, and research organizations. To evaluate FRA research in another program area, the TRB has established a Strategic Hazardous Materials Transportation Research Program.

FTA

Formed in October 2003 under the TRB, FTA’s Transit Research Analysis Committee assesses research needs and advises the agency on the Federal role in transit research, high-priority research opportunities, and processes for ensuring that FTA receives input and cooperation from stakeholders. Members represent transit authorities, community service agencies, State DOTs, research institutes, consulting firms, and equipment manufacturers. FTA has incorporated input and advice from the committee in its recently completed five-year Strategic Research Plan.

MARAD

Although MARAD receives no direct RD&T funding, the agency works closely with stakeholders to stimulate innovation through collaborative efforts such as the Marine Transportation System initiative, the Short Sea Shipping Cooperative Program, and several other cooperative research programs. This approach brings together the maritime industry, academia, and agencies to identify, coordinate, facilitate, and accomplish maritime RD&T. Recommendations for future research also come from the Marine Transportation System National Advisory Council, a Federal Advisory Council to MARAD.

NHTSA

NHTSA assures the quality and effectiveness of its research through several means, including collaborative research with manufacturers and suppliers and regular public meetings with stakeholders. Public meetings provide a forum in which researchers can present their work, respond to comments, and obtain broad input on the agency’s RD&T program. In particular, NHTSA holds meetings with auto manufacturers and suppliers to discuss developments in automotive technologies and deployment strategies, including specific recommendations for NHTSA’s New Car Assessment Program, lane departure warning systems, advanced occupant restraint systems, vehicle compatibility, and other areas.

PHMSA

PHMSA’s pipeline safety program relies on stakeholder involvement, including R&D Forums and meetings of a Blue Ribbon Panel, to make sure that RD&T is aligned with the pipeline safety mission, makes use of the best available knowledge and expertise, and considers stakeholder perspectives. Stakeholders represented include Federal and State agencies, industry, pipeline trade associations, and standards organizations. PHMSA’s Office of Hazardous Materials Safety conducts its RD&T activities in consultation with other agencies, State and local governments, international organizations, the regulated industry, and the interested public.

RITA

RITA works with stakeholders to ensure the effectiveness of its RD&T planning efforts and to identify RD&T priorities. In addition, RITA will rely on the Advisory Council for Transportation Statistics for advice on the quality and objectivity of BTS data and analyses and review by the NRC to ensure the effectiveness of RD&T strategic planning and coordination activities.

Appendix D lists website links to additional information about operating administration RD&T evaluation activities, advisory committees, and plans.