|
Chapter 10
Evaluation and Assessment of RD&T
To ensure the effectiveness of RD&T, the Department continually assesses its
research programs using three primary mechanisms: (1) systematic application of
the Administration’s R&D Investment Criteria and Program Assessment Rating Tool
(PART); (2) internal annual reviews of operating administration RD&T programs;
and (3) external RD&T coordination and review. This evaluation strategy is shown
in Table 10-1.
R&D Investment Criteria and Part
Evaluation and
Assessment of
RD&T
- R&D Investment
Criteria and PART
- Annual Program
Review
- External
Coordination and
Review
|
To guide the planning and management of research across the Federal
Government, the White House Office of Management and Budget (OMB) has established
three broad investment criteria for RD&T: relevance, quality, and performance.
Each of the criteria has both prospective and retrospective elements:
Relevance. Programs must have complete plans, with clear goals and priorities; must
articulate their potential public benefits; and must be relevant to national and customer
needs. Agencies must assess relevance periodically through both prospective
and retrospective independent review.
Quality. Programs must use clearly stated, defensible methods for awarding funding;
those allocating funds through means other than a competitive, merit-based
process must justify funding methods and document how quality is maintained.
Performance. Programs must maintain long-term objectives, with annual measures
and targets, and define appropriate outputs, outcomes, schedules, and decision
points. Agencies should express program results in terms of public benefits and document
retrospective performance annually.
The R&D Investment Criteria incorporate established best practices for research
evaluation as identified by the National Academy of Sciences, Government
Accountability Office, and others. OMB uses the PART to assess how well agencies
are implementing the criteria. Essentially a questionnaire, the PART measures all
aspects of program planning, budgeting, and management. It has four sections:
Program Purpose and Design. Assesses whether a program’s purpose and design are
clear and sound.
Strategic Planning. Assesses whether the agency has established valid long-term and
annual goals for the program.
Program Management. Assesses agency management of the program, including
financial oversight and improvement efforts.
Program Results and Accountability. Assesses whether a program is meeting its annual
and long-term goals and how effective the program is based on independent evaluations.
On the basis of the PART, OMB rates programs as Effective, Moderately Effective,
Adequate, Ineffective, or Results Not Demonstrated. To date, OMB has used the
PART to assess RD&T programs in FAA, FHWA, FRA, NHTSA, and PHMSA. Each program
has been rated Moderately Effective or better and has implemented all PART
recommendations. The PART results for FTA’s research program will be released in
February 2007.
Annual Internal Program Review
Within the Department, the primary mechanism for ensuring implementation
of the R&D Investment Criteria and PART is the annual review of modal
RD&T programs.
Through the RD&T Planning Council and RD&T Planning Team, RITA works with
the operating administrations to conduct annual reviews of RD&T programs to
assess how well they are applying the criteria, particularly whether programs are
evaluated according to established best practices. Members of a Program Review
Working Group represent each of the Department’s research-performing administrations
and the Office of the Secretary.
The objectives of these internal reviews are to:
- Continuously improve transportation research management and performance.
- Identify, share, and learn best practices.
- Identify opportunities for leveraging RD&T resources and for crossmodal
research initiatives.
- Prevent unnecessary research duplication.
RITA reports recommendations and conclusions to the RD&T Planning Council at
the end of each year’s cycle of reviews. DOT’s Program Review Working Group
assessed all operating administration RD&T programs in FY 2005 and again in
FY 2006. The working group found no research duplication and identified a
number of areas for crossmodal collaboration, including geospatial technologies,
freight capacity, security, alternative energy technologies, and advanced materials
and sensors.
External Coordination and Review
Evaluation and Assessment of
RD&T
- R&D Investment
Criteria and PART
- Annual Program
Review
- External
Coordination and
Review
|
A critical element of the Department’s RD&T evaluation strategy is regular consultation
and engagement with stakeholders. Such efforts avoid duplication, uphold the
technical quality of DOT research, and ensure that RD&T programs are wise public
investments that address critical needs.
Within the operating administrations, stakeholder input is essential for establishing
RD&T priorities, programmatic activities, and performance metrics. Of particular
importance is the conduct of regular external program evaluations. Table 10-2 summarizes the various review mechanisms employed, the results of the most
recent RD&T reviews, and program evaluations in FY 2006. The following
paragraphs provide details about these and other RD&T planning activities.
FAA
A primary method by which FAA ensures RD&T effectiveness is its Research,
Engineering, and Development Advisory Committee. Established by Congress in
1989, this committee reports to the FAA Administrator on RD&T issues and provides
a link between agency research and similar efforts in industry, academia, and
government. The committee considers aviation research needs in six key areas: air
traffic services, airport technology, aircraft safety, aviation information security,
human factors, and the environment. It meets twice a year with FAA senior
managers and annually reviews FAA’s RD&T budget. Committee members
represent corporations, universities, associations, consumers, and other agencies.
Another body, the Commercial Space Transportation Advisory Committee, advises
on RD&T in commercial space transportation safety. This committee comprises
senior executives from the U.S. commercial space transportation industry, including
entrepreneurial firms as well as large aerospace companies; space-related State government
officials; academia; and representatives from space advocacy organizations.
FHWA
Within FHWA, the TRB Research and Technology Coordinating Committee reviews
RD&T, advises on research activities, and provides policy-level recommendations on
program direction and management. The committee consists of 18 members from
the States, academia, and private sector. It provides high-level, overarching advice
on the composition and management of FHWA’s research and technology (R&T)
program in the form of letter reports to the Administrator and more in-depth TRB
Special Reports. In particular, the committee examined national highway research
and technology needs and roles in TRB Special Report 261, which urged FHWA to
focus on long-term, fundamental research; address gaps and emerging issues;
conduct policy research; and engage in technology transfer.
Another mechanism for evaluating FHWA research is the Lab Assessment Program,
which provides independent evaluation of the research performed by FHWA laboratories.
The assessment process is designed to enhance lab quality and performance
by providing feedback to managers and staff on their work. It provides a credible,
professional, and objective assessment that further improves stakeholder confidence
in the value of the work performed and the outcomes produced. The lab assessment
is a form of expert and peer review by technical and scientific experts whose knowledge
and expertise enable them to make reliable and unbiased judgments regarding
the conduct of the reviewed research. The assessment process is continuous, with
each lab assessed once every four years. Assessment criteria are based on the
Administration’s R&D Investment Criteria.
Finally, FHWA has broad interaction with R&T stakeholder groups, including the
Association of American State Highway and Transportation Officials, and engages
customers throughout the entire R&T process. For stakeholder input to the
Intelligent Transportation Systems (ITS) Program, FHWA had previously relied on
ITS America as a Utilized Federal Advisory Committee. The Department has determined
that this program has matured to the point where a formal DOT Advisory
Committee is needed and is in the process of establishing such a body.
FMCSA
FMCSA obtains input on its R&T program from various stakeholders, including the
National Transportation Safety Board, safety advocacy groups, the national enforcement
community, the motor carrier industry, commercial driver groups, truck manufacturers,
the driver training community, sleep researchers, insurance representatives,
truck manufacturers, and the motor coach industry. The agency holds annual
stakeholder forums to solicit recommendations for R&T projects and improved program
planning. This input is reflected in a recently completed five-year strategic
plan and in annual budget submissions.
FRA
FRA research is guided by inputs from both internal and external stakeholders,
including FRA’s Office of Safety, the Association of American Railroads Railway
Technology Working Committee and Tank Car Committee, the American Public
Transportation Association Research Needs Conference, and the Highway–Railroad
Grade Crossing Research Needs Conference.
Of particular importance is the TRB Committee for Review of the FRA Research,
Development, and Demonstration Programs. Initially established by Congress, this
committee conducts an annual assessment of FRA’s Railroad R&D program. Reviews
address FRA’s research management structure and approach; current direction and
allocation of funds devoted to various program areas; and whether there is an
appropriate balance of Federal, State, and private-sector input and cost sharing. The
committee includes representatives from States, railroads, labor unions, universities,
financial institutions, and research organizations. To evaluate FRA research in
another program area, the TRB has established a Strategic Hazardous Materials
Transportation Research Program.
FTA
Formed in October 2003 under the TRB, FTA’s Transit Research Analysis Committee
assesses research needs and advises the agency on the Federal role in transit
research, high-priority research opportunities, and processes for ensuring that FTA
receives input and cooperation from stakeholders. Members represent transit
authorities, community service agencies, State DOTs, research institutes, consulting
firms, and equipment manufacturers. FTA has incorporated input and advice from
the committee in its recently completed five-year Strategic Research Plan.
MARAD
Although MARAD receives no direct RD&T funding, the agency works closely with
stakeholders to stimulate innovation through collaborative efforts such as the
Marine Transportation System initiative, the Short Sea Shipping Cooperative
Program, and several other cooperative research programs. This approach brings
together the maritime industry, academia, and agencies to identify, coordinate,
facilitate, and accomplish maritime RD&T. Recommendations for future research
also come from the Marine Transportation System National Advisory Council, a
Federal Advisory Council to MARAD.
NHTSA
NHTSA assures the quality and effectiveness of its research through several means,
including collaborative research with manufacturers and suppliers and regular public
meetings with stakeholders. Public meetings provide a forum in which
researchers can present their work, respond to comments, and obtain broad input
on the agency’s RD&T program. In particular, NHTSA holds meetings with auto
manufacturers and suppliers to discuss developments in automotive technologies
and deployment strategies, including specific recommendations for NHTSA’s New
Car Assessment Program, lane departure warning systems, advanced occupant
restraint systems, vehicle compatibility, and other areas.
PHMSA
PHMSA’s pipeline safety program relies on stakeholder involvement, including R&D
Forums and meetings of a Blue Ribbon Panel, to make sure that RD&T is aligned
with the pipeline safety mission, makes use of the best available knowledge and
expertise, and considers stakeholder perspectives. Stakeholders represented include
Federal and State agencies, industry, pipeline trade associations, and standards
organizations. PHMSA’s Office of Hazardous Materials Safety conducts its RD&T
activities in consultation with other agencies, State and local governments, international
organizations, the regulated industry, and the interested public.
RITA
RITA works with stakeholders to ensure the effectiveness of its RD&T planning
efforts and to identify RD&T priorities. In addition, RITA will rely on the Advisory
Council for Transportation Statistics for advice on the quality and objectivity of BTS
data and analyses and review by the NRC to ensure the effectiveness of RD&T
strategic planning and coordination activities.
Appendix D lists website links to additional information about operating administration
RD&T evaluation activities, advisory committees, and plans.
|
|