Summary
The Department of Transportation's (DOT) research, development, and technology (RD&T) budget totaled $1.1 billion in fiscal year 2005. DOT's Research and Innovative Technology Administration (RITA)--which includes the Bureau of Transportation Statistics (BTS)--oversees DOT's RD&T activities. GAO examined (1) how RITA's responsibilities for overseeing DOT's RD&T activities differ from those of its predecessor, the Research and Special Programs Administration (RSPA); (2) RITA's practices for coordinating, facilitating, and reviewing RD&T activities; (3) the progress DOT has made in implementing GAO's 2003 recommendations on how to improve the coordination and evaluation of RD&T activities; and (4) how BTS identifies and monitors how well it serves its users. To address these issues, GAO reviewed relevant documentation and interviewed officials from RITA, BTS, and three operating administrations.
In 2005, RITA took over RSPA's responsibilities for overseeing DOT's RD&T activities when RSPA was dissolved. While RITA's mission and strategic objectives are similar to those RSPA had, RITA differs from RSPA in a number of ways. For example, RITA proposed a $2 million increase in its fiscal year 2007 budget request for the oversight of DOT's RD&T activities through its proposed Transportation Futures and Applied Technology Program, which, among other things, would provide access to technical experts to RITA on a contract basis. Additionally, RITA's responsibility for evaluation is less clearly defined than RSPA's. RITA, unlike RSPA, is not required to measure the results or evaluate the effectiveness of RD&T activities. However, RITA is not explicitly prevented from evaluating such activities. RITA coordinates, facilitates, and reviews DOT's RD&T activities through various practices. For example, RITA has two coordinating bodies--the RD&T Planning Council and the RD&T Planning Team--and conducts budget reviews, among other practices. RITA has not, however, established performance goals, a clear implementing strategy, or an evaluation plan that delineates how its coordination, facilitation, and review practices will further DOT's mission or ensure the effectiveness of its RD&T investment. Without such a strategic approach, it is difficult for RITA to ensure that DOT is making the most of its approximately $1 billion annual RD&T investment. RITA has partially implemented four of our recommendations and has not implemented the other. For example, while RITA, through its two coordinating bodies, has taken some action to review RD&T activities for duplication and opportunities for joint efforts, RITA has not established the scope of RD&T activities to be reviewed, the methodology of the review, or how the results will be used to make decisions about future RD&T activities. BTS does not have a systematic process for identifying its primary users, soliciting ongoing feedback from those users, and determining whether or how that feedback should be incorporated. For example, rather than identify specific users of BTS data products and services, BTS considers its users to be those broad categories of intended users described in federal legislation. Further, rather than routinely soliciting user feedback on all data products and services, such as through a customer satisfaction survey, BTS only solicits limited feedback from some users of specific products through conferences, workshops, or other meetings. Finally, BTS relies on its program managers to evaluate and determine how best to address feedback from its users; however, the managers are not required to--and often do not--report the results of whether or how they considered user feedback. Without a systematic process for identifying its users, soliciting ongoing feedback, and determining whether or how that feedback should be incorporated, BTS is limited in its ability to consider feedback and use it to make improvements to data products.
Recommendations
Our recommendations from this work are listed below with a Contact for more information. Status will change
from "In process" to "Implemented" or "Not implemented" based on our follow up work.
Director:
Team:
Phone:
Katherine A. Siggerud
Government Accountability Office: Physical Infrastructure
(202) 512-6794
Recommendations for Executive Action
Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate performance goals and an overall implementing strategy that delineate how the activities and results of its coordination, facilitation, and review practices will further DOT's mission and ensure the effectiveness of the department's RD&T investment into RITA's fiscal year 2008 budget process, and the annual budget process thereafter. The strategy should include an evaluation plan for monitoring and evaluating its performance against set goals to assist RITA in better allocating its resources to improve performance.
Agency Affected: Department of Transportation
Status: In process
Comments: The Research and Innovative Technology Administration (RITA) developed preliminary performance goals for its coordination, facilitation, and review practices and submitted them to the Office of Management and Budget as a component of the Performance Assessment and Rating Tool review in 2007, according to RITA officials. The final approved performance goals and an implementing strategy will be included in a supplement to DOT's transportation Research Development and Technology Strategic Plan in calendar year 2008, and in DOT's budget submission for 2010.
Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate into RITA's fiscal year 2008 budget process, and the annual budget process thereafter, common performance measures related to DOT's RD&T activities, which should be developed in consultation with the operating administrations.
Agency Affected: Department of Transportation
Status: In process
Comments: RITA's Research, Development, and Technology Planning Team--made up of the directors or associate administrators of research, development, and technology for each of the operating administrations--initiated a new effort called the Research Planning and Investment Coordination process that is addressing how DOT's research program is organized across modes, presented for funding, and evaluated. This process is being tested with the resubmission of the 2009 research budget by all operating administrations, using enhanced budget submission guidance and revised, standard submission forms and data fields. Analysis of the revised input and resulting modifications will result in an automated research budget submission process supporting coordination, collaboration, facilitation and review of all DOT research. The process will enable RITA to develop and track performance measures and targets for DOT's RD&T activities. DOT expects to complete the testing of the Research Planning and Investment Coordination process in the fall of 2008. The process is outlined in the Strategic Plan and will be detailed in separate documentation upon final approval by the Research, Development, and Technology Planning Council.
Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate a strategy for identifying and reviewing all of DOT's RD&T projects to determine areas of unnecessary duplication, overlap, and opportunities for joint efforts into RITA's fiscal year 2008 budget process, the annual budget process thereafter, and the upcoming RD&T strategic plan. The strategy should address (1) the scope of the RD&T projects or programs that will be reviewed for duplication or joint efforts, (2) the methodology for how all RD&T projects will be reviewed or how duplication or joint efforts will be identified, (3) a timeline and the frequency for reviews to occur, and (4) how the results of the reviews--the identification of duplication or an opportunity for joint effort--will be reported and used to make decisions about future RD&T activities.
Agency Affected: Department of Transportation
Status: Implemented
Comments: RITA has developed a strategy for identifying and reviewing all of DOT's RD&T projects to determine areas of unnecessary duplication, overlap, and opportunities for joint efforts. RITA's strategy consists of on-going internal reviews of all of DOT's research programs conducted by the Program Review Working Group. The objectives of these internal reviews are to continuously improve transportation research management and performance; identify, share, and learn best practices; identify opportunities for leveraging resources and for crossmodal research initiatives; and prevent unnecessary research duplication. RITA's Program Review Working Group conducts these reviews by (1) convening meetings where officials from each of the operating administrations share information about areas of on-going or planned research, looking for opportunities for joint effort, and (2) conducting annual reviews of each operating administrations research plans, looking for research duplication, among other things related to the budget process. Results from the coordination efforts are discussed during Planning Team and Planning Council meetings and have been reported in the 2006-2010 Research, Development, and Technology Strategic Plan. According to the Strategic Plan, in fiscal year 2005 and again in fiscal year 2006, the Program Review Working Group found no research duplication and identified a number of areas for crossmodal collaboration, including geospatial technologies, freight capacity, security, alternative energy technologies, and advanced materials and sensors. A RITA official told us that eight working groups have been formed with representation from the relevant operating administrations to foster collaboration in these different areas. In addition to the Strategic Plan, the results of the coordination efforts are also included in the "Research, Development and Technology Annual Funding Fiscal Years 2006-2008, A Report to Congress", expected to be delivered to Congress in mid-September 2007. The results of RITA's internal reviews are used to ensure that research programs are wise public investments that address critical needs.
Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate into RITA's fiscal year 2008 budget process, the annual budget process thereafter, and the upcoming RD&T strategic plan, a strategy to ensure that the results of all of DOT's RD&T activities are evaluated according to established best practices. This strategy should include (1) which RD&T activities of the operating administrations RITA will ensure were evaluated according to best practices, (2) the methodology for how RITA will ensure evaluation of RD&T activities took place according to established best practices, (3) a timeline for when the RD&T evaluations should occur, and (4) how the results of the RD&T evaluations will inform future research.
Agency Affected: Department of Transportation
Status: Implemented
Comments: RITA has developed a strategy to ensure that the results of all DOT's research activities are evaluated according to established best practices on an annual basis. RITA's strategy includes three primary mechanisms, (1) ensuring systematic application of the Office of Management and Budget's (OMB) Research and Development Investment Criteria (relevance, quality, and performance) and the Program Assessment Rating Tool by the operating administrations; (2) annual internal program reviews with self-reporting by the operating administrations; and (3) documenting the operating administration's external stakeholder coordination and review. The results of the fiscal year 2005 and 2006 reviews and a schedule for fiscal year 2007 reviews are included in the "Research, Development and Technology Annual Funding Fiscal Years 2006-2008, A Report to Congress", expected to be delivered to Congress in mid-September 2007. According to RITA, the fiscal year 2007 reviews have been focused on how well the operating administrations are implementing best practices, including external stakeholder involvement, merit review of competitive proposals, independent expert review, research performance measures, and external research coordination. RITA will report the results of the reviews to the Planning Council and will publish the results in the next Research, Development and Technology Annual Funding Report to Congress. This strategy, documented in the 2006-2010 Research, Development, and Technology Strategic Plan, upholds the technical quality of DOT's research activities and helps to ensure that research activities are addressing critical needs.
Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate a DOT-wide database of all of DOT's RD&T projects that will support RITA's coordination, facilitation, and review efforts and will assist in the implementation of the strategies discussed above into RITA's fiscal year 2008 budget process, the annual budget process thereafter, and the upcoming RD&T strategic plan. Information on the status of these efforts should be included in the upcoming RD&T strategic plan to be issued in the fall of 2006.
Agency Affected: Department of Transportation
Status: Implemented
Comments: RITA has designed and is now in the process of developing two database systems to inventory and track all of DOT's research activities and provide tools for querying and searching individual projects to identify potential duplication and to identify areas where operating administrations could collaborate. The first database, called the RITA Research Notification System (R2NS) captures research investments at the transactional level, allowing users to search by activity, contracts and grants, and contractor names, enabling identification of funded programs for coordination, collaboration and review. The second database, which is part of the Research Planning and Investment Coordination process, captures research at the budget request level, allowing for Department-wide transparency and coordination of proposed programs and projects. According to a RITA official, eventual combination of the two databases will be a part of the annual Research Planning and Investment Coordination process that will offer a mechanism for measuring and tracking investments from request, through funding and execution. The databases will enable RITA to adjust and leverage research activities across the Department supporting its RITA's coordination, facilitation, and review efforts.
Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate a summary of all of DOT's RD&T program evaluations conducted by the department for the past 3 years, including ongoing and completed evaluations, and a schedule of future evaluations into RITA's fiscal year 2008 budget process, the annual budget process thereafter, and the upcoming RD&T strategic plan.
Agency Affected: Department of Transportation
Status: Implemented
Comments: RITA and its predecessor, RSPA, have together published summaries of DOT's RD&T evaluations for the past three years. First, RSPA published a summary of all research program evaluations conducted in fiscal year 2004 in its fiscal year 2005 annual Research, Development, and Technology plan. Secondly, the results of RITA's fiscal years 2005 and 2006 reviews, and a schedule of RITA's planned and ongoing fiscal year 2007 reviews, was included in the "Research, Development and Technology Annual Funding Fiscal Years 2006-2008, A Report to Congress", expected to be delivered to Congress in mid-September 2007. This report to Congress also includes summaries of research program evaluations conducted by modal research advisory committees, the Transportation Research Board, and key modal stakeholders in fiscal years 2006 and 2007. At the time of this followup, a schedule for fiscal year 2008 reviews had not yet been developed.
Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate a description of RITA's process for systematically evaluating the results of its own multimodal research programs and how this process will be applied to future multimodal research programs that RITA conducts into RITA's fiscal year 2008 budget process, the annual budget process thereafter, and the upcoming RD&T strategic plan.
Agency Affected: Department of Transportation
Status: Implemented
Comments: RITA has established a systematic process for evaluating all of DOT's research, including RITA's multi-modal research programs. This process consists of reviews to identify, share, and apply best practices, identify opportunities for leveraging resources, and advance cross-modal research. RITA oversees DOT's Intermodal Advanced Research Working Groups, which have identified eight cross-modal research areas for collaboration with operating administrations including: congestion reduction; energy efficiency/alternative fuels; enhanced safety data; human factors; nanotechnology; positioning, navigation and timing; safe transport in an aging society, and system resilience/global logistics. RITA has developed an evaluation strategy to ensure that these cross-modal activities, as well as other DOT research activities, are evaluated according to best practices, including systematic application of OMB investment criteria, external stakeholder involvement, merit review of competitive proposals, independent expert review, development of research performance measures, and external research coordination. RITA officials told us that they are continuing to pursue the development and implementation of a more consistent process to be applied across all program areas, which they expect to complete by the end of fiscal year 2008.
Recommendation: To help ensure that BTS's data products meet the needs of its users, the Secretary of Transportation should direct the RITA Administrator and BTS Director to develop and implement a systematic process for BTS to identify its primary users, solicit and incorporate feedback from those users, and measure the satisfaction of its users. This process should contain the following elements: (1) that primary users of BTS's data products and services are identified and documented in a comprehensive manner; (2) that feedback on user satisfaction is solicited on a periodic basis from those users; (3) that user feedback is documented and evaluated at BTS's agencywide level and against established criteria, to ensure consistency in decisions about what improvements should be made to data products; and (4) that performance indicators that measure data users' satisfaction are developed and applied.
Agency Affected: Department of Transportation
Status: In process
Comments: RITA's Bureau of Transportation Statistics (BTS) has taken action to improve user satisfaction with its data and analysis products. BTS has developed a systematic process to identify its primary users, solicit and incorporate feedback from those users, and analyze the feedback to improve its products. The survey process is web-based, and solicits information including (1) customer type (e.g., federal, state, academia, congressional, library, for-profit, etc.), (2) product or service (e.g., Commodity Flow Survey, TranStats, etc.), and (3) interaction type (inquiry, request for technical assistance, suggestion for improvement, or expression of satisfaction). Regular reports generated from this system will provide user feedback to be incorporated into the decision-making processes for BTS products and services. Results from the survey will appear for users as a product on the BTS Web site. According to RITA officials, BTS is on track to fully implement the online customer feedback survey on the Web site by the end of October 2008.
