Detailed Information on the
Railroad Research and Development Assessment

Develop a strategic framework for managing the program and its component research projects. This would involve developing multi-year R&D program plans that contain detailed schedule and budget information; clear explanations of how research projects support FRA performance goals; standard procedures for soliciting stakeholder input on setting research agendas; and standard procedures for obtaining merit reviews of work performed and funded by FRA.

Include in FRA??s annual performance reports a numeric chart showing progress made towards achieving performance goals at the R&D project level.

Develop process for tracking "on-budget" and "on-schedule" efficiency measures.

Measure: Safety: Reduce the Rate of Rail Related Accidents and Incidents (number of rail related accidents and incidents versus train miles in millions)

Explanation:Several R&D initiatives contribute to this goal, including Railroad Systems Issues, Human Factors, Rolling Stock and Components, Track and Structures, Train and Track Interaction, Train Control, Grade Crossings, Hazardous Materials Transportation, and Train Occupant Protection. For example, FRA has directly integrated track geometry research work into the its highly successful Automated Track Inspection Program (ATIP).

Measure: Grade crossing incidents per million train-miles

Explanation:This measure tracks FRA's performance in reducing the number of grade crossing incidents per million train-miles.

Measure: Human-factors-caused train accidents per million train-miles

Explanation:This measure tracks FRA's performance in reducing the number of human-factors-caused train accidents per million train-miles.

Measure: Track-caused accidents per million train-miles

Explanation:This measure tracks FRA's performance in reducing the number of track-caused accidents per million train-miles.

Measure: Equipment-caused accidents per million train-miles

Explanation:This measure tracks FRA's performance in reducing the number of equipment-caused accidents per million train-miles.

Measure: Other train accidents per million train-miles

Explanation:This measure tracks FRA's performance in reducing the number of other train accidents per million train-miles.

Measure: Non-accident hazardous materials releases per million train-miles

Explanation:This measure tracks FRA's performance in reducing the number of non-accident hazardous materials releases per million train-miles.

Measure: Number of deliverable research products, innovations, and technologies relating to equipment and operating practices that support DOT and FRA rail safety goals.

Explanation:FRA plans to deliver 17 equipment and operating practices products, innovations, and technologies supporting railroad safety by 2009. Equipment and Operating Practices include the following initiatives: Railroad Systems Issues, Human Factors, Rolling Stock, Grade Crossings (Human Factors), Hazmat, and Train Occupant Protection.

Measure: Number of deliverable research products, innovations and technology relating to track and infrastructure that support DOT and FRA rail safety goals.

Explanation:FRA plans to deliver 17 products, innovations, and/or technologies supporting track safety by 2009. Track Research work includes the following initiatives: Track & Structures, Track/Train Interaction, Train Control, and Grade Crossings - (Infrastructure).

Measure: Organizational Excellence: Percent of projects completed within budget

Explanation:Establishes development of research products within budget

Measure: Organizational Excellence: Percent of projects completed on time

Explanation:Establishes effective, timely development of research products

Is the program purpose clear?

Explanation: FRA's primary mission is to promulgate and enforce rail safety regulations. The Office of R&D conducts research and development projects to develop technologies that support the agency's safety mission and to enhance the railroad system as a national transportation resource. FRA's R&D program conducts rail safety research in 9 areas - Rail Systems Issues, Human Factors, Rolling Stock, Track & Structures, Track/Train Interaction, Train Control, Grade Crossings, Hazardous Materials, and Train Occupant Protection. The program also maintains FRA R&D facilities and equipment and manages the construction of the Nationwide Differential Global Positioning System.

Evidence: The program's authorizing legislation is found in the DOT Act of 1966 [49 USC 103, Section 3(e)(1)], [49 USC 101 (b)(4)], [49 USC 301(6)]; FRA Safety Act [49 USC 20102]. Also see FRA Order 1100.23C. The FRA R&D website is found at www.fra.dot.gov/content3.asp?P=32.

Does the program address a specific and existing problem, interest or need?

Explanation: The hazards posed by trains are very real, with thousands of rail-related injuries and fatalities occurring annually. Moreover, derailments and accidents can result in release of hazardous materials. The R&D program provides the research and information necessary for the FRA to regulate and create standards for the industry, with the goal of reducing the number of accidents, derailments, injuries, and fatalities. Research is targeted based on input from industry stakeholders and FRA's Office of Safety. The program conducts applied research that produces technological and management solutions for adoption by the nation's railroads.

Evidence: FRA's Railroad Safety Statistics are found at http://safetydata.fra.dot.gov/officeofsafety/

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Generally, FRA is the only entity in the US conducting research on rail safety issues. However, some FRA research overlaps with industry efforts for proprietary resasons. To ensure it has complete information for developing safety regs, FRA must often conduct its own research. The rail industry trade group, the Association of American Railroads (AAR), funds its own R&D program, which is roughly one-third the size of the FRA program. FRA regularly meets with the AAR Research and Technology Working Group to ensure their work is not redundant. To the extent possible, FRA and AAR collaborate on projects, as illustrated by their shared use of the nation's largest rail research facility, Transportation Technology Center, which is owned by the FRA but operated by TTCI. (Note: TTCI is a wholly owned for profit subsidiary of the AAR.) Additionally, FRA staff participate in DOT's department-wide Research Technology Coordinating Council (RTCC) and the Human Factors Coordinating Committee (HFCC) to share information on research projects to avoid duplication of efforts.

Evidence: FRA participates in the AAR Research and Technology Working Group and various other AAR Technical Committees, the American Public Transportation Association (APTA) working group on passenger rail safety, and DOT's Research and Technology Coordinating Council and Human Factors Coordinating Committee. See http://scitech.dot.gov/research/human/.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: FRA's research program has the elements of a well designed R&D program. To set its research agenda, FRA holds research needs conferences with the rail community on a variety of subjects, including a recent Grade Crossing Research Needs Conference and a Passenger Car Research Needs conference with the American Public Transportation Association. An overall Research Needs conference with rail labor, industry, and academia is planned for 2006. To evaluate its work FRA uses the National Academy of Science's Transportation Research Board (TRB) to conduct peer reviews by members of industry, academia, and state DOTs. TRB prepares an annual report with recommendations for FRA on R&D managment issues, allocation of funds to program areas, and whether the program reflects an appropriate balance of Federal, state, private cost sharing.

Evidence: GAO notes that two characteristics leading research programs are, 1) developing research agendas through the involvement of external stakeholders, and 2) evaluating research using expert review of the quality of research outcomes. See the GAO report 'Highway Research: Systematic Selection and Evaluation Processes Needed for Research Program (GAO-02-573)' at www.gao.gov/. TRB prepares a Letter Report every May for the FRA Administrator, with the latest available at trb.org/news/blurb_detail.asp?ID=3750. The TRB rail committee web page is found at www4.nas.edu/webcr.nsf/5c50571a75df494485256a95007a091e/1a7500c9ecac742485256b89005052b0?OpenDocument

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The program targets resources to support FRA's safety regulatory and oversight functions. FRA's Office of Safety works closely with the R&D office in setting annual research agendas. FRA also elicits input from industry stakeholders, for instance through research needs conferences. The Office of Safety uses research results in rulemaking efforts, and, likewise, the FRA's Railroad Safety Advisory Committee (RSAC) use R&D results to develop regulations. FRA disseminates research results to stakeholders through a number of means, including posting research results on its website, distributing technical reports to the rail industry by mail, and presenting findings at industry technical conferences and workshops. Increasingly, FRA's ability to effectively target research dollars has been undermined by Congressional earmarks. In recent years, around 10 percent of the program's budget has been earmarked.

Evidence: FRA publication of technical reports are available at www.fra.dot.gov/Content3.asp?P=917. FRA's Railroad Safety Advisory Committee's website is found at rsac.fra.dot.gov/ASP/home.asp.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program's long term goals include 1) FRA's strategic goals for rail safety, and 2) productivity goals specific to research. The R&D program's performance is measured by FRA's overall sucess in reducing rail-related accidents, fatalities, injuries, grade crossing accidents, and hazmat releases. In addition, the program will begin measuring the number of products, innovations, or technologies it generates in support of FRA's larger strategic goals, mentioned above. This output measure indicates the program's productivity in delivering technologies that help improve rail safety. FRA will have two product deliverable measures for its two main areas of research, 1) equipment and operating practices and, 2) track research.

Evidence: See FRA's FY 2005 Budget Request to Congress; DOT Strategic Plan 2003-2008 (http://Strategicplan.dot.gov); FRA Strategic Plan (www.fra.dot.gov/about/FRAstrategic_plan_.htm); and FRA Five-Year Strategic Plan for Research, Development and Demonstrations (www.fra.dot.gov/content3.asp?p=225).

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: FRA adjusts its targets based on past performance, projected program resources, and the expectation of being able to meet targets. FRA's R&D Plan outlines how the program's work contributes the goals included FRA Strategic Plan and the DOT Strategic Plan.

Evidence: See FRA's FY 2005 Budget Request to Congress; DOT Strategic Plan 2003-2008 (http://Strategicplan.dot.gov); FRA Strategic Plan (www.fra.dot.gov/about/FRAstrategic_plan_.htm); and FRA Five-Year Strategic Plan for Research, Development and Demonstrations (www.fra.dot.gov/content3.asp?p=225).

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: FRA has annual targets for its strategic goals and for the R&D program's technology development goals. The FRA has five quantative, annual performance measures that demonstrate progress toward achieving the long-range goals. It has two product development goals in the areas of track research and equipment and operating practices research. Note that the annual production of technologies may not immediately result in a reduction of accidents, injuries, or fatalities due to the time required to deploy new technologies and other factors.

Evidence: See FRA's FY 2005 Budget Request to Congress; DOT Strategic Plan 2003-2008 (http://Strategicplan.dot.gov); FRA Strategic Plan (www.fra.dot.gov/about/FRAstrategic_plan_.htm); and FRA Five-Year Strategic Plan for Research, Development and Demonstrations (www.fra.dot.gov/Research and Development).

Does the program have baselines and ambitious targets for its annual measures?

Explanation: FRA has developed baselines and ambitious targets for its annual research measures. Going forward, FRA will track its productivity in developing useful technologies, as well as its success towards meeting its agency strategic goals. FRA adjusts its targets annually based on past performance, projected program resources, and the expectation of being able to meet targets.

Evidence: See FRA's FY 2005 Budget Request to Congress; DOT Strategic Plan 2003-2008 (http://Strategicplan.dot.gov); FRA Strategic Plan (www.fra.dot.gov/about/FRAstrategic_plan_.htm); and FRA Five-Year Strategic Plan for Research, Development and Demonstrations (www.fra.dot.gov/content3.asp?p=225).

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Program partners commit to the program goals through contracts, grant agreements, cooperative agreement work statements, and interagency agreements. Program goals are included in the Background section of the work statements. Program managers conduct regular program and project reviews to ensure compliance with technical requirements, as well as cost and schedule targets. All contracts, grants, and cooperative agreements require submission of monthly progress reports to program managers.

Evidence: Evidence includes contract, grant and cooperative agreements; monthly progress reports that compare the actual technical, schedule, and cost performance with respect to targeted goals; periodic project and program review presentations.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The National Academy of Science's TRB Committee for Review of the FRA Research, Development, and Demponstration Programs is dedicated specifically to reviewing the management and quality of work produced by FRA's R&D program. The committee considers a wide range of issues including FRA R&D managment, the allocation of funds to program areas, and whether the program reflects an appropriate balance of Federal, state, private cost sharing. The TRB Committee holds semi-annual meetings with FRA to review management issues and prepares annual letter reports for the FRA Administrator with findings and recommendations. The committee is composed of rail experts from industry, academia, and state and regional government.

Evidence: For information on the TRB Rail Committee see http://trb.org/directory/comm_detail.asp?c=B0074. The May 2004 TRB letter report to FRA is available at http://trb.org/news/blurb_detail.asp?ID=3750

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The FRA FY 2005 Budget Request to Congress directly linked its five annual performance goals to budget data. The R&D program funding is assigned to the agency's five performance goals. However, it is not completely clear how adjusting research funding impacts the achievement of R&D or FRA strategic goals.

Evidence: FRA FY 2005 Budget. See DOT website www.dot.gov/bib2005/admins.html#fra

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: In 2002 at the request of Congress, FRA published its Five-Year RD&D Strategic Plan through 2005. It outlines FRA's vision for the future of railroads and the technologies needed to support them. The plan also identifies rail issues, program priorities, program objectives, project outcomes, and project descriptions. FRA is now updating the plan for the next five years, taking into account the agency's newly developed performance goals and measures. Further, FRA uses program evalutions to help it improve the quality and delivery of service of FRA's R&D Program and Safety Program. FRA also has adopted recommendations by the TRB Review Committee for improving its strategic planning capabilties, such as developing a Program Development and Program Selection Methodology and authoring a white paper on railroad industry trends.

Evidence: The FRA Five-Year RD&T Strategic Plan is found at www.fra.dot.gov/Content3.asp?P=225; For current Program Evaluation Solicitation of the program see www2.eps.gov/spg/DOT/FRA/OAGS/DTFR53%2D04%2DR%2D00002/listing.html

If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals?

Explanation: FRA estimates the benefits of its R&D projects using an R&D Program Evaluation Methodology, which was developed by the Volpe Center for FRA R&D. This methodology uses criteria (safety, regulatory, and likelihood of success) to quantatively evaluate proposed projects in a program area. Research proposals are ranked by the model, and results are reviewed by FRA management and the TRB R&D Review Committee. FRA also solicits reaction to its research through presentations at technical meetings, in technical journals, and at meetings with stakeholders, including Research Needs Conferences.

Evidence: FRA's R&D project development and selection process is described on page 3-1 of its Five Year R&D Strategic Plan found at www.fra.dot.gov/Content3.asp?P=225. Note presentations at TRB Annual meetings, meeting with stakeholders, and presentations at technical conferences such as the American Society of Mechanical Engineers (ASME)/IEEE Joint Conference on Rail Research and the ASME International Mechanical Engineering Congress. FRA articles also appear in technical journals.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: Budget requests and spending priorities are systematically evaluated using R&D Program Evaluation Methodology discussed in question 2.8. FRA R&D also conducts periodic Research Needs Workshops that solicit stakeholder input to help identify current issues and prioritize research needs.

Evidence: FRA's R&D project development and selection process is described on page 3-1 of its five year R&D strategic plan found at www.fra.dot.gov/Content3.asp?P=225; R&D Program Evaluation Criteria; Grade Crossing Research Needs Workshops, 1995 and 2003; Railroad Dispatcher Workshops, 1998 and 2004

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Project goals are tracked and reported in monthly progress reports for each contract, grant, or cooperative agreement. These reports are reviewed by the R&D program managers, as well as FRA management, including the FRA Administrator (on a quarterly basis). The reports describe progress made in attaining the project goals, including technical, cost, and schedule milestones.

Evidence: Monthly Progress Reports for each R&D project.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Personnel evaluations of individual R&D Program Managers consider whether their programs achieved performance milestones. Program managers have targets for submitting procurement requests on-time and in obtaining results and publishing them via technical reports. Program managers also hold their contractors, grantees, and cooperative partners accountable for technical, schedule, and cost performance results. Contracts, grants, and cooperative agreements are written to require specific deliverables for a set amount of funding. FRA does not pay R&D partners that fail to meet contract terms.

Evidence: Annual performance plans for R&D program managers have targets for submitting procurement requests in accordance with the Annual Procurement Plan. Managers must also obtain research results and publish them via technical reports, present papers at technical conferences, and make them available for use by the Office of Safety for rulemaking.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The FRA has a system for tracking procurement actions and obligations that ensures that funds are not misspent. Top FRA management reviews the annual procurement plans, which includes a description of the procurement, the type, dollar estimate, office, received date, award date, actual obligation. Once the plan is approved, any addition must be approved by the Deputy Administrator if it exceeds $100,000. The Director of Procurement publishes quarterly reports for the first 2 quarters, then monthly reports for the balance of the fiscal year.

Evidence: FRA Annual Procurement Plan; Office of Railroad Development Monthly Obligations Report.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The FRA's current Five-Year RD&D Strategic Plan does not include efficiency measures and targets (percent projects completed on time and on budget). However, FRA is revising its R&D strategic plan for the next 5 years, which will include these measures. Additionally, the program attempts to make efficient use of resources by leveraging in-kind contributions from industry. This leads to better research projects and more acceptance by industry of the research results. One example is the Facility for Accelerated Service Test/Heavy Axle Load project. The FRA contributes about 1/3 of the total cost and industry contributes the remainder. Another example is the Human factors Coordinating Committee, where FRA and other agencies fund half of the project with industry contributing the other half.

Evidence: Each contract, grant, cooperative agreement, and interagency agreement requires monthly progress reports that track actual versus planned progress, cost, and schedule.

Does the program collaborate and coordinate effectively with related programs?

Explanation: FRA participates in a number of collaborative efforts with private industry to leverage program funding. Examples include the Tank Car Operating Environment, the Wayside and On-Board detector project, Passenger Rail Car Crash tests, the Facility for Accelerated Service Test/Heavy Axle Load tests, and the DOT Human Factors Coordinating Committee. Because of FRA's participation in these committees, research costs have been shared among the stakeholders and the results are available sooner for implementation by the industry. Results from these projects are being used by industry to improve safety of operations, designs, and equipment. Further, as noted previously, FRA and industry coordinate the operation of the Transporation Technology Center research facility. FRA conducts tests there, some of which FRA funds entirely (those regarding safety regulatory issues) while others are cost-shared.

Evidence: FRA Five-Year Strategic Plan; specific project plans and contracts; DOT Human Factors Coordinating Committee (See http://scitech.dot.gov/research/human/). Cost Sharing Report to Congress

Does the program use strong financial management practices?

Explanation: The Department's IG audit of FRA's financial statements and the Department's consolidated statements found that the financial statements were fairly presented in all material respects, and conformed with U.S. Generally accepted accounting principles. No material internal control weaknesses were cited.

Evidence: DOT FY 2003 Performance & Accountability Report. DOT IG Report FI-2003-018 (Jan 27, 2003) http://www.oig.dot.gov/item_details.php?item=985

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The FRA deputy administrator's office is leading efforts to improve the managment of the agency, including the R&D office. In 2000, the R&D conducted a self assessment of its organizational climate and program effectiveness, using the Baldridge Award criteria. The assesment identified the need to improve customer satisfaction, measurements of organization performance, and employee education. R&D also conducted a Workforce Planning Analysis in 2002 to assess the skills needed to meet expected changes in its future mission. This is part of an FRA agency-wide human capital assessment that is currently underway. The FRA is working on this on many other related initaitives as part of the performance scorecards in the President's Management Agenda.

Evidence: Evidence includes the Baldridge Award Criteria assessment, July, 2000; R&D Workforce Analysis, March, 2002; 360 evaluations for senior management; IT Capital Planning Program; PMA initiatives included in Performance Plan of senior managers

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: FRA invites TRB to regularly review of the management of the R&D program. TRB's latest report states, "TRB is impressed with the quality of work being in a number of areas under the Railroad R&D program..." and did not identify significant management deficiences. Further, FRA's R&D program awards the majority of its funds based on merit through a competitive process to uphold the quality of contracted work. FRA uses multi-year technical support contracts that are advertised for competition. Some funds are awarded through interagency agreements when it is determined such agreements result in the most timely and/or cost effective results. However, Congress has increasingly earmarked some R&D funds to three universities.

Evidence: The May 2004 TRB letter report to FRA is available at trb.org/news/blurb_detail.asp?ID=3750. For FRA information on FRA contracts, see FRA web site under Office of Administration, Office of Acquisition and Grantswww.fra.dot.gov/Content3.asp?P=389.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The performance of the R&D program is reflected in FRA's five primary safety indicators, which show significant improvements despite increases in rail traffic. For example, the percent change in total accidents between 2000-2003 dropped 17.7 percent, while the number of train miles increased 3.6 percent. Likewise, fatalities have decreased 8.6 percent and injuries have fallen 23.8 percent. In terms of measuring productivity, FRA will begin this year to track its development of new technologies. To date, the R&D initiatives identified in the current Five Year Strategic Plan are proceeding according to schedule, and FRA is using this information as the basis for establishing baselines for the R&D's program output measures.

Evidence: DOT Strategic Plan 2003-2008. (See website at www.dot.gov/stratplan2008/strategic_plan.htm). FRA Five-Year Strategic Plan for Research, Development and Demonstration. (See FRA website www.fra.dot.gov/Content3.asp?P=134)

Does the program (including program partners) achieve its annual performance goals?

Explanation: The FRA has come close to meeting the annual targets of its strategic goals in recent years. Additionally, FRA reports that R&D projects have, for the most part, been completed on schedule, though FRA has not systemically tracked budget or schedule performance to date.

Evidence: See the FRA FY 2005 Budget to Congress; DOT Strategic Plan (www.dot.gov/stratplan2008/strategic_plan.htm); FRA Five-Year Strategic Plan for Research, Development and Demonstration. (See FRA website www.fra.dot.gov/Content3.asp?P=134).

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: To date, FRA has not systematically tracked efficiency measures for the R&D program. However, beginning this year, two new efficiency measures will be tracked, reflecting project schedule and cost performance. [Note that FRA already monitors each of the program's numerous projects for schedule and cost performance, though not on a consistent systematic basis.] The program has also become much more adept at leveraging funds and ensuring that R&D projects include cost sharing or in-kind contributions from industry partners.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: No independent analysis shows that FRA uses industry best practices or sets the industry standard at this time. However, FRA does, to a large extent, meet the criteria of a well managed R&D program, which are to include stakeholders in the agenda setting process and to have independent reviews affirm the quality of research. The program recieves input from stakeholders through issue conferences and meetings with TRB and AAR. Further, seeks peer feedback through TRB evaluations of its R&D program management.

Evidence: See the FRA Five-Year Strategic Plan for Research, Development and Demonstration. See FRA website www.fra.dot.gov/Content3.asp?P=134

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: TRB's latest report states, "TRB is impressed with the quality of work being done in a number of areas under the Railroad R&D program..." The committee has affirmed that FRA's R&D program is generally well managed and that the quality of the science is not in doubt.

Evidence: See the TRB R&D Review Committee Reports at trb.org/news/blurb_detail.asp?ID=3750. Also, see FRA Five-Year Strategic Plan for Research, Development and Demonstration, section 1-7, for a discussion of the TRB peer review process (www.fra.dot.gov/Content3.asp?P=134).