Detailed Information on the
National Institute on Disability and Rehabilitation Research Assessment

Review and revise the NIDRR portfolio to focus on a more strategic set of priority areas that will help NIDRR achieve its goals.

Tie future funding requests to the demonstration of results.

Make grantee performance data available to the public in a transparent and meaningful manner.

Review and revise the design of NIDRR's current annual peer review panel process to improve the quality, reliability, and management utility of the expert panel reviews, along with the data produced by such reviews.

Complete expert reviews to obtain baseline performance data for the long-term performance goals and take steps to ensure complete, timely, and accurate performance information is available for funded activities.

Establish a regular schedule for announcing grant competitions.

Measure: By 2015, at least 10 percent of all NIDRR projects will be multisite, collaborative controlled trials of interventions and programs.

Explanation:

Measure: Percentage of NIDRR-supported fellows, post-doctoral trainees, and doctoral students who publish results of NIDRR-sponsored research in refereed journals.

Explanation:

Measure: The percentage of funded grant applications that received an average peer review score of 85 or higher.

Explanation:

Measure: The average number of publications per award based on NIDRR-funded research and development activities in refereed journals.

Explanation:

Measure: Percentage of new grants funded by NIDRR that assess the effectiveness of interventions, programs, and devices using rigorous methods.

Explanation:

Measure: By 2015, increase by at least 20 percent the number of accomplishments (e.g., new or improved tools, methods, discoveries, standards, interventions, programs, or devices) developed or tested with NIDRR funding that have been judged by expert panels to be of high quality and to advance the field.

Explanation:

Measure: Number of new or improved assistive and universally-designed technologies, products, and devices developed by grantees that are transferred to industry for commercialization.

Explanation:

Measure: Percentage of grant competitions for the upcoming fiscal year announced by the start of the fiscal year (October 1).

Explanation:NIDRR only had 8% of the competitions announced by Oct. 1 By tracking the % of grant competitions announced before the start of the fiscal year, this measure can help NIDRR ensure that potential grantees have sufficient day to prepare and submit quality applications.

Measure: The percentage of grant awards issued within 6 months of the competition closing date.

Explanation:

Is the program purpose clear?

Explanation: NIDRR's mission is clear and delineated in its authorizing statute: conduct research, demonstration projects and training, and related activities that improve the quality of life for individuals with disabilities. The ED strategic plan, the NIDRR long-range plan (LRP), and (to some extent) the New Freedom Initiative (NFI) guide NIDRR activities. NIDRR's new LRP for 2005-2009 is supposed to establish a framework for future research and development activities. NIDRR will solicit comments on the new plan in 2005.

Evidence: Evidence: Rehabilitation Act of 1973, Title II, NIDRR Long-Range Plan, the New Freedom Initiative, and NIDRR priority notices for grant competitions.

Does the program address a specific and existing problem, interest, or need?

Explanation: Three major developments--scientific progress that has led to longer lives for individuals with disabilities, a larger proportion of older individuals in the population, and the empowerment of persons with disabilities--have led to increased need for research and development in the disability area. The 2000 Census found that nearly 20 percent of the noninstitutionalized population age 5 and over had some type of long-lasting condition or disability.

Evidence: Demographic data on disability; research supported; results of expert reviews of key research program grants.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: NIDRR is the principal Federal agency supporting applied research and development to improve the lives of individuals with disabilities. Unlike the National Institutes for Health (NIH), which are focused primarily on basic research and on biomedical research issues, NIDRR's mission encompasses technology and the many factors that affect community and societal participation and employment for individuals with disabilities. In addition, the Rehabilitation Act of 1973 authorized the Interagency Committee on Disability Research (ICDR), a Federal interagency committee which is chaired by the NIDRR Director. The ICDR is mandated to promote coordination and cooperation among Federal departments and agencies conducting rehabilitation research programs.

Evidence: ICDR activities (see www.icdr.us); results of program reviews; the Institute of Medicine 1997 report "Enabling America."

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program does not have major flaws and it effectively supports research in many disability areas. During the last year, NIDRR revised its grant performance review procedures, and in the future it will review a larger proportion of grants. In addition, all grant applicants are notified, in the notice inviting applications, of applicable performance measures. However, NIDRR could improve program effectiveness and efficiency in selected areas. For example, NIDRR funding is split across many program and priority areas without a systematic analysis of whether this blanket approach is more effective than targeting funds on strategic priority areas. There has been no systematic study of whether alternative approaches, such as regulation or stricter enforcement of existing laws, could stimulate private sector investment in certain areas of research (e.g., public transportation accessibility; telecommunications).

Evidence: Institute of Medicine Report (1997) Enabling America: Assessing the Role of Rehabilitation Science and Engineering; program files.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: NIDRR resources are targeted through a long-range plan that sets the framework for program activities; annual priority-setting; use of specific research priorities to guide major competitions; Departmental and OMB review of priorities; publication of proposed priorities in the Federal Register and solicitation of public comment to establish the final priorities; and outreach activities. These processes help ensure that all NIDRR projects are for purposes specified in its authorizing statute. In addition, during the last year NIDRR examined its knowledge dissemination strategies and is re-designing knowledge dissemination projects in an attempt to ensure the quality of disseminated information.

Evidence: Program files that document post-award monitoring, program reviews, pre-application meetings and presentations; Long-Range Plan; priority notices published in the Federal Register.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: In 2004, NIDRR established three long-term goals: (1) To advance knowledge through capacity building, (2) to advance knowledge through research and related activities, and (3) to advance knowledge through translation and dissemination. NIDRR will measure progress towards meeting those goals through both long-term and annual performance measures. These goals specifically address three criteria - relevance, quality and performance. Progress towards meeting long-term goals will be assessed every three years.

Evidence: Department of Education Annual Program Performance Reports; Department of Education Planning & Performance Management Database.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: In 2004, NIDRR established long-term goals with associated long-term measures that have quantified long term targets for 2015. NIDRR is establishing baselines for these measures at the end of the first completed three-year cycle of portfolio assessments in 2007. The first assessment, which reviewed accomplishments from 2004, was conducted in 2005. Annual targets between these three years and 2015 will be established after the baseline years.

Evidence: Department of Education Annual Program Performance Reports; Department of Education Planning & Performance Management Database.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: NIDRR has established quantifiable annual performance measures to help assess progress towards meeting its long-term goals. Performance data are available for three of the annual measures. To assess its efficiency, NIDRR is tracking progress on measures that examine timeliness of competition announcements together with timely announcement of competition results. NIDRR believes that placing grant announcements on a regular schedule and notifying applicants in a timely manner about competition results will allow them to more efficiently manage their workloads. NIDRR's preliminary targets are to announce all competitions by a set date each year and to notify applicants within 6 months of competition closings.

Evidence: Department of Education Annual Program Performance Reports; Department of Education Planning & Performance Management Database. NIDRR developmental and internal performance indicators.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: NIDRR has established five annual measures, three of which have data and targets. NIDRR also has established a two-part efficiency measure: to announce grant competitions for the upcoming year by October 1 of each year (the beginning of the fiscal year) and to notify applicants of their award status within 6 months of competition closing dates. Having competitions on a regular schedule will allow applicants to better manage their workloads and plan research activities. For 2005, only 8% of the competitions were announced by October 1. An increase to 50% by 2007 will require significant changes in the work process for NIDRR.

Evidence: Department of Education Annual Program Performance Reports

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: NIDRR includes information on its program goals and objectives in all grant announcements and has held outcomes planning meetings with grantees. In addition, NIDRR discusses GPRA indicators at grantee meetings and during monitoring discussions. Grantees report data needed to assess progress on the performance indicators, and expert panels review the material provided on grantee activities and products. NIDRR will build collection of performance infomation into the new web-based annual performance reporting system, which is under development. In addition, NIDRR has held outcomes planning meetings to ensure that grantees are on board with program goals and measures.

Evidence: Guidelines for preparing briefing books and data and other documentation for program reviews.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: In 1997, the National Academy of Science (NAS) Institute of Medicine (IOM) conducted a comprehensive independent evaluation of NIDRR in response to a Congressional mandate to evaluate all Federal rehabilitation research programs. NIDRR used the IOM findings to improve features of its strategic planning and program management systems. This report was one in a series of major reviews in disability and rehabilitation since 1985: Injury in America (1985), Disability in America (1991), and Enabling America (1997). NIDRR is joining with the Centers for Disease Control to support an update of Disability in America, and following the completion of this study, NIDRR plans to lead an effort to update Enabling America. In addition, NIDRR has began a series of program studies and outcomes evaluations, which are expert panel studies designed to address programmatic direction and management issues for major portions of the NIDRR grant portfolio. NIDRR also funds grantee sponsored State-of-the-Science conferences to assess contributions and needs corresponding to specific research priorities identified in the Long-Range Plan.

Evidence: Enabling America: Assessing the Role of Rehabilitation Science and Engineering, National Academy of Sciences/Institute of Medicine, 1997. Department files.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget requests provide information on activities supported with program funds and include a discussion of the performance indicators, but funding decisions are not explicitly tied to accomplishment of annual and long term goals. However, the budget requests do include estimates of the cost of program administration. Funding justifications provide specific breakouts of the amount of funds proposed for each program area and, in addition, identify what new awards would be made at the request level. NIDRR uses performance information regarding projects to to help determine priority areas for future funding.

Evidence: Program review files; Congressional Budget Justifications.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: NIDRR established long-term goals and targets in 2004, taking seriously the previous PART finding that the program was Results Not Demonstrated because of the lack of long term measures. Many of the targets established are for the year 2015. While NIDRR does not have data for the long term measures, which were just added in 2004, data are available for annual measures that provide an indication of whether NIDRR is on track to meet the long term goals. NIDRR also has worked to ensure that grantees are aware of, and committed to the achievement of, NDIRR's long-range goals. Grant announcements identify performance indicators for the grant programs. NIDRR also has improved training for grant monitors.

Evidence: Long-Range Plan draft; grant monitoring manual.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: Individual grants within NIDRR's major programs (i.e., Rehabilitation Engineering Research Centers, Rehabilitation Research and Training Centers, and Model Systems) have been systematically peer reviewed to assess the quality, relevance, and utility of the work. NIDRR is revising peer review procedures to allow the inclusion of smaller grants in the reviews as well. These independent reviews provide information on the status of activities and accomplishments of topical clusters of grantees and provide recommendations for future activities.

Evidence: Program review files.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: NIDRR published its new Long-Range Plan in the Federal Register on July 27, 2005. This document, which was developed with public input, sets the general direction for future research. Specific priorities (which are linked to the Plan) for 2006 competitions have been identified and will be published for public comment. In 2006, NIDRR plans to fund new awards in the following priority areas: - Projects to improve assistive technology, including Rehabilitation Engineering Research Centers on (A) Technologies for Successful Aging; (B) Wheelchair Transportation Safety; and (C) Wireless Technologies and Small Business Innovation Research projects. - Spinal Cord Injury Model Systems grants that include funding for infrastructure support for multi-site clinical trials. - Data centers for spinal cord injury and traumatic brain injury research. - Projects to help improve employment outcomes, including Disability Rehabilitation Research Projects on improving outcomes for employees who are deaf or blind, and on way-finding technologies for the blind. - A Rehabilitation Research and Training Center grant on Independence and Community Living. - Disability Rehabilitation Research Projects on children with special health care needs, children with traumatic brain injury, disability and obesity, and assistive technology outcomes measurement. - Knowledge utilization projects. In addition, NIDRR will continue to support projects (e.g., research fellows and field-initiated grants) in which the investigators determine the topics of investigation.

Evidence: Current portfolio of work; priorities for grant competitions; Long-Range Plan.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Individual grants within NIDRR's major programs (i.e., Rehabilitation Engineering Research Centers, Rehabilitation Research and Training Centers, and Model Systems) have been systematically peer reviewed to assess the quality, relevance, and utility of the work. NIDRR has used the results from program reviews of grantees' work to help the grantees improve their work and to help set the direction of future competitions. Grantees were provided detailed feedback that allowed them to modify future work, as needed. The reviewers were external experts. NIDRR is revising its grantee reporting system so that it will provide more useful information on grant progress. In addition, NIDRR is revising peer review procedures to allow the inclusion of smaller grants in the reviews as well.

Evidence: Program review files.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ED's managers are subject to the EDPAS system, which links employee performance to success in meeting the goals of the Department's Strategic Plan. In general, managers are provided individual performance agreements where they are given responsibility for achieving relevant action steps outlined in the Strategic Plan. These action steps and other items included in managers' performance agreements are designed to measure the degree to which a manager contributes to improving program performance. For example, NIDRR managers are being held accountable for implementing GPRA plans to evaluate outcomes, recommendations to improve NIDRR grant making process, and systematic evaluations of NIDRR programs. Moreover, NIDRR monitors grantee performance on an annual basis through review and approval of annual budget plans, compliance reviews, and site visits. Grantees that do not meet Federal requirements are required to submit improvement plans and can have grants reduced or discontinued for serious or persistent failures to comply or meet performance targets.

Evidence: EDPAS criteria; NIDRR grant monitoring guidance.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: NIDRR successfully obligates funds by the end of each fiscal year, but most funds are obligated late in each fiscal year. As noted above, NIDRR is instituting changes to ensure that grant competitions are announced on a regular schedule and provide sufficient time for preparation and review of applications while allowing for the timely notification of awards. NIDRR also is using efficiency measures to track the timeliness of grant announcements and award notifications. Funds are spent for the intended purposes; this is assessed through grant and contract monitoring and intensive grant reviews for major grant programs. No improper uses of funds have been identified.

Evidence: Contract and grant files; summaries of formative and summative grant reviews.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: NIDRR has developed a two-part efficiency measure for the program. Preliminary data for these indicators will be collected in 2005. The efficiency measure is a management measure and tracks whether NIDRR announces its competitons in a timely manner and makes timely awards. We believe that announcing competitions on a regular schedule will make it easier (i.e., more efficient) for researchers to apply because they will be better able to schedule the time needed to prepare grant applications. Similarly, announcing the winners in a timely manner will allow grantees to plan activities and staffing better because award timing wil be predictable.

Evidence:  

Does the program collaborate and coordinate effectively with related programs?

Explanation: NIDRR collaborates with other agencies (e.g., HHS, VA) to plan research and data collection activities and is the lead agency for the Interagency Committee for Disability Research (ICDR). The program has not yet demonstrated, however, that the ICDR activities have produced meaningful changes in activities or resource allocation.

Evidence: http://www.icdr.us

Does the program use strong financial management practices?

Explanation: NIDRR has developed internal practices to ensure appropriate payments; e.g., staff are designated to track expenditures and NIDRR is working to improve oversight of grant activities. No internal control problems have been identified in audit reports.

Evidence: Program files.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: NIDRR has taken steps to ensure that it has in place a management system that effectively prevents problems. For example, NIDRR has separated peer review and grant oversight processes, and is examining the composition of peer review panels to ensure that all reviewers have required technical expertise. In addition, NIDRR managers conducted staff training on appropriate grant monitoring and notification procedures.

Evidence: Program files.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Most NIDRR activities are conducted by grantees, although some work is conducted by contractors. Both types of awards are made through a competitive, merit-based process. In 2004, 99% of funds went to competitively awarded grants and contracts. An analysis of funded proposals showed that 89% of the grant awards in 2004 went to applications that were rated 85 or above by the peer reviewers and 72% were rated 90 or above (on a 100 point scale). Reviewers for grant competitions are not ED employees.

Evidence: Contract and grant files; Federal Register grant announcements.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: NIDRR has routinely conducted thorough reviews of the Rehabilitation Engineering Research Centers, Rehabilitation Research and Training Centers, and Model Systems grants using expert review panels and annual performance reviews. Starting in 2005, smaller grants (e.g., field-initiated projects) will be reviewed as well. NIDRR also holds regular (at least annual) grantee meetings for most of its grantees during which NIDRR staff review program goals and requirements. NIDRR staff are responsible for monitoring all grants and contracts according to established Department procedures.

Evidence: Contract and grant files.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: NIDRR is currently revising its performance reporting and assessment system to collect systematic data from grantees that can be used by expert panels to evaluate the quality and relevance of R&D investments and outputs and assess progress towards the program's long-term and annual GPRA performance measures. The pilot phase of the new system is being implemented in FY 2005. As part of the redesign, NIDRR is identifying which data elements can be made public without compromising confidentiality and similar concerns. NIDRR plans to audit the data in the web-based system to ensure its accuracy. Program performance data for individual grantees is not publicly available, but NIDRR does provide web-based access to project descriptions and to certain grantee products (www.naric.com/research/default.cfm). NIDRR's goal is to ensure that information about major accomplishments and the results of evaluations are available on the website. Overall results for NIDRR's program performance indicators are available on the Department's website.

Evidence: On-line reporting system; materials from the program reviews; annual report of activities and accomplishments.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: This question is not applicable. Virtually all NIDRR funds are awarded competitively. A small portion of their funding is awarded as grant supplements or as interagency transfers.

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: NIDRR has established long-term performance goals with associated long-term and annual measures. All of the long-term goals have targets for the end year. Most of the annual measures have both targets and data. The program is demonstrating progress based on its annual measures, which are aligned with the long-term measures. NIDRR has anecdotal evidence that it is making progress towards meeting its goals, but is waiting to establish baselines until it can complete a thorough, systematic review of its entire grant portfolio. For example for the Goal of: By 2015, at least 10 percent of all NIDRR projects [i.e., grants] will be multisite, collaborative controlled trials of interventions and program, NIDRR staff has conducted a preliminary review of grant files and have determined that approximately 13% of grants include multisite trials. During the next 6 months, they will conduct a more thorough review to determine the number of these projects that meet the standards for the indicator. The new grantee reporting system will require grantees to provide this information. In addition to these activities, results of grant reviews indicate that funds are being used for high quality research activities that will help improve the lives of individuals with disabilities. In 2003, NIDRR began a pilot study to identify outcomes of funded projects, which resulted in NIDRR identifying specific products and tools that have been shown to contribute to improved outcomes for individuals with disabilities. Examples include new computerized technology for alignment of trans-tibial (leg) prostheses to improve the mobility of individuals with foot amputations; implementation of accessible information kiosks at the World War II Memorial in Washington, DC and by the US Postal Service; publication of a book on universal design that has been disseminated nation-wide and that was adopted by the New York City Department of Design and Construction; and the development and validation of an improved outcome measure of walking function in individuals with spinal cord injury.

Evidence: Summative reviews, program reports, internal analyses.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Many of its annual measures still need baseline data and targets before NIDRR can conclude that the program has achieved its performance goals. In one important area - the percentage of grantee research and development that has appropriate study design, meets rigorous standards of scientific and/or engineering methods, and builds on and contributes to knowledge in the field - NIDRR has met its goals. Also, at the project level, NIDRR has conducted intensive reviews of its largest grantees, and results to date indicate that it is meeting its annual goals for the program. All grant announcements provide information on how program performance will be assessed and informs grantees that they will submit information in their annual performance reports to help measure their performance and to help the Department assess the effectiveness of the program. NIDRR also has worked intensively with its grantees on grantee outcomes planning.

Evidence: GPRA data; results of program reviews; internal files.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: NIDRR has established efficiency measures that assess its performance in announcing grants competitions on a regular schedule and notifying applicants in a timely manner. A regular announcement schedule will allow potential appliants to better schedule their workload. NIDRR appears to be on schedule to announce 2006 competitions in a more timely manner than in prior years. NIDRR also is redesigning its grantee reporting system so that it will more easily capture data needed for its performance measures and will be focused on outcomes planning and reporting.

Evidence: GPRA data; results of program reviews.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: NIDRR is the only Federal program supporting applied research on disability issues but it can be compared to other research programs in government. However, no systematic evidence has been collected to compare NIDRR to other research programs.

Evidence:  

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The last comprehensive external review of NIDRR was in 1997. As noted above, NIDRR plans to lead an interagency update to this report. ED should establish a regular schedule for review of NIDRR by an independent organization to assess overall program quality, allocation of resources, and the extent to which supported research priorities meet the nation's need. NIDRR does conduct reviews of individual projects to ensure high quality, and the results of the formative and summative grant reviews show that the program is, overall, meeting its objectives.

Evidence: Results of program reviews. 1997 IOM Report. Program files.