Detailed Information on the
National Institutes of Health: Research Training Assessment

[FY09] Ensure that the proportion of pre-doctoral trainees and fellows applying for and receiving subsequent NIH support exceeds the relevant comparison groups by 12% within 10 years of graduation.

[FY09] Ensure that the proportion of post-doctoral trainees and fellows applying for and receiving subsequent NIH support exceeds relevant comparison groups by 12 percent within 10 years of termination.

Convert 25% of trainee appointment forms to be processed electronically.

[FY10] Ensure that the proportion of pre-doctoral trainees and fellows applying for and receiving subsequent NIH support exceeds the relevant comparison groups by 12% within 10 years of graduation.

[FY10] Ensure that the proportion of post-doctoral trainees and fellows applying for and receiving subsequent NIH support exceeds relevant comparison groups by 12 percent within 10 years of termination.

Convert 50% of trainee appointment forms to be processed electronically.

[FY06] Ensure that the proportion of pre-doctoral trainees and fellows applying for and receiving subsequent NIH support exceeds the relevant comparison groups by 12% within 10 years of graduation.

[FY06] Ensure that the proportion of post-doctoral trainees and fellows applying for and receiving subsequent NIH support exceeds relevant comparison groups by 12 percent within 10 years of termination.

[FY06 and FY07] Initiate x train

[FY07] Ensure that the proportion of pre-doctoral trainees and fellows applying for and receiving subsequent NIH support exceeds the relevant comparison groups by 12% within 10 years of graduation.

[FY07] Ensure that the proportion of post-doctoral trainees and fellows applying for and receiving subsequent NIH support exceeds relevant comparison groups by 12 percent within 10 years of termination.

[FY08] Ensure that the proportion of pre-doctoral trainees and fellows applying for and receiving subsequent NIH support exceeds the relevant comparison groups by 12% within 10 years of graduation.

[FY08] Ensure that the proportion of post-doctoral trainees and fellows applying for and receiving subsequent NIH support exceeds relevant comparison groups by 12 percent within 10 years of termination.

Convert 5% of trainee appointment forms to be processed electronically.

Measure: Annually, through 2012, ensure that the proportion of pre-doctoral trainees and fellows applying for and receiving subsequent NIH support exceeds the relevant comparison groups by 12% within 10 years of graduation.

Explanation:The goal is intended to show the program's success by training highly skilled researchers who utilize their expertise as researchers upon completion of their training (retention in the field). The ten year timeframe begins the year following the receipt of their Ph.D. For example, FY 2006 target will reflect # and % of NRSA trainees and fellows receiving their doctoral degrees in 1995 who have applied for NIH research support by 2005. The FY 2007 target will reflect degrees in 1996 and NIH applications for 2006.

Measure: Annually, through 2012, ensure that the proportion of post-doctoral trainees and fellows applying for and receiving subsequent NIH support exceeds relevant comparison groups by 12 percent within 10 years of termination.

Explanation:The goal is intended to show the program's success by training highly skilled researchers who utilize their expertise as researchers upon completion of their training (retention in the field). The ten year timeframe begins the same year that their training is completed. For example, FY 2006 target will reflect # and % of NRSA trainees and fellows completing their training in 1996 who have applied for NIH research support by 2005. The FY 2007 target will reflect degrees in 1997 and NIH applications for 2006.

Measure: Percentage of trainee appointment forms that are processed electronically.

Explanation:

Is the program purpose clear?

Explanation: The purpose of the National Institutes of Health's (NIH) research training programs is clear. NIH's research training and career development programs build and maintain research capacity to help ensure that diverse pools of highly trained scientists are available in adequate numbers and in research areas to address the Nation's biomedical, behavioral, and clinical research needs. NIH's research training mission is accomplished with three programs: the Ruth L. Kirchstein National Research Service Awards (NRSAs), Research Career Development Awards (K Awards), and Loan Repayments. NRSAs and K Awards are assessed in this PART. Loan Repayments are not assessed in this PART as they were PARTed in 2005 under Extramural Research programs. NRSAs support pre- and postdoctoral training and are divided into two discrete focuses: institutional training grants awarded to institutions of higher education that, in turn, select students to be supported, and individual fellowships awarded to pre- and postdoctoral scholars who apply to NIH and obtain their research experience under a mentor at a sponsoring institution. K Awards assist early, mid-career, and senior-level investigators with opportunities to hone and apply their research skills as they continue their clinical services. Loan Repayments are used to encourage intramural and extramural researchers to stay in the field, even when the financial commitment to education becomes burdensome. There are three intramural and five extramural loan repayment programs focusing on AIDS, general research, clinical research, clinical research for those from disadvantaged backgrounds, contraception and infertility, pediatrics, and health disparities.

Evidence: 1. General Authority - 42 United States Code (USC) 284(b)(1)(C) and (b)(2)(A)-(B). 2. K Awards - 42 USC 284l(a)(1)(A), (b)(1)(A), and (d)(1)(A)-(f). 3. NRSAs - PHS Act, Sec. 487 and P.L. 107-206 http://grants1.nih.gov/training/NRSA_NameChangeLanguage.rtf. 4. Loan Repayments - Intramural: PHS Act, Sec. 487A (42 USC 288-1); PHS Act, Sec. 487C; and PHS Act, Sec. 487E - Extramural: PHS Act, Sec. 487B; PHS Act, Sec. 487E; PHS Act, Sec. 487F; PHS Act, Sec. 487F; and PHS Act, Sec. 485G. 5. NIH Research Training and Career Development Strategic Plan (FYs 2004-2010).

Does the program address a specific and existing problem, interest, or need?

Explanation: The program addresses a specific and existing interest to ensure an adequate supply of biomedical researchers who are capable of responding to a range of diseases and disorders, including emerging and rare diseases and those disproportionately affecting underserved populations (i.e., minorities, rural areas, the aging, and children). NIH's training programs, while funding some traditional researchers, focuses mainly on filling gaps in emerging fields of science. For example, in 1997 a NIH convened panel identified a need for more physician scientists to focus on clinical research. In 1998, NIH initiated a clinical research component to its career development awards (K awards). In 2005, the National Research Council (NRC) of the National Academies also confirmed the need less for traditional researchers, but more for emerging research areas. NRC reported its findings on the status of biomedical and behavioral research personnel in the United States. Using workforce models, the NRC found that unemployment among trained researchers is low and most trained personnel will remain in the field. The NRC found that the research workforce is currently in reasonable balance and at least should be maintained.

Evidence: National Research Council (2005) - Advancing the Nation's Health Needs: NIH Research Training Programs pp. 1-4, 91, 103. http://darwin.nap.edu/books/0309094275/html

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: NIH's research training and career development programs are not duplicative of other Federal, local, and private efforts. They are designed to complement the work of these entities. Other Federal agencies and private philanthropies support research training; however, none provide the complete continuum of research training and career development across the biomedical, behavioral, and clinical sciences. For example, National Science Foundation (NSF) training programs focus on fundamental science and engineering outside of biomedicine and the Department of Defense (DOD) focuses on biometrics and those trained at the Masters level. A number of private philanthropies (e.g., Howard Hughes Medical Institute and Burroughs Wellcome) focus on medical research. These private groups routinely provide funding for targeted areas of training or limit eligibility for participation. NIH complements the mission of private and State colleges and universities by helping to offset the costs of research training and establishing training standards. NIH awards set a standard for research training by requiring instruction in the responsible conduct of research (i.e., conflict of interest, data management and sharing, responsible authorship, and policies for handling misconduct).

Evidence: 1. NSF- Undergraduate Students: http://www.nsf.gov/funding/education.jsp?org=NSF&fund_type=1; Graduate Students: http://www.nsf.gov/funding/education.jsp?org=NSF&fund_type=2; Postdoctoral Fellows: http://www.nsf.gov/funding/education.jsp?org=NSF&fund_type=3. 2. DOD - National Defense Science and Engineering Graduate Fellowships: http://www.asee.org/ndseg/index.cfm; Research Program Fact Sheets: http://cdmrp.army.mil/pubs/fact.htm.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The authorizations for NRSAs and K awards are broad and not prescriptive. The authorizations leave it to the discretion of NIH to design the programs. NIH's program designs are sufficiently flexible to enable Institutes and Centers (ICs) to shift training priorities in order to establish and maintain a cadre of scientific researchers to address specific problems. NIH has modified the design of its research training program when necessary to address developments in the external environment, such as economic changes, advances in technology, and shifts in public policy. Similarly, for NRSAs and K awards, the competitive NIH peer review process ensures the best applicants are selected for funding. In addition to scheduled NIH-wide assessments of the programs coordinated through the OER, within the NIH Office of the Director, NIH's ICs have the flexibility to undertake periodic, targeted evaluations to improve implementation of specific aspects of the program.

Evidence: Government Accountability Office (October 2005) - Federal Science, Technology, Engineering, and Mathematics Programs and Related Trends (pp. 16-17). http://www.gao.gov/new.items/d06114.pdf.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The NIH has developed requirements and processes to ensure that resources directed to research training and career development programs reach their intended beneficiaries efficiently and effectively. Participants who have the highest probability of success to fulfill the programs' purpose are selected through the peer review process. Program staff and national advisory councils may designate applications in areas of particular need for higher funding priority, ensuring that resources are used to address the programs' purpose. Features are in place to ensure that NIH research training and career development funds reach intended beneficiaries at the most appropriate time. Retention is fostered through time commitments and "pay back" obligations at critical junctures, ensuring that the Nation's research capacity is maintained. NIH has received 98 percent payback through the service requirement. The remaining 2 percent is handled accordingly and pursuant to the 3-year recovery period that begins 2 years after termination of the NRSA award.

Evidence: Ruth L. Kirschstein NRSA Payback Agreement: http://grants1.nih.gov/grants/funding/416/phs6031.pdf.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: NIH has developed two long-term outcome measures that meaningfully reflect the purpose of NIH-funded research training programs. The measures are intended to show the extent to which the NRSA program builds and maintains highly trained scientists. In addition, the measures strive to show that NRSA funded researchers continue to conduct biomedical research and more successfully compete for research projects, relative to non-NRSA funded researchers.

Evidence: The long-term performance measures are listed below.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The targets for the long-term measures are ambitious. In recent years, data indicate that NRSA funded researchers stayed in the field and successfully competed 12-14% more often than non-NRSA funded researchers. The 12% target requires NIH to maintain its successful track record, while also urging the program to exceed the target.

Evidence: The baselines and targets for the long-term performance measures are listed below.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: NIH has developed two annual outcome measures and an efficiency measure that meaningfully reflect the purpose of NIH-funded research training programs. The measures are intended to show the extent to which the NRSA program builds and maintains highly trained scientists. In addition, the measures strive to show that NRSA funded researchers continue to conduct biomedical research and more successfully compete for research projects, relative to non-NRSA funded researchers.

Evidence: The annual performance measures, including the efficiency measure, are listed below.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The targets for the annual performance measures are ambitious. In recent years, data indicate that NRSA funded researchers stayed in the field and successfully competed 12-14% more often than non-NRSA funded researchers. The 12% target requires NIH to maintain its successful track record, while also urging the program to exceed the target.

Evidence: The baselines and targets for the annual performance measures are listed below.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All performance measures for this program reflect the efforts of grantees to commit to provide training in a variety of biomedical research areas to help ensure a diverse and skilled research workforce. Grantees commit to these goals as a condition of the grant award. By accepting an NIH award, grantee institutions acknowledge concurrence with the terms and conditions of the award, including service payback provisions, mentoring expectations, reporting requirements, and in some cases, the area of research training foci.

Evidence: 1. NIH National Research Service Award Institutional Research Training Grants - http://www.grants1.nih.gov/grants/guide/pa-files/PA-02-109.html 2. Mentored Research Service Scientist Development Award - http://www.grants1.nih.gov/grants/guide/pa-files/PA-06-001.html 3. Ruth L. Kirchstein NRSA Individual Fellowship Progress Report for Continuation Support - http://www.grants1.nih.gov/grants/funding/416-9/phs416-9.htm 4. Paul B. Beeson Career Development Awards in Aging - http://www.grants1.nih.gov/grants/guide/rfa-files/RFA-AG-06-005.html

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Since its inception in 1974, the NRSA program has been the subject of 12 National Academies of Science (NAS) studies, which initially were conducted annually and now are completed every 4-5 years. Although these studies previously focused on the NRSA program alone, recent assessments also have addressed the NIH research career development program. The committees assembled to conduct the studies make recommendations to the NIH to address the problems, interests, and needs of the program and to help set priorities in order to build research capacity. These evaluations and subsequent recommendations provide feedback for the NIH OER and the ICs to examine current internal processes and ensure efficiency and effectiveness in program areas. The outcomes of these evaluations are used to inform priority setting; identify problems, interests, and needs; and develop new approaches and improvements to existing programs to strengthen capacity building in terms of training and career development in biomedical, behavioral, and clinical research.

Evidence: 1. National Academy of Science's National Research Council. Advancing the Nation's Health Needs: NIH Research Training Programs (2005) - http://darwin.nap.edu/books/0309094275/html. 2. National Academy of Science's National Research Council. Assessment of NIH Minority Research and Training Programs: Phase 3 (2005) - http://darwin.nap.edu/books/0309095751/html. 3. RAND Corporation. An Evaluation of the NIH Research Career Development Award (1987) - http://www.rand.org/pubs/reports/R3568/.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: At this time, NIH's budget presentation does not explicitly tie budget resource levels to long-term or annual measures and their impact on bringing new researchers into the biomedical research field or retaining them. NIH's budget requests do not detail how funding increases or decreases would impact NIH's ability to accomplish its annual targets or show the linkage between the amount of funding requested and the expected impact on the research field.

Evidence: Department of Health and Human Services National Institute of Health FY 2007 Justification of Estimates for Appropriations Committee - http://officeofbudget.od.nih.gov/ui/2007Budget_IC_Links.htm

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Each year NIH continues to take proactive steps to identifying and addressing strategic planning deficiencies in all of its programs. At the program level, NIH continues to idenitfy strategic weaknesses in budget presentations and improves its own internal use of detailed budget and planning documents. In addition, NIH is committed in its efforts to identify and address strategic planning deficiencies quickly and efficiently by redirecting and pooling existing resources to areas of high priority and new areas of research training, restructuring the strategic plan or expected milestones without affecting major completion dates or costs, and developing new policies, procedures, and data systems.

Evidence:

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: Although other Federal agencies and private philanthropies support research training, NIH research training and career development programs are unique in their biomedical focus, breadth, and depth. To ensure that programs compare favorably to and complement other research training efforts, the NIH conducts periodic assessments of potential program benefits. In one such assessment, external evaluation experts compared former NRSA participants with Ph.D.s in the biomedical and behavioral sciences who were supported by other sources. The assessment found that former NRSA pre-doctoral trainees/ fellows outperformed other Ph.D.s in several measures of career progress: educational attainment, postdoctoral training, employment in research-intensive environments, applications for and success in receiving NIH or National Science Foundation (NSF) research grants, and publications and citations. Results support the conclusion that Ph.D.s who received NRSA traineeships/fellowships are more likely to embark on careers in research. Training programs supported by foundations and other Government agencies generally address specific research areas or limit eligibility. For example: ?? The NSF mission includes support from pre-K through graduate school and beyond for all fields of fundamental science and engineering except for the biomedical sciences. ?? The Department of Defense Congressionally Directed Medical Research Programs (CDMRP) manages Congressional Special Interest Medical Research Programs on a limited number of specified areas. The CDMRP funding strategy complements awards made by other agencies and specifically avoids duplication of long-term, basic biomedical research supported by the NIH. ?? The Department of Education mission is to ensure equal access to education and to promote educational excellence throughout the Nation. The Department's offices support research on education from early childhood through postsecondary.

Evidence: 1. Pion, G. M. (2001). The Early Career Progress of NRSA Pre-doctoral Trainees and Fellows (p. xvi, Executive Summary) - http://grants.nih.gov/training/career_progress/index.htm 2. About the National Science Foundation: What We Do - http://www.nsf.gov/funding/aboutfunding.jsp 3. Overview of the Program. Department of Defense Research Program Fact Sheets - http://cdmrp.army.mil/pubs/fact.htm 4. Department of Education Mission - http://www.ed.gov/about/overview/mission/mission.html?src=rt

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The NIH OER coordinates across ICs to meet the NIH Strategic Plan and GPRA goals for capacity building and research resources, working with individual ICs and across ICs to implement, monitor, and budget for NIH research training and career development programs. NIH research training programs are guided by recommendations and other input from stakeholders and knowledgeable external experts through task forces, working groups, conferences, and workshops convened by the NIH and its component ICs. These groups are charged with reviewing the state of the training enterprise in light of continued or new needs and making program recommendations to the NIH or a given IC. The outcome serves both to inform the NIH decision-making process regarding NIH- or IC-initiated activities and to make the community aware of priorities that may be addressed through research training grants, fellowships, or career development awards. The need for greater integration of efforts across the NIH as a whole has been seen increasingly as crucial to accelerating scientific discovery. This observation has led to a set of trans-NIH crosscutting initiatives based on the need to collaborate and avoid duplication of effort. For example, in certain cases, NIH's Office of the Director Offices may direct resources to address specific trans-NIH gaps in research training programs??e.g., to strengthen training in behavioral, interdisciplinary, women's health, or health disparities research.

Evidence: 1. NIH Research Training and Career Development Strategic Plan, FY 2004-2010 2. Department of Health and Human Services National Institutes of Health FY 2007 Justification of Estimates for Appropriations Committee - http://officeofbudget.od.nih.gov/ui/2007Budget_IC_Links.htm

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: NIH actively collects award performance information through a variety of sources including grant applications, annual progress reports of the activities and achievements of grant awards, and regular ongoing feedback mechanisms, such as the NIH-wide Training Advisory Committee and the NIH Research Training Inventory process. This information drives strategic decisions in funding and managing these programs as well as provides NIH with timely and credible data for determining program performance and implementing corrective strategies. For annual progress reports for continuation grants, project managers review changes in the training program, particularly the research project, academic status, or time distribution of activities; progress in the research performed and research training obtained during the prior year; and research training planned for the requested budget period. For non-competing grant progress reports more information on actual performance is needed. The form asks for updated information to justify budget adjustments and a list of trainees who have left or completed the program, but does not explicitly ask for performance data. To provide postdoctoral NRSA participants an added incentive to stay in research training and in research careers, the NRSA Program requires them to "pay back" the benefits they receive at critical junctures by conducting health-related research or scientific teaching. Individuals who have received NRSA-supported training are required to report annually on their activities to the NIH OER, until their payback obligation has been met. NRSA participants who fail to fulfill their service obligation are required to pay back the cost of the research training received, with interest.

Evidence: 1. Ruth L. Kirschstein NRSA Payback Agreement: http://grants1.nih.gov/grants/funding/416/phs6031.pdf 2. Ruth L. Kirschstein NRSA Individual Fellowship Progress Report for Continuation Support (PHS 416-9): http://grants2.nih.gov/grants/funding/416-9/phs416-9.htm 3. Non-Competing Grant Progress Report (PHS 2590): http://grants2.nih.gov/grants/funding/2590/2590.htm

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: All Federal managers and program partners??program managers, grantees (individuals and institutions), and collaborators??involved in the management, review, conduct, and administration of NIH research training and career development programs have designated roles and responsibilities for program stewardship. All NIH employees are required to have results-oriented performance plans or contracts by which their supervisors assess their performance. Personnel performance plans for research training officers in the NIH OER and at the NIH ICs, in particular, detail responsibilities for the monitoring and oversight of research training and career development awards. The NIH Training Officer in OER also is accountable for the achievement of GPRA goals developed for the programs. All NIH employees are carefully evaluated, and sustained unsatisfactory performance, violation of Federal laws or regulations, or gross negligence may lead to suspension and/or dismissal from Federal employment. NIH managers of research training and career development programs are held accountable for the overall performance of their programs. To ensure effective and efficient accomplishment of the NIH mission, NIH ICs are required to develop and maintain adequate systems of internal controls for oversight and monitoring of NIH programs. The NIH has implemented a number of internal control measures to ensure that policies and procedures are documented and followed. NIH awarding officials also are expected to recognize high-risk applicants and employ appropriate safeguards, including use of the Department of Health and Human Services (HHS) Alert System, to become aware of instances where special action or attention is required.

Evidence: 1. HHS Performance Management Program, revised January 17, 2006: http://hr.od.nih.gov/PerfMgmt/2006/documents/FINALPMAP1-17-06.doc and Sample Performance Plan of Research Training Officer. 2. NIH Policy Manual Chapter 1750, NIH Management Control Program: http://www1.od.nih.gov/oma/manualchapters/management/1750/ and Implementation Management Controls Compliance Model for Grants Management Policy Issuances (GMAC Policy and Procedure Announcement, 2001-01 dated October 11, 2001): http://odoerdb2.od.nih.gov/gmac/sources/gmac_announce_2001_01.html. 3. NIH Extramural Management Controls??Self-Assessment Questionnaire: http://oerdb.od.nih.gov/cfdocs/mgmt_controls/mgmt_controls_self_assesment.doc. 4. NIH Grants Manual Chapter 4.1.04.204??Responsibilities of NIH Grants Management Staff: http://odoerdb2.od.nih.gov/gmac/sources/nihgam_4.1.04.204.pdf.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: In annual Financial Status Reports, NIH grantees must provide justification for any significant discrepancies between their actual and projected funding. To ensure that research training and career development funds are directed to the areas of greatest need, obligated in a timely manner, and used for their intended purposes, responsibility for funding decisions and day-to-day management of research training and career development programs are the province of the individual NIH ICs. To ensure that funds are used as intended, the NIH has instituted the following requirements and procedures: Assurances by Applicant Organizations: The signature of an Authorized Organizational Official on a grant application certifies that the applicant organization is accountable for the appropriate use of any funds awarded and for the performance of the grant-supported project or activities resulting from the application. Grantee institutions must have in place accounting and internal control systems that provide for appropriate monitoring of grant accounts to ensure that obligations and expenditures are reasonable, allocable, and allowable in accordance with Federal cost principles. Financial Status Reports (FSRs): Recipients of NIH awards are required to document grant expenditures annually using FSRs. These reports detail the financial status of the grant according to the official accounting records of the grantee organization. Final FSRs must have no unliquidated obligations and must indicate the exact balance of unobligated funds. Unobligated funds must be either returned to the NIH or reflected by an appropriate accounting adjustment (e.g., reallocated). Flexibility in Award Activation Date: To permit flexibility in the timing of research training, institutional awards may be permitted to carry over funds from one budget period to the next. ICs, however, closely monitor the level of carryover funds and retain the right to manage the research training program by offsetting a training grant's out-year budget by an amount equivalent to the level of unobligated funds. If an institutional research training grant repeatedly falls short in filling its allotted training positions, the NIH may permanently reassign training positions to other NRSA training grants or fellowships, thereby ensuring that research training funds will be used for their intended purpose and in a timely fashion.

Evidence: 1. Code of Federal Regulations (45 CFR 74.21): http://www.mfia.state.mi.us/olmweb/ex/csm/510.pdf and 2. Code of Federal Regulations (45 CFR 92.20): http://a257.g.akamaitech.net/7/257/2422/12feb20041500/edocket.access.gpo.gov/cfr_2004/octqtr/45cfr92.20.htm. 3. OMB Circular No. A-133. Audits of States, Local Governments, and Non-Profit Organizations: http://www.whitehouse.gov/omb/circulars/a133/a133.pdf.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has an efficiency measure that is intended to show that automating the receipt of trainee appointment forms will help expedite the application process. Career development award applications are slated to be submitted electronically in June 2007, research fellowships in August 2007, and institutional research training grants in September 2007. These changes are expected to speed the time from submission of a grant application to receipt by the NIH, facilitate the peer review and grants management process to make more timely awards, and improve data integrity by eliminating the possibility of errors associated with data entry. Also, NIH has introduced a number of enhancements to its information technology (IT) systems. NIH has begun to develop internal systems to make a range of grants-related information available electronically to NIH staff. While these eRA efforts are still in progress, eRA will replace the hundreds of millions of pieces of paper generated annually from grant application receipt through award with a full and complete electronic record and will reduce substantially the time required to process applications. The NIH Office of Extramural Programs is collaborating with the NIH eRA team and ICs to design, test, and deploy enhanced features that will allow training grant directors to submit information on trainees electronically that is currently transmitted to the NIH on paper forms. The new system, known as X-Train, is expected to be fully operational by Fiscal Year (FY) 2010, transforming the existing paper process into a streamlined, end-to-end electronic data flow. X-Train will save staff time and reduce the possibility of error, thereby increasing efficiency and enhancing data integrity for program monitoring and evaluation purposes.

Evidence: 1. NIH Planned Transition Dates of Mechanisms for Electronic Submission Using the SF 424 (R&R) - http://era.nih.gov/ElectronicReceipt/files/Electronic_receipt_timeline_Ext.pdf 2. Electronic Submission of Grant Applications - http://era.nih.gov/ElectronicReceipt/ 3. X-Train:User Quick Guide - http://era.nih.gov/Docs/X-Train_Quick_Guide.doc

Does the program collaborate and coordinate effectively with related programs?

Explanation: NIH research training and career development award programs are collaborative ventures that rely heavily on coordination with NIH ICs, other Federal agencies, private foundations, and State institutions such as public universities. ICs follow NIH-wide guidelines for research training and career development while focusing on and supporting activities that address their respective missions and disease areas. Where there is overlap among ICs, the NIH promotes coordination through shared initiatives such as the NIH Blueprint for Neuroscience Research. The Blueprint addresses challenges that are best met collectively with pooled resources and expertise from 15 ICs to serve the entire neuroscience community. The NIH and its grantee institutions collaborate on research training and career development activities through shared responsibilities and funding. For example, the NIH provides stipends to help defray the cost of living for pre- and postdoctoral scholars in research training; however, the stipends are not expected to cover fully the cost of living and typically are supplemented by institutions with non-federal funds. The NIH shares responsibility for the research career development of new scientists with extramural institutions by requiring that institutions make an explicit commitment to providing career development awardees with protected time, equipment, facilities, and other resources, above and beyond the financial support provided by the NIH. Where appropriate, NIH collaborates with other agencies, private foundations, and professional societies to achieve shared research training goals and coordinate effectively with related programs with a minimum of overlap. In addition, the NIH reviews current and pending support of investigators to identify and eliminate overlap to preclude duplication of funding for scientific aims, specific budgetary items, or an individual's level of effort. The OER convenes and chairs monthly meetings of the Training Advisory Committee to provide an NIH-wide forum to identify and discuss issues related to research training and career development and to provide opportunities to coordinate activities pertinent to the review, administration, management, and evaluation of training grants, fellowships, and career awards. In a related effort, the NIH is collaborating with NSF and other Federal agencies to test the feasibility of utilizing existing data from the federally funded Survey of Doctorate Recipients to track the career outcomes of research training program participants. By leveraging and building upon existing infrastructures, this collaboration is expected to achieve substantial cost efficiencies.

Evidence:

Does the program use strong financial management practices?

Explanation: The program does not use strong financial management practices. In the Financial Section of the HHS' FY 2005 Performance and Accountability Report, the independent auditor indicated that "HHS continues to have serious internal control weaknesses in its financial systems and processes for producing financial statements. ?? These long-standing issues, including system and process limitations and expertise needed in meeting evolving financial reporting requirements, simultaneously with implementing new systems, will require a sustained commitment and qualified support team to resolve in preparation for FY 2006 and future years." In addition, the report specifically cites that NIH launched the Oracle General Ledger portion of the NIH Business System (NBS) because its previous legacy NIH Central Accounting System was not designed for financial reporting purposes. The report also indicates that "[a]lthough the Oracle General Ledger became the official accounting system of record ?? NIH was required to record approximately 120,000 nonstandard accounting entries with an absolute value of $5.6 billion to adjust budgetary and proprietary accounts. Additionally, the NBS does not provide for tracking manual or non-routine entries. As a result, adjustments and corrections cannot be readily identified."

Evidence: FY 2005 Performance and Accountability Report - http://www.hhs.gov/of/library/par05/pdfmenu/.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Within its research training and career development programs in particular, NIH has been proactive in addressing several deficiencies that have been identified in the 2005 Government Accountability Office report. To improve outcome tracking will: (1) Work with other Federal agencies to develop clear principles for Federal support of graduate and postdoctoral education and training in science and engineering and (2) Participate in a collaborative effort to develop plans to test the feasibility of conducting a multi-agency outcomes evaluation of Federal programs that support the research training of pre- and postdoctoral students. NIH fiscal policies are reassessed regularly and modified to reflect changes in the research and training environment. These changes are discussed with stakeholders, and budget models are evaluated before implementation. Some of the most recent changes adopted address the needs of a changing workforce, such as the health insurance and stipend adjustments for individual NRSA fellowships. Three key program partners are essential in identifying and addressing management deficiencies across the NIH. These groups??the Office of Management Assessment, the Administrative Restructuring Advisory Committee, and the Training Advisory Committee??meet regularly, develop agendas to address management deficiencies, and work collaboratively to design and implement corrective strategies.

Evidence: 1. GAO. Higher Education: Federal STEM Programs and Related Trends. October 2005. http://www.gao.gov/new.items/d06114.pdf. 2. Addressing the Nation's Changing Needs for Biomedical and Behavioral Scientists (2000). Washington, D.C.: National Academies Press: http://www.nap.edu/books/0309069815/html/.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Selection criteria for NIH research training and career development programs identify participants who have the highest probability of success to fulfill the programs' purposes. Through the NIH system of peer review, university scientists and other experts identify the top institutional programs that and individuals who merit research training and career development funding. Following peer review, program staff and national advisory councils of each NIH IC also may designate applications in areas of particular need for higher funding priority. This process allows individual ICs to target resources to particular areas of need in the fields that fall under their purview. These key design features ensure that resources directly address the programs' purpose.

Evidence:

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: NIH actively collects award performance information through a variety of sources including grant applications, annual progress reports of the activities and achievements of grant awards, and regular ongoing feedback mechanisms, such as the NIH-wide Training Advisory Committee and the NIH Research Training Inventory process. This information drives strategic decisions in funding and managing these programs as well as provides the NIH with timely and credible data for determining program performance and implementing corrective strategies. For annual progress reports for continuation grants, project managers review changes in the training program, particularly the research project, academic status, or time distribution of activities; progress in the research performed and research training obtained during the prior year; and research training planned for the requested budget period. For non-competing grant progress reports more information on actual performance is needed. The form ask for updated information to justify budget adjustments and a list of trainees who have left or completed the program, but does not explicitly ask for performance data.

Evidence:

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: NIH actively collects award performance information through a variety of sources including grant applications, annual progress reports of the activities and achievements of grant awards, and regular ongoing feedback mechanisms.

Evidence:

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation:

Evidence:

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Both measures show adequate progress towards achieving the long-term performance goals. NIH aims to ensure that NRSA funded pre- and postdoctoral trainees and fellows are at least 12% more successful in comparison to non-NRSA funded biomedical researchers at remaining in the biomedical research field and competing for projects. NIH exceeded the goal in FY 2005 and is on track to exceed it in FY 2006.

Evidence: 1. The long-term performance measures are listed below. 2. Department of Health and Human Services National Institute of Health FY 2007 Justification of Estimates for Appropriations Committee - http://officeofbudget.od.nih.gov/ui/2007Budget_IC_Links.htm

Does the program (including program partners) achieve its annual performance goals?

Explanation: The annual measures show adequate progress towards achieving the long-term performance goals. NIH aims to ensure that NRSA funded pre- and postdoctoral trainees and fellows are at least 12% more successful in comparison to non-NRSA funded biomedical researchers at remaining in the biomedical research field and competing for projects. NIH exceeded the goal in FY 2005, and is on track to exceed it in FY 2006.

Evidence: 1. The annual performance measures are listed below. 2. Department of Health and Human Services National Institute of Health FY 2007 Justification of Estimates for Appropriations Committee - http://officeofbudget.od.nih.gov/ui/2007Budget_IC_Links.htm

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: NIH's research training efficiency measure is newly developed. At this time, the baseline is 0% and NIH expects to have performance data reflecting implementation of a new effort to automate the receipt of trainees' appointment forms from academic institutions. Data are expected to be available at the end of FY 2007.

Evidence: 1. The annual efficiency measure is listed below. 2. Department of Health and Human Services National Institute of Health FY 2007 Justification of Estimates for Appropriations Committee - http://officeofbudget.od.nih.gov/ui/2007Budget_IC_Links.htm

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Since the establishment of the NRSA Program, several reports, including the congressionally mandated reports from the NAS, have examined NIH research training programs and the NRSA Program in particular. These studies generally have confirmed the high quality of the NRSA Program. NRSA trainees fare better than their counterparts with support from other sources (both Federal and non-Federal) on measures such as: program completion (including time to completion); progress toward research careers; attaining faculty or other research-related positions in academia and industry; successfully competing for research grants; and scientific productivity. NIH collaborates and partners with other programs to enhance the research training experience or to facilitate research training opportunities. Other research training programs exist that complement NIH research training and career development efforts. Training mechanisms supported by foundations and other Government agencies tend to address specific research areas critical to their respective missions and are distinct from the NIH purpose and goals. For example: ?? The Howard Hughes Medical Institute (HHMI) sponsors research training for graduate, medical, and dental students with an emphasis on laboratory-based sciences. Funding for research career development is limited to those who participated in HHMI programs as predoctorates. Furthermore, unlike the NIH, HHMI offers limited opportunities for students interested in clinical or behavioral research. ?? The National Science Foundation (NSF) sponsors training programs that focus on the fields of fundamental science and engineering outside of medicine. ?? The Department of Defense manages Congressional Special Interest Medical Research Programs on a limited number of specified diseases. Its funding strategy complements awards made by other agencies and specifically avoids duplication of long-term basic research and research training supported by the NIH.

Evidence: 1. NSF Funding Opportunities for Graduate Students - http://www.nsf.gov/funding/education.jsp?fund_type=2 and Funding Opportunities for Postdoctoral Fellows - http://www.nsf.gov/funding/education.jsp?fund_type=3 2. Department of Defense National Defense Science and Engineering Graduate Fellowship Program - http://www.asee.org/ndseg/ 3. Department of Energy Laboratory-Graduate Research Appointments - http://www.dep.anl.gov/highered/labgrad.htm 4. Society for Advancement of Chicanos and Native Americans in Science Fellowships and Scholarships - http://www.sacnas.org/oldWebsite/fellow.html 5. United Negro College Fund Scholarships and Grants - http://www.uncf.org/scholarships/index.asp 6. National Research Council (2005). Advancing the Nation's Health Needs: NIH Research Training Programs (p. 10) - http://darwin.nap.edu/books/0309094275/html 7. Coggeshall, P.E., and P.W. Brown. (1984). The Career Achievements of NIH Pre-doctoral Trainees and Fellows (p. viii). National Academy Press 8. Garrison, H.H., and P.W. Brown. (1986). Career Achievements of NIH Post-doctoral Trainees and Fellows. National Academy Press 9. Pion, G. M. (2001). The Early Career Progress of NRSA Pre-doctoral Trainees and Fellows (p. xvi, Executive Summary) - http://grants.nih.gov/training/career_progress/index.htm 10. HHMI Graduate Science Education and Medical Research Training Program - http://www.hhmi.org/grants/office/graduate/ 11. About the National Science Foundation: What We Do - http://www.nsf.gov/funding/aboutfunding.jsp 12. Overview of the Program. Department of Defense Research Program Fact Sheets - http://cdmrp.army.mil/pubs/fact.htm 13. National Institute of General Medical Sciences (September 1998). MSTP Study: The Careers and Professional Activities of Graduates of the NIGMS MSTP - http://publications.nigms.nih.gov/reports/mstpstudy/

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The NIH research training and career development programs have been the subject of multiple independent evaluations, including more than a dozen studies by the NAS over the past 30 years. In general, the studies indicate that former participants in NIH-funded research training and career development programs outperform their peers in the biomedical, behavioral, and clinical sciences on a variety of measures, including greater involvement in research (measured by factors such as subsequent grant applications and awards and longevity in research careers) and better track records in carrying out high-quality research (measured by citations in the scientific literature). The most recent NAS report, published in 2005 indicates "quality is an essential ingredient for progress. In this regard, the NRSA Program plays a unique role?? [setting] the standards for the entire research training establishment. In addition, they attract high-quality students into research and into fields of particular need. The record of success of NRSA holders in obtaining research funding is impressive." Though the NAS studies cover NIH career development, as well as research training programs, the NIH has engaged a team of outside evaluation experts to develop a plan for a full-scale evaluation of its research career development programs. The last major assessment of these programs, completed by the RAND Corporation in 1987, found that recipients submitted higher quality research grant applications, produced more??and higher quality??publications, and had longer research careers than their peers who either (1) did not apply for career development awards or (2) applied for, but were not successful in obtaining, career development awards. Because a number of new career development mechanisms were introduced during the 1990s, particularly in patient-oriented research, it is timely to undertake another major evaluation of the career development program. Institute-specific evaluations of research training and career development supplement the regular NAS studies by considering research training needs in particular areas and often are conducted by convening independent "blue-ribbon" panels of scientific leaders from universities around the country. Other focused evaluations are conducted by evaluation experts working under contract. An example of the latter approach was the 1998 outcome evaluation of the MSTP, a series of institutional research training grants administered by the NIGMS and intended to prepare trainees to be physician-scientists through combined M.D./Ph.D. training.

Evidence: 1. National Research Council (2005). Advancing the Nation's Health Needs: NIH Research Training Programs - http://fermat.nap.edu/books/0309094275/html 2. General Accountability Office (October 2005). Higher Education: Federal Science, Technology, Engineering, and Mathematics Programs and Related Trends, GAO Report (GAO-06-114) - http://www.gao.gov/new.items/d06114.pdf 3. The National Academies Policy on Committee Composition and Balance and Conflicts of Interest for Committees Used in the Development of Reports, May 12, 2003 - http://www.nationalacademies.org/coi/bi-coi_form-0.pdf 4. National Institute of General Medical Sciences (September 1998). MSTP Study: The Careers and Professional Activities of Graduates of the NIGMS MSTP - http://publications.nigms.nih.gov/reports/mstpstudy/ 5. Carter, G.M., Winkler, J.D., and Biddle, A.K. (August 1987). An Evaluation of the NIH Research Career Development Award. RAND Corporation, R3568-NIH - http://www.rand.org/pubs/reports/R3568/ 6. Garrison, H.H., and P.W. Brown. (1986). Career Achievements of NIH Post-doctoral Trainees and Fellows. National Academy Press 7. Coggeshall, P.E., and P.W. Brown. (1984). The Career Achievements of NIH Pre-doctoral Trainees and Fellows. National Academy Press 8. Federally Funded Survey of Doctorate Recipients - http://www.nsf.gov/statistics/showsrvy.cfm?srvy_CatID=3&srvy_Seri=5