Detailed Information on the
National Institutes of Health - Extramural Research Programs Assessment

Reach goal of 40% of business processes being done electronically by 2006.

Complete goal of expanding the range of available methods used to create, analyze, and utilize chemical libraries, which can be used to discover new medicines. Specifically, use these chemical libraries to discover 10 new and unique chemical structures that could serve as the starting point for new drugs.

Complete treatment and follow-up of participants in the ACCORD trial to determine effects of glycemia, blood pressure, and blood lipid treatment approaches to prevent CVD in diabetes.

Start at least one pilot clinical trial on promising interventions based on results of previous trials and new leads for drug discovery.

Continue conversion of business processes: 80% of business processes being done electronically by FY 2009.

Report findings of the primary results of the Bari2D Trial.

Identify at least one imaging or biological marker and/or clinical or neuropsychological evaluation method that will help researchers perform less expensive, shorter, and more efficient drug trials for AD.

Continue conversion of business processes: 85% of business processes being done electronically by FY 2010.

Develop a fabrication process to miniaturize the prototype directional microphone so that is fits a hearing aid by 2006.

Reach goal of 25% of business processes being done electronically by 2005.

Integrate nanosensors and nanoparticles into a platform technology for development in applied setting by 2005.

Identify therapeutic compounds by 2005 that may be tested to determine absorption, distribution, metabolism, excretion, and toxicity.

Complete recruitment of 10,000 patients by 2005 for the ACCORD diabetes study.

Launch the Alzheimer's Disease Neuroimaging Initiative to evaluate techniques and biomarkers by 2005.

Reach goal of 40% of business processes being done electronically by 2006.

Integrate nanotechnology-based components into a system capable of detecting specific biomarkers (molecular signature) by 2006.

Fully implement the Small Molecule Repository so that it is functional and supplying molecules to screen by 2006.

Report outcome of the Look AHEAD intensive weight loss intervention trial by 2006.

Identify 1,000 new late onset Alzheimer's Disease families by 2006.

Begin development of predictive models for absorption, distribution, metabolism, excretion, and toxicity (ADME/Tox) behavior of bioactive compounds by 2007.

Complete at least 90% of the total enrollment for the Folic Acid for Vascular Outcome Reduction in Transplantation (FAVORIT) trial by 2007.

Identify and characterize molecular events that may prove to be targets for treating or preventing Alzheimer's disease through initiatives and projects focused on mechanistic and basic studies by 2007.

Continue conversion of business processes: 75% of business processes being done electronically by FY 2008.

Use chemical libraries in high-throughput biological screens.

Review and evaluate collectively, indicators of Look AHEAD's progress to date (measures such as safety-monitoring analyses, data quality, participant retention, and emerging positive or negative outcome trends) in order to determine whether the science is progressing appropriately--in accord with the clinical trial's protocol--and whether the trial will be continued.

For at least one promising drug candidate for the treatment of AD, complete at least one of the four preclinical steps necessary for regulatory approval: chemical optimization; proof of efficacy in an animal model relevant to the disease; pharmacokinetic profiling; and/or early toxicology screening.

Reach goal of 55% of business processes being done electronically by 2007.

Measure: By 2006, develop one or more prototypes for a low power, highly directional hearing aid microphone to help hearing-impaired persons better understand speech in a noisy background.

Explanation:Hearing aids currently available are not effective in restoring a listener's ability to sort out a single speech sound from competing sources. Targets: 2003 - design/test a device (diaphragm) that responds to sound; 2004 - Design/test the electronic circuitry to create a sound output from the diaphragm; 2005 - Combine diaphragm and electronic output circuitry into a directional microphone; and 2006 - Develop a fabrication process to miniaturize the prototype directional microphone so that it fits into a hearing aid.

Measure: By 2006, integrate nanotechnology-based components into a system capable of detecting specific biomarkers (molecular signature) to establish proof of concept for a new approach to the early detection of cancer, and ultimately, cancer preemption.

Explanation:Nanoscience allows scientists to measure and monitor changes within cells at the level of multiple atoms in real time. Application of nanotechnology by creating a new platform for high-throughput diagnostics would lead to early detection and prevention of cancer. Targets: 2003 - select substrate nanotechnology fabrication techniques; 2004 - Establish 1 core lab to identify the most promising applications; 2005 - Integrate nanosensors and nanoparticles into a platform technology for development in applied setting; and 2006 - Integrate nanotechnology-based components into a system capable of detecting biomarkers.

Measure: By 2009, expand the range of available methods used to create, analyze, and utilize chemical libraries, which can be used to discover new medications. Specifically, use these chemical libraries to discover 10 new and unique chemical structures that could serve as the starting point of new drugs.

Explanation:Many existing medicines are becoming ineffective due to antibiotic resistance. To speed up the discovery of new drugs, scientists need to have access to larger collections of chemicals to test. Targets: 2003 - Fund 2 additional Centers of Excellence to develop chemical libraries and high-throughput methods for screening potential therapeutic compounds; 2004 - Investigate at least 6 innovative methods to synthesize chemical libraries; 2005 - Identify therapeutic compounds; 2006 - Fully implement the Small Molecule Repository so that it is functional and supplying molecules to screen; 2007 - Develop better models for predicting absorption, distribution, metabolism, excretion, and toxicity of new compounds; 2008 - Identify 4 unique chemical structures that have gone through replication and preclinical tests and could serve as starting points for new drug development; and 2009- Identifying 10 new unique chemical structures.

Measure: Assess the efficacy of at least three new treatment strategies to reduce cardiovascular morbidity/mortality in patients with Type 2 diabetes and/or chronic kidney disease.

Explanation:For both diabetes and kidney disease, premature cardiovascular disease is the major cause of death. Due to the rate of morbidity/mortality associated with premature cardiovascular disease in diabetic and kidney disease patients, assessing the efficacy of at least three new strategies is ambitious and consistent with the goal of NIH's extramural research agenda which is "to encourage and support research, investigations, experiments, demonstrations, and studies...related to the maintenance, detection, diagnosis, treatment, rehabilitation, and prevention of human disease and disorders." Included is additional detail on the abbreviated targets. Targets: 2003 - Assess the effect of intensive vs. conventional glycemic control; 2004 - Complete recruitment of 5,000 patients for a diabetes study; 2005 - Complete recruitment of 10,000 patients for a diabetes study; 2006 - Report outcome of the Look AHEAD intensive weight loss intervention trial; 2007 - Complete recruitment of 4,000 patients for the FAVORIT trial; 2008 - Complete Phase 2 of the Look AHEAD lifestyle intervention trial; 2009 - Based on the completed BARI 2D trial, determine whether coronary revascularization provides reduction in mortality; 2010 - Determine the efficacy of 1 cardiovascular intervention being tested in ACCORD, which will inform clinical practice; and 2011 - Report FAVORIT trial results showing the effect of using multivitamin therapy to lower homocysteine levels in chronic kidney disease.

Measure: Identify at least one clinical intervention that will delay the progression, delay the onset, or prevent Alzheimer's disease.

Explanation:Alzheimer's disease is a progressive, at present irreversible, brain disease. Alzheimer's disease was the seventh leading cause of death in 2004, according to the CDC National Vital Statistics System's National Center for Health Statistics. With the aging of the U.S. population, Alzheimer's disease is an important area of extramural research and is encompassed within NIH's broad mission to improve health by supporting research into the cause, diagnosis, prevention, and cure of human diseases, physical and behavioral. Included is additional detail on the abbreviated targets. Targets: 2003 - initiate a double-blind, placebo-controlled trial of simvastatin (medication used to lower cholesterol) to determine whether it can slow AD progression; 2004 - implement strategies to facilitate drug discovery; 2005 - launch the Alzheimer's Disease Neuroimaging Initiative to evaluate techniques and biomarkers; 2006 - Identify 1,000 new late onset AD families for gene study; 2007 - Identify and characterize molecular events that may prove to be targets for treating or preventing Alzheimer's disease through initiatives and projects focused on mechanistic and basic studies; 2008 - Identify new leads for drug targets; 2009 - Start pilot trials on promising interventions; 2010 - Identify the most promising imaging and biological markers, and clinical and neuropsychology evaluation methods for drug trials; 2011 - Start new and efficient full-scale trials using the markers identified in 2010; 2012 - Identify next generation of compounds for testing in pilot clinical trials; and 2013 - Identify at least 1 clinical intervention.

Measure: By 2013, provide greater functionality and more streamlined processes in grants administration by continuing to develop a NIH Electronic Research Administration System (eRA).

Explanation:The NIH Electronic Research Administration (eRA) is NIH's infrastructure for conducting interactive electronic transactions for the receipt and review of grant applications, and the monitoring and administration of NIH grant awards to biomedical investigators. eRA aims to move internal work flows from paper-based business processes to electronic submission and receipt of grant applications. Targets: 2003 Implement electronic reporting of all 65 newly on-line institutions participating in the Federal Demonstration Partnership; 2004 Expand availability of electronic progress reporting from 145 FDP institutions to all grantee institutions; FY 2005 - Reach goal of 25% of business processes being done electronically; FY 2006 - Reach goal of 40% of business processes being done electronically; FY 2007- Reach goal of 55% of business processes being done electronically

Measure: Advance two emerging new strategies for treating muscular dystrophy to the point of preparedness for clinical trials.

Explanation:There is currently no treatment that can stop or reverse the progression of any form of muscular dystrophy. However, advances in the understanding of disease mechanisms, diagnostics, and research technologies make this an opportune time to emphasize therapeutic development.

Measure: Determine the optimal tailored treatment regimen for patients with early stage breast cancer that maximizes the benefits of chemotherapy while minimizing the side-effects of unnecessary treatment.

Explanation:The majority of women with early-stage breast cancer are advised to receive chemotherapy in addition to radiation and hormonal therapy, yet research has not demonstrated that chemotherapy benefits all of them equally. The use of a molecular profiling test in clinical decision-making may more precisely estimate a woman's risk of cancer recurrence than standard characteristics normally used to assess recurrence risk. This may spare women unnecessary treatment if chemotherapy is not likely to be of substantial benefit.

Measure: Provide at least one new or significantly improved minimally-invasive treatment for clinical use in patients using image-guided interventions.

Explanation:

Is the program purpose clear?

Explanation: Program purpose is laid out broadly by Title IVof the Public Health Service Act. The National Institutes of Health (NIH) is established to encourage and support research, investigations, experiments, demonstrations, and studies in the health sciences related to the maintenance, detection, diagnosis, treatment, rehabilitation, and prevention of human disease and disorders. Section 405 specifically authorizes NIH to enter into a contracts, grants, and cooperative agreements for research and sets technical and scientific peer review as a requirement.

Evidence: Public Health Service Act and peer review regulations from the Code of Federal Regulations; Rare Disease Act of 2002 (P.L. 107-280).

Does the program address a specific and existing problem, interest or need?

Explanation: The NIH mission to improve health is a broad one, and encompasses research into the cause, diagnosis, prevention, and cure of human diseases, physical and behavioral. The 238 disease areas tracked by NIH reflect disease burden (cancer, heart disease, stroke); emerging public health threats (SARS, biodefense, HIV/AIDS); new technologies and novel approaches for detection, treatment, and information transfer to help speed up research; and diseases uniquely addressed by NIH ("orphan" diseases affecting a small population, racial and ethnic disparities, gender differences that are seen in certain diseases and disorders).

Evidence: NIH disease funding table; guidance to applicants on the requirements for inclusion of women, minorities, and children as subjects in clinical studies; the NIH Roadmap for medical research; FY 2000 NIH Report on Disease-Specific Estimates of Direct and Indirect Costs of Illness and NIH Support.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: NIH is unique in that it is the only agency, governmental or private, that has a broad mission of improving the Nation's health through funding biomedical and behavioral research. The NIH Extramural Research program funds a wide spectrum of activities that are not typically funded by the private sector, such as basic research, research instruments and equipment, publicly accessible databases, specimen and tissue repositories, animal resources, early stage clinical trials, and development of treatment guidelines that lead to state-of-the-science standards of care.

Evidence: Title IV of the Public Health Service Act.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The NIH Extramural Research program is designed to use merit-based peer review to support grant funding decisions. NIH is one of the few Federal agencies that has a legislative requirement for peer review of grants, followed by oversight by Institute/Center advisory councils. A closely-monitored and scientifically-rigorous peer review process allows NIH to fund the most meritorious grants with the highest potential for discovery. After an award is made, NIH staff monitors grantee progress for adherence to the approved scientific research plan to appropriate cost principles. Past independent assessments have been complementary to the NIH peer review design.

Evidence: Section 492 of the Public Health Service Act; Code of Federal Regulations (42 CFR PART 52h) on Scientific Peer Review of Research Grant Applications and Research and Development Contract Projects; the NIH Grants Policy Statement; 1994 General Accounting Office Report on "Reforms Needed to Ensure Fairness in Federal Agency Grant Selection"; 1993 Institute of Medicine Report on "Strategies for managing the Breast Cancer Research Program."

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: NIH uses external advisory groups to identify scientific opportunities to ensure that resources are appropriately distributed. The NIH Extramural Research program has fourteen mechanisms of support designed to give NIH the flexibility to effectively target emerging public health concerns and scientific opportunities as they change. Investigators could acquire funding by proposing their own ideas to NIH. Other funding mechanisms allow NIH to target funds so that specific segments of the grantee population could be reached (e.g., young scientists, research with potential for commercialization, minority investigators). In situations where NIH has an identified problem, NIH publishes program announcements and Request for Applications in the NIH Guide for Grants and Contracts. Regardless of the funding mechanism used, all NIH awards are made based on the merit and the appropriateness of the grantee population.

Evidence: NIH web site on extramural research funding mechanisms: www.grants1.nih.gov/grants; Electronic Research Administration Activity Codes, Organization Codes, and Definitions Used in Extramural Programs, 2003.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Since 2003, NIH has selected 28 representative research outcome goals as proxies for NIH's overall performance. These goals are specific to a disease or a definable problem, with reference to a metric and/or a date for progress/completion, and are on a continuum of risk (i.e. likelihood to attain the goal) and time (short-, medium-, long-term). The 5 goals reviewed represent a continuum of health research conducted at NIH, ranging from basic to applied to clinical research: 1) create a low-power, highly directional microphone prototype that could be used to create better hearing aids; 2) address knowledge gaps in nanotechnology by supporting core labs to generate fabrication standards; 3) help reverse the trend of declining new drug applications and approval by using chemical libraries to identify 10 chemical structures; 4) reduce disease burden (diabetes, cardiovascular, and kidney disease) by conducting clinical trials on treatment strategies; and 5) effective intervention to prevent/delay the onset of a disease (Alzheimer's disease).

Evidence: Annual NIH GPRA Plans; Institute GPRA Goal Implementation Plans; May 1998 Report of the National Advisory General Medical Sciences Council Subcommittee for the Division of Pharmacology, Physiology, and Biological Chemistry; 2004 National Cancer Institute Cancer Nanotechnology Plan; Leon J, Cheng CK, Neumann PJ. Alzheimer's Disease Care: Costs and Potential Savings. Health Affairs (Millwood). 1998;17(6):206-16; article on statistical power needed for clinical trials of homocysteine-lowering in reducing cardiovascular disease. Arterioscler Thromb Vasc Biol. 2002; 22:488-491; NIH Roadmap overview; statistics on hearing loss.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Goal 1 has an ambitious 3 year goal to design and test a hearing device prototype. Goal 2 aims to develop nanotechnology components into a system that is capable of detecting biomarkers (molecular signature) that could lead to early cancer detection. Goal 3 provides infrastructure and support capabilities for a public collection of chemically diverse small molecules and high-throughput screening of drug compounds. Goal 4 has an ambitious 8 year target to conduct a set of four large-scale, outcome-oriented clinical trials to assess treatment strategies to reduce cardiovascular morbidity/mortality in patients with type 2 diabetes and/or chronic kidney disease. Each clinical trial has a detailed study timeline of protocol development, protocol implementation, recruitment, follow-up, and close-out/data analysis. Goal 5 will use information gathered from pre-clinical drug discovery, neuroimaging techniques, and genetic risk factors to identify the most likely clinical trial candidates for Alzheimer's disease.

Evidence: Summary of Goals: 1) To develop 1 or more low-power, highly directional hearing aid microphone prototypes by 2006; 2) To establish a proof of concept for a new nanotechnology approach for early detection of cancer and cancer preemption by 2006; 3) To use chemical libraries to discover 10 new unique chemical structures that could serve as the starting point for new drugs by 2009; 4) To assess at least 3 treatments for reducing cardiovascular morbidity/mortality in patients with type 2 diabetes and/or chronic kidney diseases by 2011; and 5) To identify at least 1 clinical intervention that will delay the progression/onset or prevent Alzheimer's disease by 2013.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Each goal has specific and measurable annual targets that represent incremental steps toward accomplishing the long-term goal. The annual targets represent the "building blocks" or "make or break" steps for achieving a goal within a specific timeframe. For instance, Clinical trials are designed by scientific and statiscal experts based on knowledge and principles of clinical resaerch methodology, including biostatistical methods, and lessons learned from previous studies. Therefore, a clinical trial cannot begin until a targeted number of patients have been recruited by a certain time frame. These "make or break" steps serve as markers/milestone for NIH to track progress.

Evidence: Annual NIH GPRA Plans; Institute Strategic Implementation Plans; NIDCD summaries of annual and monthly progress reports from grantees; 2003 NIGMS Chemistry Center Grants to Expand Drug Discovery Toolkit press release; NCI National Characterization Laboratory Business Plan; NIA protocol for the simvastatin trial; NHLBI and NIDDK clinical trial protocols for the Look Ahead (Action for Health in Diabetes) trial, the ACCORD (Action to Control Cardiovascular Risk in Diabetes) trial, the BARI (Bypass Angioplasty Revascularization Investigation in Type 2 Diabetes) trial, and the FAVORIT (Folic Acid for Vascular Outcome Reduction in Transplantation) trial.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Each goal has established annual baselines and targets for the duration of the goal. Outyear targets and baseline are re-evaluated annually and revised to reflect the latest scientific evidence and progress. The annual targets are ambitious for varying reasons, including: the short time-frame proposed to reach a goal that no other entity has been able to accomplish; the gaps in the state of scientific and medical knowledge that it would address; and usage of clinical trial implementation strategies as benchmark to track progress.

Evidence: Annual NIH GPRA Plan; Institute Implementation Plans; Institute summaries of patents files based on work done for the microphone/hearing aid goal.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Scientific targets are included in the solicitations of research grants. Corrective actions are made accordingly. The level of progress review and accounting of deliverables depend on the grant funding mechanism used (grants vs. cooperative agreements vs. contracts). As a project becomes more complex and multi-centered, NIH staff keeps current on progress made via a variety of ways, including weekly/monthly/quarterly progress reports, bi-weekly conference calls, and visits to partnering laboratories. At the minimum, all grantees are required to provide an annual progress report. A grantee's progress determines whether or not a grant/contract will be renewed or terminated.

Evidence: NIH Grants Policy Statement; NIH Policy Manual Chapter 4444 -- Evaluation of Grant Progress Reports by Program Officials; Institute Implementation Plans.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: NIH uses the PHS evaluation funds and sets aside funding to evaluate its extramural programs to improve program management and performance. For instance, a draft HHS OIG Report on late closeouts (grantee institutions that submit late closeout documentation) identified 5 findings and 4 recommendations. In response to this Report, an electronic closeout module was developed in IMPAC II, NIH's grant tracking system. All extramural Research awards are subject to frequent independent external evaluation at numerous points. Grant applications are peer-reviewed for scientific merit, reviewed by institute Advisory Council for program relevance, and monitored by a Data Safety and Monitoring Board (whenever human subjects are involved). Peer-reviewed presentations, publications, and patent applications resulting from an award also receive independent evaluation from outside reviewers.

Evidence: May 2000 GAO Report, "Improvements Needed in Monitoring Extramural Grants"; HHS OIG Draft Report on NIH Grants Management: Late Awards; Overview of Federal Advisory Committees at the NIH (Office of Federal Advisory Committee Policy).

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: NIH has a budget allocation mechanism that tracks grant awards made. At the Institute level, Advisory Councils convene three times a year to review and recommend funding of grant applications. While majority of the budget allocation is used to fund grants based on scores from the initial review group, Institute Directors have the discretion to set aside funds for what they consider as high-priority research areas. The program staff are able to track annual and total cost of grants. However, at this time, NIH's budget presentation does not explicitly tie budget resource levels to annual and long-term performance targets. The budget requests do not show how much it would cost to achieve the performance results.

Evidence: Annual Congressional Justification; Annual GPRA Plan.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Currently, NIH does not have a plan to address how the agency would revamp its budget requests for extramural research activities to explicitly tie the accomplishment of goals to resource levels.

Evidence: Insitute operations center memorandum summarizing new recruitment targets, new reimbursement plans, travel reimbursement arrangements, modification of entrance criteria for women.

If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals?

Explanation: The NIH Extramural Research Program is the largest public investment, and therefore, is not comparable to other programs. NIH is unique in that it is the only agency, governmental or private, that has a broad mission of improving the Nation's health through funding biomedical and behavioral research. The NIH Extramural Research program funds a wide spectrum of activities that are not typically funded by the private sector, such as basic research, research instruments and equipment, publicly accessible databases, specimen and tissue repositories, animal resources, early stage clinical trials, and development of treatment guidelines that lead to state-of-the-science standards of care.

Evidence:  

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: Institute priorities are developed during NIH's annual budget formulation process at annual strategic planning retreats. Priorities are set based on multiple factors, including scientific importance/relevance, emerging public health threat, and potential public health benefits. Only projects of the highest merit are approved by the Director for new/continued funding. Assessments by groups of external advisors are also often used to establish research priorities. All NIH grant applications are competitively reviewed by outside experts for quality, program relevance, and potential performance outcome. Promising concepts for new programs are reviewed and approved by Institute advisory councils. Grants are assigned a priority score based on scientific merit to guide funding decisions. Depending on the state of the science in a particular field, different funding mechanisms are used. For instance, if a field is newly emerging, NIH might use a small or exploratory/developmental grant to jump start the field.

Evidence: NIH Policy Manual 54513 -- Management and Procedures of National Advisory Councils and Board in their Review of Extramural Activities; NIH Policy Manual 1805 -- Use of Advisors in Program and Project Review and Management.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Performance data is collected when grants/contracts are peer reviewed. NIH actively monitors grantees by progress reports, correspondences, audit reports, site visits, specialized programmatic reports, publications, deliverables, and other available information. When a grantee fails to comply with the terms and conditions of the award, NIH takes enforcement action ranging from modifying the terms of the award, suspension, and termination and withholding of support. Annual performance data from grantees is used to determine the future scientific directions of the NIH Extramural Research program.

Evidence: The NIH Grants Policy Statement.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Managers for the goals are systematically and explicitly held accountable for achieving the performance goal in their performance contracts. NIH grantees and contractors are fully accountable for costs, schedule, and performance results. Contractors are bound by the delivery/reporting requirements of their contracts to provide deliverables in accordance with the prescribed delivery terms. As a management tool, NIH places grantees on the HHS Alert List to alert other grants officials to high-risk applicants.

Evidence: 45 CFR 74.51 NIH Grants Policy Statement; HHS Grants Administration Manual Chapter 1-06 HHS Transmittal 89.01, HHS Alert System; NIH Policy Manual Chapter 1750, NIH Management Control Program; NIH employee performance contracts.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Historically, obligations are made in a timely manner and funds are spent for the purpose intended. However, in FY 2003, a lapse of $7 million due to a double-entry error occurred. NIH has taken steps to ensure that appropriated funds are spent for the purpose intended. For instance, NIH's grants financial analysis function has been moved to the Office of Extramural Activities, to place it closer to the grants staff and grants information. A detailed plan has been formulated and a monthly spending status report has been designed to strengthen management and financial oversight.

Evidence: Code of Federal Regulations; NIH obligation rates.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: Through the creation of the "most efficient organization," NIH has been successful in A-76 outsourcing competition in the area of Extramural Activities Support Services. A new NIH Division of Extramural Support Activities is in place to improve organizational effectiveness, efficiency, and accountability through a consolidation of similar activities across all NIH grants offices to attain the efficiencies detailed in the A-76 competition. In the area of information technology, NIH is implementing an electronic research administration system that would enable the elimination of millions of pieces of paper generated annually from grant application receipt through award.

Evidence: NIH MEO summary of the structure of the new organization; eRA Capital Asset Plan.

Does the program collaborate and coordinate effectively with related programs?

Explanation: NIH's Early Notification System facilitates the sharing of Request for Applications, Program Announcements, and Early Concept announcements across the 27 NIH Institutes, AHRQ, NASA, and CDC. Early notification of program plans with research partners allow NIH to collaborate, coordinate, and share information. NIH's electronic Research Administration systems development is closely coordinated with other NIH, HHS, Federal, and grantee systems development efforts. Various eRA components are being used by up to 21 other agencies.

Evidence: Early Notification User List; Annual Progress Report on the Implementation of P.L. 106-107.

Does the program use strong financial management practices?

Explanation: For FY 2003, NIH was audited as part of HHS' consolidated (top-down) audit. While HHS sustained an unqualified ("clean") audit, the HHS FY 2003 Independent Auditor's Report on Financial Statements and Management Response cited "serious internal control" problems with the HHS financial systems. HHS' financial statement production processes were cited as material weaknesses. The Report noted that these weaknesses caused delays in meeting accelerated reporting deadlines and hundreds of millions of dollars of unexplained differences in reconciliations and account analyses. Until the NIH New Business System (NBS)/HHS-wide Unified Financial Management System is fully deployed, the preparation of financial statements will continue to be manually intensive and time consuming. For NIH specifically, the Auditor's Report noted that the NIH Central Accounting System was not designed for financial reporting purposes and did not apply the U.S. Standard General Ledger at the transaction level.

Evidence: HHS FY 2003 Performance and Accountability Report: Section IV - Independent Auditor's Report of Financial Statements and Management Response; NIH Corrective Action Plan.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: NIH, in conjunction with HHS, is developing the New Business System part of HHS' Unified Financial Management System, which will address many of the issues contained in the material weaknesses. NIH has a Corrective Action Plan that lays out the problem areas, the corrective action milestones, the responsible point-of-contact, a target date, an actual completion date, and current status/accomplishments.

Evidence: HHS FY 2003 Performance and Accountability Report: Section IV - Independent Auditor's Report of Financial Statements and Management Response; HHS UFMS/NIH NBS Implementation Plan.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: NIH Extramural Research awards are peer-reviewed to ensure that only research of the highest quality is supported. NIH complies with the Federal Acquisition Regulations. With a few exceptions as prescribed by the FAR, NIH announces JOFOCs (Justification for Other than Full and Open Competition) in the FedBizOps, which lists notices of proposed procurement actions available to the public, and are reviewed and evaluated in accordance with principles that generally apply to both solicited and unsolicited proposals.

Evidence: FAR 6.302; NIH Grants Policy Statement.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: All 5 representative Extramural Research performance outcomes goals are on track. Two of the goals will be completed by 2006. Markers for progress include: patents in progress and secured, prototypes designed and tested, licensing in progress and secured, new company started, partnerships formed and in progress, and peer-reviewed publications to release new knowledge. For the long-term goals, detailed implementation plans are in place to track milestones and deliverables. Markers for progress include: grants funded, acquisition and outfitting of lab space, hiring of key personnel, number of human subjects recruited, development of "pilot" projects (e.g., synthesis of a few small pilot chemical libraries in order to evaluate new methodologies).

Evidence: Annual Targets in the NIH GPRA Plan; publications; Institute progress reports.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Goal #1 Microphone for hearing aid: achieved annual goal to design and test a diaphragm that responds to sound and is based on the ears of Ormia Ochracea. Goal #2 Proof of concept using nanotechnology to detect cancer biomarkers: met goal to develop a partnership with the National Institute of Standards and Technology to perform physical and chemical characterizations of nanodevices, a necessary step for proteomic analysis. Goal #3 Chemical libraries to isolate and screen candidate drug compounds: funded 5 new centers and 7 planned grants to develop a natural products drug discovery program. Goal #4 reduce cardiovascular disease: funded the first randomized clinical trial to show that glucose levels can alter a measure of atherosclerosis (a marker for heart and cerebrovascular diseases) and published results in the New England Journal of Medicine (June 2003). Goal #5 Intervene Alzheimer's disease progression: Initiated a double-blind, placebo-controlled trial of simvastatin to determine whether it can slow down the rate of progression in AD.

Evidence: Annual Targets in the NIH GPRA Plan; publications; Institute progress reports.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: NIH recently won a Grants Management Support A-76 competition. The formation of a "Most Efficient Organization" will lead to increased productivity. Support services to be provided for grants management will be reorganized and redistributed on as-needed basis throughout the Institutes. A workforce plan has been developed to reduce duplication and overlap to ensure resource redirection toward mission-critical areas. Internet-Assisted Review reduced the time spent in review meetings by 33%. Electronic scanning of applications saved an estimated $5.5 million in FY 2003. The average operational cost per award dropped from $360 to $340.

Evidence: NIH Director's Performance Plan; NIH Most Efficient Organization Summary for NIH Extramural Activities Support Services; NIH Electronic Research Administration http://era.nih.gov.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: NIH is unique in that it is the only agency, governmental or private, that has a broad mission of improving the Nation's health through funding biomedical and behavioral research. The NIH Extramural Research program is unique in that it funds a wide spectrum of biomedical research activities that are not typically funded by the private sector, such as basic research, research instruments and equipment, publicly accessible databases, specimen and tissue repositories, animal resources, early stage clinical trials, and development of treatment guidelines that lead to state-of-the-science standards of care. No other Federal or private programs fund biomedical research with the same level of program scope.

Evidence:  

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Recent GAO, IG, IOM Reports concluded that NIH programs are working reasonably well, but have identified areas for improvement. For instance, a 2000 GAO Report concluded that improvements are needed in monitoring extramural grants. NIH has since followed the GAO recommendations to address issues such as late closeout of grants. NIH has 50 evaluation studies (financed by the PHS Evaluation Fund) in progress to evaluate program effectiveness. Specific to the 5 representative goals, none of the goals have been independently evaluated because the projects have not been completed yet. However, as a standard practice, grants are peer reviewed based on merit and all 5 Institutes plan on convening an independent board of experts to evaluate whether the goals are accomplished. Also, NIH grantee publications serve as an indicator of how effective the program is achieving results, as top-tier journals often reject 50 percent of the submitted manuscripts before review and then reject up to another 25% after review.

Evidence: GAO Reports; IG Reorts; IOM Reports.