Detailed Information on the
National Institutes of Health - Intramural Research Assessment

Initiate preclinical studies on the nature of stem cell migration in adult tissue.

(FY08) Reallocate laboratory resources based on extramural reviews of 25% of principal investigators each year by Boards of Scientific Counselors.

Reallocate laboratory resources based on extramural reviews of 25% of principal investigators each year by Boards of Scientific Counselors.

Reallocate laboratory resources based on extramural reviews of 25% of principal investigators each year by Boards of Scientific Counselors.

Reallocate laboratory resources based on extramural reviews of 25% of principal investigators each year by Boards of Scientific Counselors.

Reallocate laboratory resources based on extramural reviews of 25% of principal investigators each year by Boards of Scientific Counselors.

Reallocate laboratory resources based on extramural reviews of 25% of principal investigators each year by Boards of Scientific Counselors.

Reallocate laboratory resources based on extramural reviews of 25% of principal investigators each year by Boards of Scientific Counselors.

Complete goal of developing a knowledge base on chemical effects in biological systems using a systems toxicology or toxicogenomics approach.

Initiate preclinical studies on the nature of stem cell migration in adult tissue.

(FY09) Reallocate laboratory resources based on extramural reviews of 25% of principal investigators each year by Boards of Scientific Counselors.

Begin biologic assessment of the most likely diabetes/obesity susceptibility genes in regions of linkage/association.

HapMap III: Analyze data from samples from additional populations to assess how well the genome-wide HapMap applies to additional populations, as well as to figure out how to choose HapMap SNPs to make them most useful for additional populations.

Formulate a biocompatible cell encapsulation agent designed to protect and track mesenchymal stem cells for administration to patients to promote cell survival and engraftment.

Formulate a biocompatible cell encapsulation agent designed to protect and track mesenchymal stem cells for administration to patients to promote cell survival and engraftment.

Initiate preclinical studies on the nature of stem cell migration in adult tissue.

Initiate preclinical studies on the nature of stem cell migration in adult tissue.

Initiate preclinical studies on the nature of stem cell migration in adult tissue.

Demonstrate that encapsulated cells can be tracked non-invasively by X-ray computed tomography.

(FY10) Reallocate laboratory resources based on extramural reviews of 25% of principal investigators each year by Boards of Scientific Counselors.

Complete goal of establishing the role of genetic factors in three major diseases for which health discrepancies are noted between populations.

Test the hypothesis that encapsulated MSCs will provide increased MSC survival in normal animals.

Reallocate laboratory resources based on extramural reviews of 25% of principal investigators each year by Boards of Scientific Counselors.

Reallocate laboratory resources based on extramural reviews of 25% of principal investigators each year by Boards of Scientific Counselors.

Release Phase I core pooled data with documentation to the public, create a web utility for sharing Family Blood Pressure Program data, and hold a training workshop for the scientific community by 2006.

Enhance the Chemical Effects in Biological Systems database to allow the capture and integration of transcriptomics, proteomics, and toxicologic data for the same compound by 2006.

Complete optical imaging probe development by 2006, to help with tracking the mobility of stem cells within cardiovascular tissues.

Reallocate laboratory resources based on extramural reviews of 25% of principal investigators each year by Boards of Scientific Counselors.

Perform initial whole genome scan for prostate cancer susceptibility genes in the C-GEMS study by 2007.

Enhance electronic sharing of 'omics and biology endpoint data in the Chemical Effects in Biological Systems database by 2007.

Measure: By 2012, reallocation of laboratory resources based on extramural reviews by Boards of Scientific Counselors.

Explanation:Repeating Target: Conduct BSC review of 25% of Principal Investigators to assess quality of science in order to prioritize resources.

Measure: By 2010, establish the role of genetic factors in three major diseases for which health disparities are noted between populations.

Explanation:Understanding how genetic variations contribute to various diseases will hopefully lead to a better understanding of why individuals are at particularly high risk of developing health problems. Targets: 2005 - Collect a cumulatiave total of 5.8 million genotypes from the FUSION study. 2006 - Release Phase I core pooled data with documentation to the public, create a web utility for sharing Family Blood Pressure Program data and hold a training workshop for the scientific community. 2007 - Perform initial whole genome scan for prostate cancer susceptibility genes in the C-GEMS study. 2008 - Analyze data regarding application of SNPs from initial phases of HapMap to additional populations. 2009 - Begin biologic assessment of the most likely diabetes/obesity susceptibility genes in regions of linkage/association. 2010 - Complete goal.

Measure: By 2008, develop a knowledge base on chemical effects in biological systems using a systems toxicology or toxicogenomics approach.

Explanation:Chemicals in the environment and other air and water pollutants contribute to the burden of human disease. Targets: 2003 - Launch a prototype database project to test the design and implementation of the knowledge base components and system architecture. 2004 - Create the capability to import, export, and link molecular expression data by extending the Chemical Effects in Biological Systems (CEBS) database object model to include toxicology/pathology fields, and by creating a data portal that will load toxicology data. 2005 - Create and provide public access to a global molecular expression and toxicology/pathology database of both chemicals found in the environment and drugs that have an effect on biological systems (CEBS), featuring simple query download capability. 2006 - Enhance the CEBS to allow the capture and integration of transcriptomics, proteomics, and toxicologic data for the same compound. 2007 - Enhance electronic sharing of 'omics and biology endpoint data. 2008 - Develop capability to query entire set of databases using composite probe sequences. 2009 - Create automatic data mining for literature associated with experimental datasets. 2010 - Develop knowledge-building algorithms that will enable us to mine databases across all domains. 2011 - Implement biologically based dose-response and systems models to support risk assessment needs. 2012 - Deliver predictive toxicogenomics knowledge that supports risk assessment and environmental health understanding.

Measure: By 2012, develop and apply clinically one new imaging technique to enable tracking the mobility of stem cells within cardiovascular tissues.

Explanation:Available probes do not permit safe and effective labeling of stem cells for in vivo tracking. Targets: 2005 - Initiate stem cell labeling strategy. 2006 - Complete optical imaging probe development. 2007 - Initiate validation and toxicity studies. 2008 - Initiate preclinical studies on the nature of stem cell migration in adult tissue. 2009 - Determine the best targets for monitoring cellular differentiation. 2010 - Complete optical probes for monitoring cellular differentiation. 2011 - Initiate validation and toxicity studies of differentiation. 2012 - Develop MRI and/or PET detection systems for clinical and animal studies.

Is the program purpose clear?

Explanation: The NIH Intramural Research Program (IRP) fulfills the NIH mission to improve public health by its support of biomedical and behavioral research and by training investigators for the future. Specifically, the IRP conducts and supports research: in the causes, diagnosis, prevention, and cure of human diseases; in the processes of human growth and development; in the biological effects of environmental contaminants; in the understanding of mental, addictive and physical disorders; in directing programs for the collection, dissemination, and exchange of information in medicine and health, including the development and support of medical libraries and the training of medical librarians and other health information specialists. On behalf of the NIH Director, the Office of Intramural Research coordinates research conducted directly by NIH personnel through intramural programs. The purpose, authorities and responsibilities of the intramural research program are laid out in the Public Health Service Act, clarified and specified further in the NIH Intramural Sourcebook and NIH Policy Manuals, and closely monitored and enforced by the DDIR. Chartered Boards of Scientific Counselors and the National Advisory Councils of the individual NIH Institutes and Centers evaluate the work and accomplishments of intramural scientists using clear standards.

Evidence: ?? Mission Statements of NIH Institutes and Centers ?? Public Health Service Act ?? NIH Intramural Sourcebook of intramural policies and procedures < www1.od.nih.gov/oir/sourcebook/ > ?? Report of the External Advisory Committee of the Director's Advisory Committee (1994) ?? IOM Report on Enhancing the Vitality of the National Institutes of Health (2003)

Does the program address a specific and existing problem, interest, or need?

Explanation: The IRP conducts distinctive, high-risk, high-impact laboratory, clinical, and population-based research designed to meet specific needs and challenges, such as cancer, heart disease, mapping the human genome, and biodefense. It is poised to change research directions quickly in response to emerging problems that threaten the health of the American people. Quality, as well as distinctiveness, is important and characterizes research in the IRP that would not be feasible or readily conducted in the extramural research community. The constellation of special resources such as the NIH Clinical Center, instrumentation, and scientific talent enable the NIH IRP to address specific research problems and needs.

Evidence: ?? Mission statements ?? NIH IRP budgets by Institute and Center ?? Examples of intramural scientist citation indices ?? Selected Z01s to describe intramural research (especially in new and emerging diseases)

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The NIH IRP tries to concentrate its research on distinctive, high-risk, high-impact projects, including research on rare diseases. Though biomedical research is conducted in other agencies and private sector entities, these efforts tend to be specific to a subsector of the population (i.e., veterans), a particular need (i.e., defense medicine) or area of research (i.e., biotech companies each have their own focus area), while NIH supports biomedical research for the broad benefit of the American people. The projects are often official collaborations with academia and industry. Outside expert advisors regularly review intramural scientists and encourage unique areas of intramural research and prevent duplicative, unimaginative science.

Evidence: ?? NIH Policy Manual on Review of Intramural Research ?? Orientation Guidelines for Boards of Scientific Counselors ?? Examples of rare diseases studied at the NIH Clinical Center ?? Examples of highly collaborative research

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The IRP relies on a system of regular outside scientific reviews to assure excellence, and resources of scientists are adjusted based on these reviews. The system of research review is subject to constant refinement and improvement, and a system of management controls assures the responsible management of intramural research resources.

Evidence: ?? NIH Policy Manual on Review of Intramural Research ?? Summaries of Board of Scientific Counselors reviews ?? Sample Blue Ribbon Panel reports ?? Institute of Medicine reports ?? Annual Intramural Management Controls Questionnaire

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: NIH leadership assures effective program design and resource allocation to reach beneficiaries. The public enjoys and benefits from the medical advances of the NIH IRP while intramural scientists/trainees and outside scientific collaborators benefit from intramural biomedical research activities. Significant medical advances that benefit the public health have come from intramural research. Outside expert reviewers advise NIH on the quality and appropriateness of intramural science to guarantee responsible stewardship and appropriate benefits.

Evidence: ?? Intramural Research Advances 2004 from The NIH Catalyst ?? IRP research training opportunities website ?? Council of Public Representatives website ?? Descriptions of CRADAs, MTRAs, and U01s ?? Selected clinical research protocols to demonstrate important, high-risk studies ?? List of selected important products resulting from NIH technology transfer

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Since 2003, NIH has selected representative Scientific Research Outcome (SRO) goals as proxies for its overall performance. These goals are specific to a biomedical research question or problem and fall on a continuum of risk (likelihood of reaching the goal) and time (short-, medium-, long-term). The 3 intramural research goals reviewed represent potential advances in scientific knowledge with broad applicability to the mission of NIH and the individual Institutes and Centers (ICs). Also, these goals would be difficult to achieve in the extramural program due to their broad nature, multitude of partners/collaborators and their complexity. They are: 1) develop a knowledge base on chemical effects in biological systems using a systems toxicology or toxicogenomics approach; 2) develop and apply clinically one new imaging technique to enable tracking the mobility of stem cells within cardiovascular tissues; 3) establish the role of genetic factors in three major diseases for which health disparities are noted between populations.

Evidence: ?? Annual NIH GPRA Plans ?? Institute GPRA Goal Implementation Plans ?? Waters et al. (2003) EHP, 111:811-824 ?? Waters and Fostel (2004) Nature Reviews Genetics, 5:938-949 ?? NHLBI Strategic Plan

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Goal 1 targets and timeframes are ambitious because (1) no one has attempted to design a thorough knowledgebase that can build databases using new 'omic data and integrate across these databases with traditional toxicology/histopath data; (2) there are few database standards for 'omic technologies; and (3) bioinformatics methodologies are evolving. Goal 2 targets and timelines are ambitious because achieving them will require overcoming significant technological and scientific challenges. They were developed based on expert advice and previous research and capitalize on prior NIH research progress. However, they involve complex biological problems and must build upon and refine state-of-the-art scientific techniques. Goal 3 targets are ambitious but attainable, being based on sound science that has been carefully developed on the foundation of the Human Genome Project. The timeframe for the targets is ambitious, given the difficulty of identifying genetic factors associated with genetically-complex diseases.

Evidence: ?? Annual NIH GPRA Plans ?? Institute GPRA Goal Implementation Plans

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Each goal represents an iterative process requiring discrete steps that build on one another as teams progress toward the annual and long-term targets/goals. For example, development of the CEBS knowledgebase requires building individual databaes (microarray, tox/histopath, proteomics, metabonomics) that can be cross-referenced by chemical stressor, study design, genotype, or toxic endpoint. The project has defined times for developing individual databases and providing the technology to integrate across 2, then 3, then multiple databases.

Evidence: ?? Annual NIH GPRA Plans ?? Institute GPRA Goal Implementation Plans

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Each goal has established annual baselines and targets toward completion. Outyear targets and baselines are re-evaluated annually and revised to reflect the latest scientific evidence and achievements. Progress toward each goal relies on scientific advances that naturally build on one another. The baselines for all of the goals include the scientific progress made in each area to date - progress which has led to the establishment of these ambitious but realistic goals and timeframes - as well as the recognition of the gaps that must be filled in order for the goals to be reached.

Evidence: ?? Annual NIH GPRA Plans ?? Institute GPRA Goal Implementation Plans ?? Relevant scientific papers

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All partners - including various ICs within NIH, contractors, governmental and international agencies, and industry - are committed to and work toward achievement of the goals. Each partner commits to discrete elements that contribute to the overall common goal. Intramural research staff coordinate the efforts, with regular conference calls, meetings and reports.

Evidence: ?? Annual NIH GPRA Plans ?? Institute GPRA Goal Implementation Plans ?? Examples of grant proposals ?? Contract awards ?? Reports reflecting monitoring of contract work ?? Interagency agreements

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: In addition to the rigorous peer review process involved in scientific publication, institutes conduct regular reviews of intramural research scientists and programs. Each Institute has a Board of Scientific Counselors (BSC), a panel of expert external scientists, which performs the peer review function for the IRP. Progress toward each specific intramural goal is further evaluated as part of each goal's implementation plan. For example, independent evaluations of progress toward Goal 1 are performed at critical "decision points" of CEBS development to ensure that each database and the integration across databases address the needs of the regulatory and scientific communities. The IOM has also conducted reviews of the IRP as a whole.

Evidence: ?? Annual NIH GPRA Plans ?? Institute GPRA Goal Implementation Plans; ?? Advisory group lists ?? Letter of invitation to CEBS review with list of background materials provided ?? Boards of Scientific Counselors charters and rosters, site visit material, criteria for program evaluation and review results ?? IOM Reports

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: At this time, NIH's budget presentation does not explicitly tie budget resource levels to annual and long-term performance targets. The budget requests do not show how much it would cost to achieve the performance results. NIH's budget requests and allocations award a uniform percentage increase across ICs. At the IC level, Advisory Councils convene three times a year to review and recommend funding of extramural grant applications. Individual ICs, in concert with external advisory groups, set the intramural research agenda and determine associated resources for what they consider high-priority research areas and successful PIs. IC directors, scientific directors, PIs and program staff track ongoing costs of IRP programs, but they are not explicitly tied to budget requests.

Evidence: ?? Annual Congressional Justifications ?? Annual GPRA plans

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: With awareness of the issues noted in Question 2.RD2, the Office of the Director (OD) initiated the Roadmap for Medical Research in 2004. The Roadmap is a set of trans-NIH initiatives focusing on major public health problems that are best addressed through collaboration among multiple ICs. It is designed to identify public health and biomedical priority areas and to accelerate the pace of discovery and improve the translation of research findings into healthcare interventions for public benefit. In addition to the Roadmap, an Office of Portfolio Analysis and Strategic Initiatives is being created within the OD to provide a broad perspective on the entire portfolio of research being done across all ICs, thereby improving priority-setting across the NIH as a whole rather than just within individual ICs.

Evidence: ?? Congressional Justification language describing the Roadmap for Medical Research ?? Congressional Justification language describing the Office of Portfolio Analysis and Strategic Initiatives ?? Annual NIH GPRA Plans ?? Institute GPRA Goal Implementation Plans

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: The NIH IRP is unique in the breadth and scope of its ability to formulate a comprehensive plan to address certain broad goals and then make the substantial commitments of facilities and resources to implement plans for their achievement. It is continually assessed and evaluated by outside groups such as the Boards of Scientific Counselors to ensure that it is making unique contributions that complement other efforts in the field. The representative goals being evaluated are particularly well-placed within the IRP as they would be very difficult for an extramural investigator to coordinate. They involve multiple partners in various facilities and take advantage of the NIH IRP as the headquarters for vast stores of information and knowledge, such as that derived from the Human Genome Project. Scientific exchange in general is widespread and new findings shared publicly, thus building on each others' efforts.

Evidence: ?? Boards of Scientific Counselors charters and rosters, site visit material, criteria for program evaluation and review results ?? Advisory group lists ?? Letter of invitation to CEBS review with list of background materials provided ?? IOM Reports

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: At the IC level, scientific directors determining the annual budgets for each intramural laboratory and branch. Final budgetary authority resides with the institute director, who considers the effectiveness of each program and of the scientific director in allocating resources. Each IC's Board of Scientific Counselors performs the peer review function for the IRP, with prioritization of competing program needs, strategic plans and emerging scientifc opportunities as integral components of funding considerations. IC budget reviews are conducted on an annual basis, allowing for shifting of funds based on changing priorities and scientific achievement.

Evidence: ?? Congressional Justifications ?? Formal budget processes within the Divisions of Intramural Research ?? Board of Scientific Counselors Roster & Charter ?? NIEHS Strategic Plan, 2000 and 2003 ?? Institute GPRA Goal Implementation Plans ?? Report of the Congressionally-Established Diabetes Research Working Group entitled: "Conquering Diabetes-A Strategic Plan for the 21st Century

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Internal performance evaluations are based on annual report submissions, formal performance evaluations, promotion and tenure committees, Title 42 review groups, and the central tenure committee. External evaluations are performed at regular intervals by outside Boards of Scientific Counselors. Evaluation of performance of industry partners is evaluated by the Office of Technology Transfer. The NIH regularly uses performance information to determine the future resources (personnel, budget, space) for intramural research projects.

Evidence: ?? Orientation Guidelines for Boards of Scientific Counselors ?? Annual Performance Evaluation ?? Questionnaire on Intramural Self Assessment of Management Controls ?? NIH Office of Technology Transfer Activities

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Schedules of work, goals, and standards are established at outset of research programs via annual performance plans. Intramural Principal Investigators also submit annual reports searchable via the NIH Intramural Database (NIDB). The Unified Financial Management System (UFMS), established by the Secretary of HHS, improves the efficiency and effectiveness of agency financial, business, and operational functions.

Evidence: ?? Annual NIH GPRA Plans ?? Institute GPRA Goal Implementation Plans ?? Business Case for the NIH Intramural Database ?? Visual Status of Funds Report ?? Annual Report from the NIH Office of Technology Transfer FY 2004

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: NIH utilizes careful program and financial management procedures to ensure that appropriated funds for intramural research programs are obligated for the intended purpose, and in a timely manner. NIH ICs have accounting and internal control systems to monitor appropriated and non-appropriated intramural funds. Accounting controls ensure that obligations and expenditures conform to program plans as well as congressional and administrative requirements.

Evidence: ?? Sample Intramural Laboratory Budget Charts ?? End of Year Close-out ?? NIH Director's Statement on Management

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: NIH has many mechanisms for measuring and improving the efficiency and cost-effectiveness of its IRP. Reviews of scientific quality result in determining the level of resource support for intramural principal investigators (PIs) and projects. With respect to IT investments, the IRP has clear goals of improving efficiency by improving resource allocation, data quality, and staff response time. The NIH Clinical Center has been able to "do more with less", by improved performance management, workforce planning, and sound acquisition and inventory management. Consolidated purchasing and centralized oversight of capital equipment acquisition is another example of maximized benefits at lower costs. NIH's Intramural Research Program has two efficiency measures. The first examines the reallocation of laboratory resources based on external reviews by boards of scientific counselors (BSC). The second is to streamline business processes and automatic data movement by implementing, monitoring, and updating the Clinical Research System (CRIS).

Evidence: ?? Business Case for the NIH Intramural Database ?? List of Core Facilities ?? IT Consolidations (from CIT) ?? Clinical Center Savings ?? Measures Tab

Does the program collaborate and coordinate effectively with related programs?

Explanation: The NIH IRP collaborates effectively with many Federal agencies, academic institutions, and private organizations. Intramural scientists learn of their partners' scientific interests through publications, presentations and frequent informal interactions. Sharing of resources leverages the Federal investment in research and gives it visibility. Formal and informal agreements document these partnerships and collaborations. NIH scientific leaders are key players in the research enterprise and in research policy formulation.

Evidence: ?? Examples of collaborations with other Federal Agencies, academic institutions, and private sector. ?? Graduate Partnerships Program web site ?? NIH Office of Technology Transfer web site and model CRADA and MTA.

Does the program use strong financial management practices?

Explanation: While the IRP maintains strong financial management practices, both HHS' and NIH's overall financial statements revealed weaknesses. A FY 2004 Ernst & Young audit of HHS financial statements reported that the Department (HHS) had serious internal control weaknesses in its financial systems and processes for producing financial statements. These weakness were considered to be material. The audit recommended that HHS improve its reconciliations, financial analysis, and other key controls. The audit also cited NIH-specific needed improvements following the launch of the Oracle General Ledger portion of its New Business System (NBS). "Although the Oracle General Ledger became the official accounting system of record during the second quarter of FY 2004, [Ernst & Young] noted certain issues related to NBS that did not fully comply with the [Federal Financial Management Improvement Act of 1996]." E.g., certain parts of the statement preparation process continue to be manually intensive, time consuming, and prone to error, and the NBS does not allow NIH to track manual or non-routine entries.

Evidence: ?? Financial Statement Audit of the Department of Health and Human Services for Fiscal Year 2004 ?? Fellowship Payment System ?? NIH Manual Issuances ?? GAO and OIG reports ?? Audited Financial Statements ?? NIH governance structure

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The FY 2004 HHS audit highlighted NIH specific areas of needed improvement. As of the completion of the audit in December 2004, NIH had already corrected some deficiencies, including correcting discrepancies totaling $27.5 million between its New Business System general ledger and the previous Data Warehouse. Also, NIH recognized weaknesses in certain internal processes and controls. In addition, HHS is in the process of rolling out its Unified Financial Management System (UFMS), with full implementation expected in FY 2007. With UFMS, HHS will be able to standardize business processed for core functions including general ledger, accounts payable, accounts receivable, cost management, budget execution, and financial reporting. In addition, NIH takes a proactive approach to identify and manage deficiencies, with the Office of Management Assessment (OMA) dedicated to this task. NIH utilizes an annual Intramural Management Controls Questionnaire to identify areas that need improvement, and provides information and guidance on policies, rules and recommended practices to all staff.

Evidence: ?? Annual NIH Intramural Management Controls Questionnaire ?? NIH Orientation Program ?? NIH Guide to Preventing Conflict of Interest ?? Guidelines for the Conduct of Research ?? Guide to Training and Mentoring

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: NIH scientists in the IRP are reviewed competitively to determine allocation of research resources. All tenured and tenure-track scientists in the NIH Intramural Research Program (IRP) compete for independent Federal resources (operating budgets, personnel, and space) with which to conduct their research. Rarely does the NIH support R&D programs without full peer review, and then only with rigorous evaluation and approval to ensure high standards of program quality. Other than full and open competition contract acquisitions must be supported by a Justification for Other than Full and Open Competition (JOFOC). With few exceptions as prescribed by the FAR, JOFOCs are announced in the FedBizOps which lists notices of proposed government procurement actions available to the public, and are reviewed and evaluated in accordance with principles that generally apply to both solicited and unsolicited proposals. To ensure compliance with procurement standards, JOFOCs must be reviewed and approved at the appropriate hierarchical level at NIH, by the Competition Advocate, and for transactions above $50 million, by officials at the Department of Health and Human Services. NIH adheres to a policy that promotes fairness in the process of initiating and developing Cooperative Research and Development Agreements (CRADAs) that includes public notification and an internal review process.

Evidence: ?? JOFOC Desk Guide for Contracts ?? Federal Acquisition Regulations ?? NIH CRADA Policy ?? U.S. Public Health Service Technology Transfer Manual, Chapter 401

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Yes, each representative goal has demonstrated adequate progress in achieving its long-term performance goals. Goal 1: CEBS has met the milestones that are essential in building individual databases and the bioinformatics technology necessary to integrate and mine information across these databases. Proto-CEBS provided initial "proof of principle", two databases (microarry and tox/histopath) have been populated and integrated, and field testing by research scientists and FDA reviewers of this initial CEBS product is being sought. Goal 2: The program has already made progress in developing MRI compatible contrast agents and cell labeling strategies. Involved investigators have also pioneered the use of a new technique that can be used for optical probe detection. Goal 3 represents a new Goal as of 2005, so no specific annual targets have yet come due. Building on an extensive track record of success in the fields of genomics, diabetes, and cancer research, NIH is making excellent progress toward early targets.

Evidence: ?? GPRA report for SROs ?? Annual NIH GPRA Plans ?? Institute GPRA Goal Implementation Plans showing performance status ?? Version 1.4.1 of the CEBS database ?? CEBS SysBio Object Model available at http://cebs.niehs.nih.gov ?? CEBS SysTox Object Model available at http://cebs.niehs.nih.gov/SysTox/SysTox/ ?? Publications documenting accomplishments ?? Human Genome Project ?? Breast and Prostate Cancer Cohort Consortium progress report for the June 2005 National Cancer Advisory Board ?? Silander, K. et al. A large set of Finnish affected sibling pair families with type 2 diabetes suggests susceptibility loci on chromosomes 6, 11, and 14

Does the program (including program partners) achieve its annual performance goals?

Explanation: As mentioned above, the program and its partners are successfully achieving annual performance goals. Goal 1: Annual performance goals outlined in the CEBS implementation plan have all been met and their achievement is routinely reported via GPRA tracking. Contract performance is monitored either weekly or biweekly to assure performance goals are met. Current versions of CEBS are on a public website for review and comment. Goal 2: The NHLBI DIR has made progress in developing MRI compatible contrast agents and cell labeling strategies and a new technique for optical probe detection has been developed. It has been highly successful in its work with research partners, who can be expected to make important contributions to this effort as well. Goal 3: This goal has just begun and therefore there are no performance results to report. However, based on the success of the program to date and the the investigator's strong record of success on the Human Genome Project, they are confident that the program will achieve its annual performance goals.

Evidence: ?? GPRA report for SROs ?? Annual NIH GPRA Plans ?? Institute GPRA Goal Implementation Plans showing performance status ?? Sample biweekly report for SAIC ?? Sample weekly report for Icoris ?? Version 1.4.1 of the CEBS database and the CEBS SysBio Object Model at http://cebs.niehs.nih.gov ?? CEBS SysTox Object Model at http://cebs.niehs.nih.gov/SysTox/SysTox ?? Contracts and CRADA with previous collaborators ?? Relevant research results ?? Human Genome Project ?? publications by Boon et al., Chunhui et al., Wally et al, Rangel et al., Koga et al. and Cheng et al.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The consolidation of activities that cut across organizational lines has improved efficiencies and cut costs. In addition, co-location of related research activities in the IRP improved cost effectiveness. Labor costs were reduced by improved productivity. This includes reprogramming of staff assignments and retraining, combined with reduction in staffing levels; for example, over the past 2 years the Clinical Center has cut more than 120 positions through increased efficiencies. The Clinical Center has also realized savings of more than $4M via consolidated acquisitions, greater utilization of purchase cards, and better controls over rate of investment and maintenance. Other IRP programs have achieved savings via mechanisms such as reverse auctions in which vendors bid, offering competitive prices on products/supplies. On average, these auctions result in a 10-24% price reduction over traditional pricing methods. NIH's Intramural Research Program has two efficiency measures. The first examines the reallocation of laboratory resources based on external reviews by boards of scientific counselors (BSC). This measure was adopted in 2003. In 2003 and 2004, IRP met the annual goal of 25% of Principal Investigators who had resources evaluated for possible reallocation. The second efficiency measure is to streamline business processes and automatic data movement by implementing, monitoring, and updating the Clinical Research System (CRIS). The measure was adopted in 2004. The program met the 2004 target to implement a core hospital system and retire the 28 year-old legacy system.

Evidence: ?? Center for Inherited Disease Research ?? NIH Intramural Sequencing Center ?? NIH Scientific Advances by Year and Facility ?? Reverse Auction ?? Purchase Card Program ?? Measures Tab

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The NIH IRP is the lead participant in the global biomedical research enterprise, which includes national and international research performers, and other Federal entities. Collaborations and mutual training across disciplines and borders is the norm, rather than the exception. But taking into account its overall scope and productivity, the NIH IRP clearly plays a leadership role, and compares very favorably with other programs in terms of its scope, scientific achievements, and international recognition. The IRP regularly attracts international scientists who have fewer opportunities to pursue research in their home countries. Many outside organizations (federal and international) have come to NIH to study how the IRP is organized, operates and functions in order to improve their own systems.

Evidence: Examples of intramural scientist citation indices ?? Comparison tables: NIH, VA, and DOD ?? Websites: Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, National Institute of Standards and Technology, Medical Research Council (UK), Karolinska Institutet (Sweden), Pasteur Institute (France), Canadian Institutes of Health Research, The Wellcome Trust

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Evaluations are mixed. At the IC level, reviews conducted by outside experts, federally-chartered Boards of Scientific Counselors (BSCs), and accrediting bodies demonstrate that programs are of high quality and effective in the conduct of research. A blue ribbon panel of outside experts reviews each IC's IRP to assess its overall program, including the innovation and impact of its research. For the IRP as a whole, several weaknesses have been identified. The 2003 IOM Report on NIH expressed concerns about the degree of uniqueness represented in the IRP's portfolio, the review process for tenured scientists and scientific directors, private sector collaborations, and a number of other topics specific to the IRP. These sentiments have been echoed by BSCs, with concerns that 1) the IRP is not focusing on unique, high-risk research that is distinct from that conducted extramurally, 2) tenured scientists are not subject to the same degree of oversight as their academic counterparts, and 3) research is not synergistic enough across ICs.

Evidence: ?? Certifications from recent site visits ?? Sample BSC scientific reviews ?? Sample Institute IRP program reviews ?? Sample charters of BSC and National Advisory Councils ?? 2003 IOM Report: Enhancing the Vitality of the National Institutes of Health