The way we do science has changed, and the peer review system must meet the challenge to keep pace. Advances in basic science and clinical medicine provide unprecedented opportunities for the marriage of the life sciences with the physical sciences, mathematics, computer science, and engineering. The development of new tools and techniques offers great potential for medical advances, and there is an increasing trend in the biomedical sciences to use an integrative rather than a reductionist approach. It is thus important to assure that the peer review system has a mechanism for the proper and equitable review of research that entails engineering and the development of technology and instrumentation. To this end, Dr. Ellie Ehrenfeld, Director of the Center for Scientific Review (CSR), established the Working Group on Review of Bioengineering and Technology and Instrumentation Development Research as an ad hoc subcommittee of the CSR Advisory Committee.

The Working Group was charged to (1) identify obstacles to fair, high-quality, rigorous review of this subset of interdisciplinary research and (2) develop a set of principles to guide CSR in establishing a review infrastructure that facilitates identifying promising research across all disciplinary boundaries. The members of the Working Group, listed in Appendix I, approached their task by addressing the following questions: What are the problems? What is currently being done to address these problems? What are the ways in which shortfalls can be remedied? and What will be the impact of remedying them? The specific methods followed by the group are outlined in Appendix II. Though their charge was relatively narrow, the focus of the group was broad. Thus, the members defined bioengineering and technology as encompassing areas such as biotechnology, functional genomics, informatics, chemistry and physics, and their recommendations extend beyond individual, investigator-initiated (R01) research projects to all NIH granting mechanisms. Since the members believed the issues to be broad, this report also speaks to ways in which NIH might generally facilitate interdisciplinary research and become more agile at seizing emerging opportunities.

The Working Group identified six areas that need to be addressed in order to solve problems that impede NIH’s ability to seize emerging opportunities in interdisciplinary research.

Four of the areas indicate a need to appreciate and foster the full range of research styles, management approaches, and contributors:

1. Hypothesis-driven research

Peer review of NIH research grants has traditionally emphasized the testing of hypotheses. Although "hypothesis-driven" research could be interpreted broadly to include all the styles of research – including, using knowledge to solve important problems and developing novel instrumentation to enable knowledge generation -- the practice of NIH study sections has been to interpret it narrowly as the generation of new knowledge. While the generation of new knowledge is unquestionably vital to achieving the NIH mission, such an exclusionary interpretation of hypothesis-driven research impedes the ability of NIH to accomplish its broad charge. Clearly, the creation of new diagnostic methods and therapies calls for research where knowledge is applied to solve problems or develop novel instrumentation, yet, to obtain NIH support, the applied aspects of these projects must often be hidden behind a screen of knowledge generation. If NIH is to accomplish its full mission, all research styles must be judged on their potential to advance our understanding of biological systems, to improve the prevention and treatment of disease, and to enhance the health and well being of people.

The nature of bioengineering and technology and instrumentation development research is not well understood by the biomedical research community at large. Engineering and technology development have a powerful impact on health research, for example by developing new designs, instruments, devices and methods; providing new insights into biological mechanisms through quantitative analyses and mechanistic mathematical modeling; and generating new approaches to prevention and treatment of disease. Radical change in biomedical sciences is likely to occur from incorporating the engineering perspective into the formulation of basic questions addressed in "hypothesis-driven research." Although the rise of molecular and cell biology has made biology accessible to the tools and approaches of engineering science, most biomedical scientists are not aware of the contributions engineers can make. Indeed, the field of bioengineering has evolved to embrace molecular and cellular biology and biochemistry (e.g., tissue engineering, drug and gene delivery, and biomimetic biomaterials), while continuing to build upon its early roots in the physical sciences (e.g., magnetic resonance, computerized tomography, ultrasound imaging, and biomechanics). It is critical for biomedical scientists who review proposals at NIH to fully appreciate the ways in which engineers can contribute to the advancement of biomedical science and clinical medicine.

3. Innovation

The review system is seen as overly conservative. Study sections’ frequent avoidance of risk at the expense of innovation is well recognized. Yet, significant scientific breakthroughs most often result from calculated, aggressive risk-taking.

4. Broad program management of collaborative projects

Many in the research community fail to adequately appreciate the complementarity of top-down and bottom-up approaches of management. The NIH research portfolio is heavily weighted toward the individual, investigator-initiated research project, which the biological research community traditionally equates with hypothesis-driven research. Particularly in an era of constrained resources, investigators supported by this bottom-up approach to program management tend to deny the valuable parallel role that top-down management can play in achieving the mission of NIH. In addition, some practices may inhibit collaborations and the participation of industry. NIH’s rule of crediting only one investigator per grant as principal investigator (PI) may limit the willingness of some scientists to combine forces. This is largely true because promotions at many institutions require demonstration of scientific independence, which is often judged on the basis of PI status. Other practices may discourage industry’s full participation at a time when closer and new forms of collaboration are likely to be fruitful.

Two additional problems relate to structure and function of the CSR review system:

5. Organization and composition of study sections

The current organization and composition of study sections does not facilitate identification of all types of promising projects. For example, there are no regular study sections to serve as a home for broadly applicable technologies. Sometimes interdisciplinary applications, including those in bioengineering and technology and instrumentation development, are a small minority of those being reviewed in a given study section; hence the perception is that such applications are disadvantaged. In contrast, much of bioengineering research, ranging from biomaterials to clinical devices to biological modeling, is reviewed in one study section along with surgery-related applications. In such a construction, it is very difficult to achieve the breadth of technical expertise necessary to review incoming applications. Including two disparate fields in one study section also encourages factionalism. Given the trend toward an integrative approach, engineers, physicists, chemists, mathematicians, and computer scientists are needed on a broader range of study sections, such as those that review applications in cell biology and signal transduction. Given the growing potential for industry to make important scientific contributions, industrial experts are needed as reviewers to contribute their perspective.

6. Separation of program and review

At NIH, the principle of separating program and review functions has been interpreted strictly. This is perhaps a corollary to the greater valuation of individual-investigator initiated ("hypothesis-driven") research. However, programmatic objectives are particularly important in gauging the impact of proposed problem- and technology-based research. If SRAs are to guide reviewers in assessing the impact of such applications, they must understand and convey the programmatic context in which the applications are being considered. Conversely, if program staff are to obtain a broad perspective on the scientific frontier, they must have input from review staff. A seamless collaboration between review and program can enhance the function of both sets of science administrators without compromising the objectivity of the review process.

1. Change organizational structure and composition

To accommodate the heterogeneity of interdisciplinary applications and provide appropriate reviewer expertise, CSR should develop a flexible organizational structure.

Accordingly, CSR should charge IRG Chiefs and Working Groups to provide a suitable venue for review of interdisciplinary research within the context of the biological question it seeks to address. This may be accomplished in the following ways:

• Supplement reviewer expertise on study sections as appropriate. For example, physicists, chemists, mathematicians, computer scientists, and engineers might be members of study sections that review applications in areas such as cell biology and signaling, where integration of multiple pathways makes the whole greater than the sum of the parts. Bringing together reviewers who span a broad range of disciplines has the added benefit of cross-education. Members of industry should also be included for their valuable perspective.

• Cluster applications in specially created scientific review groups. Applications that otherwise would be scattered in small numbers across multiple study sections within an IRG should be reviewed in scientific review groups staffed with members whose expertise is tailored to the applications under review. Clustering applications in this way could obviate problems that have been noted related to evaluating a minority group of applications among a clear majority of other types of applications in the same study section.

In addition, CSR should create IRGs designed to review -- in their methodological context -- applications that are related to broadly applicable technologies.

2. Revise operating principles and practices

To facilitate identification of promising interdisciplinary projects, CSR should:

• Through SRAs, Study Section Chairs, other NIH staff, and members of the extramural community, heighten reviewers’ awareness of the important contributions of all styles of research and charge them to afford each type equal advantage.

• Develop additional formal mechanisms of interaction between program staff and SRAs to provide the opportunity for SRAs to become knowledgeable and appreciative of programmatic goals.

3. Promote system agility

Until recently, the peer review system has been seen as largely immutable. If the peer review system is to advance all fields of biomedical research and enable future opportunities, it must continue to adapt to the changing scientific landscape. IRG staff and IRG Working Group members should continually seek to enhance operations of the IRG by identifying, testing, and evaluating potential improvements in conjunction with Institute staff and members of the scientific community.

The Working Group requests that Dr. Ehrenfeld convey three additional broad recommendations to Dr. Varmus. Specifically, the NIH should:

1. Declare the importance of multiple types of research

The PHS 398 form should be modified. In particular, the requirement in the Specific Aims Section to "state the hypothesis to be tested" should be replaced with a request for the applicant to state the broad, long-term objectives of the proposed research, e.g., "to test a stated hypothesis, create a novel design, solve a specific problem, or develop new technology." In addition, NIH should develop a method of officially recognizing more than one principal investigator per grant while maintaining the essential practice of identifying one investigator responsible for fiscal and administrative matters. While the disincentive to collaborate emanates from the academic culture where investigators are often penalized if their major output comes from joint projects, NIH should take a leadership role in remedying the situation. The physics community has long recognized the value of collaborative research and credits individual participants on large projects. NIH should similarly recognize that in many partnerships, such as those between biologists and computer scientists, the contributions of investigators are complementary. NIH should also consider new ways of acknowledging industry’s needs -- for example, those related to issues of conflict of interest, intellectual property, and speed of the review and award processes -- while maintaining the integrity of the NIH peer review system.

2. Prepare to meet grand challenges

Understanding of fundamental biology is the most vital and productive endeavor in all of science today. If public health is to benefit fully from this explosion of knowledge and to advance as rapidly and cost-effectively as possible, NIH must exploit the full range of available approaches and involve the full range of participants. Although most research should and will continue to be carried out by single investigators, some specific, high-priority problems will require pooling the talents of many investigators and may involve disciplines whose role in NIH-sponsored activities should grow. "Grand challenge" problems -- mapping the human genome, the Human Brain Project, deciphering the wiring diagram for the cell, fusing living and electronic systems, functional genomics studies, understanding aging, and detecting, controlling, and reversing malignancy -- all present opportunities and difficulties of great magnitude. Their solutions will require elements of "large science" and of engineering in addition to basic biomedical science and will call on intellectual contributions from engineering, physics, and mathematics. Industry may also play an important role in undertaking such projects.

In addition, broad advances in biomedical science depend upon the research tools available. Development of these tools for widespread use is an activity that is often closer to engineering than to fundamental discovery. Applying engineering approaches to those enabling technologies (e.g., proteomics, high-throughput screening, microfluidic systems, single-cell studies, imaging, etc.) could greatly accelerate the pace of technology’s development and its application to important biomedical problems. Novel technological development will enable the solving of problems associated with grand new challenges, just as cloning, sequencing, the polymerase chain reaction, and other tools of molecular biology provided a solution to sequencing the genome.

To meet these challenges, the NIH should develop ways to implement the following general recommendations:

• Employ an aggressive planning process and develop the ability to manage large projects. NIH has the capability to develop plans for and to manage large-scale, focused, rapid development of selected areas. However, it has been the exception for NIH to undertake efforts such as the Human Genome Project. An increasing number of areas will benefit from this approach, and NIH should develop its capability in planning and managing these efforts. NIH should also seek to involve industry in the planning process in order to identify synergies and develop fruitful interfaces.

• Provide appropriate support for research and engineering to develop tools and methods. Tools and methods play a central role in advancing biomedical science. Since many of these tools originate outside of NIH, leaders in the extramural community should be invited to work with NIH staff to identify opportunities and prepare to exploit them.

• Bring together researchers from different backgrounds to solve common problems. Remarkable things happen intellectually and practically when individuals from different backgrounds work collaboratively on a common problem. Important efforts by broad-based, multidisciplinary research communities complement the valuable individual investigator approach.

• Develop mechanisms to enable innovation. Inclusion of innovation as one of five review criteria is a welcome step to promoting creativity in research. The pilot project/feasibility studies (R21) supported by several Institutes in their efforts to support creative, novel, high risk/high payoff research are also a step in the right direction. However, R21 awards provide insufficient funds ($100,000 per year) and time (two years) to explore numerous innovative ideas. Many of these projects require orders-of-magnitude more funds than what has often been provided by NIH in order to have a reasonable probability of success. Thus, NIH should set aside appropriate levels of support explicitly to fund innovative but high-risk work. It should also assign a special activity code to applications eligible for this program to distinguish them from R01 applications and specify that it is not necessary to provide extensive preliminary data, the requirement for which is often a barrier to proposing high-risk, high-reward research.

• Link equipment requests to the research for which it is used. In many cases, equipment is currently funded separately by the National Center for Research Resources. While measures should be taken to ensure that expensive equipment will be shared among multiple investigators, funding of equipment required to perform experiments in a research grant should be linked administratively to the research program and in a common time frame to facilitate execution of the research.

• Monitor and exploit activities of other government agencies and industry. NIH should monitor the efforts of agencies such as the Defense Advanced Research Projects Agency, the Department of Defense, and the Department of Energy to identify and exploit developments with potential high impact on biomedical research. Examples of such efforts include development of instrumentation and research on biological warfare. NIH should also monitor the efforts of industry and capitalize on industry’s efforts as much as possible.

3. Broaden representation on policy-making bodies

The full complement of health sciences expertise should be present as appropriate on planning and policy-making bodies to optimize federal investment in areas of increased emphasis. This includes engineers, computer scientists, mathematicians, physicists, and chemists, along with biologists and representatives of industry.

Note: The Working Group further recognizes that if SRAs are to devote the time needed to develop and implement thoughtful approaches and flexibility to the review process, their workload must be adjusted accordingly. This will require that CSR be provided adequate financial resources. Increased staffing may also be required to enable program staff to learn about and implement more work-intensive methods of program management.

The proposed recommendations should enable the CSR peer review process to identify a broader range of promising research and strategically position NIH to capitalize fully on unprecedented scientific opportunities to improve human health.

Deliberately small in number, the membership of the Working Group consists of scientists and engineers with broad perspective and vision. They have experience in diverse fields, including biomaterials, tissue engineering, medical devices, imaging, medical informatics, physiology, chemistry, physics, biochemistry, molecular biology and genetics, and genomics. Their research programs range from basic through applied studies.

The Working Group has depended heavily on input from the community to identify issues and shape recommendations to improve the system. Thus, they met at the outset with representatives of BECON to gain the perspective of program staff, and with SRAs in CSR to hear their insights. Based on the information provided by these two groups and their own experience as applicants and peer reviewers, they developed preliminary recommendations. The opportunity to comment on these preliminary recommendations was then offered to broad segments of the research community. The American Institute of Medical and Biological Engineers (AIMBE) solicited input from their Councils and segments of its 32,000 members. The opportunity to comment was also offered to industrial organizations, the small business community, and groups of genomicists and biocomputational researchers. Drawing comments from 270 respondents, the Working Group identified a list of 6 impediments and made 6 broad sets of recommendations to overcome these problems. These were outlined in a draft report circulated to approximately 50 academic and industrial opinion leaders, 25 of whom responded.

This final report reflects input from many in the broad scientific community. The Working Group members wish to acknowledge these important contributions.