Frequently Asked Questions

Question:

The report points to the outstanding success of CSR's peer review system. What is the justification for reorganizing study sections based on subjective impressions in the absence of data?

Answer:

With science advancing at an ever-increasing speed, a review system that is designed to be more agile in responding to new opportunities is clearly warranted. The peer review system has benefited over the years from ongoing improvement, and the work of the Scientific Boundaries Panel is simply a continuation of such ongoing efforts. This is the first time, however, that a systematic, comprehensive assessment of the review committee structure has been conducted. While our impressions of what needs to be changed were not based on objective data, as would be ideal, the CSR Director believes that it is important to respond to uniformly voiced concerns of the community, as well as to ensure that proposed changes are endorsed by the community before implementation.

Question:

All constituent research groups are not represented on the Panel. Why is the membership on the panel so limited in number and representation?

Answer:

The Panel's membership was deliberately limited in number, and care was taken to avoid a constituency-based group of scientists. Rather, members were selected for their broad perspective, their vision, and their contributions to the development of diverse fields of science. The 15 distinguished members were chosen for their leadership abilities and their capacity to transcend personal interests, whether scientific or constituency-based. At the same time, the Panel's approach includes multiple mechanisms for all groups to influence the outcome.

Question:

The distinction between IRGs and study sections is confusing. What is the relationship between the IRG and its component study sections? What are the functions of the IRGs and how do they differ from those of the study sections?

Answer:

The IRG is an administrative unit including a number of study sections encompassing a broad scientific domain (analogous to an academic department). Leadership and management for the IRG are provided by a Chief ("the department chair"), who coordinates the activities of several SRAs ("the faculty members"). The SRAs are in turn responsible for managing the activities of the component study sections within the IRG, each of which focuses on a specific area within the IRG's broad scientific purview. The IRG does not operate as an entity to review grant applications. Establishment of the IRG as the functional unit of review has presented opportunities to optimize the review process. With the counsel of the CSR Advisory Committee, CSR has been working to capitalize on these opportunities, which include increased chances for teamwork, flexible distribution of applications, and sharing of reviewer expertise. Initial referral of applications is now made to the IRG, with subsequent assignment to individual study sections by IRG staff. In addition, some IRGs are arranging concurrent meetings of several or all of their study sections, so that reviewers with specific expertise can participate in multiple study sections as needed.

Question:

Seven of the 20 current IRGs were recently reorganized. Will these IRGs and their component study sections be subjected to another reorganization at this time?

Answer:

Several IRGS and component study sections were created in 1998 and 1999 as a result of the reorganization the AIDs and related research, the neuroscience, and the behavioral and social science review committees. The Panel intends for these IRGS and study sections to remain intact at this time, pending evaluation of their effectiveness after they have been functioning for several years. An evaluation system for neuroscience study sections is currently under development. It is possible that some new study sections will be added to these reorganized IRGs where scientifically appropriate.

Question:

I fear that my study section is being abolished without any consultation from the community. What other changes are underway?

Answer:

No study sections have been abolished and none has been created as a result of the Panel's Phase 1 activity. The Panel has not considered which study sections should be established within each of the proposed IRGs. Study sections will be designed in Phase 2, which will be conducted by groups composed of external researchers from the relevant scientific communities, in conjunction with NIH staff. In Phase 1, the Panel is dealing only with broad concepts; details related to study sections will be dealt with in Phase 2.

Question:

How will the members of research communities be chosen to participate in Phase 2?

Answer:

The Panel will consult relevant scientific societies and NIH staff to identify appropriate members of the expert groups that will be formed to design study sections. Many who wrote thoughtful comments related to the Phase 1 report also will be called upon. While it will not be possible for all who are interested to serve on these groups, there will be an opportunity for all members of the community to contribute to the final recommendations. The organization proposed by each of these groups will be posted on the CSR web site for broad public comment.

Question:

How will the system be monitored?

Answer:

The CSR Advisory Committee is helping to develop recommendations for periodic review of study sections. Accordingly, study section members and applicants would be surveyed at frequent intervals to track study section performance, and the organizing principles and operating procedures for each study section would be assessed at approximately five-year intervals by external advisory groups as described below. Suggestions for changes and improvements would be returned to the SRAs, Chairs, and members through the IRG chiefs, and the results of the surveys will be monitored by the CSR Advisory Committee on an ongoing basis.

Question:

What exactly will the external advisory groups do? How does their function differ from that of the expert groups that will be assembled to design the study sections within each proposed IRG in Phase 2? Why are some external advisory groups being formed now when a reorganization of IRGs is imminent?

Answer:

For several years, CSR has been establishing external advisory groups for each IRG as ad hoc working groups of the CSR Advisory Committee to assess, at approximately five-year intervals, the organizing principles and operating procedures for the study sections within each IRG. These external advisory groups are asked to evaluate the appropriateness of research topics and scope of applications reviewed; the evolution of topics and scope of research; how well newly emerging research areas are being incorporated; and, in addition, the performance of the SRA, Chair and members. To date, such groups have been or are in the process of being instituted for eight of the 20 current IRGs: Cell Development and Function; Oncological Sciences; Biophysical and Chemical Sciences; Musculoskeletal and Dental; Cardiovascular Sciences; Brain Disorders and Clinical Neuroscience; Molecular, Cellular, and Developmental Neuroscience; and Integrative, Functional, and Cognitive Neurosciences. The recommendations of these external advisory groups will be provided to the expert groups responsible for Phase 2 of the Scientific Boundaries Panel activity, whose goal is to design study sections de novo within the new IRGS that are recommended in this report.

Question:

How does IRG organization affect allocation of funding? Does creation or elimination of an IRG affect the funding of the areas of science encompassed by that IRG? What is the relationship between CSR and the Institutes and Centers?

Answer:

Many researchers appear to believe that funds are allocated to IRGs and/or study sections. This is not true. Congress appropriates funds to the individual Institutes and Centers. Each of these Institutes and Centers addresses a specific health-related mission related either to diseases (e.g., the National Cancer Institute, the National Institute for Arthritis, Musculoskeletal and Skin Diseases); organ systems (e.g., the National Heart, Lung, and Blood Institute, the National Eye Institute); populations (e.g., the National Institute on Aging, the National Institute of Child Health and Human Development); or fundamental research and infrastructure (e.g., the National Institute of General Medical Sciences, the National Center for Research Resources.) CSR receives an allocation from the research management and services fund to provide a central service to these individual Institutes and Centers by managing the review process to provide evaluation of scientific merit. Approximately three-fourths of all the applications submitted to NIH Institutes for potential funding are reviewed in CSR. To do so, CSR operates study sections clustered within IRGs and organized by scientific area, not programmatic mission. In most cases, study sections evaluate applications assigned to multiple Institutes. While it is also the case that some study sections review applications assigned for potential funding to one Institute, this results from scientific coincidence and not deliberate pairing. The creation or elimination of an IRG does not affect the funding allocated to any given area of science. IRGs do not set award levels. However, to enable comparisons across study sections that may assign scores differently, priority scores are percentiled within each study section of the IRG. Thus, the organization of study sections does influence how the funds are distributed among research areas. For this reason, it is critical that there be sufficient study sections for newly emerging fields and those carrying out the highest impact work. Moreover, undeserved entitlements must not be created by perpetuating study sections that are restricted to research with relatively low impact. It is also important to continuously monitor and adjust study section boundaries as fields undergo rapid change.

Question:

Why are so many details lacking in the report?

Answer:

The Phase 1 report is a concept paper. The Panel made no attempt to be exhaustive at this time. Details will be developed during Phase 2.

Question:

There are many inconsistencies in the report. Why is this so?

Answer:

The inconsistencies noted by respondents are unavoidable, because many of the principles that the Panel believes should guide creation of study sections compete and conflict with one another. No matter which principle guides the primary distribution of research grant applications, conflicts will arise. The Panel assigned high priority to the goal of reviewing applications that apply to a given disease or organ system in the context of the biological question being addressed, because we believe that it results in the greatest net benefit. Therefore, the Panel expects those involved in Phase 2 to ensure that applications related to crosscutting areas will be clustered so that they constitute at least 30 percent of applications in a study section. The membership of each study section should also include at least 30 percent who are experts for each area to be reviewed.

Question:

How will the effectiveness of the reorganization be evaluated?

Answer:

The Panel recommends that the CSR employ, for all IRGs, the formal evaluation efforts that are currently being developed for the IRGs and study sections created as a result of the recent reorganization of neuroscience and behavioral and social science review committees. These plans, when fully developed, will be posted on the CSR web site.

Question:

What sorts of comments did the Panel receive and how did they influence the final report and recommendations?

Answer:

Most of the comments were supportive, even while offering objections or suggestions for improvement relative to the respondents' specific areas. Some felt that even greater change was in order. Comments regarding cultural changes were particularly positive. Respondents provided exactly the kind of input the Panel was counting on by indicating significant omissions and identifying a number of misunderstandings that provoked unnecessary concerns. Many of these concerns arose because of confusion regarding the distinction between IRGs and study sections, the misconception that specific study sections were to be abolished, and the assumption that failure to mention an area of science among the examples to be covered by an IRG implied its intentional omission. Most comments concerned the proposed IRGs and came from researchers involved in cross-cutting areas of research who fear being dispersed across many IRGs and those who noted the need for additional IRGs. Many comments related to Phase 2, and these will be passed on to the groups that will carry out the Phase 2 activity. As a result of these helpful comments, additions and refinements have been made to the proposed IRG organization and the report has been written to provide greater clarity of the Panel's intentions.

Question:

What is the timeline for the Panel's activity and subsequent implementation of their recommendations?

Answer:

Phase 2 will begin in 2000, when expert groups will be assembled to design study sections for several of the proposed IRGs. The remainder of these expert groups will be commissioned in 2001. Implementation will likely begin in 2001, at which time it will proceed in a careful and deliberate manner after extensive input from the extramural community.