Minutes of the Meeting of the Peer Review Oversight Group (PROG)
February 13, 1997, 9:00 am - 5:00 pm — February 14, 8:30 am - 12:00 noon
Conference Room 9104, Rockledge Center 2, Rockville, MD
Dr. Baldwin welcomed the members and gave a general overview of the agenda.
Each Peer Review Oversight Group (PROG) member was given a "Talking About
NIH" packet; Dr. Baldwin explained that these packets contain information
to assist members of the scientific community who serve on various advisory
committees and councils, and might be asked to talk generically about the NIH.
![](images7/divider_blue.jpg)
Dr. Baldwin noted that we are within the comment period for revisions to the
PHS 398 grant application form, but emphasized that any changes should decrease
rather than increase respondent burden. An example would be decreasing the
number of pages or eliminating some part of the form. There was brief discussion
of the fact that what represents a change in burden from the administrative
point of view of the Office of Management and Budget (OMB) does not always
coincide with what would represent an increase or decrease in effort for
applicants, such as a checklist to replace a page of text. Any increases would
need to be offset with equal or greater decreases in applicant burden, as
defined by OMB.
Dr. Baldwin presented an update of the information provided in the SBIR
report (available on the NIH/Office of Extramural Research homepage at
http://grants.nih.gov/grants/funding/sbir.htm). Dr. Ehrenfeld commented that at
some point the Division of Research Grants would appreciate having PROG discuss
and analyze the review of SBIR applications.
Go back to the Table of Contents
Creativity and the Rating of Grant Applications (RGA)
To begin the discussion, Dr. Baldwin pointed out that the timing for a
decision is important as we are now at a crucial point in the "life cycle"
for review and funding for the next fiscal year; therefore, there is a need for
closure on any advice the committee would like her to carry to Dr. Varmus, who
will make the decisions regarding implementation of any RGA recommendations. The
discussion focused on the use of "creativity" as an independent
criterion, not the value that NIH places on creative research. NIH does value
creativity in research and the question before the group was to recommend the
best ways to elicit and identify such applications.
Dr. Claude Lenfant shared the report of the "creativity criterion"
working group which he chaired, whose charge was to obtain the opinions of
program staff on this topic. Dr. Lenfant indicated that the findings of this
group uniformly indicated that, while all recognize the importance of funding
innovative and creative applications and agree that these issues should be
addressed in reviewers' critiques, creativity and innovation permeate all three
areas addressed by the other three criteria sets, and should not therefore be
set apart in a separate criterion. Dr. Leshner noted that he had had separate
discussions with extramural scientists who offered the same views. There was
again discussion of the public perception of whether creativity and innovation
are valued by the NIH and by scientific peer reviewers, and it was noted that
creativity and innovation should be made explicit within the other three
criteria and be incorporated into reviewers' education. Dr. Lenfant favored
persuasion rather than edicts.
Dr. Baldwin stated that NIH is strongly in favor of eliciting, identifying
and funding creative and novel research projects, but that in her view this
requires a complex intervention. She presented a six step plan for addressing
creativity and innovation, not all of which she noted would fall within the
purview of PROG. These six steps address ways to change the perception that NIH
does not want or reward creative research applications, by (1) more clearly
communicating interest in such research through the science press, world wide
web homepage activities, and speeches of our senior officials; (2) communicating
to review groups that we expect them to identify creative/innovative
applications or parts of applications; (3) assuring the applicant community that
reviewers themselves represent creative thinkers who will recognize these
qualities in the applications they review; (4) ensuring that the language used
to guide reviewers in the preparation of critiques indicates the importance NIH
places on the identification of these applications; (5) encouraging the use of
selective pay strategies to ensure that creative or novel ideas are well
represented in the Institutes' research portfolios; (6) publicizing ways that we
have succeeded in soliciting and encouraging creative and novel research. The
fourth of these six points is the one that clearly falls within the domain
addressed by the PROG.
Generally the group endorsed this approach. Dr. McGowan commented that NIH
is funding some very creative work, and agreed that an educational campaign as
outlined by Dr. Baldwin would be a good way to increase the visibility of
existing efforts and increase them, and endorsed using three criteria, with
creativity and innovation incorporated, as a good step in this direction. Dr.
Yamamoto vigorously endorsed a fourth separate criterion of creativity; he
indicated that creativity was not subsumed under other criteria but was
independent. Those recommending four criteria voiced the view that this would
send a signal to the research community that they should send in creative
applications and to the reviewer community that they needed to be alert to such
applications. Dr. Lenfant made the point that we must balance among various
kinds of research, and that some very valuable projects may not be ground
breaking with regard to novelty but are crucial to the advancement of science.
At this point Dr. Ehrenfeld indicated that she viewed Impact as the most
important of all the criteria sets discussed, stating that it should be listed
first and clearly elevated in importance. Dr. Marrett agreed, stating that
Impact is so important that it needs to be highlighted for both applicants and
reviewers. The group generally agreed. Dr. McCombs argued for changing the label
"Feasibility" to "Approach"; the group agreed that this
should be considered.
Dr. Baldwin summarized reasons to stay with the three explicit criteria sets
being discussed, with creativity/innovation incorporated within them: the strong
preference of both NIH staff and some elements of the scientific community for
this approach, the results of the pilot studies, the belief that creativity and
innovation are dispersed throughout those three areas, the view that the problem
is a complex one requiring a broader intervention (as outlined above), and the
fact that program staff have indicated that a fourth separate criterion for
creativity would not be helpful. Although there were dissenters and abstainers,
a plurality voted to recommend the adoption of three criteria. It was agreed
that this information along with the members' alternate proposals would be
presented to Dr. Varmus.
Go back to the Table of Contents
Review of Clinical Research
The Report of the NIH Director's Panel on Clinical Research was discussed,
with specific attention to the fourth recommendation: "Ensure fair and
effective reviews of extramural grant applications for support of clinical
research; panels must (a) include experienced clinical investigators and (b)
review an appreciable number of such applications." There was some
discussion of the definition of clinical research put forth in the report; this
definition was accepted and praised. Dr. Kreek indicated that she felt the
definition for clinical research was very good and extremely useful. Dr. Norman
Anderson pointed out, however, that the definition as currently written would
include all behavioral studies as clinical research. As there is a large
community of behavioral and social scientists who do not consider themselves
clinical scientists, but who study fundamental behavioral processes in healthy
individuals (e.g. learning, memory, language, social loss), he suggested that
point 1.b. of the definition should specify the types of behavioral studies that
should (and should not) fall under clinical research. Several PROG members
commented on the need to bring experienced clinical researchers into the process
of review of clinical applications. There was concern over the impact of the
time requirements and some discussion of the use of flexible mechanisms to
ensure that the broadest group of expertise could be brought to bear. It was
agreed that implementation of the review-related fourth recommendation of the
Director's Panel on Clinical Research is a suitable topic for the PROG. A
working group will be formed, chaired by Dr. David Kupfer with other PROG
members and members of the extramural clinical research community participating.
Questions to be addressed will include how to quantify what would constitute
appropriate reviewer qualifications and study section/scientific review group
composition for the review of clinical research applications, and what flexible
options for reviewer service might be employed in order to accommodate the
special needs of clinical researchers.
Go back to the Table of Contents
Electronic Reinvention Activities, Update
Mr. Geoff Grant, Director of the Office of Policy for Extramural Research
Administration, and his staff presented an update of the activities occurring at
the NIH in Electronic Research Administration (ERA), including the reengineering
of the NIH public and internal working databases for tracking grant application
referral, review, and award. Electronic demonstrations of modules in active
service, such as Edison for invention reporting, and under development, such as
the NIH "Commons", were presented. This information is presented in
the "NIH Reinvention Activities: Status Report" available on the world
wide web.
Go back to the Table of Contents
Constancy of Scoring Metric: Baseline Data
Dr. Anthony Demsey, Acting Deputy Director of the Division of Research
Grants, presented some preliminary information about study sections to examine
possible diurnal trends in the scoring metric during review meetings. Data were
collected from eight study sections whose orders of review were organized
alphabetically or numerically by IC; no distinctive differences in scoring
patterns were found for time blocks nearer the beginning or end of the meetings.
Based on these preliminary data, it appears that timing does not play a role in
scoring patterns. The group agreed that there was no indication that this issue
needed to be pursued further.
Go back to the Table of Contents
All applications reviewed in DRG were examined in terms of scoring patterns.
When examined by Initial Review Group (IRG) as well as overall, it appears that
there is an increased probability of scores averaging whole numbers (e.g. 150 or
160 rather than 147,151, or 159.) These data could serve as the background for
examining some scoring issues, such as what happens if reviewers are limited to
five or seven choices rather than having a scale in tenths of a point. How are
they using the choices they already have? What effect on scoring might a
truncated scale have? It was suggested that the group might request information
on whether the discussion at the meeting alters what reviewers initially posited
as scores, on how many reviewers actually read the applications, what parts of
the applications appear most relevant to the reviewers in assigning scores. It
was agreed that these are provocative questions that the group may want to
discuss at future meetings.
Go back to the Table of Contents
Update on Neuroscience Integration
Dr. Steve Hyman, Director of the National Institute of Mental Health, gave a
brief update on the integration of the review of neuroscience applications. He
emphasized the cooperation and collegiality with which this activity is being
implemented across Institutes both at the Directors' and staff levels. He
indicated that staff realize that this reorganization will allow for the best
assessment of science for funding, but that the referral guidelines will be
crucially important. He pointed out that one topic that will need to be
addressed as review moves to the DRG is whether several mechanisms are reviewed
within single review groups, which is currently the practice at NIMH and the
National Institute for Drug Abuse. The issue of where program projects will be
reviewed needs to be addressed thoughtfully. The review of clinical neuroscience
applications will also need to be addressed. Dr. Hyman acknowledged that the
transition period will be one which will require careful monitoring, but
concluded that the results of this major undertaking should result in benefits
to all, including the research investigators, since "in the end, funding is
all about high standards." There was some discussion of the intention to
reorganize the review of behavioral science applications next.
Go back to the Table of Contents
New Directions for the Division of Research Grants (DRG)
Dr. Ellie Ehrenfeld, the new Director of the Division of Research Grants,
spoke of the new directions she sees for DRG. She put these issues into six
broad categories: (1) issues around the reorganization of study sections, such
as entitlements, mapping of science to study sections, and keeping up with
emerging areas of science; (2) reviewer issues, such as their qualifications,
terms of service, and roles; (3) investigators' concerns in specific scientific
areas, such as clinical research, behavioral research, or emerging areas; (4)
review process issues, such as speed, efficiency, and consistency across study
sections; (5) communication issues between DRG and the ICs, and (6) staff
issues, such as Scientific Review Administrator training and recruitment. All of
these issues have both long and short term aspects, and DRG has already begun to
address some, of these, one being the reorganization of study sections, being
addressed through the Integration of Neuroscience working group. PROG members
acknowledged the importance of these issues, and Dr. Baldwin commented on the
opportunity for partnership between DRG and PROG, since many of these issues are
relevant to review across all of NIH. She asked that PROG members think about
which of these issues they might want to discuss in the future. Dr. Ehrenfeld
stated that she welcomed PROG members' thoughts on all of these issues.
Go back to the Table of Contents
Review of Multiple Mechanisms
Dr. Baldwin reminded members that part of the charge of the PROG is to
address location of review (DRG vs. IC) and how much this should be influenced
by the mechanisms involved and how much by the science. One aspect of this
question is whether multiple mechanisms can or should be reviewed within a
single study section. Dr. William Grace, an SRA from NIDA, discussed how NIDA
reviews multiple mechanisms within single study sections. The review is based on
the assumptions that each application should have the best scientific expertise
applied to it, and that reviewers can learn new material and be flexible in
applying what they learn. He pointed out that often the structure of the
application itself will cue a reviewer, such as a fellowship application which
contains a training plan and information on mentors. Meetings are also
structured to facilitate the process, by grouping like mechanisms for review.
Instructional information is provided with the applications, and the SRA orients
reviewers for each different mechanism at the meeting. Dr. Grace presented
information indicating that score distributions for the various mechanisms were
similar, and added that some reviewers held the opinion that combining
mechanisms in one review meeting enhanced the review. Typically a study section
might review as many as six different mechanisms (R01, R03, R29, K02, F31, and
F32.) When asked how he knew if the process works well, Dr. Grace replied that
he could only answer subjectively, but it is his impression that reviewers give
equal attention to each mechanism.
Dr. Jay Pettegrew, Department of Psychiatry and Neurology, University of
Pittsburgh Medical Center, and chair of an NIMH study section, also presented
views on review of multiple mechanisms within a single study section. He views
the study section as being able to be inclusive, integrative, and not elitist,
and finds having the study section broadly defined and covering a broad spectrum
of applications very useful. He added that having laboratory scientists and
clinicians together in one room is beneficial, as it enhances understanding and
communication among them, which is good for science. He sees the review of
various mechanisms not only possible but desirable. He also commented that
having a mix of senior and more junior scientists together on a panel allows
opportunities for mentoring by the more experienced reviewers. He added that the
SRA sets the tone for the study section, and that the chair and SRA must work
well together.
Dr. Marcia Litwak, an SRA from DRG, also manages a study section which
reviews multiple mechanisms. Dr. Litwak described the process by which her study
section, which was reviewing only R01 and R29 applications, gradually added
mechanisms, one mechanism at a time, over a period of years. She pointed out the
importance of preparing the reviewers in advance, educating the chair who then
worked closely with her, and orienting the group as they reviewed. She presented
similar scoring distributions for the various mechanisms, and stated that while
there is a learning process in moving from reviewing only research project
grants (R01, R29) to reviewing multiple mechanisms, the transition can be made
relatively easily.
There was some discussion among the PROG members regarding pros and cons of
mixing the review of various mechanisms, such as its contribution to breadth and
flexibility within a group, whether study sections can change from a strict
research project grant mindset, and whether the value of reviewing multiple
mechanisms within one group is worth the effort of initiating behavior change in
those who do not currently use this practice. Dr. Schachman pointed out the
value of attaining a critical mass of applications in a review group. For
example, if there are multiple review groups with very similar science, it might
be preferable for all the related fellowships to be assigned to one review group
each round. It was agreed that the solution need not be an all-or-nothing one,
and that some scientific review groups might use either strategy. Further
discussion will be around what the relative value of the practice is, and for
whom (program staff, applicants, reviewers); whether the practice be
incorporated, and in what situations; and what the limits might be.
Go back to the Table of Contents
Agenda Items for Next Meeting
The group agreed that the main issue that the PROG will address next is the
implementation of the Director's Panel on Clinical Research recommendation on
the review of clinical applications. They agreed that the working group should
proceed as soon as they could reasonably do so and not be necessarily
constrained by meeting dates of the PROG.
Another major issue is the integration/reorganization of the review of
neuroscience applications; by May the Neuroscience Integration will have held
its major meeting of extramural scientists to obtain ideas for the design of new
study sections and will be reporting that information to the PROG.
The issue of review of multiple mechanisms in study sections will be
addressed. The members have heard information from scientific review groups
which currently use this practice, and were charged to think about the
parameters of a continuum of use for this practice. Other issues that they
might address at future meetings also include reviewer and SRA education,
evaluation, and mentoring, and those aspects of Dr. Baldwin's six step plan on
creativity that relate to trans-NIH peer review.
Addition to Previous Minutes: There was one correction/addition to the
minutes from the November, 1996, meeting: In the discussion of clinical research
at that meeting, Dr. Norman Anderson commented that not all behavioral research
is clinical research, and that it is important to distinguish between clinical
and non-clinical behavioral research.
Go back to the Table of Contents
Several topics were discussed at this meeting. Dr. Baldwin noted that we
are within the comment period for revisions to the PHS 398 grant application
form, but emphasized the importance of respondent burden with any changes.
The group did not opt to take up this topic for further action. Dr. Baldwin
presented a brief summary of the SBIR meeting, noting that the report is available
on the WWW at http://grants.nih.gov/grants/funding/sbir.htm, and Dr. Ehrenfeld
noted that she would welcome future PROG input on the review of SBIR applications.
A discussion of the Rating of Grant Applications focused on whether or not
to adopt creativity as a separate criterion or incorporate it throughout the
three already recommended for adoption. Dr. Lenfant shared the report of his
subcommittee on program staff views; while the importance of funding creative
work is acknowledged, the group indicated that since creativity and innovation
permeate all of the other three areas (Impact, Feasibility, and
Investigator/Environment), the recommendation was not to have a separate fourth
criterion. Dr. Baldwin presented a six step plan for addressing the issue of
creativity: these included communicating clearly to the research community that
NIH values creativity, encouraging review groups to identify creative
applications, ensuring that review committees include reviewers who will
recognize and value creative applications, including language in the review
guidelines to indicate the importance of creativity, ensuring that creative or
novel projects are included in the Institutes' research portfolios, and
publicizing success in soliciting and funding creative research. The group
generally endorsed this approach. There was additional discussion on the value
of a separate fourth criterion, and a vote was taken on whether to recommend
three or four criteria. Although some members dissented and some abstained,
there was a plurality that voted to recommend three criteria: Significance,
Feasibility (renamed Approach), and Investigator/Environment. This information
along with alternate proposals will be presented to Dr. Varmus.
Other activities included a brief discussion of the review of clinical
research, with specific attention to the fourth recommendation in the report of
the NIH Director's Panel on Clinical Research, which addresses the review of
clinical applications. It was agreed that a working group should be formed to
address possible ways to implement that recommendation, and Dr. David Kupfer
agreed to chair the working group. Dr. Norman Anderson suggested that a bullet
be added to the definition of clinical research in the Panel on Clinical
Research report, to specify the types of behavioral studies that should and
should not be categorized as clinical (e.g. fundamental behavioral processes
such as learning, memory, language, and social loss in healthy individuals).
An informational update on NIH's Electronic Reinvention Administration was
presented by Mr. Geoff Grant and his staff. Dr. Steve Hyman, Director of the
National Institute of Mental Health (NIMH), gave an update on the Integration of
Neuroscience Review, indicating that this initiative is proceeding well, and
indicating that the transition period for implementation will require careful
monitoring but that the results of this reorganization process should ultimately
be of benefit to both investigators and NIH staff. Dr. Ellie Ehrenfeld,
Director of the Division of Research Grants (DRG), presented her views on new
directions for the DRG, including major issues around study section
organization, makeup and recruitment; reviewer and Scientific Review
Administrator (SRA) selection, education, and roles; and other important review
issues. She expressed enthusiasm for having PROG members share their thoughts
with her on many of these issues.
Data were presented on some potential issues which PROG might consider in
the future. Dr. Demsey presented some preliminary data on constancy of the
scoring method across the meeting time of study sections. As no distinctive
differences in scoring patterns were noted for time blocks nearer the beginning
or end of the meetings, the group agreed that there appeared to be no compelling
reason to pursue this issue further. Dr. Baldwin presented data on scoring
patterns across various Initial Review Groups in the DRG which indicated that
there appears to be an accelerated probability in some IRGs of reviewers
assigning whole number scores; she suggested that these data could serve as the
background for examining some scoring issues, such as number of points needed on
the rating scale and the effect a truncated scale might have. Experiences
reviewing multiple mechanisms within single study sections were shared by Dr.
William Grace, SRA at the National Institute on Drug Abuse, Dr. Jay Pettegrew,
chair of an NIMH scientific review group, and Dr. Marcia Litwak, SRA at DRG. All
were positive about having the review include various mechanisms, and Dr.
Pettegrew offered the opinion that it enhances the review. All agreed that the
SRA and chair play crucial roles and must be in synchrony for this to succeed.
Agenda items for the next meeting will include review of clinical
applications, results of the Neuroscience Integration meeting of extramural
scientists taking place in March, and further discussion of the review of
multiple mechanisms within single scientific review groups. In addition, at the
next or some future meeting the group may discuss reviewer and SRA education,
evaluation and mentoring, and peer-review related aspects of Dr. Baldwin's six
step plan on creativity.
I hereby certify that, to the best of my knowledge, the foregoing minutes
are accurate and complete.
Peggy McCardle, Ph.D.
MPH, Executive Secretary Peer Review Oversight Group
Wendy Baldwin, Ph.D.
Deputy Director for Extramural Research
Go back to the Table of Contents
Return to PROG Home Page
|