Comparison of Existing and
New Peer Review Processes |
Function |
Old |
New |
Assignment of priority scores |
Priority scores reflect reviewer judgment
of a whole application: peer review criteria are unweighted and unrelated
to the priority score. |
Changed to preliminary impact score. Assigned
reviewers also score each criterion (remains unrelated to the overall
score) Before the review meeting, each reviewer and discussant assigned
to an application will give a preliminary impact score for that application.
The preliminary impact scores will be used to determine which applications
will be discussed. For each application that is discussed, a final
impact score will be given by each eligible committee member (without
conflicts of interest). Each member’s impact score will reflect his/her
evaluation of the overall impact that the project is likely to have
on the research field(s) involved. |
Each reviewer scores to one decimal place: 1.0 is best,
5.0 worst. |
Each reviewer scores in whole numbers: 1 is best, 9 worst. |
Determination of priority scores |
To create a raw score, reviewer scores are
averaged and rounded mathematically to two decimal places, e.g., 1.34.
The result is multiplied by 100 to give an overall priority score,
e.g., 134.
The possible scores range from 100 to 500. |
The overall impact score for each discussed
application will be determined by calculating the mean score from all
the eligible members’ impact scores, and multiplying the average by
10; the overall impact score will be reported on the summary statement.
Thus, the 81 possible overall impact scores will range from 10 - 90.
(Overall impact scores will not be reported for applications that are
not discussed.)
Scoring with fewer rating options increases potential
reliability and provides sufficient range and appropriate anchors to
encourage reviewers to use the full scale. |
Streamlined applications |
Applicants get critiques from assigned reviewers. |
Streamlined applications will receive scores
on each criterion in addition to the reviewers’ critiques to help applicants
assess whether or not they should resubmit an amended application |
Determination of percentiles |
Percentile range from 0.1 (best) to 99.5
(worst). Read How Percentiles Are Determined. |
Percentiles range from 1 to 99 in whole
numbers. Rounding is always up, e.g., 12.1 percentile becomes 13. |
With almost 1,000 possible percentile rankings,
few applications are ranked the same. |
The new scoring system may produce more applications
with identical scores (“tie” scores). Thus, other important factors,
such as mission relevance and portfolio balance, will be considered
in making funding decisions when grant applications are considered essentially
equivalent on overall impact, based on reviewer ratings. |
Percentile base |
NIH calculates percentiles using applications
submitted for three review cycles. |
Unchanged, except for the first year of
the transition to the new review processes:
- First new cycle: NIH to calculate percentiles using those applications only.
- Second new cycle: NIH to calculate percentiles using applications submitted
for first and second cycles.
|
Scores for Individual Criteria |
Scores are not provided for individual critieria. |
Before the review meeting, each reviewer
and discussant assigned to an application will give a separate score
for each of five core review criteria (Significance, Investigator(s),
Innovation, Approach, and Environment). To improve information and
transparency, for all applications, even those not discussed by the
full committee, the scores of assigned reviewers and discussant(s)
for these criteria will be reported individually on the summary statement. |
Review Criteria |
Five one-word criteria plus descriptive
information. |
One-word criteria unchanged; descriptions
modified.
Reviewers will consider each of the five review criteria in
the determination of scientific and technical merit, and give a separate
score for each. An application does not need to be strong in all categories
to be judged likely to have major scientific impact (i.e. a project
that by its nature is not innovative may be essential to advance a field). |