Print | E-mail

 

Improving NIJ's Peer Review Process: The Scientific Review Panel Pilot Project

by Thomas E. Feucht and Phyllis Newton

NIJ is piloting a new grant application peer review process with standing panels designed to improve the consistency, fairness and transparency of peer review at the Institute.

Two key events in the "life course" of the National Institute of Justice occurred in 2010. In June, the National Research Council (NRC) released its evaluation of NIJ, Strengthening the National Institute of Justice; and in July, John Laub became the Director of NIJ — the first NIJ director to have a Ph.D. in criminal justice.

The release of the evaluation and Laub's appointment put NIJ in a good position to reinvigorate its science mission, one of NRC's core recommendations.[1]

NRC noted the need for NIJ to do a better job in its research program development and planning and to put in place more rigorous processes and policies related to research. Perhaps no operational issue drew more attention than the peer review of applications for research grant awards. Citing serious concerns about NIJ's current peer review processes, NRC's evaluation called for significant improvements in how peer review is conducted in order to safeguard the science and to ensure a fair, transparent and competitive process for making research grants.

Peer review lies at the heart of grantmaking. It guarantees an independent assessment of the scientific merit of the proposed research. Rigorous, fair and transparent reviews by external scientific reviewers help ensure that the competitive grant process works as it should, that the best science gets funded, and that standards of scientific excellence are consistently applied to all grant applications.

Peer Review in the Past at NIJ

For more than two decades, NIJ's peer review process involved assembling small committees (usually three or four reviewers) for each review cycle — a typical way to conduct anonymous peer reviews. But because the panels were selected anew each year, problems could arise with consistency from one year to the next. Applicants who were offered an opportunity to revise and resubmit, for example, had their applications reviewed the second time by a completely different panel. In addition, assembling a committee could be challenging when deadlines were short.

The NRC's evaluation characterized NIJ's peer review as "very weak," and urged the Institute to look to other science agencies, like the National Science Foundation and the National Institutes of Health, for good peer review models.

NIJ took the recommendation to heart and has begun a pilot program to strengthen its review processes.

Peer Review in the Future at NIJ

Starting in the review cycle for fiscal year 2012, NIJ will establish a total of five Scientific Review Panels in the following topic categories:

• Criminal justice systems
• Violence and victimization
• Forensics (two panels)
• Science and technology

Each panel will consist of 12 scientists and six practitioners. Scientific members will serve for overlapping three-year terms to provide continuity, consistency and experience. Practitioner members will serve one-year terms. The panelists, recognized authorities in their field, will be nominated by other researchers and practitioners. Final selection will be made by the appropriate NIJ Office director. The names of the panelists will be posted following the announcement of grant awards on NIJ.gov.

NIJ anticipates the need to include ad hoc panelists who can review applications that require specialized expertise. Some solicitations, for example, are likely to generate proposals for research in narrowly defined areas of study, such as some engineering and biological sciences. In such cases, NIJ will invite ad hoc members with applicable expertise to serve on the Scientific Review Panels. In most cases, ad hoc members will serve for a single review cycle.

Steps in the Review Process

NIJ's grant application review process can be summarized into four major steps. The members of NIJ's pilot Scientific Review Panels will be involved in two of the four steps:

Step 1. NIJ staff will verify that applications meet the basic minimum requirements (BMR) as stated in the solicitation. BMR verification is conducted to ensure that the documents necessary for effective scientific review are included in the application. As a rule, an application that includes a program narrative, a budget narrative, a budget detail worksheet and curriculum vitae for key personnel will meet BMR.

Step 2. Each application will be read by three members of a Scientific Review Panel (the lead and two co-leads) who will assess the application's technical merit and write narratives of their reviews. They will give each application a response of "forward to the full panel" or "do not forward to the full panel" based on scientific merit, including such factors as statement of problem, project design and implementation, capabilities/competencies, budget, impact/outcomes and evaluation, and dissemination strategies.

Step 3. All 18 members of a Scientific Review Panel will have the opportunity to review every application and participate in the consensus review. Panels will meet for two or three days once a year, during the same week every year. For each forwarded application, the lead and co-lead reviewers will give a brief overview, present their scores, outline the strengths and weaknesses, and lead the discussion. Following the discussion, each panelist will score the applications using a written ballot.

Step 4. NIJ staff will receive and review the scores and narrative written by the lead and co-leads. The staff will then make recommendations to the Director, who will make final decisions based on the scores as judged by the reviewers, how the proposed projects fit into NIJ's priorities and research agendas, and the availability of funding.

Scoring the Applications

At the meeting of the full panel, each panelist will have an opportunity to discuss the application and the lead and co-leads' narrative before giving the application an overall score for quality.

Scores will be based on a 5-point scale from 5 (excellent) to 0 (poor), in which intermediate values are treated as equal intervals on the scale.

The Scientific Review Panel's overall score for each application will be a consensus. The panel therefore must resolve disparities that arise among reviewers' overall scores. If a difference of opinion remains, a minority report will be written and submitted to NIJ.

To ensure transparency of the review process and to aid applicants in developing effective applications, applicants will receive their final scores and written technical reviews.

As Director Laub stated in his response to the NRC report, "At the core of a strong science agency is a rigorous and fair peer review process. All grants, for instance, must be awarded as the result of a fair, open, and competitive peer review process." NIJ's new Scientific Review Panels are an important step toward ensuring that NIJ's precious research resources are invested in only the best research proposals and that each proposal submitted to NIJ receives a fair and scientifically sound review.

NIJ Journal No. 268, October 2011
NCJ 235892

About the Authors

Thomas E. Feucht is NIJ's Executive Senior Science Advisor. Phyllis Newton is the Director of NIJ's Office of Research and Evaluation.

Back to the top.

For More Information

• Contact Thomas Feucht, Executive Senior Science Advisor, thomas.feucht@usdoj.gov, 202-307-2949.
• Learn how to nominate a panelist.