Peer Review Process

Overview

In accordance to the Public Health Service Act and the federal regulations governing "Scientific Peer Review of Research Grant Applications and Research and Development Contract Proposals" (42 CFR Part 52h), applications submitted to the Agency for Healthcare Research and Quality (AHRQ) will be evaluated via AHRQ peer review process to ensure a fair, equitable, and unbiased evaluation of their scientific and technical merit.  The initial peer review of grant applications involves an assessment conducted by panels of experts established according to scientific disciplines or medical specialty areas.

A Scientific Review Administrator (SRA) is the Designated Federal Official of the initial review group meeting.  Her/his role is to make sure that each application receives a review that is thorough, competent and fair.  AHRQ has established a set of principles or rules that must be followed to assure this goal, and it is the SRA's job to see that these principles are observed.  The SRA oversees, coordinates and administers the peer review process. The SRA performs initial administrative and technical review of applications to ensure completeness and accuracy. The SRA also nominates reviewers based on broad input from scientific community, AHRQ program staff, and experts who have served as peer reviewers. 

Reviewers are selected to participate in the scientific review groups (SRG)  based on a number of considerations, including:

• Authorities in their scientific fields. 
• Dedication to high quality, fair and evenhanded reviews
• Demonstration of their abilities to work collegially in a group setting
• Experience in grant/contract review.

In addition to these considerations, the SRA must ensure that race, ethnicity, gender, and geographical diversity are preserved within the SRG.

The SRA works with the study section chairperson to ensure that the peer review meeting is fair and yields high quality reviews. To accomplish this, the peer review meeting should include:

• Balanced and adequate scientific discussion that supports the review panel's recommendations.
• Consistent, criteria-based scoring.
• Adequate attention to protection of human subjects and inclusion of priority populations.

Following the peer review meeting, the SRA prepares summary statements for all applications. The summary statement contains:

• Resume and summary of discussion.
• Minimally-edited reviewers' critiques.
• Priority score and percentile ranking.
• Budget recommendations.
• Administrative notes.

The summary statement is an official feedback to the applicant conveying the issues, critiques, and/or comments that were raised during the review of his/her application.

Streamlining/Triaging

Streamlining is a practice through which less competitive grant applications, as judged by the study section members, are not discussed at the review meeting. In order for an application to be streamlined, there must be unanimous agreement amongst study section members. These applications do not receive a priority score. For the unscored applications, only summary statements integrating all of the written reviews from the assigned reviewers will be sent to the applicants.

Understanding Your Priority Score

Each grant application receives a single global score reflecting the overall rating based on scientific and technical consideration. This score is called the priority score.

The emphasis on each criterion varies from one application to another depending on the individual nature of the application. An application does not need to be strong in all categories to be judged likely to have a major scientific impact.

The priority scores, assigned by the study section members, range from 1.0 (most meritorious) to 5.0 (worst scored), with 1 decimal points.  Reviewers score applications for scientific and technical merit according to the following scale:

1.0-1.4   Outstanding
1.5-2.0   Excellent
2.1-2.4   Very Good
2.5-2.9   Good
3.0-3.9   Fair
4.0-5.0   Poor

The overall score reflects the average of individual ratings by all study section members. Abstaining reviewers and those not present during the discussion do not vote a score.

Understanding Your Percentile

A percentile, based on priority score, is an application's rank relative to other applications that are reviewed by the same scientific review group at 3 consecutive meeting sections (review meeting cycles). AHRQ calculates an application's percentile based on the combined priority scores of applications being reviewed by the study section at which the application was reviewed and by the same study section at previous two meetings. Priority scores from the combined 3 consecutive review meetings are known as the base. Percentile ratings are calculated only for scored applications reviewed by study sections. A percentile, which may range from 0.1 to 100.0; with lower numbers represent better scores, is a ranking used to provide information in the funding decisionmaking process.

Review Criteria

There are six components that make up the review criteria:

• Significance and originality.
• Methods and data.
• Organization of the project.
• Investigators.
• Budget.
• Facilities, resources, and environment.

A description of each component is provided below.

Significance and Originality

Assess the contribution of the proposed project.  Is the literature review adequate to place the project in context?  Does this study address an important problem?  If the aims of the application are achieved, how will scientific knowledge, clinical practice, or health care and services be advanced?  What will be the effect of these studies on the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?  What are the features of the project demonstrating originality or innovation?

Methods and Data

Are the conceptual framework, design, methods and analyses adequately developed, well-integrated and appropriate to the aims of the project? Are key concepts and variables clearly defined? Does the applicant acknowledge potential problems area and consider alternative tactics?

Organization of the Project

Are the plans for organizing and carrying out the project well-specified and appropriate?

Investigators

Are the plans for organizing and carrying out the project adequately specified and appropriate?  Are the roles and responsibilities of key staff, as well as the flow of the project in terms of key events and products adequately described?

Budget

Is the budget reasonable and adequate in relation to the proposed project?

Facilities, Resources, and Environment

Are the resources of the applicant institution and other study sites adequate? Does the application include documentation of agreements among participants and organizations as appropriate?  Do the proposed studies benefit from unique features of the scientific environment or subject populations, or employ useful collaborative arrangements?  Is there evidence of institutional support?

Other review considerations include:

Human Subjects Protection. Where the study includes human subjects, does the project adequately describe:

• Risks to subjects?
• Adequacy of protection against risks?
• Potential benefits of the proposed research to the subjects and others?
• Importance of the knowledge to be gained?

Inclusion of Women and Minority Subjects. As appropriate, are both genders and minorities (including relevant sub-populations) included in the study population?

Inclusion of additional AHRQ priority populations. Is adequate consideration given to AHRQ priority populations?

Importance and Impact (if applicable).  What are the importance of the problem addressed and the potential impact of the solution of this problem on health or on health care delivery? 

Data Sharing Plan (if applicable). Whether the proposed data sharing plan or the rationale for not sharing research data is reasonable?

Current as of October 2006

---

Internet Citation:

Peer Review Process. October 2006. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/fund/peerrev/peerproc.htm

---