What Is the Evidence Rating?
Every profile includes an Evidence Rating, presented in the "Snapshot" and "Did It Work?" sections, which is
an assessment of the quality and strength of the evidence that the results described in the profile are due
to the innovation and not to other factors. The Evidence Rating is one type of information that a potential
adopter of the innovation may consider when making a judgment about the likely benefits and risks of
implementing the innovation in their own organization.
How to Use the Evidence Rating
The adoption decision is complex and must take into consideration other factors besides the innovator’s
results, even when those are compelling. This is because local delivery systems, policies, resources,
and culture vary, and may have a profound impact on the results of any change in service delivery practices1.
Innovation adoption is not simply a matter of selecting an effective innovation. Any innovation must also
be implemented well in order to achieve its maximal impact, taking into account the particular needs,
history, and context in the adopting organization. The AHRQ Innovations Exchange includes a variety of
resources to guide the adoption decision and implementation process including:
- Information from the innovator in each innovation profile such as:
- Context of the Innovation – Describes the innovating organization.
- Adoption Considerations – Information on starting and sustaining the innovation.
- Planning and Development Process – Discusses the pre-innovation phase.
- Results and Evidence Rating – Describes the innovator’s evaluation methodology and findings.
- Contact the Innovator – All innovators have agreed to be contacted.
- References – Information on relevant research studies in other locations, and other material.
- In-depth information on Taking Innovations to Scale on this topical page featuring innovation spread stories, strategies,
tools, and voices from the field (http://www.innovations.ahrq.gov/takingInnovationsToScale.aspx).
- A Decisionmaker’s Guide to Adopting Innovations, a tool for assessing the “fit” of an innovation and understanding whether
it will work for you (http://www.innovations.ahrq.gov/guide/guideTOC.aspx).
- A variety of articles on innovation basics and expert perspectives (http://www.innovations.ahrq.gov/articles.aspx).
These resources can help place the Evidence Rating in context when assessing the ability of an innovation to address local problems.
Components of the Evidence Rating
The Evidence Rating is based on three types of information:
- Information about the design of the evaluation of the innovation
- The results of any quantitative evaluation of the innovation
- Non-experimental, qualitative, or anecdotal information about the effects of the innovation
The Evidence Rating draws on aspects of several established systems that implicitly or explicitly assess the strength of study
design and comprehensiveness of reporting including the GRADE guidelines for rating the quality of evidence2,
the Consolidated Standards of Reporting Trials (CONSORT) statements which are an evidence-based, minimum set of recommendations for
reporting on randomized controlled trials3, and the STROBE initiative for STrengthening the Reporting of OBservational studies in
Epidemiology (STROBE)4. These and other criteria for the conduct of health services research studies are integrated in the
Patient-Centered Outcomes Research Institute Preliminary Draft Methodology Report5, an evolving document meant to be a catalyst
for scientifically rigorous, patient-centered outcomes research that can inform decision-making. Evidence for health care policy innovations must support a
connection between the innovation and an impact on either social structures and institutions, the delivery of health care services, or health outcomes6.
Many innovations have not been around long enough to have been rigorously evaluated. In fact, by the time many ideas are rigorously
evaluated, they are no longer new or innovative. As a result, the Evidence Rating was designed to cover not only well-established,
evidence-based innovations, but also cutting-edge ideas that are not as well studied.
Three Categories for Rating the Strength of Evidence
Drawing on elements of these established systems, the Innovations Exchange uses three categories to provide
meaningful distinctions in assessing the strength of the link between the innovation and the observed results:
Strong: The evidence is based on one or more evaluations using experimental
designs based on random allocation of individuals or groups of individuals (e.g. medical practices or hospital units) to comparison groups.
The results of the evaluation(s) show consistent direct evidence of the effectiveness of the innovation
in improving the targeted health care outcomes and/or processes, or structures in the case of health care policy innovations.
Moderate: While there are no randomized, controlled experiments, the evidence includes at
least one systematic evaluation of the impact of the innovation using a quasi-experimental design,
which could include the non-random assignment of individuals to comparison groups, before-and-after
comparisons in one group, and/or comparisons with a historical baseline or control. The results of the
evaluation(s) show consistent direct or indirect evidence of the effectiveness of the innovation in
improving targeted health care outcomes and/or processes, or structures in the case of health care policy innovations. However, the strength of the evidence is limited
by the size, quality, or generalizability of the evaluations, and thus alternative explanations cannot be
ruled out.
Suggestive: While there are no systematic experimental or quasi-experimental evaluations,
the evidence includes non-experimental or qualitative support for an association between the innovation and
targeted health care outcomes or processes, or structures in the case of health care policy innovations. This evidence may include non-comparative case studies,
correlation analysis, or anecdotal reports. As with the category above, alternative explanations
for the results achieved cannot be ruled out.
If the available qualitative and quantitative information is insufficient to place the innovation in one of
the three categories above, the activity fails to meet the minimum inclusion criterion for evidence, and therefore
is not eligible for inclusion as an Innovation Profile in the AHRQ Health Care Innovations Exchange. It may,
however, qualify for inclusion as an Innovation Attempt.
If you have questions about the Evidence Rating in general, or about the rating on a specific innovation, please
contact us at info@innovations.ahrq.gov.
1. Shekelle, S et.al. 2011. Advancing the Science of Patient Safety. Annals of internal Medicine. 154(10):693-6.
2. Guyatt GH, et.al., 2011, “GRADE guidelines: A new series of articles in the Journal of Clinical Epidemiology”,
J of Clinical Epidemiology, 64:380-2.
3. CONSORT Web site http://www.consort-statement.org/ , accessed July 27, 2012.
4. STROBE Web site http://www.strobe-statement.org/index.php?id=strobe-home , accessed July 27, 2012.
5. Pre-public comment draft report of the Patient-Centered Outcomes Research Institute (PCORI)
Methodology Committee. Presented to the PCORI Board of Governors May 10, 2012 and published online June 4, 2012.
6. Burris, S., Wagenaar, A.C., Swanson, J., Ibrahim, J.K., Wood, J., and Mello, M.M. (2010). Making the case for laws that improve health: A framework for public health law research. The Milbank Quarterly, 88(2), 169-210.
Last updated: January 23, 2013.
|