Conducting a Sound Systematic Review: Balancing Resources with Quality Control

Slide Presentation from the AHRQ 2008 Annual Conference

---

On September 10, 2008, Eric B. Bass, made this presentation at the 2008 Annual Conference. Select to access the PowerPoint® presentation (100 KB).

---

Slide 1

Conducting a Sound Systematic Review: Balancing Resources with Quality Control

• Eric B. Bass, MD, MPH
  Johns Hopkins University
  Evidence-based Practice Center

Slide 2

Learning Objectives

• Identify steps in a systematic review that have greatest implications for resource needs.
• Explain how those steps are vulnerable to error/bias.
• Identify strategies for quality control when resources are limited.

Slide 3

Steps in the systematic review

1. Establish eligibility criteria for each question.
2. Search literature.
3. Review citations.
4. Extract data.
5. Evaluate study quality & applicability.
6. Summarize & synthesize evidence.

Slide 4

Focus the Question(s)

• Formulate in specific measurable terms.
• Clarify priorities.
• Be realistic about time & effort.
• Negotiate with sponsor.

Slide 5

Set Eligibility Criteria

• Study design
  • RCT only?
  • Comparison group?
  • Any observational study?
• Study population.
• Study setting.
• Size of study.
• Year of publication.
• Peer-reviewed?
• Language.

Slide 6

Search Literature

• Select highest yield sources.
  • MEDLINE®
  • EMBASE®
  • Cochrane database of reviews & trials.
• Consider other sources.
  • Use sampling to estimate incremental yield.
• Use hand searches for quality control.
  • Citations in eligible articles.
  • Table of contents of relevant journals.
  • Query experts.

Slide 7

Review Citations

• Titles:
  • Many citations obviously not relevant & can be excluded quickly.
• Abstracts:
  • Usually sufficient to determine eligibility of citations.
• Full text of articles:
  • For some studies, eligibility can only be determined by reading full text.

Slide 8

Review Citations: An Example

• Title review (n=10,475):
  • Title review: 6,863 excluded.
• Abstract review (n=3,612):
  • Abstract review: 3, 163 excluded.
• Full text review (n=449):
  • Full text review: 386 excluded.
• 63 included articles.

Slide 9

Review Citations: Quality Control

• Use independent dual reviewers.
• Assess samples early.
• Discuss discrepancies.

Slide 10

Extract Data

• Set priorities for data extraction.
  • Anticipate content of final evidence tables.
  • Resist temptation to extract everything.
• Choose data management method.
  • Systematic review software vs. Access vs. tables.
  • Consider complexity, consistency, completeness of data.
• Establish quality control.
  • Dual review vs. solo review with random checks.
  • Independent vs. sequential dual review.
  • Discuss discrepancies.

Slide 11

Evaluate Study Quality & Applicability

• Decide how evaluation will be used.
  • To determine eligibility of studies.
  • To give more or less weight to studies.
  • To improve future research.
• Focus on most important aspects of quality & applicability.
• Use established instrument(s).
  • Jadad criteria for Randomized Controlled Trials (RCTs).
• Establish quality control.
  • Independent dual review.
  • Decide how to reconcile discrepancies.

Slide 12

Summarize & Synthesize Evidence

• Assemble evidence tables.
• Reassess priorities.
• Exclude duplicative data.
• Prepare summary of evidence on each question.
• Assess strength of evidence.
• Use 2 or more reviewers.
• Decide how to reconcile differences.

Slide 13

Summary

• To conduct a sound systematic review with limited resources:
  • Pay attention to priorities when defining questions, eligibility criteria & elements of evidence tables.
  • Establish quality control measures for each step that could introduce error or bias.
    • Reviewing citations.
    • Extracting data.
    • Evaluating quality, applicability & overall strength of evidence.

Current as of January 2009

---

Internet Citation:

Conducting a Sound Systematic Review: Balancing Resources with Quality Control. Slide Presentation from the AHRQ 2008 Annual Conference (Text Version). January 2009. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/about/annualmtg08/091008slides/Bass.htm

---