Section 3 Topic Work Plan Development

When a topic is prioritized by the Topic Prioritization Work Group (TP WG) for a new or updated recommendation, the scope of the topic and approach to the review must be defined to guide the researchers undertaking the systematic review process. U.S. Preventive Services Task Force (USPSTF) leads volunteer for each topic as it begins, based on expertise and interest. An Agency for Healthcare Research and Quality (AHRQ) medical officer is assigned to oversee the topic and develops a work assignment that outlines the major issues guiding the anticipated review. A lead investigator is assigned from the evidence review team with core review staff also identified. The review team refines the work assignment using the template described below and finalizes it through discussions with the Task Force leads and the AHRQ Medical Officer. For new topics, the work plan is peer-reviewed and presented to the entire USPSTF at one of its regular meetings. The final version of the work plan is approved by the AHRQ Medical Officer. Development of a work plan generally takes from 1 to 4 months, but may take longer.

3.1 Determining Topic Scope and Review Approach

3.1.1 Types of Reviews

The USPSTF has determined that using systematic reviews where possible represents the best method for fully and fairly considering all of the evidence relevant to questions about a clinical preventive service. In order to answer the relevant questions about a clinical preventive service, the Evidence-based Practice Centers (EPCs) usually undertakes a series of related systematic reviews to answer each of the questions in the analytic framework. However, systematic reviews can be labor-intensive and expensive processes. In order to efficiently invest its limited resources in maintaining the most current evidence-based recommendations for the breadth of important clinical preventive services, the USPSTF has developed a series of approaches to defining the scope of the topic that the EPC will systematically review.

These approaches include commissioning:

• Full systematic reviews, i.e., systematic reviews of every key question in the analytic framework. This is the default approach for a topic new to the USPSTF.
• Targeted systematic reviews, i.e., systematic reviews for selected questions in the analytic framework for which the evidence is not clearly established and/or current. Many updates of USPSTF topics use this approach to review relevant evidence since the last systematic review.
• Staged reviews, i.e., systematic reviews for selected questions in the analytic framework, undertaken in a hierarchical manner in order to inform the Task Force as to whether it is possible to establish evidence insufficiency in a parsimonious fashion. In rare cases, the Task Force may see that serious gaps in evidence preclude coming to a recommendation, and a staged review may confirm these gaps and thus lead to an I statement from the Task Force.

3.1.2 Principles for Determining the Review Approach

During work plan development, the topic team considers the scope of the evidence needed for the USPSTF to make its recommendation and the most efficient way to procure the needed information. For reviews undertaken to update existing USPSTF recommendations, this process is based on:

1. Examination of the previous USPSTF recommendation(s) including populations and clinical preventive services addressed, to determine their fit with current questions about the clinical preventive service.
2. Examination of the previous USPSTF evidence review process for this topic and the review findings in order to identify established evidence, important review limitations, and evidence gaps. For example, in the 1996 Guide to Clinical Preventive Services, behavioral counseling topics had two recommendations: one for the relationship between the behavior and health outcomes (e.g., smoking and premature morbidity and mortality) and one for the efficacy of counseling. When updating such recommendations, it is often not necessary to update the behavior-health outcome link, as that is often well-established.
3. Determination of current contextual information (e.g., changes in understanding of the nature of the disease process, or changes in diagnosis, therapeutics, practice or reimbursement; controversy over any of these elements).

In order to facilitate the consistent development of the review approach across topics, the USPSTF has developed a template to guide the development of the final Work Plan. This template can be considered analogous to a protocol developed for a Cochrane Review. It is also an articulation of the rationale for the scope decisions made in framing the topic.

3.2 Methods Relevant to Work Plan Development

The work plan template (Appendix VI) is used by AHRQ in the creation of the work assignment, and by the EPC and the Task Force leads in the refinement of the work assignment to produce the final work plan. This template stimulates the thinking and guides the systematic consideration of the factors that experience has shown are important in planning a review to update an existing USPSTF recommendation. Since most reviews conducted for the USPSTF are updates, the work plan template was developed with that purpose in mind. However, the same template can be used to plan and guide the full systematic review for a new topic. Sections addressing the previous USPSTF recommendation, previous review findings, and priority evidence would obviously not be relevant or included.

3.2.1 Analytic Frameworks

The purpose of analytic frameworks is to present clearly in graphical format the specific questions that need to be answered by the literature review in order to convince the USPSTF that the proposed preventive service is effective and safe (as measured by outcomes that the USPSTF considers important). The specific questions are depicted graphically by linkages that relate interventions and outcomes. These linkages serve the dual purpose of identifying questions to help structure the literature review and of providing an "evidence map" after the review for the purpose of identifying gaps and weaknesses in the evidence. Further details about the design of analytic frameworks are provided in a paper by Woolf et al., published by AHRQ in 1994 (6).

3.2.1.1 Analytic Frameworks Are Not Causal Pathways

Analytic frameworks as used by the USPSTF are not intended to depict comprehensively all factors and variables that cause patients to receive the preventive service or all factors and variables responsible for the intermediate outcomes and health outcomes associated with a preventive service. In short, they do not depict the "mechanism of action" for a preventive service. For example, an analytic framework for cervical cancer screening that is concerned mainly with two questions (whether Pap smears detect early disease and whether early detection lowers mortality) need not specify other co-variables: e.g., the risk factors for cervical cancer, the demographic characteristics of women who are more likely to be screened, the etiologic determinants of cervical cancer, or the pathologic progression of cervical cancer from atypia (atypical cell stage) through invasive disease.

3.2.1.2 Analytic Frameworks Are Not Decision Algorithms or Flowcharts

The use of arrows and boxes gives analytic frameworks the appearance of decision trees and flowcharts but the purpose is quite different. Analytic frameworks are not intended to depict all possible outcomes of a particular event, as is expected of decision analytic models, or to calculate their probabilities. A linkage in an analytic framework for mammography screening might ask whether early detection lowers mortality. The same step in a decision analytic model would be more likely to depict each component in that step (confirmation of suspicious lesion on mammogram, follow-up mammographic views, referral to specialist, biopsy and nodal evaluation, pathologic studies, lumpectomy or mastectomy, adjuvant chemotherapy, treatment for recurrent disease) and to depict decision nodes and probabilities for each step in the process. That is not necessary for analytic frameworks because the evidence question of interest to the USPSTF in this case is whether early detection lowers mortality.

3.2.1.3 Actions Versus Outcomes

Analytic frameworks used by the USPSTF distinguish between actions (e.g., the performance of a screening test, or treatment with a drug) and outcomes such as the detection of a disease, reduced morbidity and mortality, a change in patients' behavior, or adverse effects. The performance of a screening test is not itself an outcome. Actions are depicted by arrows, whereas outcomes are depicted by rectangles (Figure 3.)

3.2.1.4 Intermediate Outcomes Versus Health Outcomes

Analytic frameworks used by the USPSTF distinguish between intermediate outcomes and health outcomes. Health outcomes are symptoms and conditions that patients can feel or experience, such as visual impairment, pain, dyspnea, impaired functional status or quality of life, and death. Intermediate outcomes are pathologic and physiologic measures that may precede or lead to health outcomes. Elevated blood cholesterol level is an intermediate outcome for coronary artery disease; patients cannot "feel" the cholesterol elevation. Myocardial infarction, or death from coronary artery disease, is a health outcome. The USPSTF gives greater weight to evidence of an effect on health outcomes than to evidence of an effect on risk factors or intermediate outcomes. The fact that a preventive service has a proven effect on an intermediate outcome does not necessarily establish that it can improve outcomes that are perceptible to patients.

When data are available, the Task Force considers data on both all-cause and cause-specific mortality in making its recommendations. (Go to Section 4.5.3 for a fuller discussion of these outcomes.)

3.2.1.5 Conventions for Graphics and Layout

The analytic framework diagram contains three types of items (population, actions, and outcomes). Below the diagram are annotated questions that correspond to specific items in the diagram. The annotated questions are designated in the diagram by superscript symbols. The conventions that follow are illustrated in the accompanying prototype (Figure 3).

The population appears at the left-hand margin of the diagram and specifies the type(s) of patients to whom the evidence about the preventive service pertains. For example, the population description in an analytic framework for cervical cancer screening might be "females who are sexually active or over age 18." Footnotes in this location refer to specific questions about the population that the evidence review must answer in order to evaluate the effectiveness of the preventive service. For example, it may be necessary to know the proportion of the population with a family history of colon cancer.

Actions, such as screening and treatment, appear as arrows linking the population to an outcome or linking one outcome to another. A curved arrow leads to adverse effects of the action. The outcome to which the arrow points should result logically from the action (e.g., early detection of disease for screening, behavior change for counseling, reduced incidence of disease for immunizations or chemoprophylaxis). The name of the action (e.g., "screening with Prostate Specific Antigen [PSA]") appears below the arrow. Each arrow is a linkage in the logical chain of evidence that connects the left side (population) and the right side (health outcomes) of the analytic framework. Overarching linkages directly connect these two sides. Each arrow (linkage) is a "key question" that must be addressed by an evidence review.

Outcomes are the intermediate outcomes or the health outcomes resulting from actions or from previous outcomes. A health outcome that follows an intermediate outcome, which typically reflects the natural progression of disease (e.g., from "retinopathy" to "visual impairment") is depicted by a dotted line (no arrowhead). Health outcomes are depicted by rectangles with squared corners; intermediate outcomes are depicted by rectangles with rounded corners.

Annotated footnotes are specific key questions that are associated with each arrow and that must be answered by the literature review. The key questions are written in detailed narrative below the analytic framework figure.

3.2.1.6 Revisions

Analytic frameworks are evolving documents that can be influenced over time by the findings of the evidence review and by other considerations. For example, an overarching linkage might later be removed if the USPSTF found no relevant evidence. New linkages may be added when new interventions, outcomes, or logical arguments emerge during the course of the review.

3.2.2 Key Questions

Key questions are an integral part of the approach to conducting systematic reviews the USPSTF uses in its recommendation process. Along with the analytic framework, these questions specify the logic and scope of the topic, and are critical to guiding the literature searching, abstracting, and analysis process.

The purpose of the following definitions is to contrast and compare three types of questions (key questions, updating key questions, and contextual questions) that are used in formulating, conducting, and reporting systematic evidence reviews to update USPSTF recommendations. Updating key questions and contextual questions have been developed to address needs in the updating process, and their use is primarily in updated topic reviews. This function is reflected in their definitions.

3.2.2.1 Definitions

1. Key Questions: Key questions, in association with the analytic framework, establish the clinical logic to support implementation of a clinical preventive service in primary care. Key questions articulate the details about relevant patients, interventions, outcomes, and comparisons—details that are essential in order to focus the review on a manageable and clinically relevant topic, and which directly guide the systematic literature review process. Each question is clearly tied to a step in the analytic framework, although certain linkages that are already well established may not have a key question that is actively answered during the review for the USPSTF. In addition, there may be reason to focus on an overarching linkage (and the associated key question) in an analytic framework rather than the intermediate linkages (and their associated key questions). All key questions are reviewed and approved by the topic team in the process of assessing and refining the topic before the detailed literature review is conducted. Key questions addressed in a systematic review are listed in the methods section and used to organize the results section of the report. Key questions are addressed using up-to-date systematic review methods, under the current guidance and methods of the USPSTF. Each key question is addressed through a distinct literature search and reported separately in the results section of the review report.
   
   
2. Updating Key Questions: Updating key questions are used by the USPSTF to direct the researchers conducting the review to focus on a limited, specific set of key questions from the analytic framework during an updated review process. Updating key questions are used to focus the review for a topic update in two main ways: (1) encouraging the researcher to examine only critical gaps in the evidence that would be required by the USPSTF in order to make a recommendation; and (2) encouraging an evaluation of new research addressing parts of the topic that might confirm or change the evidence basis for a previous recommendation. An updating key question approach is taken in order to maximize efficiency in updating when, in the context of the previous review and recommendation, the USPSTF determines that a systematic review of only a limited part of the analytic framework is necessary in order to conduct a valid update of its evidence-based recommendation.

   Updating key questions are addressed in the same manner as key questions and reported in the results section of the review report.

3. Contextual Questions: These questions are not key questions associated with the analytic framework; however, they represent issues in an updated review for which the USPSTF needs a valid but not necessarily systematic summary of current research in order to provide the context for its vote and recommendation statement. Contextual questions may elicit a range of different types of information, including: (1) updated information for a key question that is not being systematically updated; (2) contextual information on natural history, current practice, prevalence and risk groups, or other aspects of the service for which it is strongly believed there will not be information, but which are part of the Task Force's considerations (e.g., screening interval, ages when screening should be stopped; or newer technologies for screening and/or intervention); or 3) cost-effectiveness.

   Contextual questions are not necessarily addressed systematically; however, the approach taken may meet criteria for a systematic review. Comprehensive literature searches are not generally undertaken specifically to answer these questions. Information for contextual questions is gathered in a variety of ways: (1) through targeted literature searches, (2) from authoritative surveys or reviews, (3) from expert input, and (4) opportunistically, while reviewing comprehensive literature searches for key questions. Contextual questions are not listed as separate questions in the methods section of the report and are not reported in the results section. The information resulting from non-systematic review should be included as part of the introduction or in the discussion section, and related as appropriate to the results of the systematic review.

3.3 Scan of Evidence/New Evidence

During the work assignment development for a topic update slated to be undertaken by the EPC, AHRQ conducts a preparatory literature scan to estimate the volume and content of intervening research relevant to this USPSTF topic. The scan focuses particularly on locating new research for critical key questions relating to benefits and harms of the preventive service. Literature searches are conducted in the Cochrane Library databases and in PubMed. Searches incorporate the MeSH terms from the previous evidence review (if available) and cover the time period since the last review (3 months prior to the end-date of the previous search). Searches are limited to the English language, to humans, and, in the case of PubMed® searches, to the journals in the abridged Index Medicus (120 "core clinical journals"). For benefits, the search is limited to meta-analyses, systematic reviews, and CCT/RCTs. For harms, the search includes meta-analyses, systematic reviews, RCTs, cohort, and case-control studies. The results of the searches are incorporated into the work assignment as a brief summary of the findings.

At the work plan development stage (after receiving the work assignment), the EPC librarian works with the topic team to develop a strategy for searching the literature to identify existing systematic reviews and other high quality synthesized literature (such as meta-analyses). This search is the first systematic search that will be incorporated into the overall searching done by the EPC for this topic. The purpose is to locate existing synthesized evidence that should be incorporated or built upon in the current systematic review, and the current methods emphasize finding all relevant synthesized evidence.

The following databases and Web sites are searched: Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, Health Technology Assessment Database (UK), National Institute for Health and Clinical Excellence—NICE (UK), Institute of Medicine, MEDLINE® via PubMed® (using the systematic reviews search hedge developed by NLM), and when appropriate, subject-specific databases (e.g., PsycINFO). Timeframes are not limited, except in the case of databases such as MEDLINE® and subject-specific databases, which are searched beginning a year prior to the most recent, relevant citation in the previous review.

In order to identify ongoing studies that could affect review scope and/or planning, the EPC librarian and/or topic team search ClinicalTrials.gov, and, if necessary, the National Institutes of Health CRISP (Computer Retrieval of Information on Scientific Projects) clinical trials data base to identify current trials and/or studies relevant to this review. (http://crisp.cit.nih.gov)

The topic team also checks to determine whether there is a finished, in-process, or planned Community Task Force review (from the CDC's Guide to Community Preventive Services) for the clinical preventive service being reviewed. The timing of this search (work plan stage or later) is left to the discretion of the topic work team.

3.4 Previous USPSTF Review and Recommendations (for an update)

To ensure that the current work plan builds coherently upon the previous USPSTF work on the topic, this section of the work plan succinctly summarizes the conceptual clinical framework and evidence foundation built by the previous reviews and recommendation statement on the topic. The current USPSTF graded recommendations are listed here verbatim, along with the analytic framework (AF), key questions (KQ), summary of evidence table, main findings, and conclusions from the previous review. Methodological or scope limitations and evidence gaps identified in the previous review are listed in bullet form.

3.5 Current USPSTF Review Approach

3.5.1 Inclusion/Exclusion Criteria (Admissible Evidence)

The topic team clearly documents the criteria by which it chooses to admit evidence on a given key question. Such criteria might include, for example, study design (randomized clinical trials [RCTs], cohort studies, and cost-effectiveness analyses [CEAs]), setting, sample size, population studied, language(s) of publication, and year(s) of publication.

No generic criteria for admissible evidence have been established. Rather, the criteria are determined on a topic-by-topic and key-question-by-key-question basis, depending on the questions and the amount and quality of evidence available. All inclusion/exclusion criteria are reviewed and approved by the entire topic team.

One variable in exclusion and inclusion criteria relates to the timeframe of the literature search. For an update of a topic previously covered by the Task Force, the topic team establishes whether the key questions they are posing had been similarly addressed in the previous review. If they were addressed, the team may evaluate key studies previously reviewed, but would not systematically re-review the same literature. An exception to this would be if the Task Force decided to evaluate the validity of this evidence by a method different from that used in the previous review. If a key question has changed, or if the threshold for adequate evidence has changed, the team searches back in time for evidence available before the previous review was published.

In addition, the topic team searches for other systematic reviews on the topic. If another systematic review is found that is rigorous and that addresses the same key question as the topic team, the team may choose to use the data generated in that review rather than redoing the work.

An important aspect of the definition of admissible evidence is the definition of "burden of suffering" of the condition in question. This burden is the ultimate target of implementing the preventive service. Admissible evidence is evidence relevant to the burden of suffering, including the prevalence of the condition in various populations and the impact of the condition on the health of these populations. The severity of the condition as measured by such metrics as prevalence and severity (e.g., number of life-years and quality-adjusted life years lost in a population) is also an important aspect of the burden of suffering. The burden of suffering of a condition defines the maximum possible benefit from prevention of that condition.

3.5.2 Literature Search Strategy

At this point the topic team proposes its full-scale search strategy. At a minimum, all topics include a review of the English-language literature in MEDLINE® and the Cochrane Database. Other databases (e.g., nursing and allied health or psychology databases) are searched as the topic team deems necessary. To identify sources of synthesized evidence and ongoing studies that could affect review planning, the EPC librarian works with the EPC review team to develop a strategy for searching the literature to identify existing systematic reviews and other high quality synthesized literature (such as meta-analyses). The following databases and Web sites are searched: Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, Health Technology Assessment Database (UK), National Institute for Health and Clinical Excellence - NICE (UK), Institute of Medicine, MEDLINE® via PubMed® (using the systematic reviews search hedge developed by NLM), and when appropriate, subject-specific databases (e.g., PsycINFO). Searches are not limited in terms of timeframes, except in the case of databases such as MEDLINE® and subject-specific databases, which are searched beginning a year prior to the most recent, relevant citation in the previous review.

The EPC librarian and/or topic team search ClinicalTrials.gov (and CRISP, when necessary) to identify current trials and/or studies relevant to this review. The review team also checks to determine whether there is a finished, in process, or planned Community Task Force review for the clinical preventive service being reviewed. The timing of this search (work plan stage or later) is left to the discretion of the topic work team.

3.5.3 Use of Topic Experts

By design, EPC review teams consist of generalist clinicians, methodologists, and staff with various levels of content expertise. Where appropriate, the EPC review team engages content experts as consultants to advise about work plan formulation and about operational decisions to be made during the conduct of the review. To allow continuity with the previous USPSTF review, the EPC review team includes previous review team members, where possible, as members of or as consultants to the current review team.

3.6 Peer-review of Work Plan

For full systematic reviews undertaken for new USPSTF topics, the work plan is peer-reviewed ("hardened") before it is finalized. From 4 to 6 peer reviewers are chosen to provide content expertise, topical research experience, and relevant methodological or policy expertise as appropriate to the topic. Peer reviewer lists are drafted by the EPC and amended and approved by the AHRQ medical officer. The EPC coordinates the peer-review process (by telephone interview or through written communication) and incorporates peer reviewers' suggestions into the draft work plan. Peer reviewers' comments are not formally summarized. Instead, peer-reviewed work plans, the list of peer reviewers, and a synopsis of their comments and the resulting revisions are presented at a regular USPSTF meeting for final input and approval by the USPSTF as a whole.

3.7 USPSTF Approval of Final Work Plan

The USPSTF provides final approval of the work plan through its assigned topic work group (for updates) or through a vote at its meetings (for new topics). Once the work plan is approved, the EPC finalizes it and submits it to the AHRQ medical officer as a deliverable.

Return to Contents
Proceed to Next Section