II. Identifying Topic Areas, Research, and Interventions

1. Identifying Review Areas
2. Scope of the Review
3. Literature Search
4. Eligibility Screening
5. Prioritizing Interventions for Review

Since research on education covers a wide range of topics, interventions, and outcomes, a clear protocol is used to set the parameters for locating, screening, and reviewing literature in a topic area according to WWC evidence standards. Senior WWC staff, along with the PI and the Deputy PI, develop the formal review area protocol to define the parameters for the interventions within the scope of the review, the literature search, and any area-specific applications of the evidence standards. Protocols are subject to IES approval.

A. Identifying Review Areas

The WWC seeks to review the effectiveness of interventions for a wide range of educational outcomes. Topics to be reviewed are prioritized based on their potential to improve important student outcomes; applicability to a broad range of students or to particularly important subpopulations; policy relevance and perceived demand within the education community; and likely availability of scientific studies about the effectiveness of specific, identifiable interventions.

The IES selects topics based on nominations received from the public, meetings and presentations sponsored by the WWC, suggestions presented by senior members of education associations, policymakers, and the U.S. Department of Education, and reviews of existing research. A list of current topics is available on the Topic Areas page.

Top

B. Scope of the Review

The protocol includes guidance regarding the following issues:

• Topic area focus. A very brief overview of the topic area, including the outcomes of interest and key questions to be addressed by the review.
   
• Key definitions. Definitions of terms and concepts that will be used frequently within a topic area, particularly the key outcomes on which the review will focus, along with the domains in which they will be classified.
   
• General inclusion criteria. Specification of the population, types of interventions, and types of research to be included in the review, including detail on timeframe, sample, study design, and outcomes.
   
• Specific topic parameters. Specification of which studies are to be considered for review and which aspects of those studies are to be examined. Considerations include characteristics of interventions, elements of intervention replicability, issues for outcome relevance and reliability, characteristics relevant to equating groups, effectiveness of the intervention across different groups and settings, preferences for measuring post-intervention effects, identification of differential and severe overall attrition, and statistical properties important for computing effect sizes.
   
• Literature search methodology. List of the requirements for searching literature, including databases to search, parameters and keywords for the searches, and any specific instructions regarding hand searches and exploration of the gray literature. Databases typically included in the literature search are ERIC, PsychINFO, Dissertation Abstracts, Sociological Collection, Professional Development Collection, Wilson Educational Abstracts PlusText, Academic Search Premier, WorldCat, and Google Scholar. Searching gray literature typically includes public submissions, materials sent directly to the WWC website or staff, requests for research made to developers of specific interventions, prior reviews and syntheses, requests for research made via listservs, and searches of organizational websites.

The PI is responsible for assuring the topic area protocol accurately reflects the work of the review team, and reflects a comprehensive review of the topic area. The protocol may be revised and updated as needed, although all revisions must be approved by IES.

Top

C. Literature Search

Identifying and reviewing literature begins after the topic area, review protocol, and Topic Area Team leadership are approved by the IES. Studies are gathered through an extensive search of published and unpublished research literature, including submissions from intervention developers, researchers, and the public. The WWC staff use the search parameters set by the protocol to search relevant databases and store all references in the reference-tracking software for the topic area.

Trained WWC staff members use the following strategies in collecting studies:

• Electronic databases. Identify keywords for each topic and search a variety of electronic databases for relevant studies.
   
• Website searches. Search the websites of core and topic-relevant organizations and collect potentially relevant studies.
   
• Extensive outreach. Contact topic experts and relevant organizations to request studies as well as to request recommendations of other people and organizations that are able to provide studies.
   
• Submissions. Incorporate studies submitted by the public.

Top

D. Eligibility Screening

In each area, the WWC collects published and unpublished studies that are potentially relevant to the topic. Gathered studies that meet broad relevancy and methodology criteria are then screened regarding the relevance of the intervention to the topic area, the relevance of the sample to the population of interest, the timeliness of the study, the relevance and validity of the outcome measure, and other criteria specified in the topic area protocol. Across topic areas, three general criteria apply:

• Was the study published in the relevant time range? Studies need to have been published within 20 years of the beginning of the topic area review. This time frame encompasses research that adequately represents the current status of the field and of analytical methods and avoids inclusion of research conducted with populations and in contexts that may be very different from those existing today.
   
• Is the study a primary analysis of the effect of an intervention? Some research studies identified in the literature search will not be primary studies of an intervention’s impacts or effectiveness, and cannot provide evidence of the effects of the intervention for the WWC review. For example, studies of how well the intervention was implemented, literature reviews, or meta-analyses are not eligible to be included in the review of an intervention.
   
• Does the study have an eligible design? The focus of the WWC is on scientifically-based evidence. Therefore, to be included in the WWC review, a study must use one of the following designs (described in the later section on evidence standards): randomized controlled trial, quasi-experimental, regression discontinuity, or single subject.

Across topic areas, specifics of studies to be included may vary. The screening for a topic area includes four criteria.

• Is the intervention a program, product, policy, or practice with the primary focus aligned with the topic area?
   
• Does the study examine students in the age or grade range specified for the topic area?
   
• Does the study examine students in a location specified for the topic area?
   
• Does the study address at least one student outcome in a relevant domain?

Studies that do not meet one or more of these criteria are categorized as “Does Not Meet Eligibility Screens,” indicating that they are out of the scope of the review as defined by the topic area protocol. At this stage, a study is screened out if it

• Does not examine the effectiveness of an intervention.
   
• Is not a primary analysis of the effectiveness of an intervention.
   
• Does not provide enough information about its design to assess whether it meets standards.
   
• Does not use a comparison group.
   
• Does not include a student outcome.
   
• Does not include an outcome within a domain specified in the protocol.
   
• Does not occur within the timeframe specified in the protocol.
   
• Does not examine an intervention conducted in English.
   
• Does not take place in the geographic area specified in the protocol.
   
• Does not use a sample within the age or grade range specified in the protocol.
   
• Does not disaggregate findings for the age or grade range specified in the protocol.
   
• Does not examine an intervention implemented in a way that falls within the scope of the review.

Top

E. Prioritizing Interventions for Review

After the initial literature screen is completed, studies are screened and ranked to prioritize interventions to review for the upcoming review year. Only studies that relate to the protocol of the topic area (those that include the correct age range, achievement outcome measured, and so on) are included in the ranking process. Using information in the title and the abstract or introduction, the coordinator ranks the study based on internal validity, objectivity, size, and differential contrast. Once all studies are screened, the coordinator organizes the information by intervention, and interventions are ranked by their scores. After a prioritization of interventions for review has been approved, the WWC Library staff work to identify additional studies by conducting targeted searches on the named interventions.

Upon approval of the intervention ranking by the IES, the Topic Area Team can begin contacting intervention developers—the person or company that researched and created the intervention. At this point, the PI sends a letter notifying the developer of the WWC review. The letter provides a list of all WWC-identified citations related to the intervention, inquires if the list is complete, invites comment on the intervention description slated for use in the report, and requests that the developer sign an agreement not to release any information about the review. If developers have questions about the report or review process, they are encouraged to contact the WWC in writing.

Top