|
|
Methodology
Eighty-four studies on 22 dropout prevention interventions were classified for the strength of their design. To be fully reviewed, a study had to be a randomized controlled trial or a quasi-experimental design with evidence of equating between the treatment and comparison groups.
Eligibility screens and evidence standards
Quasi-experiments eligible for review include those equating through matching or statistical adjustment, regression discontinuity designs, and single case designs. No studies based on the latter two types of designs were identified for the dropout prevention review. The WWC is currently developing evidence standards for regression discontinuity designs and single case designs.
The review considered the properties of measurement instruments, the percentage of students, classrooms, or schools in the study sample that were not included in the reported results, and any sample characteristics or events that might serve as alternative explanations for the observed effect. For details please see the WWC Evidence Standards. Long-term outcomes were preferred over short-term outcomes in the WWC’s analysis of intervention effects.
The research evidence for interventions that have at least one study meeting WWC evidence standards with or without reservations is summarized in individual intervention reports posted on the WWC website. See http://ies.ed.gov/ncee/wwc/. So far, 23 studies of 16 dropout prevention interventions meet evidence standards with or without reservations. The lack of evidence for the remaining interventions does not mean that those interventions are ineffective; some interventions have not yet been studied using a study design that permits the WWC to draw any conclusions about their effectiveness. And for some studies, not enough data were reported (such as descriptive statistics of the findings) to enable the WWC to confirm statistical findings.
Rating of effectiveness
Each dropout prevention intervention that had at least one study meeting WWC standards with or without reservations received a rating of effectiveness in at least one outcome domain. The rating of effectiveness aims to characterize the existing evidence base in a given domain. The intervention effects based on the research evidence can be rated as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.
The rating of effectiveness takes into account four factors: the quality of the research design, the statistical significance of the findings, the size of the difference between participants in the intervention and the comparison conditions, and the consistency in findings across studies (see the WWC Intervention Rating Scheme).
The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. Because of these corrections, the level of statistical significance as calculated by the WWC may differ from the one originally reported by the study authors. For an explanation, see the WWC Tutorial on Mismatch. For the formulas that the WWC used to calculate statistical significance, see Technical Details of WWC-Conducted Computations. If the average effect size across all outcomes in one study in a single domain is at least 0.25, it is considered substantively important, contributing toward the rating of effectiveness. See the technical appendices of the dropout prevention intervention reports for further details.
Extent of evidence
The evidence base rating represents the size and number of independent samples that were assessed for the purposes of analysis of the intervention effects. A “medium to large” evidence base requires at least two studies and two schools across studies within one domain, and a total sample size across studies of at least 350 students or 14 classrooms. Otherwise, the evidence is considered to be “small.” The WWC is currently working to define a “large” evidence base. This term should not be confused with external validity, as other facets of external validity—such as variations in settings, important subgroups of students, implementation, and outcomes measures—were not taken into account for the purposes of this rating.
Improvement index
The WWC computes an improvement index for each individual finding. In addition, within each outcome domain, the WWC computes an average improvement index for each domain and each study and a domain average improvement index across studies of the same intervention (see the Technical Details of WWC-Conducted Computations). The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the percentile rank of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting results favorable to the intervention group. Unlike the rating of effectiveness, the improvement index is based only on the size of the difference between the intervention and the comparison conditions.
