Appendix G – Extent of Evidence Categorization

The Extent of Evidence Categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the number and sizes of studies. This scheme has two categories: small and medium to large.

The extent of evidence is medium to large if all of the following are true:

• The domain includes more than one study.
   
• The domain includes more than one school.
   
• The domain findings are based on a total sample size of at least 350 students OR, assuming 25 students in a class, a total of at least 14 classrooms across studies.
   

The extent of evidence is small if any of the following are true:

• The domain includes only one study.
   
• The domain includes only one school.
   
• The domain findings are based on a total sample size of less than 350 students AND, assuming 25 students in a class, a total of less than 14 classrooms across studies.

Each intervention domain receives its own categorization. For example, each of the three domains in character education—behavior; knowledge, attitudes, and values; and academic achievement—receives a separate categorization.

Example:

Intervention Do Good, a character education intervention, had three studies that met WWC standards and were included in the review. All three studies reported on academic achievement. There were a total of six schools across the three studies. The first study reported testing on 150 students, the second study 125 students, and the third study reported testing four classes with 15 students in each class. The extent of evidence on academic achievement for the Do Good intervention is considered “medium to large”—it met the condition for both the number of studies and the number of schools, and although the total number of students is less than 350 (150 + 125 + [4*15] =335), the number of classes exceeded 14 (150/25 + 125/25 + 4 = 15).

A “small” extent of evidence indicates that the amount of the evidence is low. There is currently no consensus in the field on what constitutes a “large” or “small” study or database. Therefore, the WWC set the indicated conditions based on the following rationale:

• With only one study, the possibility exists that some characteristics of the study—for example, the outcome instruments or the timing of the intervention—might have affected the findings. Multiple studies provide some assurance that the effects can be attributed to the intervention and not to some features of the particular place where the intervention was studied. Therefore, the WWC determined that the extent of evidence is small when the findings are based on only one setting.
   
• Similarly, with only one school, the possibility exists that some characteristics of the school—for example, the principal or student demographics—might have affected the findings or were intertwined or confounded with the findings. Therefore, the WWC determined that the extent of evidence is small when the findings are based on only a single school.
   
• The sample size of 350 was derived from the following assumptions:

– A balanced sampling design that randomizes at the student level

– A minimum detectable effect size of 0.3

– The power of the test at 0.8

– A two-tailed test with an alpha of 0.05

– The outcome was not adjusted by an appropriate pretest covariate.

The Extent of Evidence Categorization provided in recent reports, and described here, signals WWC's intent to provide at some point a rating scheme on the external validity, or the generalizability, of the findings, for which the extent of evidence is only one of the dimensions. The Extent of Evidence Categorization, in its current form, is not a rating on external validity; instead, it serves as an indicator that cautions readers when findings are drawn from studies with small size samples, a small number of school settings, or a single study.

Top