Appendix C – Clustering Correction of the Statistical Significance of Effects Estimated with Mismatched Analyses

In order to assess an intervention’s effects adequately, it is important to know not only the magnitude of the effects as indicated by the ES, but also the statistical significance of the effects. The correct statistical significance of findings, however, is not always readily available, particularly in studies in which the unit of assignment does not match the unit of analysis. The most common “mismatch” problem occurs when assignment was carried out at the cluster level (for example, classroom or school level), but the analysis was conducted at the student level, ignoring the dependence among students within the same clusters. Although the point estimates of the intervention’s effects based on such mismatched analyses are unbiased, the standard errors of the effect estimates are likely to be underestimated, which would lead to inflated Type I error and overestimated statistical significance.

In order to present a fair judgment about an intervention’s effects, the WWC computes clustering-corrected statistical significance for effects estimated from mismatched analyses and the corresponding domain average effects based on Hedges’s (2005) most recent work. As clustering correction will decrease the statistical significance (or increase the p-value) of the findings, nonsignificant findings from a mismatched analysis will remain nonsignificant after the correction. Therefore, the WWC applies the correction only to findings reported to be statistically significant by the study authors.

The basic approach to clustering correction is to first compute the t-statistic corresponding to the ES that ignores clustering and then to correct both the t-statistic and the associated degrees of freedom for clustering based on sample sizes, number of clusters, and the intra-class correlation (ICC). The statistic significance corrected for clustering could then be obtained from the t-distribution with the corrected t-statistic and degrees of freedom. In the remainder of this section, we detail each step of the process.

Compute the t-statistic for the ES ignoring clustering:

where g is the ES that ignores clustering, and n₁ and n₂ are the sample sizes for the intervention group and the comparison group, respectively, for a given outcome. For domain average ESs, n₁ and n₂ are the average sample sizes for the intervention and comparison groups, respectively, across all outcomes within the domain.

Correct the t-statistic for clustering:

where N is the total sample size at the student level (N = n₁ + n₂), m is the total number of clusters in the intervention and comparison groups (m = m1 + m2, m1 and m2 are the number of clusters in each of the two groups), and ? is the ICC for a given outcome.

The value of the ICC, however, is often not available from the study reports. Based on empirical literature in the field of education, the WWC has adopted a default ICC value of .20 for achievement outcomes and .10 for behavioral and attitudinal outcomes. The PIs and review teams may set different defaults with explicit justification in terms of the nature of the research circumstances or the outcome domain.

For domain average ESs, the ICC used earlier is the average ICC across all outcomes within the domain. If the number of clusters in the intervention and comparison groups differs across outcomes within a given domain, the total number of clusters (m) used for computing the corrected t-statistic will be based on the largest number of clusters in both groups across outcomes within the domain (that is, the largest m1 and m2 across outcomes). This gives the study the benefit of the doubt by crediting the measure with the most statistical power, so that the WWC’s rating of interventions will not be unduly conservative.

Compute the degrees of freedom associated with the t-statistics corrected for clustering:

Obtain the statistical significance of the effect corrected for clustering:

The clustering-corrected statistical significance (p-value) is determined based on the t-distribution with corrected t-statistic (t_A) and the corrected degrees of freedom (h). This p-value can either be looked up in a t-distribution table that can be found in the appendices of most statistical textbooks or computed using the t-distribution function in Excel: p = TDIST(t_A, h, 2).

Top