Identifying Estimates of Standard Errors With Large Mean Square Errors

As noted above, standard errors of average scale scores, proportions, and percentiles play an important role in interpreting student group results and in comparing the performances of two or more student groups. The jackknife standard errors reported by NAEP are statistics whose quality depends on certain features of the sample from which the estimate is obtained. In certain cases, the mean square error associated with the estimated standard errors may be quite large. This result typically occurred when the number of students upon which the standard error is based is small or when this group of students comes from a small number of participating primary sampling units (PSUs). The minimum PSU and student sample sizes that were imposed in most instances suppressed statistics where such problems existed. However, the possibility remained that some statistics based on sample sizes that exceed the minimum requirements had standard errors that were not well estimated. Therefore, in the reports, estimated standard errors for published statistics that are themselves subject to large mean square errors are followed by the symbol "!".

The magnitude of the mean square error associated with an estimated standard error for the mean or proportion of a group depends on the coefficient of variation (CV) of the estimated size of the population group, denoted as (Cochran 1977, Section 6.3). The coefficient of variation is estimated by

where is a point estimate of N and SE() is the jackknife standard error of N.

Experience with previous NAEP assessments suggests that when this coefficient exceeds 0.2, the mean square error of the estimated standard errors of means and proportions based on samples of this size may be quite large. (Further discussion of this issue can be found in Johnson and Rust 1992.) Therefore, the standard errors of means and proportions for all groups for which the coefficient of variation of the population size exceeds 0.2 are marked as described above. In the Report Cards and tables in the NAEP Data Explorer, statistical tests involving one or more quantities that have standard errors, confidence intervals, or significance tests so flagged should be interpreted with caution.

Last updated 25 February 2008 (TS)

Printer-friendly Version