Examining the Population-Structure Models Used in NAEP

			Proportion of Variance of the Group-Defining Contrasts Accounted for by the Principal Components Used in NAEP Student Group Distribution Models Proportion of Scale Score Variance Accounted for by the Population-Structure Models Comparison of Mean Scale Scores and Mean Item Scores by Jurisdiction

The population-structure models are examined in three ways. First, for national main, state, and combined national and state assessments, the proportion of variance accounted for by the principal components is provided for each contrast of the group-defining variables. When these values are close to one, most of the variability in the group-defining variable is accounted for by the principal components used in the population-structure model, and the population-structure model reflects membership in the group well.

Second, the proportion of scale score variance accounted for by the population-structure models is provided for each sample. These values report the proportion of variance among scale scores explained by group membership for each sample. When these values are larger than zero, the population-structure model is needed to estimate summary statistics for the population and subgroups represented by the sample.

Finally, for state assessments, the mean scale score for each jurisdiction is plotted against the mean item score for each jurisdiction in a plot. While the correlation between the mean scale score and mean item score for jurisdictions is not expected to be one, large differences in the ordering of the jurisdictions on these two measures might indicate that the use of Item Response Theory (IRT) models and/or population-structure models is problematic.

Last updated 09 July 2008 (FW)

Printer-friendly Version