Data Analysis and Interpretation

Due to standard errors, caution is warranted when drawing conclusions about the size of one population estimate in comparison to another or about whether a time series of population estimates is increasing, decreasing, or staying about the same. Although one estimate may be larger than another, a statistical test may find that there is no measurable difference between the two estimates because of a large standard error associated with one or both of the estimates. Whether differences in means or percentages are statistically significant can be determined using the standard errors of the estimates.

Readers who wish to compare two sample estimates to see if there is a statistical difference will need to estimate the precision of the difference between the two sample estimates. This would be necessary if one wanted to compare, for example, the mean proficiency scores between groups or years in the National Assessment of Educational Progress. To estimate the precision of the difference between two sample estimates, one must find the standard error of the difference between the two sample estimates (sample estimate A, or E_A, and sample estimate B, or E_B). Expressed mathematically, the difference between the two estimates E_A and E_B is E_A-E_B.

The standard error of the difference (or se_A-B) can be calculated by taking the square root of the sum of the two standard errors associated with each of the two sample estimates ( se_A and se_B) after each has been squared. This can be expressed as

After finding the standard error of the difference, one divides the difference between the two sample estimates by this standard error to determine the “t value,” or “t statistic,” of the difference between the two estimates. This t statistic measures the precision of the difference between two independent sample estimates. The formula for calculating this ratio is expressed mathematically as

The next step is to compare this t statistic to 1.96, which is a statistically determined criterion level for making a decision as to whether there is a difference between the two estimates. If the t statistic is greater than 1.96, then there is evidence that there is a difference between the two populations. Note that one cannot say for certain that the two estimates are different, only that there is evidence that the difference in estimates is not due to sampling error alone. If the t statistic is equal to or less than 1.96, then one is less certain that the observed difference is not due to sampling error alone. This level of certitude, or significance, is known as the “.05 level of (statistical) significance.”

As an example of a comparison between two sample estimates to determine whether there is a statistically significant difference between the two, consider the data on the performance of 12th-grade students in the reading assessment of the 1992 and 2005 National Assessment of Educational Progress (see table 12-1). The average scale score in 1992 was 292, and the average scale score in 2005 was 286. Is the difference of 6 scale points between these two different samples statistically significant? The standard errors of these estimates are 0.6 and 0.6, respectively (see table S12-1). Using the formula above, the standard error of the difference is 0.85. The t statistic of the estimated difference of 6 scale points to the standard error of the difference is 7.07. This value is greater than 1.96—the critical value of the t distribution for a 5 percent level of significance with a large sample. Thus, one can conclude that there was a statistically significant difference in the performance of 12th-graders between 1992 and 2005 in reading and that the reading score for 12th-graders in 2005 was lower than the reading score for 12th-graders in 1992.

For all indicators reporting estimates based on samples in The Condition of Education, differences between estimates (including increases or decreases) are stated only when they are statistically significant. To determine whether differences reported are statistically significant, two-tailed t tests, at the 0.05 level, are typically used. The t test formula for determining statistical significance is adjusted when the samples being compared are dependent. When the difference between estimates is not statistically significant, tests of equivalence will often be run. An equivalence test determines the probability (generally at the .15 level) that the estimates are statistically equivalent; that is, within the margin of error that the two estimates are not substantively different. When the difference is found to be equivalent, language such as x and y “were similar” or “about the same” has been used; otherwise, the data will be described as having "no measurable difference".

When the variables to be tested are postulated to form a trend, the relationship may be tested using linear regression, logistic regression, or ANOVA trend analysis instead of a series of t tests. These other methods of analysis test for specific relationships (e.g., linear, quadratic, or cubic) among variables.

A number of considerations influence the ultimate selection of data years to feature in The Condition of Education. To make analyses as timely as possible, the latest year of data is shown if available during report production. The choice of comparison years is also based on the need to show the earliest available survey year, as in the case of the National Assessment of Educational Progress and the international assessment surveys. In the case of surveys with long time frames, such as for enrollment, the decade’s beginning year (e.g., 1980 or 1990) starts the trend line. Intervening years are selected in increments to show the general trend in the figures and tables. The narrative for the indicators typically compares the most current year’s data with those from the initial year and then with those from a more recent period. The narrative may also note years in which the data begin to diverge from previous trends where applicable.