NAEP Long Term Trend - Interpreting Long-Term Trend Results

Interpreting NAEP Long-Term Trend Results

Overview of the Assessment
Reporting the Assessment—Scale Scores and Performance Levels
Results Are Estimates
NAEP Reporting Groups
Statistical Significance
Exclusion Rates and Assessment Results
Cautions in Interpretations

Overview of the Assessment

One of the primary objectives of NAEP is to track trends in student performance over time. The NAEP long-term trend assessments in reading and mathematics were administered throughout the nation in the 2007–2008 school year to students aged 9, 13, and 17. Because the long-term trend program uses substantially the same assessments decade after decade, it has been possible to chart educational progress since 1971 in reading and 1973 in mathematics. Measuring trends in student achievement, or change over time, requires the precise replication of past procedures. Therefore, the long-term trend instruments do not evolve based on changes in curricula or in educational practices; in this way, the long-term trend assessments differ from main NAEP. It is not possible to compare results from national or state main NAEP with those of the long-term trend assessment.

Beginning with the 2004 administration of the long-term trend assessment in reading and mathematics, several changes were made to the assessment design. Any time changes are made in a long-term trend assessment, studies are required to ensure that the results can continue to be reported on the same trend line—that is, that they are validly comparable to earlier results. Analyses were needed to ensure that the 2004 results under the new design were comparable to the results from 1971 through 1999, under the design that existed earlier. Therefore, two assessments were conducted in 2004. The revised assessment used the new design, and the original assessment replicated the former design. Comparisons of the results could then detect any shifts in results due to changes in test design. The original assessment linked the old assessments to the new one.

NAEP assesses student performance by administering samples that are representative of the nation's students. The content of the NAEP long-term trend assessment is determined by a set of objectives incorporating expert perspectives about the measurement of reading and mathematics. Read how the long-term trend assessment was developed, who took the assessment, and how the assessment was administered. Read more about what the long-term trend reading and long-term trend mathematics assessments measure.

The results presented on the website are based on representative samples of students for the nation. In 2008, approximately 27,700 students took the reading assessment, while about 27,500 took the mathematics assessment. Although students attending both public and private schools were sampled, response rates for reporting data separately for private schools were met at ages 9 and 13, but not at age 17. The participation rate threshold was met by Catholic schools at all three ages; therefore, separate results for Catholic schools are included in this report.

Reporting the Assessment—Scale Scores and Performance Levels

The results of student performance on the long-term trend assessment are presented on this website in three ways: as average scores, scores at selected percentiles, and percentages of students performing at or above each performance level. Student performance in each subject area is summarized as an average score on a 0-to-500 scale. For each year in which the assessments were administered, achievement in a particular subject area is described for a group of students by their average scale score. Trends in student achievement are determined by examining the average scale scores attained by students in the current assessment year and comparing them to the average scale scores in other assessment years. While the score ranges in both subjects are identical, the scale was derived independently for each subject. Therefore average scale scores between subjects cannot be compared.

Student performance is also described in terms of the percentages of students attaining specific levels of performance. These performance levels correspond to five points on the reading and mathematics scales: 150, 200, 250, 300, and 350. For each subject area, the performance levels from lowest to highest are associated with increasingly advanced skills and knowledge. Examining the percentages of students in each year who attained each performance level provides additional insights into student achievement. Read more about the long-term trend performance levels.

Results are Estimates

The average scores and percentages presented on this website are estimates because they are based on representative samples of students rather than on the entire population of students. Moreover, the collection of subject-area questions used at each grade level is but a sample of the many questions that could have been asked. As such, NAEP results are subject to a measure of uncertainty, reflected in the standard error of the estimates. The standard errors for the estimated scale scores and percentages in the figures and tables presented on this website are available in the NAEP Data Explorer.

NAEP Reporting Groups

Results are provided for groups of students defined by shared characteristics—gender, race or ethnicity, parents' education level, and type of school. Based on participation rate criteria, results are reported for groups of students only when adequate school representation and sufficient numbers of students are present. The minimum requirement is at least 62 students in a particular student group from at least five primary sampling units (PSUs). However, the data for all students, regardless of whether their group was reported separately, were included in computing overall results. Explanations of the reporting groups are presented below.

Gender

Results are reported separately for males and females. Gender was reported by the school.

Race/Ethnicity

Results are presented for students in different racial/ethnic groups according to the following mutually exclusive categories: White, Black, and Hispanic. Results for Asian/Pacific Islander and American Indian (including Alaska Native) students are not reported separately because there were too few students in the groups for statistical reliability. The data for all students, regardless of whether their racial/ethnic group was reported separately, were included in computing the overall national results.

In NAEP long-term trend assessments, data about student race/ethnicity have been collected in three ways: through observation, school records, and student self-reports. The 2004 original assessment and all previous assessment results were based on observed race/ethnicity, while the 2004 revised assessment results and the 2008 results were based on school-reported race/ethnicity.

Type of School

Results are reported by the type of school that student attends—public or private. Private schools include Catholic schools and other private schools.

Parents' Education Level

Students were asked to indicate the extent of schooling for each of their parents—did not finish high school, graduated from high school, had some education after high school, or graduated from college. The response indicating the highest level of education for either parent was selected for reporting. The questions were presented only to students at ages 13 and 17. (Results for parental education are not reported at age 9 because research has shown that students' reports of their parents' education level are less reliable at this age.) Although students in previous long-term trend assessments were asked about their parents' level of of education, the wording of the question in the 2004 revised and 2008 reading trend assessment booklets was different from previous years. Consequently, trend results are reported only for the mathematics assessment.

Exclusion Rates and Assessment Results

Some students selected for participation in the NAEP long-term trend assessments were identified as English language learners (ELL) or students with disabilities (SD). In previous long-term trend assessments, if it was decided that a student classified as SD or ELL could not meaningfully participate in the NAEP assessment for which he or she was selected, the student was, according to NAEP guidelines, excluded from the assessment. See tables that summarize the percentage of students identified, excluded, and assessed in long-term trend.

For each student selected to participate in NAEP who was identified as either SD or ELL, a member of the school staff most knowledgeable about the student completed an SD/ELL questionnaire. Students with disabilities were excluded from the assessment if an IEP (individualized education program) team or equivalent group determined that the student could not participate in assessments such as NAEP; if the student's cognitive functioning was so severely impaired that he or she could not participate; or if the student's IEP required that the student be tested with an accommodation or adaptation not permitted or available in NAEP, and the student could not demonstrate his or her knowledge of the assessment subject area without that accommodation or adaptation.

A student who was identified as ELL and who was a native speaker of a language other than English was excluded if the student received instruction in the assessment subject area (e.g., reading or mathematics) primarily in English for less than three school years, including the current year, or if the student could not demonstrate his or her knowledge of reading or mathematics in English without an accommodation or adaptation.

In recent years, changes in policy and legislation pertaining to civil rights have resulted in assessment accommodations being permitted so that SD and ELL students selected to participate in NAEP can do so under revised conditions. Beginning with the 2004 revised trend assessments, NAEP long-term trend assessments offered accommodations for these students. For consistency with the long-term trend assessments administered in past years, however, accommodations were not offered to students in the 2004 original assessment samples.

Statistical Significance

The differences between scale scores and between percentages discussed in the results on this website take into account the standard errors associated with the estimates. Comparisons are based on statistical tests that consider both the magnitude of the difference between the group average scores or percentages and the standard errors of those statistics. Throughout the results, differences between scores or between percentages are discussed only when they are significant from a statistical perspective.

All differences reported are significant at the 0.05 level with appropriate adjustments for multiple comparisons. The term "significant" is not intended to imply a judgment about the absolute magnitude or the educational relevance of the differences. It is intended to identify statistically dependable population differences to help inform dialogue among policymakers, educators, and the public.

Cautions in Interpretations

Users of this website are cautioned against interpreting NAEP results as implying causal relationships. Inferences related to student group performance or to the effectiveness of particular classroom practices, for example, should take into consideration the many socioeconomic and educational factors that may also impact performance.

The NAEP long-term trend scales make it possible to examine relationships between students' performance and various background factors measured by NAEP. However, a relationship that exists between achievement and another variable does not reveal its underlying cause, which may be influenced by a number of other variables. Similarly, the assessments do not reflect the influence of unmeasured variables. The results are most useful when they are considered in combination with other knowledge about the student population and the educational system, such as trends in instruction, changes in the school-age population, and societal demands and expectations.

Comparisons of the 2008 results to the 2004 original or previous trend results should be interpreted with caution, bearing in mind the differences in assessment accommodations and changes to assessment procedures.