The Status of Achievement Levels

The 2001 reauthorization law requires that the achievement levels be used on a trial basis until the Commissioner of Education Statistics determines that the achievement levels are "reasonable, valid, and informative to the public" (see the No Child Left Behind Act of 2001, P.L. 107-110, 115 Stat. 1425 [2002]). Until that determination is made, the law requires the Commissioner and the National Assessment Governing Board to state clearly the trial status of the achievement levels in all NAEP reports.

The most recent congressionally mandated evaluation conducted by the National Academy of Sciences (NAS) relied on prior studies of achievement levels, rather than carrying out new evaluations, on the grounds that the process has not changed substantially since the initial problems were identified. Instead, the NAS Panel studied the development of the 1996 science achievement levels. The NAS Panel basically concurred with earlier congressionally mandated studies. The Panel concluded that "NAEP's current achievement level setting procedures remain fundamentally flawed. The judgment tasks are difficult and confusing; raters’ judgments of different item types are internally inconsistent; appropriate validity evidence for the cut scores is lacking; and the process has produced unreasonable results."¹

A proven alternative to the current process has not yet been identified. NCES and the Board continue to call on the research community to assist in finding ways to improve standard setting for reporting NAEP results. The NAS Panel accepted the continuing use of achievement levels in reporting NAEP results on a developmental basis, until such time as better procedures can be developed. Specifically, the NAS Panel concluded that "...tracking changes in the percentages of students performing at or above those cut scores (or, in fact, any selected cut scores) can be of use in describing changes in student performance over time."² The National Assessment Governing Board urges all who are concerned about student performance levels to recognize that the use of these achievement levels is a developing process and is subject to various interpretations. The Board and NCES believe that the achievement levels are useful for reporting trends in the educational achievement of students in the United States. However, based on the congressionally mandated evaluations so far, NCES agrees with the National Academy's recommendation that caution needs to be exercised in the use of the current achievement levels. Therefore, NCES concludes that these achievement levels should continue to be used on a trial basis and should continue to be interpreted and used with caution.

---

Last updated 03 February 2005 (JM)