NAEP Scoring

			Geography Scoring Mathematics Scoring Reading Scoring Science Scoring U.S. History Scoring

NAEP assessments include multiple-choice items, which are machine-scored by optical mark reflex scanning, and constructed-response items, which are scored by trained scoring personnel using an image-based scoring system that routes student responses directly to each rater. Focused, explicit scoring guides are developed to match the criteria emphasized in the assessment frameworks. Rater consistency is monitored throughout the process through ongoing reliability checks and frequent backreading.

Throughout the scoring process, three types of personnel make up individual scoring teams:

• raters are professional scorers who are hired to rate the individual student responses; 
• supervisors lead teams of raters throughout the scoring process on a daily basis; and
• trainers provide training for the scoring raters, continually monitoring the progress of each scoring team.

Team members are required to have, at a minimum, a baccalaureate degree from a four-year college or university. An advanced degree, scoring experience, and/or teaching experience is preferred. Scoring teams use the training process to determine whether each individual scorer is sufficiently prepared to score. Following training, each scorer is given a pre-scored "qualification set" and expected to attain 80 percent correct in order to proceed.

All scoring is carried out via image processing. To assign a score, raters click the mouse over a button displayed in a scoring window. Since buttons are included only for valid scores, there is no editing for out-of-range scores. Two significant advantages of the image-scoring system are the ease of regulating the flow of work to raters and the ease of monitoring scoring. The image system provides scoring supervisors with tools to determine scorer qualification, to backread raters, to determine scorer calibration, to reset trend items, to monitor trend item scoring through t-statistics reports, to monitor interrater reliability, and to gauge the rate at which scoring was being completed.

The scoring supervisors monitor work flow for each item using a status tool that displays the number of responses scored, the number of responses first-scored that still need to be second-scored, the number of responses remaining to be first-scored, and the total number of responses remaining to be scored. This allows the scoring directors and project lead to monitor accurately the rate of scoring and to estimate the time needed for completion of the various phases of scoring.

Last updated 18 June 2008 (TS)

Printer-friendly Version