Scoring NAEP Geography Assessments
The NAEP geography items that are not scored by machine are constructed-response items—those for which the student must write in a response rather than selecting from a printed list of multiple choices. Each constructed-response item has a unique scoring guide that identifies the range of possible scores for the item. To measure longitudinal trends in geography, NAEP requires trend scoring—replication of scoring from prior assessment years—to demonstrate statistically that scoring was comparable across years. Students' constructed responses are scored on computer workstations using an image-based scoring system. This allows for item-by-item scoring and online, real-time monitoring of geography interrater reliabilities, as well as the performance of each individual rater. In the 2001 assessment, some of these items—those that appeared in large-print booklets—required scoring by hand. The 2001 geography assessment included 57 discrete constructed-response items. The total number of constructed responses scored was 381,477. The number of raters working on the geography assessment and the location of the scoring are listed here:
Each constructed-response item has a unique scoring guide that identifies the range of possible scores for the item and defines the criteria to be used in evaluating student responses. During the course of the project, each team scores the items using a 2-, 3-, or 4-point scale as outlined below: Dichotomous Items Short Three-Point Items Extended Four-Point Items In some cases student responses do not fit into any of the categories listed on the scoring guide. Special coding categories for the unscorable responses are assigned to these types of responses. These unscorable categories are only assigned if no aspect of the student's response can be scored. Scoring supervisors and/or trainers are consulted prior to the assignment of any special coding category to an item. The unscorable categories used for geography are outlined as follows.
Last updated 15 April 2008 (TS) |
|||||||||||||||||||||||||||||||||||||||
