Calibration of Scoring RatersFor new assessment items, the scoring supervisor of each team invokes calibration as needed from the tool used in backreading. During backreading, the scoring supervisor has a pool of 300 responses for each item to use in the calibration process. The scoring supervisor views samples of these responses together with the scores assigned by the first, and if applicable, second rater. From this pool, the scoring supervisor chooses some responses to put into the pool—responses that have been scored correctly and are a good measure to keep scoring on track. From this pool, the supervisor builds sets with the desired number of responses, usually between five and ten. These sets are then released on the image-based scoring system for raters to score. When raters invoke the calibration window, all raters receive the same responses and score them. After all raters have finished scoring this pool, the scoring supervisor can look at reliability reports, which include only the data from the calibration set just run. This process serves to refresh training and avoid drift in scoring. If pre-scored paper calibration sets already exist, these can be used to calibrate raters instead of the image-based sets created by the scoring supervisor. In general, each team scores calibration sets whenever they take a break longer than fifteen minutes, such as when returning from lunch or at the beginning of a shift. Raters scoring trend rescore items are calibrated using trend rescore responses that are already loaded into the system with the scores given during prior year scoring. Last updated 15 December 2007 (TS) |
