NAEP Item Development Process

The National Assessment of Educational Progress (NAEP) is based on content frameworks and test specifications developed by the National Assessment Governing Board. The National Center for Education Statistics (NCES) oversees the development of the assessment items and scoring rubrics. Both agencies employ extensive reviews by multiple groups at various stages. The Governing Board and NCES play crucial roles in ensuring NAEP’s quality and that it is developed through an extensive process.

Phase I. The Governing Board Framework and Test Specification Development

The Governing Board is responsible for developing the framework and test specifications that serve as the blueprint for the assessment. The Governing Board utilizes panels of business representatives, members of the general public, local and state policymakers, curriculum specialists from state and local education agencies, practitioners, noted researchers in the disciplines, and educators. Three committees (a steering committee for policy guidance, a planning committee for content expertise, and a technical advisory committee to deal with psychometric issues relevant to large-scale assessments) produce two documents:

• the content framework that defines the content and skills that should be covered; and,
• test specifications that describe how the assessment instrument should be constructed (i.e., percentage of items in each sub-domain, percentage of items by cognitive skill, test length, percentage of multiple choice and constructed response items).

These documents are reviewed in multiple public forums by educators and the general public. State curriculum specialists and testing directors also provide reviews. Both the Governing Board Assessment Development Committee and the full Board approve the framework and assessment specifications.

Phase II. Item Development

NCES calls on many constituents to guide the process and review the assessment. The NAEP contractor begins this process by developing more than twice the number of needed items. This item pool is then subjected to:

• an internal contractor review with content experts, teachers, experts on political sensitivity and on bias;
• pilot tests with small groups of students and targeted cognitive labs; and,
• under NCES guidance, refinement of items and scoring rubrics.

Then a standing committees of content experts, state and local education agency representatives, teachers, parents, and representatives of professional associations reviews the items. Each standing committee considers:

• the appropriateness of the items for the particular grade;
• the representative nature of the item set;
• the match of the items with the framework and test specifications; and
• the quality of items and scoring rubrics.

This is followed by a state item review where further feedback is provided. Items are then revised and submitted to the Governing Board assessment development committee for approval prior to field testing.

The field test is used to finalize the testing instrument. Items are revised based on student data, and the item set is once again subjected to review following the same procedure described above. Then, a final set of test items is assembled for NCES and the Governing Board review and approval.

Phase III. Assessment

After the main assessment, items are once again examined.  In the rare cases where item statistics indicate any remaining problems, an item is dropped from the assessment.  The remaining items are secured for reuse in future assessments, with a subset of items publicly released.

---

Last updated 26 January 2007 (MH)