The Nation's Report Card:
Technical Report of the NAEP Mathematics Assessment in
Puerto Rico: Focus on Statistical Issues
|
 |
September 2007
Authors: G.P. Baxter, S. Ahmed, E. Sikali, T. Waits, M. Sloan, and S. Salvucci
 Download the complete report in a PDF file for viewing and printing. (671K PDF)
Executive Summary
In 2003, a trial NAEP mathematics assessment was administered in Spanish to public school students at grades 4 and 8 in Puerto Rico. Based on preliminary analyses of the 2003 data, changes were made in administration and translation procedures for the 2005 NAEP administration in Puerto Rico. This report describes the content and administration of the trial NAEP mathematics assessments in Puerto Rico in 2003 and 2005, problems with item misfit in the 2003 data, results of a special validity analysis, and plans to integrate Puerto Rico into the national sample in future administrations.
The 2003 trial NAEP mathematics assessment in Puerto Rico was administered in Spanish. Preliminary analysis of the 2003 Puerto Rico data raised concerns that the items were not functioning as they did in other jurisdictions. In Puerto Rico, there were larger amounts of missing data, fewer correct responses than expected for every content area, and a more frequent mismatch between expected and actual student performance on items (item misfit) compared to other jurisdictions.
To improve data quality, modifications to translation and administration procedures were made for the 2005 assessment in Puerto Rico. These changes included revision of the administration script, provision of an additional 10 minutes for each of the two timed sections of the assessment, and enhanced translation procedures. Analysis of the 2005 data showed fewer missing responses and a higher percentage of correct responses compared to 2003. Despite these improvements in data quality, there was concern about the validity of reporting Puerto Rico results on the NAEP scale.
To address this concern, a validity study was conducted using the 2003 and 2005 results in Puerto Rico. The analysis involved eliminating items that exhibited misfit in the Puerto Rico sample, thus yielding a restricted scale. Performance on the reduced set of items (restricted scale) was then compared to the corresponding performance on the full set of items (full scale) for the nation and Puerto Rico. The two scales agreed to within three-tenths of a scale point in the nation, but there was a 2- to 3-point difference between the scales in Puerto Rico. However, this difference does not change the ranking of Puerto Rico among other participating jurisdictions. These findings indicate that Puerto Rico results could be reported on the NAEP 0-500 scale.
All jurisdictions receiving federal Title 1 funds, including Puerto Rico, are required to participate in NAEP in fourth-and eighth-grade every other year beginning in 2003. The 2003 and 2005 NAEP administrations in Puerto Rico were considered trials and results were not reported with those of other jurisdictions. In future NAEP administrations, the intent is to include Puerto Rico as part of the national sample. For the 2007 administration, NCES increased the involvement of Puerto Rico educators in the development and translation review process of the NAEP mathematics assessment. Steps are in place to move Puerto Rico toward full integration into the NAEP sampling, data collection, and reporting for the 2009 administration.
Download the complete report in a PDF file for viewing and printing. (671K PDF)
NCES 2007-462rev Ordering information
Suggested Citation
Baxter, G.P., Ahmed, S., Sikali, E.,Waits, T., Sloan, M., and Salvucci, S. (2007). Technical Report of the NAEP Mathematics Assessment in Puerto Rico: Focus on Statistical Issues (NCES 2007-462rev). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, D.C.
For more information, see an overview of NAEP mathematics in Puerto Rico.
Last updated 19 December 2007 (RF)
|
