Summary
In September 2003, the Head Start Bureau, in the Department of Health and Human Services (HHS) Administration for Children and Families (ACF), implemented the National Reporting System (NRS), the first nationwide skills test of over 400,000 4- and 5-year-old children. The NRS is intended to provide information on how well Head Start grantees are helping children progress. Given the importance of the NRS, this report examines: what information the NRS is designed to provide; how the Head Start Bureau has responded to concerns raised by grantees and experts during the first year of implementation; and whether the NRS provides the Head Start Bureau with quality information.
The Head Start Bureau developed the NRS to gauge the extent to which Head Start grantees help children progress in specific skill areas, including understanding spoken English, recognizing letters, vocabulary, and early math. Due to time constraints and technical matters, the Head Start Bureau adapted portions of other assessments for use in the NRS. Head Start Bureau officials have responded to some concerns raised during the first year of NRS implementation, but other issues remain. For example, the Head Start Bureau has modified training materials and is exploring the feasibility of sampling. However, it is not monitoring whether grantees are inappropriately changing instruction to emphasize areas covered in the NRS. Head Start Bureau officials have said NRS results will eventually be used for program improvement, targeting training and technical assistance, and program accountability; however, the Head Start Bureau has not stated how NRS results will be used to realize these purposes. Currently, results from the first year of the NRS are of limited value for accountability purposes because the Head Start Bureau has not shown that the NRS meets professional standards for such uses, namely that (1) the NRS provides reliable information on children's progress during the Head Start program year, especially for Spanish-speaking children, and (2) its results are valid measures of the learning that takes place. The NRS also may not provide sufficient information to target technical assistance to the Head Start centers and classrooms that need it most.
Recommendations
Our recommendations from this work are listed below with a Contact for more information. Status will change
from "In process" to "Implemented" or "Not implemented" based on our follow up work.
Director:
Team:
Phone:
Cornelia M. Ashby
Government Accountability Office: Education, Workforce, and Income Security
(202) 512-6778
Recommendations for Executive Action
Recommendation: To help ensure that the NRS successfully and efficiently achieves its purposes, the HHS Assistant Secretary for ACF should take steps to better monitor some aspects of NRS implementation and examine means of improving its efficiency, including monitoring the effects of the NRS on local Head Start instructional practices.
Agency Affected: Department of Health and Human Services
Status: Implemented
Comments: As part of the National Reporting System (NRS) Quality Assurance effort, an independent contractor is collecting information from a national sample of 36 Head Start programs on how well they are implementing the NRS and what changes they seem to be making in curriculum and instructional practices in response to NRS requirements. Observed changes in instructional activities are documented in the periodic reports of the Quality Assurance contractor, which are carefully reviewed by Administration for Children and Families staff. Feedback on how programs can use the NRS to stimulate constructive changes has been and will be provided to local programs through the memoranda that accompany each report on NRS results and in nationwide satellite television broadcasts that focus on NRS data collection.
Recommendation: To help ensure that the NRS successfully and efficiently achieves its purposes, the HHS Assistant Secretary for ACF should take steps to better monitor some aspects of NRS implementation and examine means of improving its efficiency, including improving the management and accuracy of its data on the number of children eligible for and participating in the NRS.
Agency Affected: Department of Health and Human Services
Status: Implemented
Comments: The Office of Head Start has requested that a National Reporting System contractor continue tracking major discrepancies between the number of children actually assessed and the number presumably eligible for assessment per the Program Information Report (PIR) and other sources of data.
Recommendation: To help ensure that the NRS successfully and efficiently achieves its purposes, the HHS Assistant Secretary for ACF should take steps to better monitor some aspects of NRS implementation and examine means of improving its efficiency, including working with the Technical Work Group to determine the feasibility of sampling options for administering the NRS, including documentation of their costs and benefits.
Agency Affected: Department of Health and Human Services
Status: Implemented
Comments: The HHS contractor that developed the NRS prepared information regarding the costs and benefits of sampling options for review by HHS' Advisory Committee on Head Start Accountability and Educational Performance Measures. The Advisory Committee completed its work in February 2007, including a recommendation that HHS integrate the National Reporting System (NRS) into local assessment approaches. Congress has subsequently passed bills reauthorizing Head Start (H.R. 1429 and S. 556) that would suspend the NRS as currently implemented and ask the National Academy of Sciences to form a committee addressing developmental outcomes and assessments for young children.
Recommendation: The Assistant Secretary for ACF should reduce uncertainty about the appropriate uses of the NRS by determining how the NRS data will be used for the purposes of accountability and targeting training and technical assistance, and clearly communicate this information to grantees.
Agency Affected: Department of Health and Human Services
Status: In process
Comments: In December 2007, the Head Start National Reporting System (NRS) was suspended by the Improving Head Start Act for School Readiness Act of 2007, Sec. 649(j)(4). Before that date, some action had been taken related to our recommendation. In February 2007, the Secretary's Advisory Committee on Head Start Accountability and Performance Measures completed its work and recommended, in part, that HHS make clear to Head Start teachers the link between NRS results and technical assistance goals and that HHS determine appropriate gains and outcome levels so programs could meet their intended goals. HHS expressed the intention to consider these recommendations as it continued to improve the utility of the National Reporting System. Separately, P.L. 109-149 set aside funds for establishing a panel of independent experts and directed the Office of Head Start to sponsor a study by the National Academy of Sciences (NAS) to review and provide guidance on appropriate outcomes and assessments for young children. In August 2008, NAS issued its guidance in "Early Childhood Assessment: Why, What, and How?"
Recommendation: The Assistant Secretary for ACF should reduce uncertainty about the appropriate uses of the NRS by using the first year of NRS results to conduct further study to ensure that the results are reliable and valid for both the English and Spanish versions and that the results are appropriate for the intended purposes.
Agency Affected: Department of Health and Human Services
Status: Implemented
Comments: An extensive series of analyses was conducted with the first year fall and spring data from the national implementation of the National Reporting System (NRS), as well as parallel data from the second year of national implementation. Data from the first year of implementation of the NRS suggest that the NRS measures have strong internal consistency reliability. Further, data from the original field test of the NRS suggest that these data also have strong inter-rater reliability and strong test-retest reliability. Taken together, these findings suggest that the NRS instruments are reliable. Similarly high reliability has been found for both the English and Spanish language versions of the NRS. These findings are currently being compiled in annual technical reports for the NRS that will be made available to the public upon completion. Further, a report on the predictive validity of the NRS, which is examined using FACES data, is currently being prepared. Preliminary results suggest that the NRS has strong predictive validity. Finally, a quality assurance study, which has been conducted every year since the implementation of the NRS, has shown high levels of fidelity in the implementation of the NRS direct child assessment. The findings of this report have been made available to grantees through the Computer-Based Reporting System.
Recommendation: The Assistant Secretary for ACF should reduce uncertainty about the appropriate uses of the NRS by compiling detailed technical information on the NRS, including appropriate uses, in a single, well-organized document and make this information publicly available.
Agency Affected: Department of Health and Human Services
Status: Implemented
Comments: By the end of FY06, the contractor had prepared a technical report for the government's review that detailed both the process used to create the NRS, as well as findings from the NRS field test. As of August 2007, HHS officials indicated that the technical manual prepared by HHS' contractor was still in clearance. In 2008, HHS reported that the manual is publicly available upon request. Separately, P.L. 109-149 set aside funds for establishing a panel of independent experts and directed the Office of Head Start to sponsor a study by the National Academy of Sciences (NAS) to review and provide guidance on appropriate outcomes and assessments for young children. In August 2008, NAS issued its guidance in "Early Childhood Assessment: Why, What, and How?"
