• Introduction
• Technical Characteristics of Results
• Students With Disabilities
• Students With Limited English Proficiency
• References

• Share studies and research that are developmental in nature.

• Share results of studies that are on the cutting edge of methodological developments.

• Participate in discussions of emerging issues of interest to researchers.

These reports present results or discussion that do not reach definitive conclusions at this point in time, either because the data are tentative, the methodology is new and developing, or the topic is one on which there are divergent views. Therefore, the techniques and inferences made from the data are tentative and are subject to revision.

This study grew out of concerns about the under- representation of students with special needs in the National Assessment of Educational Progress (NAEP) assessments. The term "special needs students" is sometimes used to include both students with disabilities and students who are limited English proficient (LEP).In the 1996 NAEP assessment samples, 10 percent of 4th-graders, 9 percent of 8th-graders, and 5 percent of 12th-graders were identified by their schools as students with disabilities. In the same assessment year, 4 percent of 4th-graders and 2 percent of 8th- and 12th-graders were identified by their schools as students with limited English proficiency (Reese et al. 1997, 67). Schools participating in NAEP have been permitted to exclude individuals they identify as special needs students from the assessment, in accordance with criteria provided by the program at that time. In fact, at least half of all special needs students in the NAEP samples were excluded from NAEP assessments in 1992 and 1994. This exclusion has raised concerns that some special needs students who could be meaningfully assessed are being excluded from NAEP. Moreover, there is an additional concern that variations across locales in exclusion practices may introduce biases in NAEP results.

In recent years, a number of policy, legislative, civil rights, and technical considerations have caused the NAEP program to look more closely at its administration and assessment procedures and to consider changes that can increase participation among students with disabilities and LEP students. Based on previous studies (National Academy of Education 1993) as well as recommendations from various offices in the U.S. Department of Education, program procedures have been modified with the aim of increasing participation among special needs students. Modifications were made in two areas (Olson and Goldstein 1996). First, inclusion criteria for the NAEP 1996 assessment were revised with the intention of making them clearer, more inclusive, and more likely to be applied consistently across jurisdictions participating in the state assessment program. Second, for the first time in NAEP, a variety of assessment accommodations were offered to (1) students with disabilities whose Individualized Education Plan (IEP) specified such accommodations for testing; and (2) LEP students who, in the opinion of their instructors, required an accommodation in order to take the assessment in English.

This report presents in-depth analyses of the effects on inclusion rates of the above efforts to increase the participation of special needs students in NAEP. It also contains an analysis of selected technical characteristics of the assessment results and a review of descriptive results of the background characteristics and educational experiences of students with disabilities and LEP students who participated in the NAEP 1996 national assessments in mathematics and science.

In particular, data are presented on

• the possible effect on the NAEP proficiency scales of including greater percentages of special needs students;

• the comparability of results from nonstandard administrations (i.e., administrations in which accommodations were allowed) to results obtained under standard conditions; and

• the effect of nonstandard administrations on NAEP's capacity to provide accurate comparisons of trends over time.

In addition, it is important to be clear on what this report does not contain:

• This report does not provide an in-depth examination of the performance on NAEP of students with disabilities and LEP students.

• The relatively small sample sizes obtained in the study did not allow disaggregation of students with disabilities and LEP students in many of the statistical analyses that dealt with the effects on NAEP scales.

• This report does not separate students with disabilities from LEP students in the Differential Item Functioning (DIF) analyses.

• This report does not look at performance results or inclusion rates for students with disabilities and LEP students by state.

An experiment was designed for the 1996 assessments in mathematics and science, which permitted analysis of data relevant to the issues above. In addition, a questionnaire was included that was designed to obtain information on student background and educational experiences. The questionnaire was to be completed for all sampled students with disabilities and for all sampled LEP students.

The design of the NAEP 1996 assessment included three distinct national samples of schools. In the first of these school samples (denoted S1), the assessment was conducted using the same inclusion criteria used during the 1990 and 1992 NAEP assessments in mathematics and science. In the second school sample (denoted S2), revised inclusion criteria were used. No assessment accommodations or adaptations were offered to students in S1 or S2 schools. In the third sample (denoted S3), the assessment was conducted using inclusion criteria that were identical to those used in S2 schools. The S3 sample was distinguished, however, by the availability of a variety of assessment accommodations and adaptations. To ensure sufficient amounts of data for planned analyses, students with disabilities and LEP students were oversampled in national S2 and S3 schools, and all students in S3 who received an accommodation at a given grade were administered the same NAEP assessment booklet.

The findings of the current research on technical characteristics of the assessment results, based on the combined data from all special needs students, include the following:

• For two of the three grades in science, there is some evidence to suggest that test results obtained using accommodations and adaptations cannot be fit with the same Item Response Theory (IRT)^* model as results obtained under standard administration conditions. The evidence for the mathematics assessment was less conclusive. Because small sample sizes necessitated the combination of students with disabilities and LEP students for IRT and scaling analyses, it is not yet clear whether future NAEP reports will need to report these categories of students separately. A future report using larger samples (combined state data) from the 1998 reading assessment should shed further light on this question.

• Despite the finding above, the inclusion of data from nonstandard administrations had no discernable effect on aggregate NAEP scaling results in mathematics and science at any of the three grades. Differences in test characteristic curves and test information curves plotted with and without the inclusion of such data differed no more than would be expected due to sampling variability.

• Proficiency means were estimated for the NAEP mathematics and science scales, with and without the inclusion of students with accommodations in testing, at each of grades 4, 8, and 12. There were no significant differences in the overall means or in the means for significant subgroups at any of the three grades.

• The results reported here suggest that the procedural changes being considered would not significantly affect the NAEP scale score results. If so, it may be possible for the NAEP program to achieve its joint goals of increasing inclusion while maintaining trend lines. However, additional research is necessary to determine the generality of these findings across content areas and over time, as state policies and procedures with respect to inclusion evolve.

Data from background questionnaires did allow separate analyses for students with disabilities and LEP students pertaining to background characteristics, educational experiences, and inclusion rates. Major findings for these analyses are summarized below.

• Learning disability was by far the most frequently reported category for students with disabilities, with close to three of four students so identified at each of the three grades.

• About half of the students at each grade were described as having mild disabilities. The remaining half at each grade were almost all categorized with moderate to severe disabilities. Very few students receiving special education services at schools participating in NAEP (1 percent at grades 4 and 8, and 3 percent at grade 12) were judged to have profound disabilities.

• Regardless of grade level, about half of all students with disabilities were mainstreamed in academic subjects at least 80 percent of the time.

• In reading/language arts, half or fewer of the students with disabilities received instruction that was at grade level. In mathematics and science, the situation was slightly better at the two lower grades. More than half of the grade-4 and grade-8 students with disabilities received grade-level instruction, and over 70 percent of these students received grade-level instruction in science.

• Almost all students who received instruction that was at or above grade level received the same curriculum content as their nondisabled peers. In contrast, fewer than half of those students with disabilities who received below grade-level instruction were taught the same curriculum content as their nondisabled peers.

• In all three grades, more than 75 percent of students with disabilities were judged by school personnel to be performing below grade level in reading/language arts. Reported performance levels in mathematics and science were somewhat higher than those in reading/language arts at grade 4.

• Across the three grades, respondents reported that 42 to 44 percent of students with disabilities received some form of accommodation or adaptation in testing.

• Comparisons of questionnaire results with actual participation rates from the 1996 mathematics assessment suggest that (1) increases in the percentages of students with disabilities participating in NAEP are not likely to result solely from revisions to inclusion criteria; and (2) a further expansion of accommodations or adaptations permitted by NAEP, or a change in NAEP guidelines as to who is eligible for special testing conditions, could result in further small increases in inclusion percentages.

• Most exclusion decisions were made on the basis of what is stated in the IEP, and relatively few exclusion decisions were made on the basis of a judgment of severe cognitive impairment, absent corroborating direction from the IEP. However, results also suggest that, for substantial percentages of excluded students, neither a determination by the IEP team nor the presence of cognitive impairments was given as a reason for exclusion.

• Some students who do not regularly receive accommodations or adaptations were offered them in NAEP and others who should not have been tested were, in fact, included. These results suggest that incorrect decisions regarding inclusion or testing conditions may have been made or that incorrect questionnaire data may have been provided.

• The largest proportion of LEP students spoke Spanish as their native language (74 percent at grade 4, 72 percent at grade 8, and 54 percent at grade 12). The most frequently encountered other languages were Vietnamese, Hmong, Chinese, Russian, and Pacific Island languages.

• Forty-four percent of grade-4 LEP students, 47 percent of grade-8 LEP students, and 65 percent of grade-12 LEP students had received academic instruction primarily in English for 3 or more years.

• At grades 8 and 12, few students received native-language instruction in academic areas. At grade 4, the percentages of LEP students who received native- language instruction in reading/language arts, mathematics, and science were 22, 27, and 26 percent, respectively.

• Among LEP students receiving English-language instruction, the majority received instruction at grade level at all three grades.

• The vast majority of LEP students at all three grades (87 percent of grade-4 LEP students, 80 percent of grade-8 LEP students, and 81 percent of grade-12 LEP students) received some special academic instruction in English or in their native language. At grades 8 and 12, such special instruction appears to have been predominantly in English.

• Although most LEP students were receiving English-language instruction at grade level, a significant percentage were judged to be performing below grade level in English. In reading/language arts, where one might expect the impact of limited language proficiency to be most pronounced, 70 percent of grade-4 and 62 percent of grade-8 LEP students were judged by school personnel as performing below grade level in English; at grade 12, 50 percent were so judged. In science, the percentages reported performing below grade level ranged from 30 percent at grade 12 to 44 percent at grade 8. In mathematics, the percentages ranged from 33 percent at grade 12 to 46 percent at grade 8.

• Respondents indicated that 37 percent of grade-4 LEP students, 27 percent of grade-8 LEP students, and 22 percent of grade-12 LEP students used accommodations and adaptations in achievement testing in their schools.

• The operational criteria used in NAEP from 1990 to 1996 indicated that LEP students enrolled in schools where English is the primary language of instruction for 2 or more years were to be included in the assessment. At least 85 percent of 4th-grade LEP students, 67 percent of 8th-grade LEP students, and 83 percent of 12th-grade LEP students had been enrolled for 2 or more years in schools where English was the primary language. Historically, NAEP inclusion rates for LEP students have been below the ideal minimums suggested by these questionnaire results.

• As was the case for students with disabilities, comparisons of questionnaire results with assessment inclusion rates for LEP students suggest that (1) increases in the percentage of LEP students are not likely to result solely from revisions to inclusion criteria that do not also involve the provision of accommodations; and (2) further modest improvements in inclusion might still be possible if the list of permitted accommodations and adaptations can be expanded.

• Analyses of inclusion rates by the length of time students were enrolled in schools where English is the primary language of instruction provided some evidence that, when implemented without the provision of accommodations and adaptations, the revised criteria actually resulted in less inclusion among LEP students than did the original criteria. This evidence was strongest at grade 4.

• Under the revised criteria, all LEP students receiving academic instruction in English for 3 or more years were to be included in NAEP. Analyses based on questionnaire responses as to the number of years students were receiving academic instruction in English indicated that this ideal was not quite achieved. Inclusion rates among students with 3 or more years of academic instruction in English were high, but total inclusion was not achieved, even where accommodations and adaptations were provided.

• Some LEP students who do not usually receive accommodations in testing were apparently provided accommodations in the NAEP assessment. The percentages of LEP students in this category were small (10, 6, and 5 percent in grades 4, 8, and 12, respectively).

• Questionnaire results suggest that the procedural modifications made to NAEP had their primary impact on inclusion rates at grades 4 and 8 among students who would be tested in their native language if this accommodation were available. Participation rates for these students were higher when accommodations were available.