One challenge in comparing assessment data from countries around the world is determining the extent that variations in the characteristics of student and adult populations relate to achievement scores. For example, restrictions in attrition rates as students move through the educational system, the economic and social status of students and their families, and parental levels of education may each affect the comparability of findings both within and across assessments. In developing international assessments, the challenge of making student populations comparable is generally dealt with in two ways.

First, countries that participate in international assessments such as TIMSS, PIRLS, ALL, and PISA are required to select national probability samples from all students or adults in a particular grade or of a particular age. Exclusions are strictly limited, must be clearly documented, and are reported along with participation rates at each level of sampling. Countries with exclusion rates that are above established levels or with samples that are not representative of the population being assessed run the risk of being eliminated from reports.

Second, in the school-based assessments, the grades or ages selected for assessment are chosen to maximize the likelihood of youth being enrolled in school; for example, PISA samples are drawn from the population of 15-year-old students enrolled in school. In 2003, the most recent year for which data are available, the percentage of the population ages 5–14 enrolled in school was 90 percent or higher in most developed countries, including the United States, and 80 percent or higher in most developing countries that participated in international assessments (OECD 2004a, table C1.2). The percentage of the U.S. population ages 15–19 enrolled in public or private school was 75 percent, which is comparable to or below that of most other industrialized countries. Comparisons of graduation rates from upper secondary school (high school in the United States) paint a similar picture: the U.S. graduation rate (73 percent) is comparable to or below that of most industrialized countries, where 80 percent or more of students finish upper secondary school (OECD 2004a, table A2.1).

Further differences among countries in terms of their student population characteristics, especially those found to be significantly related to achievement, can also be evaluated and explained in comparative analyses. Research has established that students’ economic and social characteristics, such as their immigrant status and family income, are associated with academic achievement (Coleman et al. 1966; Entwisle and Alexander 1993; Shavit and Blossfield 1993). Moreover, research has shown that these factors are often interrelated, further complicating the picture (McLanahan and Sandefur 1994; Schmid 2001). For example, minority status, family income, language ability, and family structure are associated with students’ achievement in the United States (Coleman et al. 1966; Jencks et al. 1979; McLanahan and Sandefur 1994; Schmid 2001), and such relationships are also found in many other countries (Buchmann 2002). The uneven distribution of students’ economic and social factors across countries, as well as the potential cross-national variation in the relationship between student achievement and these factors, may affect the outcomes of cross-national comparisons.

Recent comparisons of PISA 2003 data have explored how variations in student population characteristics across countries may affect the reported outcomes of international studies. For example, it is true that some characteristics of the U.S. student population are different from those of student populations in countries like Japan and Korea, where there are few foreign-born students; however, student populations in other countries are often not measurably different from the U.S. student population in terms of the distribution of salient social and economic factors (figures 1 and 2; Hampden-Thompson and Johnston 2006). For example, 48 percent of 15-year-old students in the United States reported having at least one parent who had a college degree or a postsecondary vocational qualification (figure 1). When the United States was compared to the other 19 countries in this study, 11 countries were found to have a smaller percentage of students with postsecondary-educated parents when compared with the United States. Seven countries had a higher percentage of 15-year-old students who reported that at least one of their parents was educated to the postsecondary level (Australia, Belgium, Canada, Denmark, Finland, Norway, and Sweden). Also, the data show that 9 percent of U.S. 15-year-olds did not speak the language of the test at home (i.e., English; figure 2). Of the 19 other countries, 6 had a greater percentage of 15-year-olds who did not speak the language of the test at home, and 8 countries had a lower percentage.

Cross-national comparisons of student populations and their social and economic contexts show that the United States shares many of the same educational challenges as other countries. For example, while the strength of the association may vary, many studies report a fairly consistent relationship between lower socioeconomic status and lower student achievement (Buchmann 2002). The cross-national comparisons of achievement displayed in the sections that follow have not been adjusted for socioeconomic or other factors.