• Introduction
• Purpose and Content of This Report
• Field Test
• Sampling
• Data Collection and Processing
• Questionnaire Data
• Coding of the Video Data
• Weighting and Variance Estimation
• References

The Third International Mathematics and Science Study (TIMSS) 1999 Video Study was conducted under the auspices of the International Association for the Evaluation of Educational Achievement (IEA) and managed by the U.S. Department of Education’s National Center for Education Statistics in cooperation with its study partner, the National Science Foundation. This complex and ambitious study examined classroom teaching practices through in-depth analysis of videotapes of eighth-grade mathematics and science lessons. Over a period of 4 years, researchers collected, transcribed, translated, coded, and analyzed hundreds of hours of videotapes of eighth-grade mathematics and science lessons in the seven participating countries.

The design of the TIMSS 1999 Video Study built on the foundations established by the TIMSS 1995 Video Study, but was improved and carried out through a collaborative process that involved individuals around the globe. Both of the TIMSS video studies investigated nationally representative samples of classroom lessons from relatively high achieving countries. The video studies were designed to supplement the information obtained through the TIMSS 1995 and 1999 mathematics and science assessments.

This first volume of the Third International Mathematics and Science Study 1999 Video Study Technical Report provides details about every aspect of the planning, implementation, processing, analysis, and reporting of the mathematics component of the TIMSS 1999 Video Study. The report is a record of actions and documentation of outcomes that can be used as a reference for interpreting results and planning future studies. Results of the mathematics component of the study are detailed in Hiebert et al. (2003).

Individual chapters of the report cover the field test process, sampling strategies and participation rates in each participating country, data collection and processing, questionnaire development and coding, coding of video data by an international team of coders as well as several specialist groups, and weights and variance estimates used in analyzing the data. Ten appendixes present study materials and additional details on study personnel and procedures.

The field test for the TIMSS 1999 Video Study helped to generate improvements in data collection, processing, and analysis. Some of the most important modifications involved creating videotaping procedures for two cameras, updating data storage software to include two video tracks, generating transcription/translation protocols to incorporate five languages, and generating hypotheses and coding ideas to describe teaching in a wide range of countries.

The field test analysis team consisted of representatives from each of the countries participating in the field test study. These representatives spent several months in 1998 studying the data from all the countries. As one task, they selected a “typical lesson” from their own country and then viewed the “typical lessons” from all the other countries. Structured group discussions about these lessons led to preliminary theories about the characteristics of instruction within each country as well as important differences in teaching across countries. These theories paved the way for more intensive work on code development.

The TIMSS 1999 Video Study aimed to expand on the TIMSS 1995 Video Study by examining instruction in more countries—in particular, in a larger selection of countries that outperformed the United States on the TIMSS 1995 assessments. Besides the United States, the countries for which mathematics lessons were analyzed in the TIMSS 1999 Video Study included Australia, the Czech Republic, Hong Kong SAR,* Japan, the Netherlands, and Switzerland. All these countries outperformed the United States on the TIMSS 1995 mathematics assessment administered to eighth-grade students.

The TIMSS 1999 Video Study was designed to provide comparable information about nationally representative samples of mathematics and science lessons in participating countries. To make the comparisons valid, it was necessary to devise a sampling design for each country that called for uniformity in sampling procedures but also allowed participating countries to account for differences in their education systems, as well as implementation limitations.

The samples of classrooms for the study were selected using two-stage probability sampling methods. The first stage of selection was the sample of schools. All participating countries were required to include at least 100 schools in their initial selection of schools. For each subject area (mathematics and science), the second stage involved the random selection of one eighth-grade classroom. The full report presents details about sampling strategies, participation rates, and the nature of participating schools in each country.

Videotapes, questionnaire responses, and other supplementary materials were processed using a sophisticated database management and tracking system. Both videographers and transcribers followed well-defined protocols in order for videotaping and transcription/translation procedures to be standardized across countries. Specific quality control measures were in place to carefully monitor both groups. Video data and corresponding English transcripts were entered into vPrism, a multimedia database software developed for the TIMSS 1995 Video Study and enhanced especially for the TIMSS 1999 Video Study. Coders used this software to watch the teacher and student tapes, see running English translations, access supplementary materials, and enter codes into the database.

To help understand and interpret the videotaped lessons, questionnaires were collected from teachers and students in each lesson. The teacher questionnaire was designed to elicit information about the professional background of the teacher, the nature of the mathematics course in which the lesson was filmed, the context and goal of the filmed lesson, and the teacher’s perceptions of its typicality. The questionnaire included a number of open-ended items, for which a coding scheme was developed and applied. The full report presents justifications for each item on the teacher questionnaire, along with reliability statistics for the open-ended items.

The student questionnaire was designed to elicit information about the demographic characteristics of the students, their home environment, and their educational expectations. Both the teacher and student questionnaires were approved by a review panel, and then country-appropriate versions were created under the direction of the national research coordinators in each country.

Most codes were developed by an international code development team and applied by an international coding team. Drawn from all participating countries, members of these teams were cultural insiders and fluent in the language of the countries they represented. However, not all of them were experts in mathematics or teaching. Therefore, several specialist groups were employed to create and apply codes regarding the mathematical nature of the content, the pedagogy, and the discourse.

The international code development team, with the aid of the national research coordinators and a steering committee, created 45 codes that were applied to the video data in seven passes by the international coding team. For each code, the percent correct procedure (described in Bakeman and Gottman [1997]) was used to compute initial and midpoint reliability statistics, which in all cases exceed 85 percent.

Four specialist groups were enlisted to analyze portions of the TIMSS 1999 Video Study mathematics data. The mathematics problem analysis group and the problem implementation analysis group studied the mathematical problems in the lessons, and the mathematics quality analysis group made judgments about the nature of the mathematics presented in the lessons. Each of these three groups consisted of individuals with particular expertise in mathematics, mathematics education, and teaching. A fourth group, the text analysis group, created and implemented specially designed software to study the nature of the classroom talk (or discourse) in the lessons.

Analyses of the TIMSS 1999 Video Study data were conducted using data weighted with survey weights. These weights were calculated specifically for the classrooms in this study. The full report presents details about calculation of classroom base weights to reflect the probability of selection, rates of nonresponse among selected schools, and adjustments made to the weights in order to account for nonresponse. In addition, an explanation of the jackknife technique is provided, along with a description of how to use the weights when conducting analyses using these data.

Bakeman, R., and Gottman, J.M. (1997). Observing Interaction: An Introduction to Sequential Analysis. Second Edition. Cambridge: Cambridge University Press.

Hiebert, J., Gallimore, R., Garnier, H., Givvin, K.B., Hollingsworth, H., Jacobs, J., Chui, A.M.-Y., Wearne, D., Smith, M., Kersting, N., Manaster, A., Tseng, E., Etterbeek, W., Manaster, C., Gonzales, P., and Stigler, J. (2003). Teaching Mathematics in Seven Countries: Results From the TIMSS 1999 Video Study (NCES 2003–013). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

*For convenience, Hong Kong SAR is referred to as a country. Hong Kong is a Special Administrative Region (SAR) of the People’s Republic of China.