Research Reports

Practical Considerations in Sample Size Selection for Behavior Coding Pretests

Andrew L. Zukerberg, Dawn R. VonThurn and Jeffrey C. Moore

KEY WORDS: Behavior coding, Pretesting

ABSTRACT

Despite some initial work (e.g. Presser and Blair, Cannell et al.), few objective guidelines currently exist to suggest how many cases are needed when using behavior coding as a pretest technique. In practice, time and budget factors often coalesce to reduce the number of cases which are finally behavior coded. This paper attempts to establish a practical number of cases to maximize knowledge gained from behavior coding while holding dollar and time cost to a minimum. It also examines these issues for questionnaire items categorized by expected level of respondent and interviewer difficulty. Results of the research may be useful to survey organizations wishing to conduct efficient behavior coding pretests to identify questionnaire design problems before fielding a survey on a large scale. The paper first describes the impetus for the research and briefly reviews the relevant literature. Second, we discuss our methods and analytic procedures. Next, we present our findings, which suggest that a relatively small number of cases may typically provide questionnaire designers with the same information - leading to identical conclusions - that they would have obtained with a much larger sample at much greater cost and effort. Finally, we discuss the implications of these findings and suggest next steps.

Citation: 1995, Proceedings of the Section on Survey Research Methods, Alexandria, VA: American Statistical Association, pp. 1116-1121.