Vignettes and Respondent Debriefings for
Questionnaire Design and Evaluation
Elizabeth A. Martin
KEY WORDS: pretesting, questionnaire design, comprehension, recall
ABSTRACT
Recent decades have seen theoretical and empirical advances in understanding the cognitive sources of measurement error introduced by failures of comprehension or retrieval. This chapter describes how two methods, vignettes and respondent debriefing questions, can be used to identify measurement problems and craft and test questionnaire designs to address them. The focus is on their application in field-based tests of interviewer-administered questionnaires, although they also are used in laboratory and qualitative studies (the latter use is discussed) and with other types of questionnaires, such as self-administered ones. The chapter draws on research (much of it hitherto unpublished) conducted for the redesign of several Census Bureau surveys as well as other studies. To illustrate and illuminate problems and strategies for addressing them, the paper draws upon research (much of it hitherto unpublished) conducted for the redesign of several Census Bureau surveys. Four types of applications of vignettes to questionnaire design are described and illustrated, drawing on research conducted to address problems of interpretation and comprehension in the Current Population Survey. The application of respondent debriefing questions is explained, drawing on research undertaken to redesign instruments for the National Crime Victimization Survey to reduce recall and reporting problems. The advantages and disadvantages of the two methods are compared and contrasted and evidence pertaining to their validity and consistency with other measures is summarized.
CITATION: Elizabeth Martin. (2004) “Vignettes and Respondent Debriefing for Questionnaire Design and Evaluation,” Chapter 8 in S. Presser, J. M. Rothgeb, M. C. Couper, J. T. Lessler, E. Martin, J. Martin, and E. Singer (eds.), Methods for Testing and Evaluating Survey Questionnaires. New Jersey: Wiley.
