A data collection instrument that a respondent self completes through the visual channel, such as on paper or over the Web, is visually administered. Traditional methods of evaluating visually administered instruments, such as cognitive interviewing, usability testing, and experiments, cannot directly identify information respondents perceive, or in what order they observe the information. Consequently, eye-tracking equipment developed at the University of Virginia for use with computer monitors was adapted to track the eye movements of respondents answering three paper questionnaires, which differed in the visual designs of their branching instructions. Twenty-five respondents answered one of the three questionnaires. The study revealed that it was not whether respondents perceived a branching instruction that determined if they executed it correctly, but when they perceived it. If respondents did not observe the instruction immediately prior to or after marking their answer, they misexecuted the instruction. This is a very insightful finding and an encouraging lead, one that could not be drawn from any other method. Thus, eye-movement analysis does appear to be a promising new tool for evaluating visually administered questionnaires. However, it currently has drawbacks as well, which we discuss in depth and make recommendations for improving.
