Challenging Research Issues in Statistics and Survey Methodology at the BLS
Issue: The Pros and Cons of Questions Containing Cue Lists
Key Words: cues, data quality, mode of administration, questionnaire design, recall error and response alternatives
Contact for further discussion:
Monica Dashen
Office of Survey Methods Research, PSB 1950
Bureau of Labor Statistics
2 Massachusetts Avenue NE
Washington, DC 20212
Telephone: (202) 691-7530
Fax: (202) 691-7426
E-mail: Dashen.Monica@bls.gov
Background:
A challenging research issue is that of questions which contain non-exhaustive cue lists (response alternatives). Such lists help shape and align respondents' interpretation of the question with that of the survey designer by providing respondents with subsets of acceptable (included) and unacceptable (excluded) answers. Without cue lists respondents are forced to devise their own inclusion criterion and as a result may not provide acceptable answers (Dashen and Fricker, 2001).
Questions containing cue lists can be seen in a variety of surveys, including establishment surveys (Annual Refiling survey) and household survey (Consumer Expenditure Survey). The table below provides examples of questions containing cue lists.
| Survey & Respondent Expectations |
Example of Questions with Cue Lists |
| In the Annual Refiling Survey, respondents are expected to verify whether their establishment fits an industry or not. |
Child day care services of infants or children. Generally care is for preschool children, but may include caring for older children when they are not in school and may also offer prekindergarten educational programs. Examples include, but are not limited to: |
| Child day care services of infants or children. Generally care is for preschool children, but may include caring for older children when they are not in school and may also offer prekindergarten educational programs. Examples include, but are not limited to: |
| * Child day care babysitting services |
* Nursery schools |
| * Child or infant day care centers |
* Preschool centers |
| DOES NOT INCLUDE kindergarten educational programs provided in elementary and secondary schools. |
| In the Consumer Expenditure- Diary, respondents are expected to record their expenses for 7-days. |
Food and Drinks Away from Home (selected cues) |
| Breakfast buffet |
pizza delivery |
beer at happy hour |
croissant from café |
| Carry-out lunch |
Chinese takeout |
pretzels at ballgame |
ice cream from truck |
| Dinner/cocktails at restaurant |
child's school lunch |
wine at tavern |
wedding reception caterer |
Cue lists are often non-exhaustive because of space limitations and other reasons. To signify this non-exhaustive status, survey designers often use phrases like "illustrative," "e.g.," or "not limited to." Survey designers often assume that the cues on the list will aid in the recollection of related items, thereby assisting the respondent in generating items. For example, Consumer Expenditure (Diary) respondents might remember taking out Mexican food after having read the Chinese Takeout cue. The number of items in a cue list can vary greatly and can be provided across all modes of administration. Cues can be either presented in a list or a sentence-like format. Cues can also be presented in a general manner (e.g., fresh produce) or a specific manner (e.g., bananas, oranges, and lettuce).
Issues Pertaining to Cue Lists:
- Do cue lists help or hurt in the respondents' recollection of non-mentioned cues? Although it may seem counter-intuitive, cue lists can actually inhibit recollection of non-mentioned cues. In a classic study, Brown (1968) demonstrated that when asked to recall all of the US states, respondents did better when given a blank sheet of paper than when given a list of all the eastern states. This part-set cuing finding has been replicated in many different situations; see Nickerson (1984) for a thorough and well-cited review of the literature, and see Marsh, Dolan, Balota, and Roediger (2004) for more recent research.
- Is there such a thing as an optimal number of cues? Survey designers are often faced with the challenge of deciding what is too few items vs. too many. Will too many items overburden respondents causing them acquiesce? Or would too few cues fail to meet the needs of the designer to illustrate the question?
- Does the mode of administration affect how well individuals remember the items on a list? Krosnick and Alwin (1987), for example, found that when the list was presented visually, as in a self-administered survey, respondents tended to remember more items at the beginning of the list than those toward the end. In contrast, when the list was presented auditorily, as in a telephone survey, people tended to remember more of the items toward the end of the list compared to those toward the beginning.
- Does the cue list format (list- or sentence-like) matter? Are respondents more likely to read cues presented in a list-like format than say a sentence-like format?
- Do excluded items help or hurt the respondents ability to fit their establishment into an industry? Anecdotal reports from the Annual Refiling Survey field staff indicate that respondents are led astray by exclude statements. For example, some respondents incorrectly reject their own industry descriptions because their establishment performs an excluded activity.
- Does the level of cue generality matter? Do more general cues elicit more responses than specific ones? See Tucker (1992) for related research on the Consumer Expenditure Diary.
- Is a rule-based approach an alternative to cue lists? Rather than itemizing the contents of a categorical question, survey designers in some situations may find it more effective to provide a rule for inclusion of items. To distinguish between limited and full-service restaurants, for example, CE designers provide the following rule to guide respondents in their choices: fast food category is where you pay before you eat, whereas, you pay after you eat at the full service places.
- Are cue lists conducive to the conversational-interviewing approach where interviewers are expected to clarify responses (Schober and Conrad, 1997)? A situation may arise where a respondent asks the interviewer about the acceptability of an answer that is not mentioned on the cue list.
References:
Brown (1968). Reciprocal facilitation and impairment in free recall. Psychonomic Science, 10, 41-42.
Dashen, M. & Fricker, S. (2001). Understanding the Cognitive Processes of Open-Ended Categorical Questions and their Effects on Data Quality. Journal of Official Statistics. 17(4), 457-77.
Krosnick, J. & Alwin, D. (1987). An evaluation of a cognitive theory of response order effects in survey measurement. Public Opinion Quarterly, 51, 201-19.
Marsh E.J., Dolan, P.O. Balota, D.A., and Roediger, H.L. (2004) Part-set cuing effects in younger and older adults. Psychological Aging. 19(1), 134-44.
Nickerson (1984). Retrieval inhibition from part-set cuing: a persisting enigma in memory research. Memory and Cognition. 12(6), 531-52.
Schober, M.F. & Conrad, F.C. (1997). Does Conversational Interviewing Reduce Survey Measurement Error? Public Opinion Quarterly. 61(4), 576-602.
Tucker, C. (1992). The Estimation of Instrument Effects on Data Quality in the Consumer Expenditure Diary Survey. Journal of Official Statistics, 8(1), 41-61.
Last Modified Date: January 06, 2006
Last Modified Date: July 19, 2008
