Challenging Research Issues in Statistics and Survey Methodology at the BLS

Problem Statement: How do Respondent Cognitive Processes and Respondent Interviewer Interaction Affect Data Quality in the American Time Use Survey (ATUS)?

Key Words: time-use diary, respondent cognitive processes, respondent recall strategies, behavior coding, respondent-interviewer interaction, conversational interviewing.

Contact for further discussion:
Polly Phipps
Office of Survey Methods Research, PSB 1950
Bureau of Labor Statistics
2 Massachusetts Avenue NE
Washington, DC 20212
Telephone: (202) 691-7513
Fax: (202) 691-7426
E-mail: Phipps.Polly@bls.gov

Background and Definitions

Since 2003 the ATUS, sponsored by the Bureau of Labor Statistics, has produced annual estimates of the amounts of time people spend in activities such as paid work, childcare, volunteering. The ATUS collects information about time spent in activities beginning at 4:00 am on the previous day up until 4:00 am of the interview day. Interviewers use a set of scripted open-ended questions to walk respondents chronologically through the prior 24-hour day. Respondents are asked to recall their daily activities, how long each activity took, who was in the room with them, and where the activity took place. This 24-hour recall diary is used in tandem with computer-assisted telephone interviewing (CATI) for data collection.

Early BLS research on the 24-hour recall diary explored the cognitive processes that respondents used to reconstruct their prior day and how cognitive processes were related to data quality. BLS and Westat researchers testing the 24-hour recall diary instrument were concerned about possible variation in data quality associated with differences in how respondents interpret "activities," the effects of highly salient activities, routine activities, and schemata (or everyday activities) on recall; and heuristic or rule of thumb strategies for estimating activity duration (Forsythe, 1997). Findings from cognitive interviews indicated that participants recalled activities and time durations more easily when their activities and times were atypical and/or when they had an appointment or time commitment. Some participants used a visualization technique as part of the recall process, while others used a decomposition strategy — recalling a larger block of time or major activity to set time boundaries, then recalling what happened during the time period. Participants who lived by the clock reported by the clock, but others had difficulty identifying starting and stopping times, and reported time durations instead.

Since this early testing, BLS has not conducted further formal research on the cognitive processes associated with the questions used and tasks undertaken in completing the 24-hour recall diary. Additional research on respondents' cognitive processes might aid in developing instrument and interviewer interventions to assist respondents in their task and ultimately reduce measurement error.

In addition, research on respondent and interviewer interaction during diary data collection is another important and unexplored area of methodological study. ATUS interviewers are trained in conversational interviewing techniques, which allow for interventions with a respondent to help him or her stay on track when remembering the day's activities, and activity sequences and timing. For example, after a respondent says that he or she ate dinner for an hour, the interviewer might check back, "that brings us up to 7:00 pm, does that sound about right?" Interviewers are also trained in ways to facilitate recall; one such technique is called working backwards. If a respondent can't remember what he or she did right after dinner, interviewers can ask, "what is the next thing you remember doing (and what time did that take place)?" and then prompt respondents to think of what he or she did in between. This working backwards technique may help respondents fill in the gaps, as may visualization techniques, such as an interviewer asking respondents to picture where they were in order to remember forgotten activities. Research to evaluate how conversational interviewing and specific recall techniques are used by interviewers and whether the techniques are successful in helping respondents reconstruct their day could help refine ATUS procedures and data quality.

One method that might be used to study both respondent cognitive processes and respondent and interviewer interaction is behavior coding. This technique has been successfully utilized with event history calendar data collection (Belli, 2004) to understand how interviewers ask questions and provide clarification and feedback to respondents, how respondents interpret questions and recall answers, and how interviewers and respondents interact during the survey task.

Issues

1. How do respondents reconstruct their prior day during the 24-hour recall diary data collection?
2. What role do interviewers and interviewer techniques have in shaping the 24-hour recall diary tasks?
3. How are respondent and interviewer behaviors and interactions related to data quality?

Related Questions:

1. How do respondents comprehend the 24 hour recall task, what strategies are used to recall activities and times, and how do respondents judge the adequacy of a response and communicate it to the interviewer?
2. Does the traditional cognitive model as generalized to surveys (question comprehension; ability to recall, judge, and communicate answers), assist us in the understanding of respondent's cognitive process when completing time diaries?
3. Do other theories or models from cognitive psychology or other disciplines provide insight into understanding respondent understanding and execution of the diary task?
4. If and how do interviewers use conversational interviewing and recall techniques on which they are trained during diary data collection? Are the techniques useful in helping respondents reconstruct their day?
5. Does greater use of some respondent processes or interviewer techniques produce better quality data, as measured by respondent understanding, item nonresponse, or number of activities reported?
6. How can behavior coding be used to evaluate data accuracy and quality in the ATUS? Can it identify difficult tasks and concepts or successful interviewing practices?

References:

Belli, R., Lee, E.H., Stafford, F.P., and Chou, C.H. (2004). "Calendar and Question-List Survey Methods: Association Between Interviewer Behaviors and Data Quality." Journal of Official Statistics 20: 185-218.

Forsythe, B. (1997). "Cognitive laboratory research plan for collecting time use data." Washington, D.C.: Bureau of Labor Statistics, Office of Survey Methods Research. Horrigan, M. & Herz, D. (2004). "Planning, designing, and executing the BLS American Time-Use Survey." Monthly Labor Review, 127, 3-19

Juster, F.T. & Stafford, F.P. (Eds.). (1985). Time Goods and Well Being. Ann Arbor, MI: Institute for Social Research.

Juster, F.T. (1986). "Response Errors in the Measurement of Time Use." Journal of the American Statistical Association 81, 390-402.

National Academy of Sciences. (2000). Time-Use Measurement and Research: Report of a Workshop. Washington, DC: National Academy Press.

Phipps, P. and Vernon, M. (2008). "24 Hours: An Overview of the Recall Diary Method and Data Quality in the American Time Use Survey." In Robert F. Belli, Frank P. Stafford, and Duane F. Alwin (Eds.), Calendar and Time Diary Methods in Life Course Research. Thousands Oaks, CA: Sage. Expected Publication 2008.

Robinson, J.P. (1985). "The validity and reliability of diaries versus alternative time use measures." In F.T. Juster and F. P. Stafford (Eds.), Time Goods and Well Being (pp. 33-62). Ann Arbor, MI: Institute for Social Research, University of Michigan.

Robinson, J.P. (1999). "The Time-Diary Method: Structures and Uses." In W.E. Pentland, A.S. Harvey, M.P. Lawton, and M.A. McColl (Eds.), Time Use Research in the Social Sciences (pp. 47-87). NY: Kluwer Academic/Plenum

 

Last Modified Date: July 30, 2008