Analysis of Questionnaire Errors in Survey Measurements of Census Coverage
Elizabeth A. Martin, Robert E. Fay, and Elizabeth A. Krejsa
KEY WORDS: questionnaire design, reliability, mobility, measurement error, validation, duplications
ABSTRACT
Recent U.S. censuses have produced relatively accurate population counts, but the considerable importance of the results has driven the efforts to study and possibly correct the errors of coverage that occur. The 2000 Accuracy and Coverage Evaluation (A.C.E.) attempted to measure the net error of the census, originally with the intention to correct the census counts for all purposes other than the apportionment of the House of Representatives. In turn, the accuracy of the A.C.E. was assessed by several evaluation studies, including the Evaluation Followup. Although many of the evaluations implied that the A.C.E. was generally successful, the Evaluation Followup (EFU) indicated that the A.C.E. had seriously underestimated some types of erroneous enumerations in the census, including persons who lived elsewhere on census day. In October 2001, the U.S. Census Bureau decided not to incorporate the A.C.E. findings into the Census 2000 results.
Although there has been extensive statistical research to develop and improve dual-system estimates of census undercount, there has been less attention to the accuracy of the questionnaire measurements on which the statistical models are based. In this paper, we examine the reliability and validity of questionnaire responses by linking three sources of information about individuals’ residence at the time of the census: census records, reports given in A.C.E. interviews (or follow-ups conducted as part of the A.C.E), and reports given in EFU interviews. The duplication rates (based on census records) provide evidence independence of both surveys, which we use to assess the quality of the questionnaire classifications.
The results show high levels of unreliability in measurements of moves in and out of households. For both A.C.E. and EFU, we find higher rates of duplication for people whose responses indicate they were enumerated in error at a sample address, indicating that responses are valid. Neither questionnaire was clearly superior to the other, and each questionnaire added information about errors that were not identified by the other.
Our results support the need for more research to develop more accurate measures of residence status, and to better understand sources of reporting error, including recall error and lack of knowledge on the part of proxy respondents.
CITATION: Elizabeth A. Martin, Robert E. Fay, and Elizabeth A. Krejsa. 2002. “Analysis of Questionnaire Errors in Survey Measurements of Census Coverage.” Proceedings of the American Statistical Association (Survey Research Methods Section): 2260-2265.
