Computer Assisted Interviewing for SAMHSA's National Household Survey on Drug Abuse

11. Effect of Interviewers on Data Quality

In this chapter, we report the results of two investigations of the effect of the interviewers on data quality. We examine the relationship between interviewer characteristics and performance measures and look at the effects of the mode of interview on interviewer variance, which is the increase in variance due to the correlation of response errors within interviewers.

In this section, we describe the characteristics of the 1997 field experiment's field interviewers (FIs) and examine the relationship of those characteristics to a number of FI performance measures. This chapter describes an exploratory study to determine whether the quality of the NHSDA interviewer workforce can be improved by selecting them on the basis of their background characteristics.

There were 165 FIs involved in the 1997 field experiment. For most of the interviewers, data from a personal data sheet (PDS) were available. For the purposes of this study, the following variables were created from the PDS: gender, age in years, a four-category ethnicity variable (black, white, Hispanic, and other), and a three-category education-level variable (high school, some college, and 4-year college graduate).

Eighty-one percent (133) of the interviewers were female. Their average age was 50 years, with a standard deviation of approximately 12. Most were white (73%), 17% were black, and the remaining 10% were Asian, Pacific Islander, American Indian, Alaskan Native, or Hispanic. Some 21% attended high school or were high school graduates; 41% had some college or a 2-year degree; and 36% were college graduates or had postgraduate degrees.

The other set of background variables related to field interviewing experience were as follows: a six-level ordered categorical variable reflecting years of survey interviewing experience (none, up to 1 year, 1 to 2 years, 2 to 5 years, 5 to 10 years, more than 10 years). Of the 143 interviewers who provided information on survey interviewing experience, only 21% had no experience, 48% had up to 5 years of experience, and 31% had 5 or more years of experience.

Five binary indicator variables indicated whether the FI had experience as a data collector or supervisor of data collectors in the following five areas: household surveys with face-to-face interviews; listing area segments; computer-assisted personal interviewing (CAPI); computer-assisted telephone interviewing (CATI) or other telephone interviewing; and field reporting with a microcomputer. These field interviewing experience variables are summarized in Exhibit 11.1.1.

Most of the FIs had either household survey face-to-face interviewing experience (70%) or CAPI experience (58%). Also, 79% of the interviewers had either household survey or CAPI experience. Some 49% had experience in both areas.

Each interviewer completed a 23-item computer beliefs and attitudes questionnaire at three time points: prior to training, immediately after training, and at the end of the 1997 field experiment study. The time between the pre- and post-administrations was approximately 2 weeks, and the time between the post- and end-of-study administration was approximately 3 months. The 23 items were scored using a 5-point Likert scale: 5 (strongly agree), 4 (agree), 3 (neither agree nor disagree), 2 (disagree), and 1 (strongly disagree).

Twenty-one of these items came from an instrument used by the U.S. Bureau of the Census to study their FIs. Research Triangle Institute (RTI) added two items concerning the use of the Newton in screening. We decided to drop these two items so that our results might eventually be compared with those of the Census Bureau. The 21 items measured, for the most part, beliefs and attitudes about the role of computers in general and specifically with field interviewing. The correlation matrices for the 21 items were factor analyzed separately for each of the three time points. For each administration, two through five factors were extracted and rotated by Varimax. We felt that the best criterion for selecting a particular factor solution (i.e., number of factors) at each time point should be based on how effectively they predicted the FI performance measures.

The factor analyses of the beliefs and attitude questionnaires were undertaken (a) to reduce the number of independent variables in the interviewer performance regression models, (b) to address the problem of multicollinearity, and (c) to simplify the interpretation of results from the regression analyses. The unit of analysis was the interviewer. About 140 interviewers met the criteria to be included in the regression analyses. Therefore, we wanted to minimize the number of independent variables to be included in the regression models.

To model interviewer performance, we needed to control for the segment characteristics in which each interviewer worked. Nine segment characteristics were selected for this purpose:

1. Quarter 1 through 3 successful screening percentage (number of households screened divided by number of eligible households),

2. Quarter 1 through 3 successful interview percentage (number of completed interviews divided by the number of people who were selected to be interviewed),

3. median rent,

4. median value of owner-occupied dwelling,

5. percentage of noninstitutionalized Hispanics 12 years of age or older,

6. percentage of noninstitutionalized black, non-Hispanics 12 years ofage or older,

7. percentage of urban population in segment,

8. a binary urbanized area indicator, and

9. an ordered three-level population density variable.

The performance measures for the interviewers were based upon a single segment, which was the first clean segment in which the interviewer worked. A clean segment was one in which only one interviewer worked. The reason for this definition was to avoid contaminating the performance measures for a particular interviewer with the performance of one or more other interviewers. In most cases, an interviewer worked in only one segment, and that segment was usually clean.

The following interviewer performance measures were used as the five dependent variables in the regression modeling: the percentage of eligible dwelling units that were successfully screened; the percentage of completed interviews from selected persons; the probability of the interviewer having trouble setting up the computer as judged by the respondent; the probability of the interviewer having trouble with the computer during the interview as judged by the respondent; and the number of days from assignment of segment to beginning work in the segment.

Logistic regression analysis was used to model the four binomial outcome variables. For each interviewer, we had the number of trials (e.g., the number of persons selected for an interview) and the number of successful outcomes or events (e.g., the number of completed interviews). All of the logistic regression analyses were adjusted for overdispersion because the interview completion events within interviewers were not independent. A Cox proportional hazards model was used to model the time from assignment to beginning work in the assigned segment.

Because there were eight different factor solutions (two- to five-factor solutions for both the pre-training and post-training beliefs and attitudes questionnaires), we modeled each of the five interviewer performance measures as a function of each of the eight sets of factor scores. Thus, there were a total of 40 regression analyses. In general, the attitude and belief factors were not very predictive of interviewer performance. Overall, the post-training five-factor solution seemed to be the best set of factors for predicting the five performance measures, perhaps because these factors were measured after training and because they were also measured closer to the fieldwork activities on which the performance measures were based. Also, five factors explain more variation in the 21-questionnaire items than two-, three-, or four-factor solutions.

The five rotated factors accounted for 60% of the total variation in the 21 attitude and belief questions. The first factor had high loadings from 10 of the 21 items ranging from 0.55 to 0.78. It explained the most variation in the attitude items. We interpret this factor as a general expression of positive beliefs and attitudes toward computers both in general and specifically withrespect to their utility for interviewing.

The remaining four factors explained considerably less variation and had only between two and four variables loading high (i.e., about 0.50 and above) on them. Factor 2 reflected a disagreement with negative beliefs, such as beliefs that computers threaten interviewers' jobs and the feeling that the interviewer is being watched all the time by using a computer. Factor 3 was defined by three questionnaire items that reflected not being afraid of the responsibility for taking care of the computer and other equipment. Factor 4 was defined by three items that reflected lack of concern about computer breakdowns and keying in data. The last factor, Factor 5, was defined by two items that indicated that the interviewer believed that the respondents will enjoy the interview and that carrying the computer around will not be tiring.

Two logistic regression models, described below, were developed for each of the following four performance measures: the probability of a completed interview, the probability of a successful screening, the probability of having trouble setting up the computer, and the probability of having trouble with the computer during the interview. The independent variables came from three variable sets: nine segment characteristics, eight interviewer background and experience characteristics, and the five-factor post-training belief and attitude factors. Using the same independent variables, the Cox proportional hazards model was used to model the number of days from segment assignment to start of work in the segment.

Because there were a large number of independent variables relative to the number of observations, the following strategy was adopted for the first logistic regression model. The five post-treatment belief and attitude factors were included in all of the models. CAPI experience also was included in all four models. A stepwise logistic regression was run separately for the nine segment characteristics to select segment characteristics for inclusion in one of the two final models for each performance measure. Likewise, a stepwise logistic regression analysis was run on the eight interviewer background and experience variables to select interviewer characteristics for inclusion in one of the two final models for each performance measure. Variables significant at the 0.10 level for each of the two stepwise regression analyses along with the five attitude and belief factors (as well as CAPI experience) were included in a final model for each of the four performance measures.

The second model-building strategy was to do a single stepwise logistic regression analysis for each of the four interviewer performance measures. The variables were selected from the entire pool of 22 independent variables (i.e., the nine segment characteristics, the eight interviewer background and experience characteristics, and the five post-training factors). For all of the stepwise regressions, the 0.15 significance level was chosen for entry into the model, and 0.20 was chosen for removal from the model.

Thus, for each of the four performance measures, two models were developed. One model involved selecting the variables for inclusion in the final model (henceforth referred to as Model 1) by selecting the significant variables separately from the segment and interviewer background characteristics sets by stepwise logistic regressions and then combining the significant variables from these two stepwise regressions with the five post-training attitude and belief factorsand CAPI experience to estimate a final model. The other final model (henceforth referred to as Model 2) involved selecting the variables from a single stepwise logistic regression analysis using the single pool of all 22 variables from all three variable classes.

For modeling the probability of completing an interview, both models indicated that the most important variables were segment characteristics. In both models, there was a highly significant negative relationship between the percentage of Hispanics in the segment and the probability of a completed interview; moreover, there was a highly significant positive relationship between the percentage of black, non-Hispanics in the segment and the probability of a completed interview. For Model 1 (see Exhibit 11.1.2), there were no significant parameters for any of the interviewer background variables or for any of the five belief and attitude factors. For Model 2 (see Exhibit 11.1.3), two interviewer variables were significant: education level and household survey experience. Education level was positively associated with the probability of a completed interview, and household survey experience was negatively associated. (The significance levels were 0.04 and 0.05, respectively.) Factor 2, reflecting disagreement with the belief that computers are watching the interviewer, and reflecting disagreement with the belief that computers may threaten interviewers' job security, was negatively associated (p=.02) with the probability of a completed interview. The direction of two of these relationships (household survey experience and Factor 2) was counterintuitive.

For modeling the probability of completing a household screening, once again both models indicated that the most important variables were segment characteristics. For Model 1 (see Exhibit 11.1.4), percentage Hispanic and median rent were negatively associated with the probability of a completed screening, while median value of owner-occupied housing and the Quarters 1 to 3 completed screening rate were positively associated with the screening rate. None of the interviewer background variables was significant. However, Factor 2 was once again significant, but in the expected direction. Interviewers who disagreed with the negative belief statements had a higher probability of completing household screenings. For Model 2 (see Exhibit 11.1.5), percentage Hispanic and median rent were negatively associated with the probability of a completed screening as in Model 1. The percentage of the segment that was urban had a positive relationship with the screening rate. In this model, interviewers who had computer-assisted interviewing experience had about two times the odds of a completed screening (p=.04) compared to those with non-CAPI experience. Factor 2 was once again significant (p=.01) in the expected direction. A one-scale point increase in Factor 2 increased the odds of a completed screening by about 44%.

Model 1 (see Exhibit 11.1.6), for modeling the probability of having trouble in setting up the computer for the CAPI, indicated that none of the 22 candidate variables was significant. The overall level of significance of the model was only 0.44. For Model 2 (see Exhibit 11.1.7), only two variables entered the model (percentage urban in the segment and interviewer's age). However, neither was significant at the 0.05 level.

For Model 1 (see Exhibit 11.1.8), in modeling the probability of having computer troubleduring the interview, only two segment characteristics were marginally significant. Percentage Hispanic had a negative parameter (p=0.07), and population density had a positive parameter (p=0.09). Factor 5 had a positive relationship (p=0.03) with the probability of having computer problems during the interview. For Model 2 (see Exhibit 11.1.9), percentage Hispanic and population density had negative and positive parameter, respectively, and were significant at the 0.09 and 0.02 levels, respectively. Age and Factor 5 had positive relationships with computer problems. (The significance levels were 0.03 and 0.01, respectively.) Older interviewers had a higher probability of experiencing computer trouble. Factor 5 was defined primarily by the belief that respondents will enjoy the interview and the belief that carrying computers around will not be tiring. It is puzzling why this factor should be positively associated with computer troubles during the interview.

The Cox proportional hazards models indicated that the time from segment assignments to starting work in the segment could not be predicted from these same independent variables.

Segment characteristics were the most important predictors of the probability of a successful screening and of the probability of a completed interview. In general, the six survey experience variables were not predictive of interviewer performance. The only exception was the finding that CAPI experience was positively related to the successful screening rate. The five attitude and belief factors were marginally predictive, but in some cases the direction of the relationship was counterintuitive.

In general, there was not a lot of variation in the dependent variables. The screening rates were generally in the 0.90s, the interview rates were in the high 0.70s, and the two computer problem variables were in the 0.10s. Nevertheless, the screening and interview rates were fairly predictable from the segment characteristics. One of the reasons that the survey experience variables were not predictive of performance is that five of the six variables simply indicated whether or not they had any experience in that particular area, but did not indicate the amount of experience with respect to either years or number of surveys in each of the five areas.

Another problem was the large number of independent variables relative to the number of interviewers. With four performance measures and 22 independent variables, we would expect four or five parameters to be significant at the 0.05 level just by chance. The situation is even worse when variables are selected by stepwise procedures. This could be a reason why the relationship of the attitude and belief factors to the performance measures is in some instances counterintuitive. Because the independent variables came from different sources, missing data further reduced the sample sizes for the regression analyses.

HU=housing unit.

Source: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment.

DU=dwelling unit; HU=housing unit.

Source: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment.

DU=housing unit.

Source: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment.

HU=housing unit.

Source: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment.

Source: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment.

Source: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment

Source: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment.

Source: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment.

Interviewer variance or correlated response variance refer to an increase in the variance of estimates due to correlation of response errors within interviewers. It has often been assumed that this correlation arises because the "interviewers influence respondents to make response errors of a similar type" (Groves, 1991, p. 16). Well-developed self-administered questionnaires (SAQs) should be subject to less influence by the interviewers. This is because the interviewer would have less interaction with the respondent during the question-answering process and, thereby, less chance to influence his or her response errors. Thus, as we become more successful in developing questionnaires that respondents can complete without help from the interviewer, we should expect interviewer variance to decrease. ACASI does allow the respondents to operate more independently than the paper answer sheets. Other chapters of this report document the fact that ACASI increases reporting of sensitive behaviors and that respondents find it to be a more private response mode. We also examined how it affected interviewer variance.

Ideally, to assess the impact of mode of interview on correlated measurement errors, we would use a design in which subsamples of the sample segments were randomly assigned to interviewers and, within interviewers, randomly assigned to an interviewing mode (either ACASI or PAPI). The 1997 field experiment did not meet these requirements; however, because the sample segments within primary sampling units (PSUs) were randomly assigned to one of four quarters of data collection and because we were using segments from Quarters 1 through 3 for the 1997 field experiment, the assumption that segments were randomly assigned to mode of interview within PSUs is valid. Segments were not randomly assigned to interviewers.

For evaluating the impact of the interviewing method on the interviewer variance component, we fit a general linear mixed model with fixed and random effects of the form

where  Y  is an outcome variable, represents fixed effects with respect to explanatory variables  X , represents random effects with coefficients Z, and represents random error in each individual observation. For fixed effects, we used the following independent variables: (a) stratum, (b) gender, (c) race, and (d) age category. For the random effects, we considered the following components: (a) PSU and (b) field interviewer ID. We initially tried fitting a single model, which included mode of interview. Because of the fact that overall people gave different responses under ACASI and PAPI interviews, however, the individual or residual variance components for the same response variable were different for the two interview types. Therefore, the model was applied separately for ACASI and PAPI data. We examined several outcome variables using the SAS procedure MIXED. The results are displayed in Exhibit 11.2.1. For the mental health-related outcomes, individuals under the age of 17 years were dropped and the educationvariable was added to the model.

It is not possible to compare the interviewer components directly because the individual and PSU components are different for the two interview types. Thus, for each method, we computed the intraclass correlation to estimate the relative effect of interviewers on variance. This represents the proportion of the population variance that is accounted for by interviewers.

Of the 15 variables examined, for 14 of the variables correlation for PAPI was greater than the correlation for ACASI. For only one of the variables, percentage who did not gain or lose weight when depressed, the PAPI correlation was slightly smaller than ACASI, but both values were very close (0.09869 and 0.09207).

The results indicate that the ACASI method reduced the influence of interviewers on the variance. This is another positive impact of the ACASI methodology.

Because of the difficulties associated with this analysis and the fact that the design was not as well suited as it might be to investigate this issue, we recommend that this analysis be repeated with 1999 NHSDA split-sample data.

ACASI=audio computer-assisted self-interviewing; PAPI=paper-and-pencil interviewing; PSU=primary sampling unit.

Source: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment.

This page was last updated on June 16, 2008.

SAMHSA, an agency in the Department of Health and Human Services, is the Federal Government's lead agency for improving the quality and availability of substance abuse prevention, addiction treatment, and mental health services in the United States.