2001 National Household Survey on Drug Abuse

Appendix C: Effects of Changes in Survey Protocol on Trend Measurement

C. 1 Background

The 2001 National Household Survey on Drug Abuse (NHSDA) results showed some unexpected increases in trend measures, particularly in the lifetime use of marijuana. As a result, a review of any methodological changes and their potential impact on estimates of prevalence was conducted concurrently with preparation of this report. The ultimate focus of this review centered on two methodological issues. The first was an embedded experimental design studying the impact of two alternative monetary incentive procedures. The second was the implementation of a field interviewer (FI) observation plan that led to the implementation of a continuing training and supervision program whose aim was the improvement in compliance with the intended data collection protocols.

Comparable protocols for data collection, data processing, sample design, and statistical analysis applied to each annual survey are essential for effective measurement of trends in substance use. Although a major shift in survey methodology occurred in 1999 with the introduction of computer-assisted data collection and a new 50-State sample design, the goal since then has been to maintain a consistent protocol in all areas following that transition. However, this goal did not appear to be inconsistent with maintaining or improving response rates, implementing procedures to ensure compliance with the established protocols, or implementing a general program of data quality improvement.

Due to concerns with response rates, an experimental study of the impact of monetary incentives was designed and implemented in the first two quarters of 2001. The design involved a sample of 251 FI regions (out of a total of 900 FI regions nationally). During the first quarter, one of the two monetary incentives ($20 or $40) was offered to respondents for completing the computer-assisted interviewing (CAI) questionnaire in one randomly selected segment in each FI region. There was no incentive offered in the other sample segment. During the second quarter, the other incentive amount was offered to respondents for completing the CAI questionnaire in one randomly selected segment. The sample of FI regions selected for the incentive experiment was selected to be nationally representative (with proper weighting) and to include a higher proportion of areas known to have historically low response rates. Because the experiment was embedded in the national sample, the incentives offered had some influence on national response rates. The embedded experimental design and the total sample design are summarized in Table C.1 at the end of the appendix. The FI regions involved in the incentive experiment constituted about 28 percent of all FI regions, but the sample area segments where any incentive was offered constituted only about 7 percent of all area segments.

A program of FI observation was initiated in quarters 1 and 2. An initial 39 FIs were observed between February 3^rd and April 15^th. An additional 111 FIs were observed between July 1^st and August 11^th.

As a result of the field observations, an emphasis was placed on conforming with established study protocols. Special telephone training sessions were developed emphasizing correct screening and interviewing procedures and the need to follow established protocols. A guidance document, Steps to Maximize Data Quality, was reviewed with all FIs in early July. Additional guidelines for training interviews, Reviewing NHSDA Procedures, were developed and used by field supervisors in a series of six weekly conference calls with interviewers over the period from October 22^nd through November 26^th. Session topics included screening, transition from screening to interview, front- and back-end computer-assisted personal interviewing (CAPI) portions (two sessions), properly administering audio computer-assisted self-interviewing (ACASI), and verification and wrap-up. Although this special training did not define any change in protocol, it did enforce the need to follow established protocol and, as a result, could have influenced the comparability of 2000 and 2001 data primarily for the last 6 months of the year.

A number of special analyses were initiated to investigate potential explanations for the observed 2000 to 2001 change in prevalence measures. These can be grouped as follows:

• review of postsurvey data-processing procedures (editing, imputation, and weighting);
• analysis of the incentive experiment effects;
• further analysis of FI experience effects;
• further analysis of historic response rate and changes in response rate;
• analysis of proxy measures of FI behavior (timing, debriefing questions, etc.);
• alternative measures of change based on retrospective data;
• focused analysis on first two quarters of 2000 and 2001; and
• questionnaire changes.

C.2 Postsurvey Data Processing

The effects of editing, imputation, and weighting on prevalence measures were examined by comparing estimates before and after processing. Comparable estimates were produced by quarterly subsamples and by age groups to identify any unusual impacts limited to shorter time periods or to a subset of the data.

An important set of initial analyses focused on unedited respondent data. These analyses investigated whether increases in prevalence in 2001 might be due in part to increased reporting of drug use by respondents prior to the data being edited or imputed. Trends in unedited data among specific age groups were examined by quarter for 1999 to 2001, both on an unweighted and weighted basis. To promote consistency in the examination of the trends, data resulting from changes to the instrument in 2000 and 2001, such as the addition of those described in Section C.9, were generally not taken into account.

For most of the drug use measures and age groups, the unedited trend data indicated that the significant differences in drug use estimates between 2000 and 2001 were due to higher percentages of respondents reporting drug use in at least some quarters of 2001. These results held for both weighted and unweighted data. Therefore, the unedited, unimputed, and unweighted data indicated the same trends as the fully processed data.

Figure C.1 shows the overall impact of the editing, imputing, and weighting processes on lifetime marijuana and cocaine use estimates for persons aged 18 to 25. Plots across quarters are shown for raw unweighted data, for raw weighted data, and for edited, imputed, and weighted data. Note that the raw (unedited) weighted data and the edited, imputed, and weighted data track very tightly across quarters. The raw unweighted data exhibits a different level than the other two measures in most quarters, but shows the same general trend over longer periods of time.

C.3 Incentive Experiment Effects

The results of the incentive experiment were reported by Eyerman and Bowman (2001). Selected conclusions from their report are as follows:

• The $20 and $40 incentive payments each produced about a 10-point gain in overall response rates when compared with the $0 control group. The overall response rate was significantly higher for $40 than the $20 incentive within many of the subgroups addressed in the analysis.
• Both incentive payment treatments more than paid for themselves due to decreased costs of follow-up and more productive screening resulting from the improved response rates.
• Some significant differences in prevalence rates were noted in comparisons between the $40 treatment and the control in some of the age, race, and historical response rate groups: two cases of significantly higher past month alcohol use and one case of significantly lower past month cigarette use.

Wright, Bowman, Butler, and Eyerman (2002) conducted additional analysis of the 2001 incentive experiment. By adjusting the weights for predicted response propensity based on incentive treatment (and other covariates), applying regression models to the full sample data, and combining $20 and $40 as a single treatment level, they obtained statistically significant incentive effects on prevalence measures for past year use of marijuana (a positive effect with p = .027) and for past month use of cocaine (a negative effect with p = .033). Past month marijuana use showed a marginally significant positive effect for incentives (p = .055). Surprisingly, no relationship (after adjustment for other covariates) was found between incentives and lifetime use of marijuana. This may indicate that most persons were willing to report lifetime marijuana use without an incentive, possibly because lifetime use is not as stigmatized. All four sets of regression analyses also showed a negative relationship of prevalence measures with historic response rates.

Preliminary data review indicated some possible carryover effects of the experiment into quarters 3 and 4 of 2001. To study these potential effects, the data for 1999, 2000, and 2001 were partitioned based on the 251 FI regions involved in the incentive experiment and the remaining 649 non-incentive experiment FI regions (see Table C.1). The special weight developed for analyzing the incentive experiment was applied to the 251 incentive experiment FI regions; a pseudo-weight was developed for the 649 non-incentive experiment areas. Both sets of weights were adjusted to estimate the total population. To partially calibrate the weights for the two subpopulations, poststratification by gender and five age categories was implemented.

Although prevalence rates between the incentive FI regions and non-incentive FI regions were not the primary interest, statistical tests were applied to determine whether the incentive regions produce higher prevalence rates. Prevalence estimates for five substances (marijuana, cocaine, cigarettes, alcohol, and hallucinogens) at three recency of use levels (lifetime, past year, and past month) and for three dependency measures (illicit drugs, alcohol, and marijuana) were analyzed. Based on four quarters (even though the incentive experiment was only fielded in the 251 FI regions in quarters 1 and 2) of data across all age groups, only lifetime marijuana showed a statistically significant difference between incentive and non-incentive FI regions in 2001; the higher estimate was measured for the incentive FI regions.

Additional statistical tests were applied to trend measures (year-to-year change) for the two samples. Estimates of change from 2000 to 2001 for these 15 prevalence and 3 dependence measures were positive in every case for both subsamples. Both subsamples showed statistically significant positive trend for five measures: lifetime marijuana, past year marijuana, past year cocaine, past year alcohol, and past year hallucinogens. For these five measures, the estimated change based on the incentive FI region subsample was always higher or at least as great as the estimated change based on the non-incentive FI region subsample. Only the incentive subsample showed a statistically significant trend measure in seven other cases: lifetime cocaine, lifetime alcohol, lifetime hallucinogens, past month marijuana, past month alcohol, dependence on illicit drugs, and dependence on marijuana. Only the non-incentive subsample showed a statistically significant trend measure in one case: past month hallucinogens. These results offered little direct evidence of carryover effects of the incentive experiment to quarters 3 and 4.

C.4 Field Interviewer Experience Effects

In examining the trend from 1998 to 1999 based on paper-and-pencil interviewing (PAPI), the FI experience was found to be a factor in explaining the observed trend (see Appendix D in Office of Applied Studies [OAS], 2000b). It was discovered that the more experience the interviewers gained, the less likely it was that the respondent would report drug use. Because 1999 was the final year of PAPI, an adjustment procedure was developed through special weight calibration procedures to remove the FI experience effect from the 1998 to 1999 trend measures. It was necessary to do this because the distribution of interviewers by their prior experience was much different in 1999 from what it had been in 1998. This difference in experience occurred because the size of the interviewing staff increased to accommodate an increase in planned sample size from about 25,500 in 1998 to 70,000 in 1999. Table C.2 shows the distribution of interviewers by interviewer's experience in prior years for 1999, 2000, and 2001. Because the size of the interviewing staff required decreased in 2000 and remained stable in 2001, the experience distribution changed again from 1999 to 2000, but remained fairly stable from 2000 to 2001. The 2000 Summary of Findings (OAS, 2001b, pp. 83-90) discusses the potential impact of the change in FI experience on selected measures of prevalence. Because 2000 was the second year in a continuing series of annual estimates based on the new CAI methodology and the impacts of FI experience were small, no adjustments to 1999 or 2000 data were initiated to reflect the change in FI experience. However, had such adjustments been implemented, some of the substance use measures that showed a small, not statistically significant, decrease may have been adjusted to show a small, not statistically significant, increase in 2000 (OAS, 2001b, p. 86 and Table B-17, p. 91).

Analysis of interviewer experience conducted in 1999 and 2000 used a two-part experience variable based on (1) NHSDA experience in a prior year and (2) order of interview in the current year (1-19, 20-39, 40-59, 60-99, 100 or more). A number of analyses were conducted using these variables to see whether the experience effect was diminishing over time. The analyses showed fewer significant interviewer experience effects in 2001 compared with 1999 or 2000, but some effect remained. A single comprehensive measure of interviewer experience was developed that focused primarily on the number of interviews completed since the introduction of CAI in 1999. Three categories were defined as follows:

• Inexperienced: 0-39 interviews since January 1, 1999 (and, for the 1999 survey only, no NHSDA experience prior to 1999);
• Experienced: 40-99 interviews since January 1, 1999 (and/or, for the 1999 survey only, some NHSDA experience prior to 1999); and
• Highly experienced: 100 or more interviews since January 1, 1999.

Based on this definition of prior experience, the distribution of interviews by interviewer experience is shown in Table C.3. The proportion of interviews conducted by highly experienced interviewers continued to grow due to year-to-year retention. The proportion of interviews conducted by inexperienced interviews declined slightly in 2001, while the proportion of interviews conducted by interviewers in the experienced (but not highly experienced) category declined by almost one half.

Because the incentive experiment FI regions were considered to have influenced reported substance use prevalence, a logistic regression analysis restricted to the non-incentive experiment areas was conducted using 1999, 2000, and 2001 data from these areas. Also examined was whether the experience effect may have diminished over the 3-year period, but no clear evidence was found to support this. Improved compliance with the prescribed study protocols might have had a positive (but not detectable) influence on reducing any interviewer experience effects in the last half of the 2001 data collection year. Using the data from all 3 years (649 non-incentive regions only), Table C.4 shows how adjustment for interviewer experience would have affected the odds ratios (ORs) for trends in reported substance use. Unadjusted ORs are based on a simple main effects model (i.e., only the variables designating the survey year) with no covariates. Adjusted ORs are based on the main effects for year-after adjustment for interviewer experience (the three levels shown above), Census region, gender, age group, race/ethnicity, population density, and gender by age interaction. As might be expected due to the continuing shift toward more highly experienced interviewers shown in Table C.3, adjustments for interviewer experience tended to increase the ORs. This general effect also was supported by some a limited number of tabled estimates produced using only data from interviews conducted by inexperienced interviewers.

The relative experience levels of FIs can vary over time in response to the demands of the survey. In addition, the impact of FI experience on the quality of the data can be subtle and thus difficult to control. The higher proportion of inexperienced interviewers in 1999 was the direct result of interviewer staff additions required by the increase in sample size by about threefold in 1999 to accommodate the large sample required for the 50-State design and a sample supplement completed using the 1998 and prior year PAPI mode. Since then, the size of the interviewing staff has stabilized and declined somewhat as the most productive interviewers have been retained, but experience has continued to accumulate resulting in a higher proportion of highly experienced interviewers (those having completed 100 or more interviews since January 1, 1999).

Adjustments in trend measures for the changes in interviewer experience distributions had the effect of increasing selected substance use estimates for 2000 relative to 1999 and for 2001 relative to 2000. However, it needs to be noted that the estimated experience effect in this model was based on an average across all 3 years and that training effects in 2001 may have resulted in significantly reducing the experience effect, especially in the second half of the year. Some of the training and supervision methods implemented in 2001 were precisely what was needed to make sure that experienced interviewers continued to follow the proper survey protocol long after their initial comprehensive training. The fact that they were successful is supported by the data showing the reduction in the percentage of short interviews discussed in Section C.6.

C.5 Changes in Response Rates

Final analytic weights are adjusted for nonresponse and calibrated to agree with Census projections for geographic and selected demographic population distributions. Unadjusted, but design-based, weights were used to examine quarterly response rates by age, gender, and population density to see whether patterns of nonresponse were changing in any systematic way. The unadjusted weights also were used to examine the quarterly weighted distributions of study respondents by gender, race, Hispanic origin, population density, marital status, education, employment status, and income and program participation. No large or unusual shifts in distributions were noted across quarters. It should also be noted that some of the variations by quarter in these distributions were, subsequently, removed by the weight calibration process.

The incentive experiment clearly showed that incentives increased response rates in 2001 among those cases receiving a $20 or $40 incentive. Increased response rates also occurred in 2000, but these were attributed to more adequate interviewer staffing, a general improvement in interviewer performance as a result of continuing interviewers accumulating experience and improving interviewing skills, retention of the interviews with successful records, and fine-tuning of training and supervisory practices. If the offering of incentives to respondents improves response and concurrently increases some prevalence measures, the reason for the increase in prevalence measures could be explained in at least two ways:

1. Persons who responded with incentives, but would not have responded without them, are different and have higher substance use than persons who would respond with or without incentives.
2. Incentives motivate (or obligate) respondents to admit to substance use that they might not have admitted without the incentive.

In the modeling work done to evaluate incentive effects discussed above, historic response rates were found consistently to be negatively related to substance use prevalence. Because the historic response rate is observed and not controlled in any experimental fashion, this relationship does not imply causation and could simply indicate that other unknown factors lead to both lower response rates and higher substance use.

To try to understand the impact that changing response rates might have on prevalence rates, the 900 FI regions were classified by three levels of historic response rates and three levels of annual change in response rate. The change in reported prevalence rates for these nine subgroups were then measured for 1999 to 2001. The historic response rate levels were as follows:

• Low: Less than 63 percent response rate in the initial year;
• Midrange: 63 to 77 percent response rate in the initial year; and
• High: Above 77 percent response rate in the initial year.

The annual changes in response rates were classified as follows:

• decrease by 5 percent or more,
• little change (less than 5 percent), and
• increase by 5 percent or more.

Twelve measures (lifetime, past year, and past month reported use of any illicit drug, any illicit drug except marijuana, marijuana, and psychotherapeutics) were studied. All 12 measures showed statistically significant increases from 2000 to 2001. Only one statistically significant change from 1999 to 2000 was detected for these same 12 measures, and it was a negative change. Table C.5 summarizes an analysis of the observed changes from 1999 to 2000 and from 2000 to 2001. Surprisingly, the largest relative increases in prevalence measures occurred in 2001 in areas where the 2000 response rate was already high and was then increased even more; in this group of FI regions, the average relative increase in the 12 substance use measures was over 47 percent compared with about 15 percent over all regions.

The pattern of change from 1999 to 2000 is less clear perhaps as a result of the several reasons for poor response that occurred in 1999, the startup year for the expanded 50-State sample design.

Although Table C.5 shows the relationship between response rates and prevalence levels for 2000 and 2001, it needs to be noted that overall response rates remained fairly constant at 68 percent. The overall implication of Table C.5 is that the increases in prevalence occurred in almost all cells without regard to historic or current response rates.

C.6 Field Interviewer Behaviors

As noted above, empirical results adjusted for respondent characteristics show that respondents interviewed by experienced interviewers report lower substance use measures than respondents interviewed by inexperienced interviewers. Mean times required to complete interviews were considered, but did not appear to be a fair measure of interviewer behavior or interviewer influence with the respondent. Given the branching patterns of the CAI instrument, it is inevitable that respondents reporting more substance use will require more time to complete the questionnaire. However, extremely short interview times might indicate some shortcuts or inappropriate prompting of the respondent. An unusually short interview was defined as one completed in 30 minutes or less for the entire questionnaire or 5.8 minutes or less for the core questions completed privately by the respondent. For this analysis, an inexperienced interviewer is defined as one who had completed 20 or fewer CAI interviews since January 1, 1999.

Comparisons of the percentage of short interviews by experience of interviewer were done quarterly for both the entire questionnaire and for the core sections. Quarterly averages are shown in Table C.6 for 1999, 2000, and 2001. Because of some changes to the questionnaire in the modular sections, annual changes in the percentage of short questionnaire times based on the full questionnaire do not accurately reflect any trend. The timing data for the core questionnaire, which remains relatively stable, does allow interpretation of annual changes. The important finding is that the difference between experienced and inexperienced interviewers declined from year to year for both the entire questionnaire and the core sections. In addition, the core questionnaire timing data show that the percentage of questionnaires with short interview times declined by a factor of about 3 for both experienced and inexperienced interviewers between 1999 and 2001. This is important because the core sections of the questionnaire are where questions are asked about substance use and recency of use. Thus, the decrease in short interviews between 2000 and 2001, especially in the core sections, could be a contributory factor to the increased prevalence rates in 2001. This would especially affect the lifetime prevalence rates because the first question always asks the respondent whether he or she has ever used the substance.

In 1999, 2000, and 2001, two comparable interviewer debriefing questions were asked:

Was it necessary for you to assist the respondent in completing the ACASI portion of this interview?

How often did this respondent let you know what his or her answers were as he or she completed the ACASI portion of the interview?

1 = None of the time-I do not know what any of the answers are.
2 = A little of the time-I know what a few of the answers are.
3 = Some of the time-I know what some of the answers are.
4 = A lot of the time-I know what a lot of the answers are.
5 = All of the time-I know what all of the answers are.

Table C.7 shows the unweighted responses given by interviewers to these questions in 1999, 2000, and 2001. The proportion of respondents receiving assistance remained fairly low in all 3 years, but was highest in 2001 at 3.50 percent. The proportion of interviews for which the interviewer knew a little to all of the answers decreased from 1999 to 2001 with the largest decrease (over 2 percent) occurring between 2000 and 2001. This decrease in the overall percentage of cases where the interviewer reported knowledge of the respondent's answers to the ACASI questions occurred in spite of the increase in the number of respondents receiving some assistance from the interviewer.

C.7 Retrospective Measures of Change in Lifetime Use

The 1999, 2000, and 2001 estimates of the number of lifetime users of marijuana and cocaine show decreases from 1999 to 2000 and unusually large increases from 2000 to 2001. Analysis of data on initiation of use suggests intermediate increases in lifetime use in both 2000 and 2001.

Better measures of change in substance use measures could be obtained with longitudinal samples. Longitudinal data permit one to identify the proportion of people who change their behavior in some way, causing the level of key estimates to increase or decrease. Another method of getting the same information is through retrospective questions that ask the respondent to report current status of substance use and compare it with his/her status of substance use some time earlier, say, a year earlier. Because of problems with memory, particularly related to times that certain behaviors may have begun or ended, the retrospective method may be difficult to implement. For lifetime use measures, it is currently possible to construct an indicator variable that specifies whether the respondent was already a lifetime user a year earlier. Respondents are asked their age at the time of first use, and, if that age is within 1 year of their current age, the respondent also is asked for the month and year of first use. This information, along with the date of the interview, can be used to determine whether the respondent first became a lifetime user during the past year. The current questionnaire does not identify the respondent's earlier status as a past year or past month user except that he/she must have been a lifetime user to qualify as a past year or past month user.

Some preliminary estimates were constructed for annual change in lifetime use status based on the retrospective data derived from current status and date of first use as described above. The methodology ignored the effects of mortality and may understate the change for older age groups where some lifetime users a year earlier are not represented in the change because of death prior to the current survey. Tables C.8 and C.9 compare estimates of change in the number of lifetime users of marijuana and cocaine based on the retrospective estimates from current year data versus differences between current estimates and estimates obtained a year earlier. For both substances, the retrospective method shows an increase in the number of lifetime users for both 2000 and 2001, with the larger increase occurring in 2001. As noted above, the annual-estimates approach shows an overall decrease from 1999 to 2000 and much larger increase from 2000 to 2001.

The increases in the numbers of lifetime users among the older age groups (35 to 49 and 50 or older) is primarily caused by lifetime users from younger cohorts aging into the higher age categories; only very small portions of the increases in these age groups are due to initiation of use during the past year by persons in these age groups. Although more initial users are found among persons aged 26 to 34, the cohort shift is much larger and actually has had the effect of reducing the number of lifetime users in this age group over the 2-year period.

C.8 Analysis Focused on First 6 Calendar Months

One of the final analyses conducted was to produce a subset of the summary tables using data from only the first 6 months of each year and only from the set of FI regions that were not involved in the incentive experiment. The first 6 months were selected to avoid any possible impact of the telephone training procedures on compliance with survey protocols initiated in July 2001. The non-incentive FI regions were chosen to exclude any direct or indirect effects of the incentive experiment. Table C.10 shows some selected comparisons with the full sample data for persons aged 18 to 25. In general, the data for the first 6 months in the non-incentive FI regions showed smaller measures of change with fewer statistically significant trend measures than those based on complete samples for both years. Some of the reduction in statistically significant findings was, of course, due to the reduction in sample size when looking at a subset of the total data. Some of the reduced change is due to limiting of the sample to the non-incentive regions and to the first half of the year when the training effect was less. However, because the change based on the first 6 months was generally only slightly smaller than for the full sample, strong evidence remained for concluding that substance use increased for many of the substances measured.

C.9 Questionnaire Change

Changes to the questionnaire in 2001 also were examined to assess whether some increases in drug use prevalence in 2001 might be attributable to the addition of new questions. However, not all increases in drug use prevalence could be attributed to questionnaire changes. In particular, the content of the sections for marijuana, cocaine, and cigarettes were exactly the same in 2000 and 2001. Thus, the increase in lifetime marijuana use in 2001 that was shown in Figure C.1 for adults aged 18 to 25 could not be explained by changes to the questionnaire.

One change to the questionnaire in 2001 was that follow-up probes were added to persuade respondents to reconsider their answers if they initially refused to indicate whether they had ever used Ecstasy (MDMA) or methamphetamine, or if they refused all questions pertaining to lifetime use of inhalants, pain relievers, tranquilizers, stimulants, or sedatives. However, no respondents who initially refused all questions about lifetime use of inhalants, pain relievers, tranquilizers, stimulants, or sedatives indicated on follow up that they had ever used these drugs. Similarly, no respondents who initially refused to answer the question about lifetime methamphetamine use indicated use on follow-up and only two respondents who initially refused the lifetime Ecstasy question indicated use on follow-up. Therefore, the significant increases in estimates of lifetime use should not be explained by the addition of these new follow-up probes in 2001.

Another important change to the questionnaire in 2001 involved the addition of new questions pertaining to the initiation and recency of use of the hallucinogen Ecstasy. As in 2000, respondents in 2001 also were asked questions about their initiation and recency of use of LSD or PCP. If respondents in 2001 reported more recent use of a specific hallucinogen (i.e., LSD, PCP, or Ecstasy) than what they reported for their recency of use of any hallucinogen, they were prompted to resolve this inconsistency in their answers. If respondents did not resolve the inconsistency (i.e., by changing their general hallucinogen recency to indicate more recent use or by changing the recency for LSD, PCP, or Ecstasy to indicate less recent use), the editing procedures that had been in place since 1999 favored the information that indicated the most recent use of a hallucinogen. Suppose, for example, that a respondent indicated use of Ecstasy in the past 30 days and indicated use of any hallucinogen more than 30 days ago but within the past 12 months. The respondent would be alerted that these two answers disagreed. If the respondent on follow-up again indicated last using Ecstasy in the past 30 days, the editing procedures logically inferred that this respondent had last used any hallucinogen in the past 30 days. Thus, the new question about recency of use of Ecstasy provided respondents an additional opportunity to indicate more recent use of any hallucinogen. The new questions about Ecstasy use also provided additional data that were not available in 2000 for use in logically editing the hallucinogen recency of use variable.

Table C.11 shows some comparisons of estimates with and without additional questions or follow-up probes. To produce the estimates without the additional questions, the data were re-edited and re-imputed without taking into account information present in these new questions. The largest changes in the estimates occurred for hallucinogens and any illicit drugs other than marijuana for persons aged 18 to 25. However, the differences in estimates of hallucinogen and any illicit drug use other than marijuana between 2000 and 2001 were still significant for this age group when the new hallucinogen questions were not taken into account.

The addition of the new hallucinogen questions in 2001 did affect some estimates of use of hallucinogens and any illicit drug except marijuana for the population aged 12 or older and for age groups other than 18 to 25 year olds. The difference in the estimate of past month use of hallucinogens among the population aged 12 or older was significant between 2000 and 2001 when the estimate for 2001 took into account the new questions but was not significant when the new questions were disregarded. Similarly, past year use of hallucinogens among adults aged 26 or older was significantly higher in 2001 when the new questions were taken into account but was not significantly different between the 2 years in the absence of these new questions. For these estimates, it is safer to conclude that some of the change in levels of estimates should be attributed to the questionnaire changes in 2001 that pertained to Ecstasy.

As substance use phenomena change, it can often become necessary to adjust the measuring instrument to reflect those changes. Changes to the questionnaire in 2001 to obtain more and better data about the use of Ecstasy were implemented in this spirit. An analysis that assumed the 2000 form of the questionnaire for both years showed that some of the increases in the estimates for hallucinogens and for any illicit drug other than marijuana were the result of the questionnaire change.

C.10 Summary

It appears safe to conclude that part of the change in substance use indicated by the 2000 and 2001 annual estimates may be a result of noncomparable data collection methodology, including the implementation of an incentive experiment in a subset of the total sample and the steps taken to ensure better compliance with the intended survey protocol. The intention of both was to obtain higher quality data. If these changes increased the level of the estimates in 2001, it is probably safe to say that any bias in the level of these estimates has been reduced, not increased. For comparison with 2001, this type of improvement in the quality of current year data concurrently creates a bias in the measures of change. The long-term solution to this problem should be to maintain the higher level of data quality in future surveys and concurrently produce quality estimates both of level and of change.

The review of interviewer experience effects (Section C.4) and the analysis of retrospective measures of lifetime use (Section C.7) both support the conclusion that the reductions in lifetime and past year marijuana use in 2000 may have been overstated and that some small increases from 1999 to 2000 were the more likely reality. The restricted comparisons of 2000 to 2001 using the non-incentive areas and the first 6 months of data (Section C.8) continue to support an increase from 2000 to 2001, but of a somewhat smaller magnitude than the complete data would indicate.

Table C.1 Sample Distribution, by Incentive Experiment Treatments

Table C.3 Distribution of Interviews, by Interviewer Experience: Unweighted and Weighted

Table C.4 Unadjusted and Adjusted Trend Odds Ratios Based on 1999, 2000, and 2001 Data from the 649 Non-Incentive FI Regions

^a Odds ratio is statistically significant at the .05 level when compared with an odds ratio of 1.00, which would indicate no change.
^b Odds ratio is statistically significant at the .01 level when compared with an odds ratio of 1.00, which would indicate no change.
N/A = Not available due to a change in the definition of dependence.

Table C.5 Changes in Prevalence Measures, by Response Rate (Historic and Change) Groups

Table C.6 Percentage of Short Interviews, by Interviewer Experience

Table C.7 Self-Reported Interviewer Behaviors: 1999-2001

Table C.8 Estimates of Change in Lifetime Use of Marijuana in Thousands of Users

Table C.9 Estimates of Change in Lifetime Use of Cocaine in Thousands of Users

Table C.10 Comparison of Full Sample Trends with Trends Based on First 6 Months for Non-Incentive Regions Only: Numbers of Users Aged 18 to 25

^a Difference between estimate and 2001 estimate is statistically significant at the .05 level.
^b Difference between estimate and 2001 estimate is statistically significant at the .01 level.

Table C.11 Percentages of Past Year and Past Month Users of Illicit Drugs with and without Additional Questions among Persons Aged 12 or Older: 2000 and 2001

* Low precision; no estimate reported.
^a Difference between this estimate for 2001 and the estimate for 2000 is statistically significant at the .05 level.
^b Difference between this estimate for 2001 and the estimate for 2000 is statistically significant at the .01 level.
¹ Any Illicit Drug includes marijuana/hashish, cocaine (including crack), heroin, hallucinogens, inhalants, or any prescription-type psychotherapeutic used nonmedically. Any Illicit Drug Other Than Marijuana includes cocaine (including crack), heroin, hallucinogens, inhalants, or any prescription-type psychotherapeutic used nonmedically.
Source: SAMHSA, Office of Applied Studies, National Household Survey on Drug Abuse, 2000 and 2001.

This page was last updated on June 16, 2008.

SAMHSA, an agency in the Department of Health and Human Services, is the Federal Government's lead agency for improving the quality and availability of substance abuse prevention, addiction treatment, and mental health services in the United States.