Appendix C: NSDUH Changes and Their Impact on Trend Measurement

C.1 Background

Although the design of the 2002 National Survey on Drug Use and Health (NSDUH) is similar to the design of the 1999 through 2001 surveys, there are important methodological differences between the 2002 NSDUH and prior surveys:

• The name of the survey was changed in 2002 from the National Household Survey on Drug Abuse (NHSDA).
• Incentive payments of $30 were given to respondents beginning in 2002.
• Improved data collection quality control procedures were introduced in the survey during 2001 and 2002.
• Population data used in NSDUH sample weighting procedures are based on the 2000 decennial census for the first time in the 2002 NSDUH.

These NSDUH changes clearly improved the quality of the data provided by the survey. In particular, there were significant improvements in response rates beginning in January 2002, which probably reduced nonresponse bias in NSDUH. The weighted interview response rate increased from 73.3 to 78.9 percent in 2002. The higher response rates had been expected, based on an experiment conducted in 2001 that showed that incentives increased response rates with no significant impact on prevalence rates. However, the results of the 2002 survey, as well as more recent analyses of data from the 2001 experiment, suggest that the incentive, and possibly the other survey changes, did have an impact on the estimates produced from the 2002 survey. Estimates of rates of substance use, dependence and abuse, and serious mental illness (SMI) were significantly higher in 2002 than in 2001. Analysis of the 2001 and 2002 data has shown that many of these "increases" could not possibly be real and are likely to be artifacts of the changes in the survey procedures. A key finding was that the increases in rates of lifetime use were inconsistent with rates of new use. For example, NHSDA data for recent years have consistently shown fewer than 3 million persons had tried marijuana for the first time each year; however, the estimated number of persons who had ever used marijuana, based on the 2002 NSDUH, was 10.5 million greater than the estimate from the 2001 survey.¹

Results of these analyses were presented to a panel of survey methodology experts, who concluded that, because of the survey improvements, 2002 estimates should not be compared with 2001 and earlier estimates. The panel also concluded that, because of the multiple changes made to the survey simultaneously, it would not be possible to measure the effects of each change or to develop a method of "adjusting" pre-2002 data to make them comparable for trend assessment. The panel also recommended that the Substance Abuse and Mental Health Services Administration (SAMHSA) continue its analyses of the 2001 and 2002 data to learn as much as possible about the impacts of each of the methodological improvements. This appendix summarizes these analyses to date.

C.2 Description of Changes

C.2.1 Name Change in 2002

In 2000, the Secretary of the Department of Health and Human Services (DHHS) requested that SAMHSA change the name of the survey to make it more representative of the topic and content of the survey. There was concern that the original name might have been misleading or threatening to potential respondents. In particular, there were three ways in which the name could have been affecting respondent attitudes and behaviors:

1. Use of the term "drug abuse" implied a focus on hard-core users of illicit drugs. This could increase refusals or suppress the recall of past use among casual users, past users, and users of legal drugs who could think their experiences were not applicable to the study.
2. The term "drug abuse" also might have been threatening to users of illicit drugs who feared investigation and arrest by law enforcement.
3. A title including only drug use fails to convey the scope and variety of general-interest topics included in the survey.

To aid in selecting a new name for the study, project staff conducted an "Evaluation of Potential Name Change" in March 2001. Supervisory, office, and interviewing staff responded to questionnaires and participated in conference calls that solicited their opinions on the new names and terminology under consideration. Following this study, SAMHSA decided on the NSDUH name.

Beginning with the 2002 survey, references to NHSDA were replaced with NSDUH in all project materials, including the computer-assisted interviewing (CAI) and screening questionnaires, informed consent documents, all supplementary respondent materials, all in-house documentation, and all project reports.

C.2.2 Incentive Payment of $30

There was a considerable decline in NHSDA response rates between 1998 and 1999 due mainly to the transition from a national probability sample to a much larger sample designed to yield State-level estimates. The increase in sample size required a large expansion in the field interviewer (FI) staff, resulting in difficulties in recruiting FIs in some areas, as well as increasing the proportion of inexperienced interviewers. Several management actions were made to improve the response rates. These actions included a reduction in interviewer caseload, an increase in the management-to-staff ratio, additional training sessions for field staff, interviewer pay adjustments, use of sample waves, site visits by field supervisors, and the use of traveling interviewers (Eyerman, Odom, Wu, & Butler, 2002). In general, the adjustments were successful, and a recovery was made from the 1999 decline. However, the rates remained below the project target rate and the historical average. Therefore, an incentive payment to respondents was considered as an option for addressing the downward trend in respondent cooperation. In an effort to understand the risks and benefits associated with a respondent incentive, an incentive experiment was conducted to compare the effectiveness of $40 and $20 incentive payments with a $0 control group. The experiment was overlaid on the NHSDA main study data collection sample and conducted during the first two quarters of 2001.

The results of the incentive experiment showed that the $20 and $40 treatments produced significantly better interview response rates than the control group (Office of Applied Studies [OAS], 2000d). Both treatments had significantly lower refusal rates than the $0 group, and the $40 treatment had significantly lower noncontact rates than the $0 group. FIs reported that the incentives reduced the amount of effort required to complete a case and that the payments influenced the respondent's decision to cooperate. Furthermore, both the $20 and the $40 treatments more than paid for themselves, each resulting in a lower data collection cost per completed case, including incentive payment, than the control.

The initial analysis of the incentive experiment showed little impact on the population estimates of past month alcohol, cigarette, or marijuana use. The prevalence rates for past month use of these substances by respondents in the treatment groups were not significantly different from those reported by respondents in the control group. Based on the results of this study, a $30 incentive payment was offered to respondents beginning with the 2002 NSDUH. The lead letter, study description, informed consent item of the screening script, interview introduction and informed consent documents, and question-and-answer brochure were altered to include the information that, at the conclusion of the CAI interview, the respondent is given the $30 incentive payment and one copy of an interview payment receipt. Information about the incentive also was added to the videos sent to managers of properties to which the interviewers could not gain access.

C.2.3 Improved Data Collection Quality Control Procedures

A series of methodological analyses was conducted in 2000 to evaluate the impact of the 1999 redesign of the survey on a variety of measures, including response rates, data quality, and prevalence estimates (Gfroerer, Eyerman, & Chromy, 2002a). Analyses revealed that the interviews completed by newer field staff yielded slightly higher but statistically significant prevalence rates than those completed by the more experienced staff (Hughes, Chromy, Giacoletti, & Odom, 2002). Anecdotal evidence suggested that the newer interviewers were following the survey protocol more closely than the veteran staff, which resulted in higher self-reports of substance use by respondents. These findings led to an increased awareness of the potential for field staff to affect the outcome of the survey. The project team responded by implementing a series of changes to the field procedures to reduce the potential for an interviewer to bias the prevalence rates. These changes included the following:

• field staff monitoring by management staff through in-person observations of the screening and interviewing process was conducted during two periods—February through early April 2001 and July through early August 2001;
• beginning in July 2001, interviewers were provided additional guidelines ("Steps to Maximize Data Quality") to reinforce and encourage compliance with study protocols (these were combined with conference calls hosted by field managers to review the contents of the guidelines);
• during October and November of 2001, a series of weekly training sessions were held with FIs to review specific topics on survey protocol;
• beginning in January 2002, new training modules were added to the veteran and new-to-project training sessions that placed additional emphasis on respondent rights, gaining cooperation, data quality, and project protocols;
• beginning in January 2002, a certification process was used to evaluate the adherence of each interviewer to the project protocols during mock screening and interview sessions; and
• beginning in January 2002, an electronic home study was completed by all veteran interviewers prior to the training session.

These changes were implemented to reduce the influence of individual interviewers on the survey process. However, it is possible that increased emphasis on protocol may have improved respondent recall and reduced underreporting of socially undesirable substance use behaviors. For example, it is possible that some veteran interviewers were intentionally or unintentionally shortening the interview by placing less emphasis on survey instructions and confidentiality statements. This may have reduced the effectiveness of recall tools, such as the reference calendar, and increased respondent concerns about reporting sensitive issues. The increased emphasis on protocol should have improved the performance of interviewers and resulted in a higher prevalence rate closer to the true value in the population.

It should be noted that the first three of the six changes listed above were introduced at different times during 2001 and were analyzed as temporal interventions. The final three changes were implemented at the beginning of the 2002 survey year, along with the introduction of the $30 incentive payment and the new name of the survey. As a result, it is very difficult to isolate the impact of these changes.

C.2.4 Use of 2000 Census-Based Control Totals

Estimates of substance use from NSDUH depend on person-level weights calibrated to match population projections based on the latest census data available. The 1999, 2000, and 2001 surveys provide estimates of substance use that depend on weights based on projections from the 1990 decennial census. The 2002 survey used projections from the 2000 census data to calculate the appropriate weights. It is possible that changes in the demographic distribution in the 2000 census may have yielded an annual change in the substance use estimates. If so, this change would be an artifact of the weighting procedures and not a true change in the number of substance users in 2002.

C.3 Analyses of the Impact of Changes

The analyses of the impact of the methodological changes began in 2002 after preliminary runs from the 2002 NSDUH indicated significant improvements in response rates and unexpected shifts in substance use prevalence rates. Of particular concern were large increases in rates of lifetime use of marijuana, cocaine, and other substances. In addition, subsequent extended analysis of the incentive experiment data indicated that the prevalence estimates may have been influenced by the use of the incentive (Wright, Bowman, Butler, & Eyerman, 2002).

Early analyses were presented to the panel of survey design experts convened on September 12, 2002. The panel recommended a series of additional analyses to inform an understanding of the methods' effects. Based on the panel's recommendations, an analysis plan was developed to assess the impact of the methodological changes on the reported prevalence rates. Preliminary results from five of these analyses are presented in this appendix:

• retrospective cohort analysis;
• response rate pattern analysis;
• response rate impact analysis;
• analysis of the impact of new census data; and
• model-based analysis of protocol changes, name change, and incentives.

The purpose of the retrospective cohort analysis was to evaluate the changes in the estimates of lifetime use reported in the 2002 survey. As stated above, estimates of lifetime use were noticeably higher than in past years and were inconsistent with rates of new use. This analysis provides a more detailed evaluation the change by comparing lifetime prevalence estimates with the retrospectively reported lifetime use estimates.

The purpose of the response rate pattern analysis was to assess the impact of the methodological changes on the response rates of different demographic subpopulations. A series of tables was generated to assess screening and interview response rate changes associated with various geographic and demographic characteristics, reasons for refusal, and FI characteristics.

The purpose of the response rate impact analysis was to assess the potential levels of substance use prevalence under different assumed scenarios about the behavior of the respondents "added" as a result of the higher response rates under the new methodological conditions. It is possible that the increase in 2002 was due to "added" respondents in 2002 that have high rates of drug use.

The purpose of the analysis of the impact of new census data is to determine if any part of the increases in substance use observed in 2002 is due to the transition from 1990 census data to 2000 census data for weight calculations. This analysis examines the effect of the two censuses on estimates of lifetime use and the impact on trend analysis. It includes an assessment of the changing demographic base from 1990 to 2000 and an examination of lifetime use of any illicit drug, marijuana, cocaine, cigarettes, and alcohol.

The purpose of the model-based analysis is to improve an understanding of how much each of the methodological changes that can be measured might be influencing the comparisons of 2001 and 2002 data. This analysis attempts to control for as many of the confounding influences as possible in order to isolate the impact of each of the methododological changes on the substance use estimates.

C.3.1 Retrospective Cohort Analysis

Changes in the number of lifetime users of specific drugs were examined based on direct survey estimates, as well as the numbers of new users occurring within the preceding year. The 2001 and 2002 direct survey estimates of the number of lifetime users of marijuana and cocaine showed unusually large increases from 2001 to 2002, but analysis of data on initiation of use within each survey year suggests increases in lifetime use that are more stable (Table C.1).

Based on the current questionnaire, it is possible to construct a retrospective measure that specifies whether the respondent was already a lifetime user a year earlier. Respondents are asked their age at the time of first use, and if that age is within 1 year of their current age, the respondent also is asked for the month and year of first use. This information, along with the date of the interview, can be used to determine whether the respondent first became a lifetime user during the past year. The questionnaire does not identify the respondent's earlier status as a past year or past month user except that he or she must have been a lifetime user to qualify as a past year or past month user.

The retrospective approach gives insight into the nature of the change in lifetime use for different age groups. It accounts for how much of the change is due to initiation of use in the past year, and how much is due to a cohort shift out of lower and into higher age groups in the past year (see Table C.1). The data show that most new users occurred in the 12 to 17 and 18 to 25 age groups, but this is masked in the 12 to 17 age group by the large negative cohort shift (i.e., large numbers who turned 18 and small numbers who turned 12 in the past year). By contrast, in the 26 or older age group, most of the change in lifetime use was driven by cohort shifts, and this is most noticeable in the 50 or older age group. This analysis demonstrates that the increases in lifetime substance use rates in 2002 could not be due to an increase in new initiates or cohort shift. From this it can be concluded that the 2002 data are not comparable with data from previous surveys.

C.3.2 Response Rate Pattern Analysis

Table C.2 provides the quarterly distribution of screening and interviewing results from 2000 to 2002. The screening response rate (labeled "complete" in the table) was largely unaffected by the methodological changes. The screening response rate was around 93 percent in 2000, 92 percent in 2001, and 91 percent in 2002. The decline in the screening response rate was unexpected and probably was a result of the unfamiliarity of the field staff with the new procedures. The rate was lowest in the first quarter and rose consistently throughout the year (e.g., from 89.8, 90.5, 91.2, to 91.4 percent in 2002). The interview response rate, on the other hand, showed its largest increase between quarter 4 of 2001 and quarter 1 of 2002 (71.97 and 80.78 percent, respectively). Interview response rates remained high for the remainder of the 2002 survey year.

The increase in response rate between quarter 4 of 2001 and quarter 1 of 2002 was consistent across geographic and demographic subgroups (see Table C.3). The Northeast and West regions showed the largest increases-9.88 and 9.10 percent, respectively. Interview response rates increased across population densities, with metropolitan statistical areas (MSAs) having fewer than 1 million people showing the largest effect (10.24 percentage points higher in quarter 1 of 2002 than in quarter 4 of 2001). Finally, the most affected demographic groups were the younger age groups (increases of 10.06 and 13.67 percent, respectively, for youths aged 12 to 17 and persons aged 18 to 25) and non-Hispanic blacks (+9.95 percent). Males and females responded approximately the same to the methodological changes. However, the gains for the older age groups in quarter 1 of 2003 attenuated as the year progressed. The 50 or older age group initially realized gains of about 7.6 percent in quarter 1 of 2002, but dropped down nearly to the 2001 level by quarter 4 of 2002. The improvement in response rates was less among the older adults than the other age groups. The response rate only improved from 69.92 to 71.54 percent for the 50 or older age group (see Table C.4). This is due in part to the change in the pair selection algorithm in 2002 that increased the pairs selected in this age group.

Table C.4 provides interview response rates by selected age group pairs. This table suggests that the incentive may have attenuated or reversed the negative impact of the pair sampling on response rates that was observed in prior surveys. The increase from 2001 to 2002 was seen among all age group combinations. Table C.5 and Table C.6 provide breakdowns of screening and interview results by final disposition. The screening refusal rate increased between 2001 and 2002 (from 4.93 to 5.86 percent, respectively). "Nothing in it for me" was the most popular reason for refusing the screener and the interview across all 3 years; however, this reason for refusal decreased among interview respondents from 2001 to 2002 (7.06 to 5.52 percent).

Finally, screening and interview response rates were examined with respect to interviewer characteristics. In 2001, interviewers with some experience achieved a 73.80 percent interview response rate, while those with no experience achieved 70.57 percent rate. In 2002, these figures were 78.78 and 76.39 percent for experienced and inexperienced interviewers, respectively. The gap between experienced and inexperienced interviewer performance in obtaining interviews at screened households closed by nearly 1 percentage point.

In conclusion, the methodological changes implemented in 2002 were accompanied by substantial increases in interview response rates. These increases were seen across all geographic and demographic subgroups and for both experienced and inexperienced interviewers. Screening response rates, however, were largely unaffected. Despite the increase in interview response rates, the weighted distribution of respondents across population densities and demographic characteristics stayed the same (see Table C.12).

C.3.3 Response Rate Impact Analysis

Prevalence estimates of most substance use measures showed increases from the 2001 to 2002 surveys across all age categories. There also was a corresponding increase in response rates across all age categories. The increased response rates may be due to several reasons, such as the name change from NHSDA to NSDUH, or the application of a $30 incentive to all respondents from January 2002. The objective of this investigation is to see the extent to which the higher prevalence rates may be attributed to the increase in response rates and associated reporting of higher prevalence levels among new respondents who participated because of the survey changes as opposed to increased reporting among respondents who would have participated without the survey changes but now admit to more substance use because of the change.

The weighted overall response rates for 2001 and 2002 for different age categories are given in Table C.7 to Table C.11. This investigation focused on how the "additional" respondents in 2002 may have affected the prevalence rates, by calculating the "marginal" prevalence rates of those additional respondents. Following the method of Deming (1953), the 2002 marginal prevalence rates of the additional respondents are computed (in percentages) as follows:

     D

where and , respectively, are the weighted overall response rates for 2001 and 2002, and and , respectively, are the 2001 and 2002 reported prevalence rates in percentages. If falls outside the interval [0, 100], then the change in reported prevalence rates from 2001 to 2002 is due to factors other than simply the increase in response rates. This calculation of the marginal prevalence rate assumes that all of the difference between and is due to the "new" responders. An implication of the marginal prevalence rate falling between 0 and 100 is that it is possible that the higher (or lower) prevalence rate in 2002 could be due to the marginal respondents if it iswe believed that group could have reported the marginal prevalence rate .

Table C.7 shows the 2001, 2002, and marginal prevalence estimates for various substance use and mental health measures among respondents aged 12 or older. The measures include lifetime, past year, and past month use of any illicit drug, marijuana, cocaine, cigarettes, and alcohol; illicit drug dependence or abuse; illicit drug specialty treatment; alcohol or drug dependence or abuse; serious mental illness (SMI); and perceived risk of using marijuana once a month. Table C.8 to Table C.11 show the same information for the remaining age categories.

For the 12 or older age category in Table C.7, marginal estimates of lifetime use for all substances except cocaine exceed 100. This means that, except for cocaine, increases in lifetime use cannot be fully explained by the increase in response rates. Marginal estimates of measures other than lifetime use exceed 100 only twice (past year and past month use of alcohol), but these estimates are typically much larger than the 2001 or 2002 estimates, suggesting that the annual increases also are due to factors beyond the increased response rates. For one of the measures (perceived risk of using marijuana), the marginal estimate is negative, meaning that the corresponding annual decrease cannot be attributed to the increased response rate alone.

For the 12 to 17 age group in Table C.8, the marginal estimates of most measures are usually higher than the 2001 and 2002 estimates, but they are within the plausible range of 0 to 100. This is due to a combination of the factors that the increases in prevalence rates were relatively smaller and the increases in response rates were relatively larger in this age group. The marginal estimate of one measure (perceived risk of using marijuana) is negative.

These results must be interpreted with caution because of the underlying assumption that all of the difference between the 2001 and 2002 prevalence rates was due to methodological effects. Of course, there may have been true increases or decreases between those years, implying that the actual estimates of marginal rates made above could be somewhat smaller or larger in reality. This problem can be illustrated with two examples where the true level of change is better known. For adults aged 50 or older, the change for lifetime use of marijuana between 2001 (17.3 percent) and 2002 (21.8 percent) was 4.5 percent. The marginal rate was estimated to be 453.2, under the assumption that all of the increase was due to the "added" respondents. However, the prevalence of lifetime marijuana use in this age group is believed to be increasing by approximately 2 percent a year simply because of the aging of cohorts with high rates of lifetime use into this age group, estimated as follows. About half of the persons aged 49 in 2001 had used marijuana in their lifetime. This cohort entered the 50 or older age group in 2002, adding about 4 million persons and 2 million lifetime users to this age group. Approximately 2 million deaths per year occur in this age group, but only about 5 percent would have ever used marijuana, based on age-specific prevalence rates and death rates. Thus, the rate of lifetime marijuana use in this age group would be expected to rise from about 17.3 percent in 2001 to about 19.3 percent in 2002. Assuming that about 2 percent of the 4.5 percent change in prevalence between 2001 and 2002 was the "true" trend, a revised marginal prevalence for "additional" respondents aged 50 or older would be calculated by assuming = 19.3. This results in a revised marginal rate of 261.5—still indicating that the additional respondents in 2002 could not have accounted for all of the prevalence increase.

Another example is the estimated rate of past month cigarette use among youths aged 12 to 17. For 2001 and 2002, the NHSDA/NSDUH estimates of past month cigarette use among youths were 12.9 and 13.0 percent, respectively. However, results from other youth surveys indicate that youth cigarette use has been declining for a number of years. For example, the Monitoring the Future data (Johnston, O'Malley, & Bachman, 2003a, 2003b) show a continued decrease in past month use of cigarettes among youths from 1997 through 2002. Between 2001 and 2002, the rate of decline in cigarette use among 8^th and 10^th graders (see Table 10.1 in Chapter 10 of this report) was about 15 percent (from 16.8 to 14.2 percent). Assuming a "true" rate of decline of 15 percent among youths aged 12 to 17 (i.e., assume = 11.0), the resulting marginal prevalence rate is 47.3 instead of the 14.7 that results from assuming no change from 2001 to 2002 (see Table C.8).

Despite this limitation, the analysis provides results that are useful in determining the extent to which the higher response rate achieved in the 2002 survey was the cause of the higher prevalence estimates observed in 2002. In general, it appears unlikely that the additional respondents participating in 2002 could be solely responsible for all of the higher prevalence rates, especially among older adults. The methodological changes introduced in 2002, particularly the incentives and name change, apparently had a variable effect on the reporting of substance use and other measures among a large part of the sample.

This conclusion is further supported by Table C.12 in which the weighted distribution of the sample is given for 1999 through 2002. The weighted sample distribution for such factors as gender, race, and Hispanic origin is controlled through the weight calibration process and shows some changes over time reflecting the census projections used in the weight calibration process. The weighted distributions for other factors, such as population density, marital status, education, employment, and income, are not directly controlled by the weight calibration process. Similar moderate changes over time are shown in Table C.12 for these variables, supporting the conclusion that the "additional" respondents in 2002 did not significantly change the weighted distribution of the sample on these selected demographic factors and should, therefore, have had little impact on substance use estimates that are correlated with these demographic factors.

C.3.4 Impact of New Census Data

This analysis examined the extent to which demographic shifts between the two censuses, not accounted for in intercensal projections, may have on estimates of substance use. Estimates of lifetime and past month use of five key measures (any illicit drug, marijuana, cocaine, cigarettes, and alcohol) were examined for 2001 and 2002 in terms of both counts and rates. The direct estimate of the census effect, available from the 2001 survey, also was examined.

Between 1990 and 2000, there is clear evidence of a relative surge in the Hispanic population. There is also clear evidence of a population "bulge" at ages 25 to 34 in 1990 that shifted a decade in 2000. These changes beg several questions: For example, how well do intercensal population projections based on the 1990 census capture these demographic changes? And if the intercensal projections fail to adequately capture those changes, what effects do those errors have on survey estimates?

Annual estimates of lifetime and past month use (counts and rates) of five key measures (any illicit drug, marijuana, cocaine, cigarettes, and alcohol) were examined for 2001 and 2002. The 2001 estimates were based on 1990 census-derived weights, and the 2002 estimates were based on 2000 census-derived weights. The 2001 estimates also were recalculated using the 2000 census data, giving a direct estimate of the census effect in 2001.² Survey estimates of lifetime and past month use counts and rates for the five measures from 2001 and 2002 are given in Table C.13.

In terms of lifetime use, there is evidence of a positive census effect in all five measures in 2001, indicating that the 2001 estimates based on 1990 census projections may have been underestimated. This effect is small but discernible for any illicit drug, marijuana, and cocaine. However, for cigarettes and alcohol, the effect is fairly large relative to the sizes of the annual increases. Similar conclusions can be drawn for the past month use estimates.

Table C.13 also shows that the census effect all but disappears for estimates of lifetime and past month use rates. This is not surprising because the prevalence rate is calculated as the lifetime or past month use count divided by the total population count. Moreover, because both the numerator and denominator in a ratio of this kind are estimates based on weights derived from the same intracensal population projections, one might expect the census effect to largely cancel out.

The census effect for different race categories in terms of counts and rates is shown in Table C.14 by comparing 2001 lifetime and past month use estimates using weights based on projections from both censuses in question. The census effect in lifetime use counts is largest among Hispanics for all five measures. For all other race categories, the effect is small or even negative in some cases. On the other hand, the census effect in lifetime use rates among Hispanics has largely disappeared. There also is a small negative census effect among whites for all five measures. Estimates of past month use show analogous results.

The census effect for different age categories in terms of counts and rates is shown in Table C.15 by comparing 2001 lifetime and past month use estimates using weights based on projections from both censuses in question. The census effect in lifetime use counts is largest in the 26 to 34 age group for all measures. For all other age categories, the effect is small or even negative in some cases, except for cigarettes and tobacco, where the census effect among the 50 or older age group is relatively large. Contrast this with the negative census effect in the 50 or older age group for the other three measures. The census effect in lifetime use rates has been significantly reduced for all age categories and measures. Similar conclusions hold for past month use estimates.

C.3.5 Model-Based Analysis

To focus on the impact of methodological changes on reported substance use, it was necessary to focus on subsets of the data. These data subsets were used to obtain comparable measures taken before and after implementation of some methodological intervention or to limit the analysis to experimental comparisons involving only a portion of the total sample. To help control for factors unrelated to the methodological interventions, model-based analytic procedures were used. Model-based analysis of the effects of interviewer monitoring and training interventions and of the combined effects of incentives and name change are discussed below.

C.3.5.1 Impact of Interviewer Monitoring and Training

As described earlier, a number of interviewer monitoring and training interventions were implemented during the 2001 survey. Special analyses were conducted to assess the potential impact of these interventions.

The first intervention involved nonrandom field interview observations carried out during two periods: February through early April and July through early August 2001. The purpose of the interview was to note any deviations from the specified protocol in screening or interviewing and provide feedback to the FI. To assess the impact of this intervention, the observed interview and any subsequent interviews conducted by the observed interviewer were considered to have received this intervention. Because not all interviewers were observed, three levels of the treatment variable were defined:

• interviewer not observed during 2001;
• interviewer observed during 2001, with interview conducted before any observed interview; and
• interviewer observed during 2001, with interview observed or completed after first observed interview.

The key comparison for evaluating the impact of observation was based on the odds ratio for the third treatment to second treatment measures (i.e., the effect of observation among those interviewers who were observed at least once during 2001).

Because the observation process took place over an extended period, the entire 2001 sample was used to evaluate this intervention. An assumption made in the analysis was that there was no consistently increasing or decreasing trend during this period that would result in consistently higher or lower prevalence estimates unrelated to having been observed. A logistic regression model was used to compute adjusted odds ratios within six age groups (12 to 17, 18 to 25, 26 to 34, 25 to 49, 50 to 64, and 65 or older). Because the comparison groups were not randomly selected, additional covariates were included in the model to reduce nonrandom noise: race/ethnicity (white, black, Hispanic, and other), population density (MSAs with 1 million or more population, smaller MSAs, and other), gender, interviewer experience subsequent to January 1, 1999 (0 to 39 interviews completed, 40 to 100 interviews completed, and more than 100 interviews completed), historic response rate (three levels: greater than or equal to 75 percent, greater than or equal to 57 percent but less than 75 percent, and less than 57 percent), and historic marijuana use (five levels). The odds ratios shown in Table C.16 are based on the population-weighted average effects over the six age groups. Five lifetime use measures (cigarettes, alcohol, any illicit drug, marijuana, and cocaine) were studied with three showing odds ratios greater than or equal to 1.00 and two showing odd ratios less than 1.00. Preliminary analysis shows that the 35 to 49 age group had the only significant change, a decrease in lifetime alcohol use from 91.9 percent before observation to 87.2 percent after observation. None of the other results was statistically significant either for the individual age groups or for the combined 12 or older age group.

The second intervention (labeled "initial special training" in Table C.16) involved the development and distribution of additional guidelines reinforcing compliance with survey protocol and followed by a short telephone conference with each interviewer. For analysis purposes, the treatment was defined as having occurred on July 5^th, and all interviews occurring on or after July 5^th were considered to have been treated by this intervention. To avoid any seasonal impacts or the aftermath of September 11^th, the sample analyzed for this intervention was limited to quarters 2 and 3.³ The logistic modeling approach applied to the impact of field observations was repeated for this intervention. Odds ratios for comparing the after with the before intervention measures are shown in the middle rows of Table C.16. Only lifetime marijuana use among persons aged 65 or older displayed a statistically significant change, from 2.2 percent before the guidelines to 4.9 percent after the guidelines. For all persons aged 12 or older, three odds ratios were less than 1.00 and two were greater than 1.00; none was statistically significant.

The third 2001 intervention was carried out during October and November as a series of weekly training sessions reviewing prepared topical guidelines that further reinforced compliance with prescribed survey procedures. Because this intervention accumulated during quarter 4, the intervention effect was assessed by comparing interviews conducted early in the quarter with those conducted later in the quarter. Because half the interviews had been completed by October 23^rd, this date was chosen to separate the earlier from the later interviews. The same logistic regression modeling approach used for the earlier interventions was used here to measure and assess the impact of this intervention. The results for all persons aged 12 or older are shown in the lower third of Table C.16. For the five measures studied, all of the odds ratios were 1.00 or greater, most were close to 1.00, and none was statistically significant.

Based on these analyses, it appears that the three observation and training interventions implemented during 2001 had little appreciable impact on the reporting of lifetime substance use among persons aged 12 or older. Even though the treatment effects were small in each case and not significant, it is possible that the combined effect could be statistically significant if an expanded model were used.

The logistic regression was used to control for the nonrandom nature of the interventions; however, few statistically significant effects have been found so far. It is important to note that further analysis is ongoing and other models that included added variables to control for nonrandom noise could result in more statistically significant differences in the future.

C.3.5.2 Analysis of the Incentive and Name Change

Two methodological interventions that occurred at the beginning of the 2002 survey year (January 1, 2002) were (a) implementation of a $30 per completed interview incentive plan for all respondents, and (b) the name change from NHSDA to NSDUH. Although the potential impact of the name change had been widely discussed and evaluated in focus groups, it had not been evaluated experimentally. It was hoped that eliminating the word "abuse" from the title and adding the word "health" would result in an introduction to the survey that was less threatening, thus encouraging higher response rates and more honest reporting.

In contrast to the other changes implemented on January 1, 2002, the potential impact of offering respondent incentives was carefully evaluated in a three-level field experiment that was embedded in the 2001 survey during quarters 1 and 2. The experimental levels were (a) no incentive, (b) $20 for a completed interview, and (c) $40 for a completed interview. The initial analysis of these data showed a very positive impact on response rates, but little impact on reported substance use (OAS, 2002d). Wright et al. (2002) in subsequent analysis showed some moderate effects of incentives (when combining the $20 and $40 treatments), but not all in the same direction. Some of the same data are reevaluated (again combining the $20 and $40 treatment levels) in this appendix as a means of correcting for the incentive effect when looking at changes from 2001 to 2002.

As discussed earlier, initial analyses have indicated little, or no, impact of the three interviewer monitoring and training interventions on reported drug use. Only one of them (field observations) was implemented during the first two quarters of 2001, and it involved only a small number of interviewers. Because the incentive experiment was carefully designed with random assignment of the treatments, any impact of the field observation was equally likely to affect the three experimental treatments. If one assumes that the average effect of a $20 or $40 incentive would be roughly equivalent to the effect of a $30 incentive, the 2001 incentive experiment provides an appropriate basis for evaluating the impact of incentives and their contribution to the changes in lifetime substance use observed as of January 1, 2002. To further control for sample variation within treatment levels, response-propensity-adjusted weights were used to adjust for the higher response rates obtained by the $20 and $40 incentives. These weights then were poststratified to the 2001 age-by-gender control totals within each of the two treatment levels. The unadjusted comparisons are based on direct estimates and are presented in Table C.17. To control for additional variables that might vary across experimental incentive treatments, the data also were analyzed using a logistic regression model that incorporated race (four levels), gender, interviewer experience (three levels), number of persons selected, population density, the incentive treatment, and six age groups. Predicted marginal means were produced by incentive treatment and age group. Incentive treatment contrasts were computed for all persons aged 12 or older and for each age group (RTI, 2001, pp. 665–667). The adjusted measures are shown in Table C.18.

The analysis of lifetime use measures for cigarettes, alcohol, any illicit drug, marijuana, and cocaine presented in Table C.17 and Table C.18 also found a few statistically significant differences in the reporting of lifetime use that could be attributed to the use of incentives. The unadjusted effects revealed no statistically significant incentive effects for persons 12 or older treated as a combined group, but statistically significant effects for cigarettes and cocaine for persons aged 35 to 49 and for any illicit drug, marijuana, and cocaine in the adjusted results for persons aged 35 to 39. As is clear from the tables, both the unadjusted and adjusted differences indicate significantly different patterns in the 2001 incentive effects on lifetime use depending on the age group and, to a lesser degree, on the substance. Because the age group effects switch from positive to negative, it is not surprising that the 12 or older age groups show incentive effects that are rather small and nonsignificant. One might expect that a monetary incentive should produce similar effects across all age groups. The adjusted measures have been adjusted mostly for demographic differences in the samples compared and, therefore, would seemingly be the stronger measure of the incentive effect. However, even for the adjusted incentive effects, the results appear unstable, perhaps reflecting an overall sample size in the experiment that was too small within age groups to detect relatively small differences or a need to add more variables in the model to control for more of the "nonrandom" noise.

A second approximation of the incentive effect combined with any effect of the name change and other training effects occurring as of January 1, 2002, can be approximated by limiting the data to the quarters immediately surrounding that date: quarter 4 of 2001 and quarter 1 of 2002. At most, only small temporal trends would be expected over any 6–month period, particularly in lifetime substance use. In addition, two of the three 2001 interventions were implemented before the 2001 survey's quarter 4 began. The third intervention was spread over a period of several weeks with consecutive weekly telephone training sessions implemented during October and early November 2001. Note that at the beginning of quarter 1 of 2002, all veteran interviewers attended centralized refresher training. For purposes of this analysis, the assumption of current training and training reinforcement at nearly comparable levels appears tenable for interviews conducted over this 6–month period.

Table C.17 and Table C.18 show the overall January 1, 2002, effect with this measure. All five substances showed statistically significant effects for the 12 or older estimates and most within-age-group comparisons. The model-adjusted results were somewhat stronger than the unadjusted effects.

Given a fairly small sampling error in the age 12 or older results from the 2001 incentive experiment, these estimates appear more comparable with the January 1, 2002, effect with the exception of the small negative result for alcohol. For cigarettes, any illicit drug, marijuana, and cocaine, the lifetime prevalence rates for the incentive groups are slightly larger for the January 1, 2002, estimates, perhaps indicating some effect beyond the incentive. For the individual age groups, the two sets of estimates do not appear to be comparable.

C.4 Summary of Analyses

Analyses to date of the impact of the methodological improvements made in the 2002 NSDUH have provided some important insights, but a number of questions still remain.

Comparisons of the changes in lifetime prevalence with trends based on retrospective reporting demonstrate that the increases in lifetime substance use rates are not due to an increase in new initiates or a cohort shift; therefore, the 2002 data are not comparable with data from previous surveys. Concurrent with the upward shift in prevalence in 2002, there were substantial increases in interview response rates across all geographic and demographic groups. One group that experienced less of an increase in response rate was the population aged 50 or older. Analysis of the connection between the response rate increases and the prevalence increases showed that the "additional" respondents in 2002 did not solely account for the increases in prevalence, indicating that the changes in methods did affect the level of reporting of some behaviors among survey respondents. This finding was strongest in the 50 or older age group, where the increase in the response rate was small but the increase in prevalence was large.

The effect of the switch from the 1990 to the 2000 census-based weights was very small for NSDUH estimates of rates, but somewhat larger for some estimates of the number of persons using substances. Unlike the other changes implemented in 2002, the impact on the results can be precisely estimated subject to the sampling error of the data and thus do not adversely affect the ability to measure trends by themselves.

Model-based analyses were used to attempt to identify and quantify the impact on the prevalence of each of the separate NSDUH methodological improvements. Results are inconclusive at this point. The impact of each of the interviewer monitoring and training interventions in 2001 and 2002 appears to be small. More analysis will be pursued in this area by adding more predictor variables to models and by developing a single combined analysis of all three interventions to determine whether the combined effects of the interventions might be significant.

A comparison of the estimated incentive effects from the 2001 incentive experiment to the increases in prevalences between quarter 4 of 2001 and quarter 1 of 2002 shows that there is too much noise in the results of the experiment to draw conclusions from it. The results of the experiment appear to be most stable for the combined 12 or older age group because the sample is largest there. Those results mostly agree with the results of the effects of the introduction of the $30 incentive in 2002, except in the case of alcohol. Because the 2002 effects are larger than those from the 2001 experiment in the 12 or older age group, those results also may reflect other changes that occurred at the same time, such as the name change, further training of field staff, or seasonal or secular trends. Further analyses are planned.

Table C.1  Estimates of Change in Thousands of Lifetime Users

¹ First use earlier than within past year.
² Former users who aged into age group within past year.
³ Former users who aged out of age group within past year.
⁴ Former users who aged in minus those who aged out within past year.
⁵ First use within past year.
⁶ New users plus cohort shift.

Table C.2  2000–2002 NSDUH Quarterly Distribution of Weighted Screening and Interview Disposition

Table C.3  2000–2002 NSDUH Quarterly Weighted Interview Response Rates, by Region, Population Density, Age, Gender, and Race/Ethnicity

MSA = metropolitan statistical area.

Table C.4  2001–2002 NSDUH Weighted Interview Response Rates, by Age Group and Number of Persons Selected Per Household

Table C.5  2000–2002 NSDUH Weighted Final Disposition of Eligible Screenings

Table C.6  2000–2002 NSDUH Weighted Final Disposition of Eligible Interviews

Table C.7  Prevalence and Marginal Substance Use, Dependence and Abuse, and Serious Mental Illness Rates among Persons Aged 12 or Older, by Year

Table C.8  Prevalence and Marginal Substance Use, Dependence and Abuse, and Serious Mental Illness Rates among Youths Aged 12 to 17, by Year

Table C.9  Prevalence and Marginal Substance Use, Dependence and Abuse, and Serious Mental Illness Rates among Persons Aged 18 to 25, by Year

Table C.10  Prevalence and Marginal Substance Use, Dependence and Abuse, and Serious Mental Illness Rates among Persons Aged 26 to 49, by Year

Table C.11  Prevalence and Marginal Substance Use, Dependence and Abuse, and Serious Mental Illness Rates among Persons Aged 50 or Older, by Year

Table C.12  1999–2002 NSDUH Weighted Population Distributions

Table C.13  Estimates of Lifetime and Past Month Use Counts and
Rates of Any Illicit Drug, Marijuana, Cocaine, Cigarettes,
and Alcohol

Table C.14  Census Effect, by Race Category: 2001 Estimates of Lifetime and Past Month Use Counts and Rates of Any Illicit Drug, Marijuana, Cocaine, Cigarettes, and Alcohol Based on 1990 Versus 2000 Census Projections

Table C.15  Census Effect, by Age Category: 2001 Estimates of Lifetime and Past Month Use Counts and Rates of Any Illicit Drug, Marijuana, Cocaine, Cigarettes, and Alcohol Based on 1990 Versus 2000 Census Projections

Table C.16  Adjusted Odds Ratios for Reported Lifetime Use for Three 2001 Field Interventions

Table C.17  Unadjusted Measures of the Incentive Effect and the January 1, 2002, Effect

Table C.18  Adjusted Measures of the Incentive Effect and the January 1, 2002, Effect