Go to the Table Of ContentsSkip To Content
Click for DHHS Home Page
Click for the SAMHSA Home Page
Click for the OAS Drug Abuse Statistics Home Page
Click for What's New
Click for Recent Reports and HighlightsClick for Information by Topic Click for OAS Data Systems and more Pubs Click for Data on Specific Drugs of Use Click for Short Reports and Facts Click for Frequently Asked Questions Click for Publications Click to send OAS Comments, Questions and Requests Click for OAS Home Page Click for Substance Abuse and Mental Health Services Administration Home Page Click to Search Our Site

Worker Substance Use and Workplace Policies and Programs

Appendix B: Statistical Methods and Measurement

B.1 Target Population

An important limitation of estimates of drug use prevalence from the National Survey on Drug Use and Health (NSDUH) is that they are only designed to describe the target population of the survey—the civilian, noninstitutionalized population aged 12 or older. Although this population includes almost 98 percent of the total U.S. population aged 12 or older, it excludes some important and unique subpopulations who may have very different drug use patterns. For example, the survey excludes active military personnel, who have been shown to have significantly lower rates of illicit drug use. Also, persons living in institutional group quarters, such as prisons and residential drug use treatment centers, are not included in NSDUH, yet they have been shown in other surveys to have higher rates of illicit drug use. Also excluded are homeless persons not living in a shelter on the survey date; they are another population shown to have higher than average rates of illicit drug use. Since this report is largely focused on the U.S population aged 18 to 64 who were employed full time in the past year, the exclusion of the aforementioned subpopulations has minimal impact. Members of these subgroups are typically not present in the general U.S. workforce.

B.2 Sampling Error and Statistical Significance

The national estimates, along with the associated variance components, were computed using a multiprocedure package, SUDAAN® Software for Statistical Analysis of Correlated Data. SUDAAN was designed for the statistical analysis of data collected using stratified, multistage cluster sampling designs, as well as other observational and experimental studies involving repeated measures or studies subject to cluster correlation effects (RTI International, 2004). The final, nonresponse-adjusted, and poststratified analysis weights were used in SUDAAN to compute unbiased design-based drug use estimates.

The sampling error (i.e., the standard error [SE]) of an estimate is the error caused by the selection of a sample instead of conducting a census of the population. The sampling error may be reduced by selecting a large sample and/or by using efficient sample design and estimation strategies, such as stratification, optimal allocation, and ratio estimation.

With the use of probability sampling methods in NSDUH, it is possible to develop estimates of sampling error from the survey data. These estimates have been calculated in SUDAAN for all estimates presented in this report using a Taylor series linearization approach that takes into account the effects of the complex NSDUH design features. The sampling errors are used to identify unreliable estimates and to test for the statistical significance of differences between estimates.

B.2.1 Variance Estimation for Totals

Estimates of means or proportions, image representing p hatd, such as drug use prevalence estimates for a domain d, can be expressed as a ratio estimate

Equation B-1,     D

where image representing y hatd is a linear statistic estimating number of substance users in the domain and image representing n hatd is a linear statistic estimating the total number of persons in domain d (both users and nonusers). The SUDAAN software used to develop estimates and their SEs produces direct estimates of image representing y hatd and image representing n hatd and their SEs. The SUDAAN application also uses a Taylor series approximation method to estimate the SEs of the ratio estimate image representing p hatd.

When the domain size, image representing n hatd, is free of sampling error, an appropriate estimate of the SE for the total number of users is

Equation B-2,     D

This approach is theoretically correct when the domain size estimates, image representing n hatd, are among those forced to match their respective U.S. Bureau of the Census population projections through the weight calibration process (Chen et al., 2005). In these cases, image representing n hatd is not subject to sampling error. For a more detailed explanation of the weight calibration process, see Section A.3.2 in Appendix A.

For estimated domain totals, image representing y hatd, where image representing n hatd is not fixed (i.e., where domain size estimates are not forced to match the U.S. Bureau of the Census population projections), this formulation may still provide a good approximation if it can be assumed that the sampling variation in image representing n hatd is negligible relative to the sampling variation in image representing p hatd. This is a reasonable assumption for most cases in this study.

For a subset of the estimates produced from the 2002, 2003, and 2004 data, the above approach yielded an underestimate of the variance of a total because image representing n hatd was subject to considerable variation. In these cases, the SEs for the total estimates calculated directly within SUDAAN are reported. Using the SEs from the total estimates directly from SUDAAN does not affect the SE estimates for the corresponding proportions presented in the same sets of tables.

B.2.2 Suppression Criteria for Unreliable Estimates

As has been done in past NSDUH reports, direct survey estimates produced for this study that are considered to be unreliable due to unacceptably large sampling errors are not shown in this report and are noted by asterisks (*) in the tables containing such estimates. The criteria used for suppressing all direct survey estimates were based on the relative standard error (RSE) (defined as the ratio of the SE over the estimate) on nominal sample size and on effective sample size.

Proportion estimates (image representing p hat) within the range [0 < image representing p hat < 1], rates, and corresponding estimated number of users were suppressed if

RSE[-ln(image representing p hat)]> 0.175 when image representing p hat ≤ 0.5

or

RSE[-ln(1 - image representing p hat)]> 0.175 when image representing p hat > 0.5.

Using a first-order Taylor series approximation to estimate RSE[-ln(image representing p hat)] and RSE[-ln(1 - image representing p hat)], the following was obtained and used for computational purposes:

Equation B-3 > 0.175 when image representing p hat ≤ 0.5     D

or

Equation B-4 > 0.175 when image representing p hat > 0.5.     D

The separate formulas for image representing p hat ≤ 0.5 and image representing p hat > 0.5 produce a symmetric suppression rule (i.e., if image representing p hat is suppressed, then 1 - image representing p hat will be as well). This ad hoc rule requires an effective sample size in excess of 50. When 0.05 < image representing p hat < 0.95, the symmetric property of the rule produces a local maximum effective sample size of 68 at image representing p hat = 0.5. Thus, estimates with these values of image representing p hat along with effective sample sizes falling below 68 are suppressed. See Figure B.1 for a graphical representation of the required minimum effective sample sizes as a function of the proportion estimated.

A minimum nominal sample size suppression criterion (n = 100) that protects against unreliable estimates caused by small design effects and small nominal sample sizes was employed. Prevalence estimates also were suppressed if they were close to 0 or 100 percent (i.e., if image representing p hat < 0.00005 or if image representing p hat ≥ 0.99995).

Estimates of other totals (e.g., number of initiates) along with means and rates that are not bounded between 0 and 1 (e.g., mean age at first use and incidence rates) were suppressed if the RSEs of the estimates were larger than 0.5. Additionally, estimates of the mean age at first use were suppressed if the sample size was smaller than 10 respondents. Also, the estimated incidence rate and number of initiates were suppressed if they rounded to 0.

The suppression criteria for various NSDUH estimates are summarized in Table  B.1 at the end of this appendix.

Below is a graph, click here for the text describing this graph.

Figure B.1 Required Effective Sample as a Function of the Proportion Estimated

Figure B.1

B.2.3 Statistical Significance of Differences

This section describes the methods used to compare prevalence estimates in this report. Customarily, the observed difference between estimates is evaluated in terms of its statistical significance. Statistical significance is based on the p value of the test statistic and refers to the probability that a difference as large as that observed would occur due to random variability in the estimates if there were no difference in the prevalence estimates for the population groups being compared. The significance of observed differences in this report is generally reported at the 0.05 and 0.01 levels. When comparing prevalence estimates, the null hypothesis (no difference between prevalence estimates) was tested against the alternative hypothesis (there is a difference in prevalence estimates) using the standard difference in proportions test expressed as

Equation B-5,     D

where image representing p hat1 = first prevalence estimate, image representing p hat2 = second prevalence estimate, var(image representing p hat1)= variance of first prevalence estimate, var (image representing p hat2) = variance of second prevalence estimate, and cov (image representing p hat1, image representing p hat2) = covariance between image representing p hat1 and image representing p hat2. In cases where significance tests between years were performed, the 2003 prevalence estimate becomes the first prevalence estimate and the 2004 estimate becomes the second prevalence estimate.

Under the null hypothesis, Z is asymptotically distributed as a normal random variable. Therefore, calculated values of Z can be referred to the unit normal distribution to determine the corresponding probability level (i.e., p value). Because the covariance term is not necessarily zero, SUDAAN was used to compute estimates of Z along with the associated p values using the analysis weights and accounting for the sample design as described in Appendix A. A similar procedure and formula for Z were used for estimated totals.

When comparing population subgroups defined by three or more levels of a categorical variable, log-linear Chi-square tests of independence of the subgroups and the prevalence variables were conducted first to control the error level for multiple comparisons. If the Chi-square test indicated overall significant differences, the significance of each particular pairwise comparison of interest was tested using SUDAAN analytic procedures to properly account for the sample design. Using the published estimates and SEs to perform independent t tests for the difference of proportions usually will provide the same results as tests performed in SUDAAN. However, where the significance level is borderline, results may differ for two reasons: (1) the covariance term is included in SUDAAN tests whereas it is not included in independent t tests, and (2) the reduced number of significant digits shown in the published estimates may cause rounding errors in the independent t tests.

B.3 Other Information on Data Accuracy

Errors can occur from nonresponse, coding errors, computer processing errors, errors in the sampling frame, reporting errors, and other errors not due to sampling. These types of errors are reduced through data editing, statistical adjustments for nonresponse, close monitoring and periodic retraining of interviewers, and improvement in various quality control procedures.

Although these types of errors often can be much larger than sampling errors, measurement of most of these errors is difficult. However, some indication of the effects of some types of these errors can be obtained through proxy measures, such as response rates and from other research studies.

B.3.1 Screening and Interview Response Rate Patterns

In 2002, 2003, and 2004, respondents received a $30 incentive in an effort to improve response rates over years prior to 2002. Of the 142,612 eligible households sampled for the 2004 NSDUH, for example, 130,130 were successfully screened for a weighted screening response rate of 90.9 percent (Table  B.2). In these screened households, a total of 53,331 persons aged 18 to 64 were selected, and completed interviews were obtained from 43,053 of these sample persons, for a weighted interview response rate of 77.2 percent (Table  B.3). Weighted screening response rates for 2002 and 2003 were 90.7 percent in each survey year (Table  B.2). Weighted interview response rates were 78.9 percent in 2002 and 77.5 percent in 2003 (Table  B.3).

The overall weighted response rate, defined as the product of the weighted screening response rate and weighted interview response rate, was 70.2 percent in 2004. Nonresponse bias can be expressed as the product of the nonresponse rate (1-R) and the difference between the characteristic of interest between respondents and nonrespondents in the population (Pr - Pnr). Thus, assuming the quantity (Pr - Pnr) is fixed over time, the improvement in response rates in 2002 through 2004 over prior years will result in estimates with lower nonresponse bias.

B.3.2 Inconsistent Responses and Item Nonresponse

Among survey participants, item response rates were above 99 percent for most drug use items. However, inconsistent responses for some items were common. Estimates of substance use from NSDUH are based on responses to multiple questions by respondents, so that the maximum amount of information is used in determining whether a respondent is classified as a drug user. Inconsistencies in responses are resolved through a logical editing process that involves some judgment on the part of survey analysts. Additionally, missing or inconsistent responses are imputed using statistical methodology. Editing and imputation of missing responses are potential sources of error.

In addition to reporting substance use prevalence among the full-time employed population as a whole, this population was further divided into subgroups based on responses to workplace questions presented in the noncore employment section of the NSDUH questionnaire. These finer categories included self-reported characteristics of their employer's substance testing policies and treatment programs, as well as respondent's opinions on working for employers who test for substance use at random and during the hiring process. Respondents were further classified into occupational and industry groups using the 2000 Standard Occupational Classification (SOC) and the North American Industry Classification System. For all these workplace measures, item nonresponse was present. Respondents had unknown information as a result of refusing to answer certain questions or being unable to answer. While standard NSDUH logical editing procedures were implemented, unknown responses to these noncore questions were not imputed. For this report, all reported estimates pertaining to a workplace-related characteristic are based on only those respondents who had complete data for all of the workplace items. That is, respondents with unknown information for a given workplace measure or categorization were excluded from any and all analysis regarding that workplace topic.

B.3.3 Validity of Self-Reported Use

Most drug use prevalence estimates, including those produced for NSDUH, are based on self-reports of use. Although studies have generally supported the validity of self-report data, it is well documented that these data often are biased (underreported or overreported) by several factors, including the mode of administration, the population under investigation, and the type of drug (Bradburn & Sudman, 1983; Hser & Anglin, 1993). Higher levels of bias also are observed among younger respondents and those with higher levels of drug use (Biglan, Gilpin, Rorhbach, & Pierce, 2004). Methodological procedures, such as biological specimens (e.g., urine, hair, saliva), proxy reports (e.g., family member, peer), and repeated measures (e.g., recanting), have been used to validate self-report data (Fendrich, Johnson, Sudman, Wislar, & Spiehler, 1999). However, these procedures often are impractical or too costly for community-based epidemiological studies (SRNT Subcommittee on Biochemical Verification, 2002). NSDUH utilizes widely accepted methodological practices for ensuring validity, such as encouraging privacy through audio computer-assisted self-interviewing (ACASI). Comparisons using these methods within NSDUH have been shown to reduce reporting bias (Aquilino, 1994; Turner, Lessler, & Gfroerer, 1992).

B.4 Measurement Issues

A measurement issue associated with the 2004 NSDUH that may be of interest and is discussed in this section includes the methods for measuring substance dependence and abuse.

B.4.1 Illicit Drug and Alcohol Dependence and Abuse

The 2004 NSDUH CAI instrumentation included questions that were designed to measure dependence on and abuse of illicit drugs and alcohol. For these substances,1 dependence and abuse questions were based on the criteria in the DSM-IV (APA, 1994).

Specifically, for marijuana, hallucinogens, inhalants, and tranquilizers, a respondent was defined as having dependence if he or she met three or more of the following six dependence criteria:

  1. Spent a great deal of time over a period of a month getting, using, or getting over the effects of the substance.

  2. Used the substance more often than intended or was unable to keep set limits on the substance use.

  3. Needed to use the substance more than before to get desired effects or noticed that same amount of substance use had less effect than before.

  4. Inability to cut down or stop using the substance every time tried or wanted to.

  5. Continued to use the substance even though it was causing problems with emotions, nerves, mental health, or physical problems.

  6. The substance use reduced or eliminated involvement or participation in important activities.

For alcohol, cocaine, heroin, pain relievers, sedatives, and stimulants, a respondent was defined as having dependence if he or she met three or more of seven dependence criteria, including the six standard criteria listed above plus a seventh withdrawal symptom criterion. The seventh withdrawal criterion is defined by a respondent reporting having experienced a certain number of withdrawal symptoms that vary by substance (e.g., having trouble sleeping, cramps, hands tremble).

For each illicit drug and alcohol, a respondent was defined as having abused that substance if he or she met one or more of the following four abuse criteria and was determined not to be dependent on the respective substance in the past year.

  1. Serious problems at home, work, or school caused by the substance, such as neglecting your children, missing work or school, doing a poor job at work or school, or losing a job or dropping out of school.

  2. Used the substance regularly and then did something that might have put you in physical danger.

  3. Use of the substance caused you to do things that repeatedly got you in trouble with the law.

  4. Had problems with family or friends that were probably caused by using the substance and continued to use the substance even though you thought the substance use caused these problems.

Criteria used to determine whether a respondent was asked the dependence and abuse questions included responses from core substance use and frequency of substance use questions, as well as noncore substance use questions. Unknown responses in the core substance use and frequency of substance use questions were imputed. However, the imputation process did not take into account reported data in the noncore (i.e., substance dependence and abuse) CAI modules. Responses to the dependence and abuse questions that were inconsistent with the imputed substance use or frequency of substance use could have existed. Because different criteria and different combinations of criteria were used as skip logic for each substance, different types of inconsistencies may have occurred for certain substances between responses to the dependence and abuse questions and the imputed substance use and frequency of substance use as described below.

For alcohol and marijuana, respondents were asked the dependence and abuse questions if they reported substance use in the past year but did not report their frequency of substance use in the past year. Therefore, inconsistencies could have occurred where the imputed frequency of use response indicated less frequent use than required for respondents to be asked the dependence and abuse questions originally.

For cocaine, heroin, and stimulants, respondents were asked the dependence and abuse questions if they reported past year use in a core drug module or past year use in the noncore special drugs module. Thus, inconsistencies could have occurred when the response to a core substance use question indicated no use in the past year, but responses to dependence and abuse questions indicated substance dependence or abuse for the respective substance.

A respondent might have provided ambiguous information about past year use of any individual substance, in which case these respondents were not asked the dependence and abuse questions for that substance. Subsequently, these respondents could have been imputed to be past year users of the respective substance. In this situation, the dependence and abuse data were unknown; thus, these respondents were classified as not dependent on or abusing the respective substance. However, the respondent was never actually asked the dependence and abuse questions.

Table B.1 Summary of 2004 NSDUH Suppression Rules
Estimate Suppress if:
SE = standard error; RSE = relative standard error; deff = design effect.
Source: SAMHSA, Office of Applied Studies, National Survey on Drug Use and Health, 2004.
Prevalence rate, image representing p hat, with nominal sample size, n, and design effect, deff (1) The estimated prevalence rate, image representing p hat, is < 0.00005 or ≥ 0.99995, or

(2) The ratio of two quantities is greater than 0.175. The numerator of the ratio is the standard error of p hat divided by p hat. The denominator is the negative of the natural logarithm of p hat when image representing p hat ≤ 0.5, or

The ratio of two quantities is greater than 0.175. The numerator of the ratio is the standard error of p hat divided by 1 minus p hat. The denominator is the negative of the natural logarithm of 1 minus p hat when image representing p hat > 0.5, or

(3) Effective n < 68, where Effective Effective n is the ratio of n over the design effect or

(4) n < 100.

Note: The rounding portion of this suppression rule for prevalence rates will produce some estimates that round at one decimal place to 0.0 or 100.0 percent but are not suppressed from the tables.
Estimated number (numerator of image representing p hat) The estimated prevalence rate, image representing p hat, is suppressed.

Note: In some instances when image representing p hat is not suppressed, the estimated number may appear as a 0 in the tables. This means that the estimate is greater than 0 but less than 500 (estimated numbers are shown in thousands).
Mean age at first use, image representing x bar, with nominal sample size, n

(1) RSE(image representing x bar) > 0.5, or

(2) n < 10.
Incidence rate, image representing r hat (1) The incidence rate, image representing r hat, rounds to < 0.1 per 1,000 person-years of exposure, or

(2) RSE (image representing r hat) > 0.5.
Number of initiates, image representing t hat (1) The number of initiates, image representing t hat, rounds to < 1,000 initiates, or

(2) RSE (image representing t hat) > 0.5.
Table B.2 Weighted Percentges and Sample Sizes for 2002, 2003, and 2004 NSDUHs, by Screening Result Code
  Sample Size Weighted Percentage
2002 2003 2004 2002 2003 2004
Source: SAMHSA, Office of Applied Studies, National Survey on Drug Use and Health, 2002, 2003, and 2004.
Total Sample 178,013 170,762 169,514 100.00 100.00 100.00
Ineligible cases 27,851 27,277 26,902 15.27 15.84 15.76
Eligible cases 150,162 143,485 142,612 84.73 84.16 84.24
Ineligibles 27,851 27,277 26,902 15.27 15.84 15.76
Vacant 14,417 14,588 15,204 51.55 52.56 56.24
Not a primary residence 4,580 4,377 4,122 17.36 17.07 15.54
Not a dwelling unit 2,403 2,349 2,062 8.16 8.08 7.51
All military personnel 289 356 282 1.08 1.39 1.07
Other, ineligible 6,162 5,607 5,232 21.86 20.90 19.65
Eligible Cases 150,162 143,485 142,612 84.73 84.16 84.24
Screening complete 136,349 130,605 130,130 90.72 90.72 90.92
No one selected 80,557 74,310 73,732 53.14 51.04 50.86
One selected 30,738 30,702 30,499 20.58 21.46 21.53
Two selected 25,054 25,593 25,899 17.00 18.22 18.53
Screening not complete 13,813 12,880 12,482 9.28 9.28 9.08
No one home 3,031 2,446 2,207 2.02 1.68 1.55
Respondent unavailable 411 280 259 0.26 0.18 0.18
Physically or mentally incompetent 307 290 265 0.20 0.18 0.17
Language barrier—Hispanic 66 42 51 0.05 0.03 0.04
Language barrier—Other 461 450 391 0.35 0.39 0.32
Refusal 8,556 8,414 8,588 5.86 5.98 6.10
Other, access denied 471 923 660 0.30 0.81 0.67
Other, eligible 12 12 10 0.01 0.01 0.01
Resident < 1/2 of quarter 0 0 0 0.00 0.00 0.00
Segment not accessible 0 0 0 0.00 0.00 0.00
Screener not returned 15 16 15 0.01 0.01 0.01
Fraudulent case 479 6 14 0.21 0.00 0.02
Electronic screening problem 4 1 22 0.00 0.00 0.02
Table B.3 Response Rates and Sample Sizes for Persons Aged 18 to 64 in the 2002, 2003, and 2004 NSDUHs, by Demographic Characteristics
  Selected Persons Completed Interviews Weighted Response Rate
2002 2003 2004 2002 2003 2004 2002 2003 2004
Source: SAMHSA, Office of Applied Studies, National Survey on Drug Use and Health, 2002, 2003 and 2004.
Total 51,129 52,726 53,331 42,215 42,708 43,053 78.85% 77.53% 77.23%
Age in Years                  
18-25 27,216 27,259 27,408 23,271 22,941 23,075 85.16% 83.47% 83.87%
26-34 7,672 8,060 8,052 6,191 6,371 6,366 79.41% 78.69% 78.61%
35-49 12,076 12,604 12,907 9,616 9,829 9,927 78.95% 77.20% 75.96%
50-64 4,165 4,803 4,964 3,137 3,567 3,685 73.89% 73.12% 73.61%
Gender                  
Male 24,676 25,432 25,838 19,721 19,943 20,279 76.17% 74.98% 74.83%
Female 26,453 27,294 27,493 22,494 22,765 22,774 81.47% 79.97% 79.53%
Race/Ethnicity                  
Hispanic 6,582 7,061 7,273 5,345 5,687 5,869 79.62% 79.07% 78.25%
White 35,387 36,437 36,364 29,189 29,515 29,209 78.85% 77.59% 77.13%
Black 5,702 5,769 5,888 4,884 4,824 5,010 82.60% 79.71% 82.68%
All other races 3,458 3,459 3,806 2,797 2,682 2,965 70.46% 69.36% 66.73%
Region                  
Northeast 10,521 10,837 10,884 8,544 8,625 8,626 76.44% 76.06% 75.47%
Midwest 14,283 14,666 14,794 11,861 12,028 11,899 80.39% 78.47% 77.73%
South 15,514 15,857 16,133 12,927 12,915 13,246 80.25% 78.48% 79.06%
West 10,811 11,366 11,520 8,883 9,140 9,282 77.13% 76.31% 75.30%
County Type                  
Large metro 20,637 23,866 24,382 16,637 18,804 19,188 77.06% 75.42% 75.74%
Small metro 18,145 18,083 18,017 15,141 15,018 14,819 79.90% 80.16% 78.48%
Nonmetro 12,347 10,777 10,932 10,437 8,886 9,046 81.92% 79.96% 80.07%

End Notes

1 Substances include alcohol, marijuana, cocaine, heroin, hallucinogens, inhalants, pain relievers, tranquilizers, stimulants, and sedatives.

Go to Top of PageGo to the Table of Contents

This is the page footer.

This page was last updated on June 19, 2008.

SAMHSA, an agency in the Department of Health and Human Services, is the Federal Government's lead agency for improving the quality and availability of substance abuse prevention, addiction treatment, and mental health services in the United States.

Yellow Line

Site Map | Contact Us | Accessibility Privacy PolicyFreedom of Information ActDisclaimer  |  Department of Health and Human ServicesSAMHSAWhite HouseUSA.gov

* Adobe™ PDF and MS Office™ formatted files require software viewer programs to properly read them. Click here to download these FREE programs now

What's New

Highlights Topics Data Drugs Pubs Short Reports Treatment Help Mail OAS