Sample Design

The NSSRN 2004, the eighth in the series, continued to oversample nurses in small States in order to better support HRSA’s National Center for Health Workforce Analysis’ State level supply and demand projections for registered nurses.  The basic design was enhanced by using sample design optimization methodology developed by Chromy ^[1] to determine the sample allocation to the States that would simultaneously satisfy variance constraints defined by the 51 States and the total U.S. 

In the original sample design, and in the 1988 redesign, the universe of RNs was sorted alphabetically by last name and approximately equal-sized clusters of RNs were constructed by partitioning the alphabetically ordered list into 250 alpha-segment clusters with equal (or nearly equal) numbers of RNs.  An alpha-segment was defined as all alphabetically adjacent names falling within pre-specified boundaries. For example, all names beginning with the lower boundary, up to but not including the name that defined the upper boundary. 

From the frame of 250 equally divided alpha-segments, a total of 40 alpha-segments were randomly selected, representing a 16 percent sampling rate overall.  Registered nurses are selected in the sample based on their name, with an RN being included in the sample if the name of licensure falls into one of the alphabetic segments that are in sample for that State. 

Although each State had 40 sample segments, the sample size of each State differed in size depending on the State’s sampling rate.  While uniform-sampling rates would have produced the best national estimates, the resulting sample sizes for the smallest States would have been inadequate to support State-level estimates.  Since both national and State-level estimates are required for the 2004 NSSRN, as was done is prior surveys, sampling rates were increased in the smaller States to obtain larger State-level sample sizes.  While this disproportionate sampling improved the precision of estimates in the smaller States, it also reduced precision of national estimates due to unequal weighting effects.  .

To accommodate the differing State sampling rates, a planned variation in the size of the segments, i.e., “portions of alpha segments” was used.  Each of the 40 alpha-segments selected for sample was divided into ½-, ¼-, 1/8-, 1/16-, and 1/32- portions.  These fractions indicate the size of the alpha segment portion relative to the size of the basic alpha-segment. 

The sampling rate for a particular State was achieved using a combination of the alpha-segment portions. As a result, each State contains some sample (i.e., a portion) from each of the 40 alpha-segments, depending on the sampling rate for the State.  For example, selecting the entire 40 complete alpha segments on a State list is expected to constitute a 16 percent sampling rate (40 ÷ 250 = 0.16) in the State. This is because each alpha segment contained an expected 0.4% of the State’s RN names (40 X 0.4 percent = 16 percent).  Likewise, the sample for a State with an 8 percent sampling rate consisted of the 40 ½ portion selections.  Several sampling rates use a combination of portions for each alpha-segment in sample (rather than one fractional portion for all alpha-segments).  For example, a 5 percent sampling rate was achieved by first randomly dividing the 40 alpha-segments into two groups, the first containing 30 alpha-segments and the other containing 10; and by using the ¼ portions from the first group and the ½ portions from the second group (0.4 percent x [(30 x ¼) + (10 x ½)] = 5 percent). 

To identify and account for nurses appearing in more than one of the 51 State lists, the portions were constructed such that each portion was “nested” (or included) in the boundaries of the larger portion.  As a result, the alpha segment clusters from the States with lower sampling rates (typically larger States) were automatically included in the alpha segment clusters selected from the States with higher sampling rates (typically smaller States).  

As a result, a RN who was licensed under the same name in two States with identical sampling rates was selected (or not selected) for both States, since the alphabetic name boundaries defining the portions are the same for both States.  However, if the RN was licensed under the same name in two States that are sampled with different sampling rates, then, if the RN was sampled in the State with a lower sampling rate, they were also included in the sample for the State with the higher sampling rate (as the alphabetic name boundaries defining the portions for the State with the lower sampling rate are nested within those of the State with the higher sampling rate).  This nesting property of the sample design maximizes the chances that the RN will be selected in all States that they have an active license in.  A nurse that is licensed in two or more States under the same name will have a probability of selection corresponding to the State with the highest sampling rate.

Sample design optimization techniques developed by Chromy (1996) were used to determine how to allocate the sample of 54,000 RNs to the 51 State lists.  This sample size was then converted to a sampling rate, and the rate was rounded to one of the admissible rates for the nesting design.  For example, the original rate for the State of Washington was 1.59%, the closest admissible rate was 1.5%.   Rates were rounded down only such that the change in sampling rate still left their effective sample size at or above the 1996 NSSRN level. 

After determination of frame sizes and expected sampling rates, the States were assigned a priority order to properly determine selection probabilities for nurses appearing on more than one of the 51 State lists.  Traditionally, States were ordered by size, with larger States having lower sampling rates and smaller States having higher sampling rates.  However, as in the 2000 NSSRN, States were priority ordered based on their sampling rate.  As such, it is mostly, but not necessarily, the case that States with larger RN populations had lower sampling rates.

Essentially the same procedure was followed for sample selection for all States. Once a State provided a licensure file containing all appropriate names of individuals with active RN licenses and meeting all specifications, the required sample names in that file were selected.  Regardless of the way a State alphabetized and standardized the names in its files, the sample names were selected according to the standards established by the survey design.  That is, sample selections ignored blanks and punctuation in the last names (except a dash in hyphenated names) and ignored titles (e.g.,”Sister”). 

Registered nurses were selected in the sample on the basis of name, with an RN being included in the sample if the name of licensure fell within a specific alpha-segment portion as defined by the State sampling rate.  In other words, the sample for a given State consisted of all RN names falling into any one of the State’s pre-designated 40 alphabetic portions that corresponded to the State sampling rate (one portion from each of the complete 40 alpha-segments in sample). 

The pairs of names that defined the alpha-segment portion constituted the lower and upper boundaries corresponding to the sampling rate.  Thus, the membership of the alpha-segment portion was defined by all names, beginning with the lower boundary (i.e., the last name in alphabetical order of all the names included in that segment), up to but not including a name that defined the upper boundary.  This latter name fell into the next alpha-segment.  As was done in the NSSRN 2000, any deviations of more than 8 percent were candidates for either an increased or decreased rate. 

Because the survey is longitudinal in nature, a panel structure was constructed to allow for several of the sample alpha-segments to be systematically replaced each survey.  Under the original survey design, the 40 sample alpha-segments were arranged in alphabetical order and then partitioned into eight groups of five successive alpha-segments each.  One segment from each group was randomly assigned to each panel, so that each panel consisted of segments that spanned the entire alphabet.  For each successive survey, a new panel (consisting of eight new alpha-segments or 20 percent of the sample) was entered into the sample, replacing one of the five panels from the previous survey.  Under this scheme, a nurse who maintained an active license in the same State(s) could be retained in the sample for up to five surveys. 

The planned NSSRN 2004 sample size was 54,000 cases, similar to that of the NSSRN 2000, and up from the 45,000 used in previous studies.  Planned sampling rates ranged from 1.125 percent in several of the largest States to 15 percent in Wyoming.  This translated into planned sample sizes ranging from 3,225 RNs in California to approximately 796 in Wyoming.  The initial round of sampling, however, yielded a much smaller sample than expected due to the variable size of the alpha-segments in each State.  Thus, a second round of sampling was done by increasing the sampling rates from 1 to 1.125 in the eleven largest States and “adding to” the sample selected in the first round, yielding a total of 56,917 sample cases.  After eliminating cross-State duplications, the expected the sample size to be fielded was still approximately 54,000 cases. 

Table B-1 in Appendix B shows the sampling rates and sample sizes that were planned and actually obtained for the 51 States in the survey.  Differences between planned and actual sampling rates result from State-specific variation in the distribution of nurses’ names. States are priority ordered by sampling rate and size.

Because many nurses are licensed in more than one State, their names could be selected in the sample more than once.  In accordance with the sample design, we ensured that each sampled RN was retained in the outgoing sample file exactly once to avoid multiple questionnaires being sent to nurses.  If we identified an exact duplicate, the nurse in the lower priority State was coded as a duplicate of the sample member in the higher priority State.  For example, an Alaska record was coded as a duplicate to the sample record in Wyoming.  Following data collection, these expected duplicates were reviewed to ensure that the nurse reported a license in both of the States. 

Table B-1. State Sampling Rates and Sample Sizes (Priority Ordered)

Weighting Procedures

The probability sample design of the survey permits the computation of unbiased estimates of characteristics of the RN population at the National and State level.  These estimates are based on weights that reflect the complex design and compensate for the potential risk of nonresponse bias to the extent feasible.  The weights that are assigned to each sample nurse may be interpreted as the number of nurses in the target population that the sample nurse represents. The sampling weight for an RN is the reciprocal of the nurse’s probability of selection in her/his priority State, adjusted to account for nonresponse and multiple licenses.

Before computing the weights, the original State frame sizes (shown above) were adjusted to account for duplicate licenses within States and ineligible licenses (i.e., frame errors) found in the sample.  Most within-State duplicates were identified at the time of initial list processing, but a few were identified after sample selection.  The ineligible licenses were identified in the process of reconciling the State and nurse reported licenses.  Some of the inconsistencies between the State reported data and the nurse reported data are due to the time period that elapsed between frame construction and data collection (a period during which changes and license expirations naturally occur).  Other differences are due to errors in either the State list or the nurse’s questionnaire.  Cases that could not be reconciled by Gallup were sent to the State Boards of Nursing for resolution.

In both cases, the frame total is computed by subtracting the estimated number of ineligible and duplicate licenses from the State’s original frame count.  The adjusted frame total used to compute the resulting weights for State i can be computed as:

where:

N_i = the total number of licenses on State i list,

= the estimated number of within-State duplicates in State i,and

= the estimated number of frame errors in State i (e.g., licenses listed by State that were not reported by a responding nurse).

Each responding nurse was assigned a weight corresponding to their unique ‘priority State’; that is, the State with the highest sampling rate from which he or she was licensed and selected into the sample.In other words, the weight is reflective of the probability of selecting the sampled nurse in their “priority” State.  All nurses with the same priority State have an equal probability of being selected and, consequently, have equal initial sampling weights.  The sum of the weights for all nurse respondents assigned to a specific priority State will equal, approximately, the total number of active licenses on the list (at the time the sample was drawn) less the number of those licenses assigned to higher priority lists.

The weights were computed sequentially for each State A, B, etc., where A was the highest-priority State, and B the next-highest-priority State.  The weight for an RN sampled from the highest priority State, State A, was the ratio of the adjusted count of licenses in the sampling frame for State A to the number of eligible respondents licensed in State A.  For State B, and the remaining States, the numerator and denominator of this ratio were adjusted to account for State A and other higher-priority States.  To describe the basic method, the following terms are defined:

N(_i) = total number of licenses for State i (adjusted for within-State duplicates and frame errors)

m(_i) = number of  eligible respondents  for   State i that did not have a license in a higher-priority State

n(_i,j) = number of eligible respondents with a license in both State i and State j [note n(_i,i) denotes the number of eligible respondents with a license only in State i]

W(_i) = the adjusted weight for eligible respondents who were assigned to the higher priority State i

The weight for State A was computed as follows:

W(A) = N(A) / m(A).

For the State B weight, W(B), the numerator was the adjusted frame count of licenses for State B, N(B), after removing the estimated total count of State B nurses who were also licensed in State A (i.e., W(A) n(A,B)).  Similarly, the numerator of W(C) excluded State C nurses who were also licensed in either State A or State B (i.e., W(A) n(A,C) + W(B) n(B,C)).  That is, for the State B weight and the State C weight, the computations were:

W(B)  = [N(B) - W(A) n(A,B)] / m(B)

W(C)  = [N(C) - W(A) n(A,C) - W(B) n(B,C)] / m(C) .

In either case, the denominator was the number (m(B) or m(C)) of respondents in the State not licensed in a higher-priority State.

In general, the numerator of a State I weight, W(I), was the total adjusted frame count of RN licenses in State I after removing the estimated total count of State I nurses also licensed in higher-priority States.  The denominator, m(I), was the number of State I respondents not licensed in a higher-priority State.  This weighting scheme incorporated both a nonresponse adjustment that inflated the respondents’ data to account for those that did not respond to the survey and a duplication adjustment to account for duplication in the sampling frame across States.  These final analysis weights will serve to differentially weight responding nurses to reflect the level of disproportionality in the final respondent sample relative to the population.

Estimation Procedure

Final NSSRN estimates can be computed using the final set of sampling weights, W_k (for sample nurse k).  For example, an estimate of the total number of RNs working in a particular State is based on the following indicator variable, X_k:

X_k = 1 if nurse k worked in a particular State,

= 0 otherwise.

The desired estimated total may then be written as

the sum being over all sample nurses.

Estimates of ratios and averages are obtained as the ratio of estimated totals.

Sampling and Nonsampling Errors

To the extent that samples are sufficiently large, relatively precise estimates of characteristics of the licensed RN population of the United States can be made because of the underlying probability structure of the sample data.  Such estimates are, sometimes, an imperfect approximation of the truth.  Several sources of error could cause sample estimates to differ from the corresponding true population value.  These sources of error are commonly classified into two major categories: sampling errors and nonsampling errors.

A probability sample such as the one used in this study is designed so that estimates of the magnitude of the sampling error can be computed from the sample data.  In addition, nonsystematic components of nonsampling error are also reflected in the sampling error estimates.

Nonsampling Errors

Some sources of error, such as unusable responses to vague or sensitive questions; no responses from some nurses; and errors in coding, scoring, and processing the data are, to a considerable extent, beyond the control of the sampling statistician.  They are called “nonsampling errors” and also occur in cases where there is a complete enumeration of a target population, such as the U.S. Census. Among the activities that were directed at reducing nonsampling errors to the lowest level feasible for this survey included careful planning, keeping nonresponses to the lowest feasible level, and coding and processing of the sample data.

If nonsampling errors are random, in the sense that they are independent and tend to be compensating from one respondent to another, then they do not cause bias in estimates of totals, percents, or averages.  Furthermore, the contribution from such nonsampling errors will automatically be included in the sampling errors that are estimated from the sample data.  However, correlations or relationships in cross-tabulations are often decreased by such errors, and sometimes substantially.  Thus, random errors that tend to be compensated for in estimates of simple aggregates or averages may (but not necessarily will) introduce systematic errors or biases in measures of relationships or cross-tabulations.

Nonsampling errors that are systematic (rather than random and compensating) are a source of bias for sample estimates. Such errors are not reduced by increasing the size of the sample, and the sample data do not provide an assessment of the magnitude of these errors.  Systematic errors are reduced in this study by such efforts as careful wording of questionnaire items, respondent motivation, and well-designed data-collection and data-management procedures.  However, such errors sometimes occur in subtle ways and are less subject to design control than is the case for sampling errors.

Nonresponse to the survey is one of the largest sources of nonsampling error because a characteristic being estimated may differ, on average, between respondents and nonrespondents.  For this reason, considerable effort has been expended in this survey to obtain a high response rate by respondent motivation and follow-up procedures.  A high response rate reduces both random and systematic nonsampling errors.  After taking into account duplicates and frame errors, the overall response rate to this survey was  70.47  percent.  State-level response rates ranged from 61.98 percent to 81.57 percent except for the District of Columbia where the response rate (46.12 percent) was significantly lower.

Sampling Errors

All sample survey estimates are subject to sampling error.  The magnitude of the sampling error for an estimate, as indicated by measures of variability such as its variance or its standard error (the square root of its variance), provides a basis for judging the precision of the sample estimates.

Systematic sampling, which was the selection procedure used in choosing the alpha-segments for this study, is convenient from certain practical points of view, including providing for panel rotation.  However, it does not permit unbiased estimation of the variability of survey estimates unless some assumptions are made. Thus, standard errors were estimated based upon the assumption that the systematic sample of 40 alpha-segments is equivalent to a stratified random sample of two alpha-segments from each of 20 strata of adjacent alpha-segments.  Ordinarily, this assumption should lead to overestimates of the sampling error for systematic sampling, but in this case (with alpha-segments as the sampling units) the magnitude of the overestimate is believed to be trivial.

Regarding the sample as consisting of 20 pairs of alpha-segments (thus obtaining 20 degrees of freedom) for variance estimation, the probability is approximately 0.95 that the statistic of interest differs from the value of the population characteristic that it estimates by not more than 2.086 standard deviations.

Specifically, a 95 percent confidence interval for an estimated statistic  takes the form:

where  is the estimated standard error for .

Direct Variance Estimation

Similar to prior cycles of the NSSRN, direct estimates of sampling variance were obtained for a set of important variables for each State and for the United States using the jackknife variance estimation procedure with 20 replicates of the sample.  Variance estimates using the jackknife approach require the computation of a set of weights for the full sample and a set for each replicate using the established weight computation procedure (i.e., 20 additional sets of weights).  Having 20 sets of weights permits construction of 20 replicate estimates to compare with the estimate produced from all of the data; each replicate estimate is based on about 39/40ths of the data.

Each replicate was formed from 19 pairs of alpha-segments (38 alpha-segments total) and 1 alpha-segment from the 20th pair.  Alpha-segments were randomly removed from each pair to form the replicate estimates.  This procedure was performed 20 times, once for each pair of alpha-segments.  Thus, actual respondent count in the included segments for a particular replicate was approximately 39/40^ths of the full respondent sample and was weighted to represent the full population.

The variance of , Var , is estimated by computing:

where:

=    an estimated total for replicate i associated with alpha-segment pair i, and

   =  an estimated total obtained over the full sample.

If the estimate of interest is a ratio of two estimated totals (e.g., the total number of RNs resident in Florida between 25 and 29 years old to the total number of RNs resident in Florida), the variance estimate for the estimated ratio would be of the following form:

Following the example, the and measurements would be full sample and replicate estimates, respectively, of the number of RNs resident in Florida who were 25 to 29 years old, while and would be the corresponding estimates of the total number of RNs resident in Florida.  The variance of any other statistic, simple or complex, can be similarly estimated by computing the statistic for each replicate.

The jackknife variance estimator can use either the full sample estimate, or the average of the replicate estimates.  While usually little difference exists between the two estimates, the estimator, was used which tends to provide more conservative estimates of variance.

Direct estimates of the variance were computed for a variety of variables.  These variables were chosen not only due to their importance, but also to represent the range of expected design effects.  The average of these design effects (on a State-by-State basis) provides the basis for the variance estimate for variables not included in the set for which direct variance estimates were computed. Table B-2 in Appendix B presents direct estimates of the standard error (the square root of the variance) for a selected set of variables. Table B-3 in Appendix B shows the estimated population of nurses in each State and the standard error of these population totals.

Design Effects and Generalized Variances

The generalized variance is a model-based approximation of the sampling variance estimate, which is less computationally complex than the direct variance estimator but is also less accurate.  The generalized variance equations use the national-level or State-level estimates of the design effect and, for some estimates, the coefficient of variation (CV) to estimate the sampling variance.  The design effect, F, for an estimated proportion  is determined by taking the ratio of the estimated sampling variance, obtained by the jackknife method, to the sampling variance of the  in a simple random sample of the same size. This design effect, F, can be computed as follows:

where n is the unweighted number of respondents used to determine the denominator of .

Direct estimates of the design effect were computed for a set of variables for each State.  The median of the design effects was then computed for each State and the nation.  These median design effects can be used in formulas for estimating generalized variances or standard errors.  This procedure uses median design effects for a class of estimates instead of calculating direct estimates (with a resulting economy in time and costs), at the sacrifice generally of some accuracy in the variance estimates.

A generalized standard error estimate for an estimated proportion, for a State or for the United States, is provided by the equation:

(1)

where n is the number of survey respondents used to determine the estimate .  The multiplier F, the median^²­­­­ design effect, depends upon the State for which the estimated proportion was generated.  The median design effects are listed on Table B-4 in Appendix B.

Generalized estimates of standard errors can also be computed  for  estimated  numbers (or totals) of RNs in a State with a particular characteristic  (such as those employed in hospitals).  The estimate  is a subtotal of the estimate , the estimated total of RNs working and/or living in the State.  Note that the standard error and coefficient of variation of  (represented by were determined for the nation and for each State (see Table B-3). 

To calculate the standard error of a total, one must first compute the relative variance (or square of the coefficient of variation) of the ratio of  to  (called .  The relative variance can be calculated as:

where F is the design effect for the State of interest and n is the number of respondents to the survey that were weighted to obtain the estimate

Then, from the relative variance of the ratio, one can approximate the relative variance of the total denoted by using:

This approximation is based on the first-order Taylor series approximation to the variance of a product and the assumption of zero correlation between the estimate of ratio and the denominator of the ratio.

Finally, the standard error of the total can be estimated by multiplying the estimate by the square root of the relative variance defined above.  The standard error of    is thus estimated as:

     (2)

The standard error of an estimated percentage for a region of the United States depends upon a linear combination of the variance of the same estimated percentages for the States making up that particular region.  The estimated proportion for the region is:

Here, h is the number of States in region R, and  and  are estimates for a particular State.  The formula used to approximate the standard error of an estimated proportion for a region is:

   (3)

where represents the standard error of the estimated proportion for the States and the standard errors are estimated from equation (1) or from direct estimation.

The direct standard error for an estimated number for a region of the United States also depends upon a linear combination of the variance of the same estimated numbers for the States that make up the region.  The formula used is

    (4)

where the standard error  of the  estimated number  is  available  either  from  the direct procedures or from equation ⁽²⁾.

Table B-2.  Estimates and Standard Errors (S.E.) For Selected Variables of U.S. Registered Nurse Population

Table B-3.   Direct Estimates of State Nurse Population, Standard Error, and Coefficient of Variation by State, 2000

Table B-4.  Median Design Effects for Percentages Estimated from the Eighth National Sample Survey of Registered Nurses, 2004

^[1] Chromy, James R. “Design Optimization with Multiple Objectives”. American Statistical Association of the Section on Survey Research Methods, Arlington, VA., pp A4-199

^[2] Since the actual distribution of names differs for each State from the frame distribution used to develop the 250 alpha-segments, some variation occurs between the planned and actual sampling rates.

^²The median design effect was based on all design effects for estimates of proportions computed on selected variables.  Using a median instead of mean value avoids the effects of extreme estimates of standard errors, which can occur for some relatively rare attributes.  In prior years, an average (mean) design effect was computed for selected variables.  Given that the distribution of design effects is skewed to the right, it is expected that the true median be less than the true mean.