|
||||||||
National Center for Chronic Disease Prevention and Health Promotion Behavioral Risk Factor Surveillance System BRFSS Home | Contact Us |
|
|
|
|
|
|
2000 BRFSS Summary Data Quality Report
This report provides selected statistical indicators of data quality in the Behavioral Risk Factor Surveillance System (BRFSS). The report presents data on four types of measures by state1:
(1) Outcome measures, including response rates, which are based on disposition codes.
(2) Selection biases with respect to gender, age, and race/ethnicity.
(3) Missing values on income.
(4) Sampling frame coverage.
The target audiences for this report are researchers and other analysts who will be presenting results from BRFSS data. The measures in this report are designed to document the quality of BRFSS data.2 "Data quality" in this report refers to the accuracy of BRFSS data.
Although data quality can be indicated by outcome measures, selection biases, and missing values, it is also affected by the sampling frame used. The BRFSS sampling frame includes more household telephone numbers than are usually included in telephone survey designs. Although the differences are small nationwide, the magnitude of the differences vary by state. Thus, this report will also include tables documenting the additional coverage of the target population coming from the more inclusive telephone sampling frame used in the BRFSS.
Outcome Measures
The factors affecting the distribution of disposition codes by state may be grouped into differences in telephone systems, sample designs, surveyed populations, and data collection processes. Different outcome measures are variously affected by differences in these factors. Ideally, outcome measures that are used as indicators of data quality would be affected only by characteristics of the surveyed populations and data collection processes. Characteristics of telephone systems and sample designs would have no effect. In practice, outcome measures, including response rates, can be affected by characteristics of telephone systems and sample designs. These extraneous (with respect to data quality) influences should be taken into account in any interpretation of the results.
Among the data collection processes affecting outcome measures is the accuracy of the disposition code assignments. Outcome measures are only as valid as the accuracy of the disposition code assignments. The accuracy of disposition code assignments should also be taken into account in any interpretation of the results.
Table 1 presents the frequency distributions of individual disposition codes by state. The percent completes by state is the Efficiency Rate.
Table 2 presents brief definitions of each disposition code.
Table 3 presents median BRFSS Cooperation Rates by state from 1996 to 2000. The categories used in the calculations of the outcome rates in Tables 3 and 4 are defined below.
Categories of Case Outcomes |
||
Category | Disposition Code Definition | Format in Formulas |
Completed Interview | 01 | COIN |
Eligible and able to complete interview | 01+02+07+09 | Elig HH |
Unknown Eligibility | 04+10 | Unknown |
Ineligible or unable to complete interview | 03+05+06+08+11 | Ineligible |
The BRFSS Cooperation Rate is an outcome rate with the number of completes in the numerator and the number of eligible respondents who are capable of completing the survey in the denominator.3 Records indicating a respondent who does not speak English—or in some states, English or Spanish—or a respondent with a physical or mental impairment are excluded from the denominator. The denominator of the Cooperation Rate consists of records with disposition codes of 01, 02, 07, or 09. Thus, the formula for the BRFSS Cooperation Rate is
.
Table 4 presents median BRFSS CASRO (Council of American Survey Research Organizations) Rates from 1996 to 2000. A response rate is an outcome rate with the number of completes in the numerator and an estimate of the number of eligible units in the sample in the denominator. The BRFSS CASRO Rate calculation treats telephone numbers used by someone aged 18 years or older who speaks English—or in some states, English or Spanish—well enough to complete the interview and with no physical or mental impairment that prevents completion of the interview as eligible units. These are records with disposition codes of 01, 02, 07, or 09. Added to this total is an estimate of the number of households among the records whose eligibility is undetermined. These are records with disposition codes of 04 or 10. The BRFSS CASRO Rate calculation assumes that the unresolved numbers contain the same percentage of eligible households as the records whose eligibility or ineligibility are determined. Thus, the formula for the BRFSS CASRO Rate is
A response rate is an indicator of the potential for bias in the results of a survey. It does not indicate the actual amount of bias. The actual amount of bias can be conceptualized as a function of two factors—the amount of non-response, which is measured by a response rate, and the differences between the responders and the non-responders. A response rate does not address the latter factor. Thus, a survey with a low response rate but very little difference between responders and non-responders will have results with only a small amount of bias whereas a survey with a relatively high response rate but a large difference between responders and non-responders will have results with a large amount of bias. These considerations should be taken into account in any interpretation of the CASRO Rates.
Figure 1 presents median BRFSS CASRO, Cooperation, and Efficiency Rates from 1996 to 2000.
Table 5 presents frequency distributions of individual household disposition codes by state. The percent completes by state may be termed the Household Completion Rate. It is the percent of completes among identified households. Of special interest here are the dispositions 06, 08, and 11—those treated as ineligible households in the Cooperation and CASRO rate calculations. To the extent that telephone numbers actually ringing into eligible households have been inaccurately dispositioned as ringing into ineligible households, the Cooperation and CASRO rates will be artificially inflated. Disposition code 06 should only be assigned to telephone numbers ringing into households with no one aged 18 or over to teen phones which no one aged 18 or over will use. Disposition codes 08 and 11 should be assigned as defined in Table 2. In cases where the sum of dispositions 06, 08, and 11 is deemed excessive, the Household Completion Rate may be treated as an alternative measure of data quality.
Table 6 presents frequency distributions of individual disposition codes by state for listed household telephone numbers.4 These distributions may be less subject to the influences of characteristics of the telephone systems and of sample designs than the distributions of all sample records. By definition, all of these numbers were listed household numbers at one time. While states will differ in the extent to which these numbers will have changed status since being listed, the resulting differences should be smaller than the differences among numbers that began with different statuses. Thus, the percent of completes among listed household numbers may be considered an alternative measure of data quality.
Selection Biases
Tables 7 through 14 present data on the differences between BRFSS and population data with respect to gender, age, and race/ethnicity by state. In these tables, BRFSS data are weighted for the characteristics of the sample design—disproportionate sampling by geographic and density strata (where they exist), number of telephones, and number of adults in the household.5 Because these factors are built into the sample design, they should be adjusted for before comparing survey distributions to population distributions. No definitive standards exist with respect to what constitutes a substantial difference between survey and population percentages. One approach would be to examine the distribution of discrepant values with the purpose of identifying extreme values, or outliers, which may indicate biased data.
Income Missing Values
Table 15 presents the percent missing (Don’t know/Not sure or Refused or either) income by state. Income is the variable in the survey data with the largest percentage of missing values. A larger percentage of missing values for income implies lower quality data for income and, by extension, for other variables.
Sampling Frame
Table 16 presents data on the weighted percent of completes in the final BRFSS by density stratum by state.
"Density stratum" refers to whether a telephone number is in a zero block or a one-plus block. Hundred blocks are sets of 100 telephone numbers with the same area code, prefix, first two digits of the suffix, and all possible combinations of the last two digits. Commercial sample providers have access to databases of all telephone numbers available to be assigned by a telephone company and of listed household telephone numbers. Thus, they can determine the number of listed household telephone numbers for each hundred block of telephone numbers available to be assigned by a telephone company. "Zero blocks" are hundred blocks with no listed household numbers. "One-Plus blocks" are hundred blocks with one or more listed household numbers.
Zero block numbers are excluded from most telephone survey samples. Their inclusion in the BRFSS is a net addition to the response rate relative to the target population. In other words, a given response rate in a BRFSS survey actually reflects less non-response relative to the target population than the same response rate from a survey that excludes zero block numbers because the BRFSS sampling frame contains less non-coverage. For many states, the estimates in Table 16 understate the additional coverage provided by zero block numbers because the Household Completion Rate is frequently substantially smaller among zero block numbers than among one-plus block numbers and there is currently no explicit adjustment for these differences in the BRFSS weights. Table 17 presents data on the percent of households and the Household Completion Rate, weighted for probability of selection, by density stratum and state. The weighted percent of households suggests that the weighted percent of completes in zero blocks in the final BRFSS data set would be substantially larger in some states if the Household Completion Rates were equal in both sets of strata.
The weighted percent of completes from zero block numbers is (an
understated) measure of the additional coverage provided by the BRFSS
sampling frame compared to typical telephone survey sampling frames.
Figure 1. Median BRFSS Cooperation, CASRO, and Efficiency Rates: 1996-2000
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
* Links to non-Federal organizations are provided solely as a service to our users. Links do not constitute an endorsement of any organization by CDC or the Federal Government, and none should be inferred. The CDC is not responsible for the content of the individual organization Web pages found at this link.
Policies and Regulations | BRFSS Home | Contact Us Page last reviewed: 04/14/2008
United States Department of Health and Human Services |
|