|
|
BRFSS Data Quality, Validity, and Reliability
BRFSS Data Quality and National Estimates
There have been nearly 20 separate studies that have examined issues
related to the reliability and validity of the BRFSS and the system’s ability to
provide both valid national estimates and comparisons across states (see
reference list below). While many of these studies look at particular topic
areas, three provide original and secondary analyses across topic areas and
a fourth provides an overview of the BRFSS, including data quality,
challenges for the system, and recommendations for future action:
(1) Nelson DE, Holtzman D, Bolen J, Stanwyck CA, Mack KA. Reliability and
validity of measures from the Behavioral Risk Factor Surveillance System
(BRFSS). Social and Preventive Medicine, 2001;46Suppl 1:S03-S42.
This is the most comprehensive BRFSS validity and Reliability study
conducted to date. The authors review and summarize more than 200
reliability and validity studies of measures on the BRFSS and studies of
measures that were the same or similar to those on the BRFSS from other
studies. Measures determined to be of high reliability and high validity
were current smoker, blood pressure screening, height, weight, body mass
index (BMI), and
several demographic characteristics. Measures of both moderate reliability
and validity included when last mammography was received, clinical breast
exam, sedentary lifestyle, intense leisure-time physical activity, and fruit
and vegetable consumption. Few measures were of low validity and only one
measure was determined to be of low reliability. Several other measures were
of high or moderate reliability or validity, but not both (see article
referenced below for more information). The reliability
and validity could not be determined for some measures, primarily due to
lack of research. They concluded that most questions on the core BRFSS
questionnaire were at least moderately reliable and valid, and many were
highly reliable and valid.
(2) Nelson DE, Powell-Griner E, Town M, Kovar MG. A comparison of
national estimates from the National Health Interview Survey and the
Behavioral Risk Factor Surveillance System. American Journal of Public
Health, 2003; 93:1335–1341.
The study compares national estimates from the National Health Interview
Survey (NHIS) and the BRFSS.
The authors compared data from the two surveys on smoking, height, weight,
BMI, diabetes, hypertension, immunization, lack of insurance
coverage, cost as a barrier to medical care, and health status. Overall the
national estimates were similar for 13 of the 14 measures examined. Small
differences according to demographic characteristics were found for height
and BMI, with larger differences for health status. These
conclude that although estimates differed within subgroups, the BRFSS
provided national estimates comparable to those of the NHIS and that BRFSS
national data could provide rapidly available information to guide national
policy and program decisions.
(3) Ronaldo Iachan, Jane Schulman, Eve Powell-Griner, David E. Nelson,
Peter Mariolis, and Carol Stanwyck. “Pooling state telephone survey health
data for national estimates: The CDC Behavioral Risk Factor Surveillance
System, 1995 Pp. 221–226 in Conference on Health Survey Research Methods
(7th:1999:Williamsburg, VA.), Seventh Conference on Health Survey
Research Methods, 2001, Marcie L. Cynamon and Richard A. Kulka (eds.),
Hyattsville, MD: DHHS Publication No. (PHS) 01–1013.
Several objectives were addressed in this study, including (1)
investigating the appropriateness of combining data across states in terms
of variations in sample design as well as sampling and nonsampling errors;
(2) comparing the usual method of computing BRFSS national estimates to a
newer method that may take sample design more explicitly into account; and
(3) assessing the correspondence between national estimates for a select set
of health measures from the BRFSS with those of the NHIS. They concluded
that combining data across states is feasible but depends on each state’s
ability to minimize both sampling and nonsampling errors. [Note: changes in
the BRFSS design over the past 3 years, in part in response to this report,
have led to greater standardization in the sampling designs across states
and in closer monitoring of state-level data quality]. Comparing two methods
for combining data, they concluded that although there is very little
difference between the national estimates obtained by using a pooling
approach versus a restratification approach. Finally, they found that the
BRFSS and NHIS gave consistent results for most of the items examined. In
addition, they caution that because the BRFSS and NHIS use different
methodologies and cover somewhat different populations, they cannot
reasonably be expected to yield identical results.
(4) Mokdad AH, Stroup DF, Giles WH. Public health surveillance for
behavioral risk factors in a changing environment: recommendations from the
Behavioral Risk Factor Surveillance team. MMWR, 2003;52 (RR-9):1–12.
The article provides an overview of the BRFSS and describes the
challenges for BRFSS in effectively managing an increasingly complex
surveillance system that serves the needs of numerous programs while facing
changing telecommunication technology and the greater demand for more
local-level data. To determine options and recommendations for how best to
meet BRFSS future challenges, a 2-day strategy workshop was held and
attended by survey research specialists. The workshop featured presentations
on the current system; emerging technologic challenges; telephone surveying
techniques; program perspectives of CDC, partner organizations, and states;
and recommendations for change. The report summarizes the recommendations
resulting from that workshop. [Note: of the 42 specific recommendations made
at the May 2003 meeting, 37 have been implemented or are in process of
implementation, while the remaining 5 were determined not to be feasible for
BRFSS. A second expert panel meeting was held November 2004. The
Overview of the BRFSS 2004 Expert Panel Review and Recommendations
contains specific recommendations for continual improvement of BRFSS.]
Other References for BRFSS-Related Reliability and Validity Studies
(5) Anda RF, Dobson DL, Williamson DF et al. Health promotion data for
state health departments: telephone versus in-person survey estimates of
smoking and alcohol use. Am J Health Promotion 1989;4:32–6.
(6) Arday DR, Tomar SL, Nelson DE et al. State smoking prevalence
estimates: a comparison between the Behavioral Risk Factor Surveillance
System and Current Population Surveys. Am J Public Health 1997;87:1665–9.
(7) Battelle. Evaluation of the Behavioral Risk Factor Surveillance
System (BRFSS) as a source for national estimates of selected health risk
behaviors: final report. Baltimore, MD. Battelle, December 1999.
(8) Bowlin SJ, Morrill BD, Nafziger AN et al. Reliability and changes in
validity of self-reported cardiovascular disease risk factors using dual
response: the Behavioral Risk Factor Survey. J Clin Epidemiol 1996;49:511–7.
(9) Bowlin SJ, Morrill BD, Nafziger AN et al. Validity of cardiovascular
disease risk factors assessed by telephone survey: the Behavioral Risk
Factor Survey. J Clin Epidemiol 1993;46:561–71.
(10) Brownson RC, Eyler Aa, King AC et al. Reliability of information on
physical activitiy and other chronic disease risk factors among US women
aged 40 years or older. Am J Epidemiol 1999;149:379–91.
(11) Brownson RC, Jackson-Thompson J, Wilkerson JC et al. Reliability of
information on chronic disease and risk factors collected in the Missouri
Behavioral Risk Factor Surveillance System. Epidemiology 1994;5:545–9.
(12) Gentry E, Kalsbeek W, Hogelin G et al. The behavioral risk factor
surveys: design, methods, and estimates from combined state data. Am J Prev
Med 1985;1(6):9–14.
(13) Jackson C, Jatulis DE, Fortmann SP. The Behavioral Risk Factor
Survey and the Stanford Five-City Project Survey: a comparison of
cardiovascular risk behavior estimates. Am J Public Health 1992;82:412–16.
(14) Martin LM, Leff M, Calonge N et al. Validation of self-reported
chronic disease and health services data in a managed care population. Am J Prev Med 2000;18:215–18.
(15) Shea S, Stein AD, Lantigua R et al. Reliability of the Behavioral
Risk Factor Survey in a triethnic population. Am J Epidemiol 1991;133:489–500.
(16) Stein AD, Courval JM, Lederman RI, et al. Reproducibility of
responses to telephone interviews: demographic predictors of discordance in
risk factor status. Am J Epidemiol 1995;141:1097–106.
(17) Stein AD, Lederman RI, Shea S. Reproducibility of the women’s module
of the Behavioral Risk Factor Surveillance System questionnaire. Ann Epidemiol 1996;6:47–52.
(18) Stein AD, Lederman RI, Shea S. The Behavioral Risk Factor
Surveillance System questionnaire: its reliability in a statewide sample. Am
J Public Health 1993;83:1768–72.
|
|
