![](https://webarchive.library.unt.edu/eot2008/20090508162453im_/http://www.cdc.gov/navimages/spacer.gif)
![](https://webarchive.library.unt.edu/eot2008/20090508162453im_/http://www.cdc.gov/brfss/images/spacer.gif)
|
![](https://webarchive.library.unt.edu/eot2008/20090508162453im_/http://www.cdc.gov/navimages/spacer.gif)
|
![](https://webarchive.library.unt.edu/eot2008/20090508162453im_/http://www.cdc.gov/navimages/spacer.gif)
Publications, Research, and Methodology
Improvements to
BRFSS Methodology, Design, and Implementation
The original print version of
the following information is available as a printable PDF,*
Improvements
to the Behavioral Risk Factor Surveillance System (BRFSS) Methodology,
Design, and Implementation.
Background
The Behavioral Risk Factor Surveillance System (BRFSS) is a state-based
system of health surveys that was established in 1984 by CDC and state
health departments. These surveys obtain information about health risk
behaviors, clinical preventive health practices, and health care access,
primarily related to chronic disease and injury, from a representative
sample of adults in each state. For the majority of states, BRFSS is the
only source for this type of information. Data are collected monthly in
all 50 states, the District of Columbia, Puerto Rico, and the U.S.
Virgin Islands. Approximately 350,000 interviews of adults are
completed each year, making BRFSS the largest health survey conducted by
telephone in the world.
The challenge for BRFSS is effectively managing an increasingly
complex surveillance system that serves the needs of multiple programs
while adapting to changes in communications technology (increased use of
cellular telephones and call screening devices), societal behaviors
(concerns about privacy and declining participation in surveys), and
population diversity (growing number of languages spoken in the United
States along with greater cultural and ethnic diversity). To address
these challenges, BRFSS maintains an ongoing program of improvement and
adaptation that involves
- Designing and conducting innovative pilot studies to improve the
current BRFSS methodology and provide a foundation for the
implementation of future methodologies.
- Identifying and addressing potential threats to the validity and
reliability of BRFSS data that might affect survey participation and
data quality.
- Expanding the utility of the surveillance system by implementing
special surveillance projects, including rapid response surveillance
efforts and follow-up surveys.
These efforts are critical for improving the quality of BRFSS data,
reaching populations previously not included in the survey, and
expanding the utility of the surveillance data. Pilot studies are
conducted in collaboration with the states, and the information garnered
from these studies is widely disseminated in reports, conference
presentations, and peer-reviewed publications. By addressing current
challenges and keeping an eye on future issues, these studies help
prepare the surveillance system for design and implementation changes
when needed. In this way, the BRFSS team ensures that public health
surveillance efforts meet the highest scientific standards, use the most
effective and cost-efficient approaches, and produce valid and reliable
data and results.
BRFSS Expert Panel
Meetings
Guidance on system improvements comes from a variety of sources,
including state partners, other CDC Centers and Programs, and other
outside experts in the fields of survey research, statistics, and
epidemiology. In 2002, BRFSS held its first biannual BRFSS Expert Panel
Meeting, inviting approximately 20 survey statisticians, methodologists,
and operational experts to a 2-day meeting to discuss the challenges
facing the field of survey research and implications for the BRFSS.
Repeated in 2004 and 2006, the goal of these meetings is to develop
options and prioritize recommendations for maintaining data quality in
the face of societal and technological changes.
At the most recent meeting in November 2006, the panel made a number
of specific recommendations, including using advance letters by all
states, conducting pilot studies with cellular telephone users, and
identifying the appropriate mix of sampling frames and survey modes to
maintain the validity of BRFSS estimates. These and other
recommendations made by the panels are critical for improving BRFSS,
ensuring the quality and validity of the data, and reducing the
potential for bias in BRFSS estimates.
[Return to Top]
Overview
and Outcomes of Selected Projects
Based in part on recommendations from the Expert Panel Meetings, BRFSS
has undertaken a number of innovative and informative pilot studies and
analyses, including the following:
Use of prenotification letters and messages on answering machines.
Advance letters can improve participation in telephone surveys, like
BRFSS. When tested in a number of states, letters improved response
rates, on average, 6 percentage points. The letters were also cost
efficient in that the cost of obtaining a fixed number of completed
surveys using advance letters was lower than the cost without letters.
As a result, advance letters are recommended for use with the BRFSS in
all states. Messages left on the answering machines of potential
respondents did not, however, improve response rates significantly. This
is likely due to the relatively small percentage of sample members who
remembered hearing the message and who found the message to be effective
in persuading them to participate in the survey.
Assessing the impact of the Do Not Call Registry. More than
100 million telephone numbers have been listed on the National Do Not
Call Registry since it began in 2003. To assess the potential impact of
the registry on participation rates in BRFSS, case outcomes were
examined from nearly 4.5 million telephone numbers called between
January 1, 2002, and June 30, 2005. Using trend analyses and time series
modeling, the findings indicated that once pre-DNC Registry trends in
response rates and other potential covariates were accounted for, the
do-not-call rules appeared to have had no significant impact on
state-level response rates in either a positive or negative direction.
Use of real-time telephone survey interpreters. Real-time
interpretation during a survey can expand the number of languages in
which surveys are offered. A detailed assessment of the quality of this
approach was conducted as part of the BRFSS in California using behavior
coding of interviews conducted with respondents who otherwise would have
been finalized as “language barrier nonrespondents.” Interviews were
recorded and behavior coded, quantifying for each question (1) the
accuracy of the question interpretation, (2) the accuracy of the
interpreted response, (3) the degree of difficulty administering the
question, (4) the number of times the question was repeated, and (5) the
number of times the interpreter and respondent engaged in conversation
that was not relayed to the interviewer. The approach produced favorable
results, with less than a 4% error rate for interpretation of the
questions and a 1% error rate in interpretation of survey responses.
Use of Web and mail questionnaires. Web and mail versions of
the BRFSS questionnaire were administered to potential respondents drawn
from the standard BRFSS telephone sampling frame and reverse-matched to
identify valid mailing addresses. Telephone survey follow-up was
conducted with Web and mail survey nonrespondents. The findings suggest
that self-administered modes, when used in conjunction with telephone
follow-up, can improve levels of participation, but may also increase
differences between respondents and nonrespondents on certain measures
of interest such as respondent demographic characteristics and key
health and risk measures.
Use of Address-Based Sampling (ABS). Advances in electronic
record keeping have allowed researchers to develop and sample from a
frame of addresses, which appears to provide coverage that rivals that
obtained through RDD sampling methods. A pilot study conducted in 2005
compared use of traditional RDD telephone survey methodology to an
approach using a mail version of the questionnaire completed by a random
sample of households drawn from an address-based frame. The findings
indicate that the mail survey approach can achieve higher response rates
in low-response-rate states (< 40%) than RDD (particularly when two
mailings are sent). Additionally, the address frame with mail survey
design provides access to households with cellular telephones only and
offers cost savings over the telephone approach.
Improving the current BRFSS weighting methodology. Post-survey
adjustments are becoming an increasingly important means of maintaining
the representativeness of survey data. Using statistical raking
techniques, the approach to weighting BRFSS data is being re-evaluated.
The new approach adjusts the data not only in terms of respondents’ sex
and age, but also race (in a more consistent manner), education, and
telephone coverage—variables all found to be significantly related to
key health and risk outcomes on BRFSS.
[Return to Top]
Current and Future Pilot Studies
Health surveillance in the future will be much more complex and involve
multiple ways of collecting public health data. Although telephone
surveys will likely remain the mainstay of how BRFSS data are collected,
it is likely that some additional modes of interviewing will also be
necessary. To prepare for this future, BRFSS currently has major pilot
studies underway in the following areas:
Mixed-mode survey approaches. Studies of mixed-mode surveys
involving mail surveys with telephone follow-up are underway, comparing
samples of telephone numbers drawn using RDD methods to a sample of
addresses drawn using U.S. Postal Service records. The study is being
conducted in California, Florida, Massachusetts, Minnesota, South
Carolina, and Texas.
Surveying cellular telephone users. To meet the challenge
posed by the growing number of cellular-telephone-only households, BRFSS
is conducting interviews with individuals sampled from known cellular
telephone exchanges. Conducted in Georgia, New Mexico, and Pennsylvania,
the study will help determine the feasibility of conducting BRFSS
interviews by cellular telephone, their cost, and the impact on survey
estimates of including such interviews.
Obtaining physical measurements. BRFSS is piloting an approach
for collection of direct health measures (e.g., height and weight, blood
pressure, cholesterol) from a subset of respondents to the ongoing BRFSS
surveys. The data collection will be used to adjust statewide data
collected from the ongoing BRFSS survey, facilitate validation of key
BRFSS interview questions, and assess the feasibility of collecting
physical measure data on an ongoing basis.
[Return to Top]
Selected Publications
M. Link, A. Mokdad, D. Kulp, and A. Hylon. (2006) “Has the National
Do Not Call Registry Helped or Hurt Survey Research Efforts?” Public
Opinion Quarterly 70(5):794–805.
M. Link, M. Battaglia, M. Frankel, L. Osborn, and A. Mokdad. (2006).
“Address-Based Versus Random-Digit Dialed Surveys: Comparison of Key
Health and Risk Indicators.” American Journal of Epidemiology
164:1019–1025.
M. Link and A. Mokdad. (2006). “Can Web and Mail Survey Modes Improve
Participation in an RDD- ased National Health Surveillance?” Journal
of Official Statistics 22:293–312.
M. Link, A. Mokdad, H. Stackhouse, and N. Flowers. (2006). “Race,
Ethnicity, and Linguistic Isolation as Determinants of Participation in
Public Health Surveillance Surveys.” Preventing Chronic Disease
[serial online] 2006 Jan. Available at
http://www.cdc.gov/pcd/issues/2006/jan/05_0055.htm.
M. Link and A. Mokdad. (2005). “Use of Prenotification Letters: An
Assessment of Benefits, Costs, and Data Quality.” Public Opinion
Quarterly 69:572–587.
M. Link and A. Mokdad. (2005). “Use of Alternative Modes for Health
Surveillance Surveys: Results from a Web/Mail/Telephone Experiment.”
Epidemiology 16:701–704.
M. Link and A. Mokdad. (2005). “Effects of Survey Mode on
Self-Reports of Adult Alcohol Consumption: Comparison of Web, Mail, and
Telephone.” Journal of Studies on Alcohol 66(2): 239–245.
M. Link and A. Mokdad. (2005). “Leaving Answering Machine Messages:
Do They Increase Response Rates for the Behavioral Risk Factor
Surveillance System?” International Journal of Public Opinion
Research 17: 239–250.
*You will need
Acrobat Reader
(a free application) to view and print the PDF document.
To navigate through the guide’s chapters and sections, open the document
and click the Bookmarks tab at the left side of the browser window. The
expandable chapter menu features links to specific sections of the guide.
[Return to Top]
|
![](https://webarchive.library.unt.edu/eot2008/20090508162453im_/http://www.cdc.gov/navimages/spacer.gif)
|