2008

Behavioral Risk Factor Surveillance System 

Data Quality Report Handbook

Version 1.0.0
October 22, 2008

2008 BRFSS Data Quality Report Handbook

Introduction

This document discusses the tables found in the 2008 Year-to-Date Data Quality Report. The main target audience is State BRFSS Coordinators, especially those who do not have day-to-day responsibility for data collection.  The data may be collected by a contractor or in the state health department.

The rest of this document is divided into four sections.  Section B, Definitions of Variables, defines unfamiliar variables.  General Questions the Tables below Can Help Answer lists several general questions and identifies the tables that can help answer them.  The next section, Standards of Comparison, identifies the types of standards that can be used to evaluate the data in the tables.  Next is Specific Guidelines for Interpretation of 2008 BRFSS Year-to-Date Data Quality Report Tables, which identifies things to look for and, as much as possible, standards for comparison for each table.  Finally, the Appendix lists the title of each table in the report, as well as explanations of charts in the report and definitions of the response rates and other outcome rates.  

Definitions of Variables

Assigned Month.  Assigned month is the month of the file in which a record was submitted or the month of the file in which a related pre-screened record should have been submitted but was not. BRFSS protocol calls for states to submit related pre-screened sample records that they never called along with sample records that they did call.  Sometimes states fail to do so.  For states that receive their sample through BSB, we are able to identify related pre-screened records they may have failed to send and to add them to their data files.  

File Month.  Each data file submitted to CDC contains the name of a month in the filename.  The file month of a record denotes the month in which the data file containing the record was submitted to CDCs.

Household Roster Status.  
None=Number of adults, number of men, number of women are all missing.  
Partial=One or two of number of adults, number of men, number of women are missing.  
Inconsistent (But Complete)=Number of adults, number of men, number of women are all non-missing but number of adults does not equal the sum of number of men and number of women.  
Consistent (And Complete)=Number of adults, number of men, number of women are all non-missing and number of adults equals the sum of number of men and number of women.

Number of Residential Telephone Numbers.  
Missing=“Do you have more than one telephone number in your household?” or “How many residential telephone numbers do you have?” was refused.  
Otherwise, the reported number of residential telephone numbers.

Density Status.  Categorized as Listed or Not listed one-plus block.

Disposition Codes  

The 2008 final BRFSS disposition codes are

1.	Interview

110	Complete
120	Partial Complete

2.	Eligible, Non-Interview

210	Termination within questionnaire
220	Refusal after respondent selection
230	Selected respondent never reached or was reached but did not begin interview during
 interviewing period
240	Selected respondent away from residence during the entire interviewing period
250	Language problem after respondent selection
260	Selected respondent physically or mentally unable to complete an interview during the entire 
	 interviewing period
270	Hang up or termination after number of adults recorded but before respondent selection
280	Household contact after number of adults recorded but before respondent selection

3.	Unknown Eligibility, Non-Interview

305	Household members away from residence during entire interviewing period
310	Hang-up or termination, housing unit, unknown if eligible respondent
315	Household contact, eligibility undetermined
320	Language problem before respondent selection
325	Physical or mental impairment before respondent selection
330	Hang-up or termination, unknown if private residence
332	Contact, unknown if private residence
335	Telephone answering device, message confirms private residential status
340	Telecommunication technological barrier, message confirms private residential status
345	Telephone answering device, not sure if private residence
350	Telecommunication technological barrier, not sure if private residence
355	Telephone number is no longer in service or has been changed
360	No answer
365	Busy
370	On never call list

4.	Not Eligible

405	Out-of-state
410	Household, no eligible respondent
420	Not a private residence
430	Dedicated fax/data/modem line with no human contact
435	Cellular telephone
440	Fast busy
450	Non-working/disconnected number

5.	Interim Disposition Codes

505	Refusal:  hang-up or termination
510	Appointment
515	Language problem
520	Physical or mental impairment
525	Answering machine, message confirms residential status
530	Technological barrier other than answering machine, message confirms residential status
535	Answering machine, not sure if private residence
540	Technological barrier other than answering machine, not sure if private residence
545	Phone number temporarily out of service
550	No answer
555	Busy
560	Fax/data/modem
565	Fast busy
570	Possible non-working number
575	Circuit busy
580	Null attempt
585	Requires supervisor attention

In the tables below, these codes are categorized as follows:

Household records are records with disposition codes of 110, 120, 210, 220, 230, 240, 250, 260, 270, 280, 305, 310, 315, 320, 325, 330, 332, 335, 340, 355, 370, 410.

Definitions and Labels for Disposition Code Categories

Category	Definition	Format in Tables	
Completed interview	Disposition Codes (110, 120)	Completed Interview	
Terminations and Refusals	Disposition Codes (210, 220)	Terminate, Refusal	
Known Household, Possible Eligible, Non-Interview	Disposition Codes (230, 240, 250, 260, 270, 280, 305, 310, 315, 335)	Known HH	
Probable Households	Disposition Codes (320, 325, 330, 332, 340, 355, 370)	Prob HH	
Answering Machine Unknown	Disposition Codes (345, 350)	Machine Unknown	
Ineligible Households	Disposition Code = 410	HH, No Elig Resp	
Non-Contact	Disposition Codes (360, 365)	Non-Contact	
Business Non-Residential	Disposition Code = 420	Business	
Non-Working Out-of-Scope	Disposition Codes (405, 430, 435, 440, 450)	Non-Working	

General Questions the Tables Below Can Help Answer

The tables are generally organized around questions that a state BRFSS Coordinator, as the ultimate state person responsible for assuring and assessing the quality of BRFSS data, should ask.  

Is There Evidence of Significant Bias in the Data?  The tables in Section II address this question. Specific guidelines for assessing biases are given in Section E of this handbook.

Is the Data Collector Calling Numbers Frequently Enough and According to the BRFSS Callback Rules?  Tables III.1 to III.5 address these questions.  The discussion for these tables in the Specific Guidelines for Interpretation section, below, identifies the patterns for which to look.  

How Consistent Is the Data Collection Effort From Month to Month?  Every table that is run by month can help answer this question.  The primary ones to look at are Tables III.09, III.11, and III.13.  The percentage of records in each disposition code category in Tables III.13 should vary by less than two points from month to month.  If these show less consistency than you think appropriate, verify that the same patterns exist in Tables III.09 and III.11.  These latter tables should show more month-to-month variability, since their bases are smaller.

Is the Data Collector Dispositioning Numbers According to Their Definitions?  There are a number of different ways, which vary by disposition code, to approach this question.  The first place to look is in Section IV, Proper Assignment of Disposition Codes.  By looking at the disposition codes of various sets of numbers, you can identify codes that are being assigned to more or fewer records than they should be.  Another way to assess if disposition codes are being appropriately assigned is to determine the consistency between the household roster status and the disposition code.  A third way is to determine the consistency between the number of attempts and the disposition code. 

Disposition code 410 ‘No eligible respondent at this number’ is especially prone to being assigned when another code would be more appropriate.  This code should be applied to well under 2% of households.  Also, all records assigned a disposition code of 410 should have no household roster.

Are the Interviewers Adequately Trained, Supervised, and Monitored?  Sections VI and IX can be used to address this question.  The standards for comparison in this case are standing relative to other states.  Other indications that interviewers are not adequately trained, supervised, or monitored would be:  (1) Relatively large percentages of records missing income or weight in Section VI.  (2) A relatively large percentage of Hispanics coded “Other race” and relatively large biases in Section IX.

Are There Particular Interviewers Who Seem to Be Deviating from Sound Practices?  The tables in Section IX can address this question.  The easiest way to approach it is to look, first, for outliers (Tables IX.01, IX.02, and IX.03, and interviewers with 0s or *s next to their records in the box-plots).  Because, however, a flat distribution can mask true outliers, you should also look for records with values that are separated from the others, even if they are not marked as outliers. Please remember that there may be valid reasons for deviations from a statistical norm, so these findings should be treated more as indicators of a need for further investigation rather than conclusive proof of inappropriate interviewer practices.

Standards of Comparison  

Various standards of comparison are appropriate for different tables.

In many cases, the standard is defined by BRFSS protocol.  For example, 100% of records dispositioned ‘No answer’ should have received at least 15 call attempts.   

In other cases, a standard cannot be exactly determined but logic or data may indicate that only values within a fairly narrow range seem reasonable.  In such cases, I have usually set an exact standard as a guideline, based on knowledge about the population of households with telephones and an arbitrarily defined precise threshold.  

An outside standard is appropriate in bias measures.  Bias is measured by the difference between a sample value and a population value.  For example, a sample with 60% female respondents drawn from a population that is 52% female shows a selection bias of 8 percentage points.  The 2008 Year-to-Date Data Quality Report contains population data that serve as comparison standards for selection bias measures.

There are cases where the best that can be done is to determine if a state is an outlier in a distribution.  For example, there is no particular standard that can be determined for the percent of completes among household records by examination of the BRFSS calling rules and protocols. The best that can be done is to see if a state is an outlier compared to other states.

Finally, regardless of how a standard is determined for an individual measure, consistency in that measure is important.  A measure may or may not be consistent at an acceptable level but inconsistency itself can be an indicator of inconsistent or poor quality data collection practices.

Standards are indicated by bold print.

Specific Guidelines for Interpretation of 2008 BRFSS Year-to-Date Data Quality Report Tables

I.  Sample Generation, Release, and Submission

Table I.1.  Density Status, by Assigned Month.  This table presents the percentages of listed and not-listed one-plus block numbers in the sample.  Other things equal, increases in percent listed should increase efficiency.  This table will help in assessing the extent to which changes in sample composition may be responsible for changes in efficiency.

Table I.2.  Number of Records in Replicate.
Telephone sample records should be released by replicate and all records, including those pre-screened as non-working or business, should be sent to BSB.  Large numbers of replicates with fewer than fifty records indicate that one or the other protocol is not being followed.  Small numbers of replicates with fewer than fifty records probably indicate processing problems, either in the CATI software or in post-data collection processing.
Standard:  All replicates should contain 50 records.  This table is only generated when this standard has not been met.

Table I.3.  Interview Month, by File Month.  BRFSS protocol states that data collectors should attempt to complete a monthly survey within the prescribed month but that it is more important to call numbers fully according to the BRFSS callback rules than it is to finish within a prescribed month.  The BRFSS standard is that 100% of records in a given file month should be in the identical interview month.  Cells in this table containing records that do not meet this standard are shown in a bold font.  Nevertheless, because of the priority of calling telephone numbers fully according to the BRFSS callback rules, an occasional few percentage points below 100% is not a matter for concern.  A substantial deviation from the 100% standard should, however, be an occasion for an inquiry.  A chronic deviation from the 100% standard is an indicator that the data collector needs to devote more resources (for example, more hours of calling per month or more interviewers) to the BRFSS in order to complete the survey on time.

Table I.4.  File Month, by Assigned Month.  This table shows whether the data collector is submitting sample records prescreened by GENESYS as non-working or business.  BRFSS Policy Memo 98-3 specifies that such records should be submitted to BSB.
Standard:  100% of the records in an assigned month should be in the identical file month.  This table is only generated when this standard has not been met.

II.  Bias

     Is There Evidence of Significant Bias in the Data?  

Table II.1A/B, II.2A/B, II.3A/B.  Discrepancy in Sex, Age, and Race/Ethnicity between 2008 Claritas Population Estimates and Unweighted BRFSS Data, Year-to-Date.  These tables show the percentages among completes of sex, age, and race/ethnicity compared to the 2008 population estimates from Claritas.  The A tables show the unweighted BRFSS data and the B tables show the data with the design weights applied.  The B tables show the true representativeness of the BRFSS data considering the sample design and before post-stratification is done to adjust for selection bias by age, sex, and race/ethnicity.  Large selection biases are a strong indicator of possible biases in the data.  They do not, however, indicate anything about the source of the possible biases.  The source could be any source of non-sampling error, which causes data to not be representative of the sample.  Some sources of non-sampling errors under the control of the data collector are not working the sample hard enough or according to BRFSS protocol, interviewer misconduct (for example, fabrication of interviews, recording one adult in a household in order to be able to interview the person on the phone), or a staff of interviewers without the training or skill to induce hard to interview respondents to complete an interview.  

Standards:  Acceptable ranges for sex, age, and race/ethnicity biases are given below.  A value above or below the acceptable range should trigger a search for possible causes and remedies.  These ranges are based on observed biases in 2000.  Cells in this table containing values that do not meet these standards are shown in a bold font.

Variable	Category	Acceptable Bias Range	
Sex	Female	3% to 9%	
Age	18-24	-4% to 1%	
Age	25-34	-5% to 3%	
Age	35-44	-1% to 3%	
Age	45-54	-1% to 3%	
Age	55-64	-1% to 3%	
Age	65+	-3% to 1%	
Race/Ethnicity	White/Non-Hispanic	-2% to 3%	

Table II.4.  Race by Hispanic Origin.  Hispanics tend to approach race from a different perspective than non-Hispanics.  For this reason, it is difficult to get Hispanics to name one of the standard race categories as their race.  As a result, many Hispanics receive a race of Other.  In the 2000 Census, about 42% of Hispanics indicated their race as Other.
Standard:  More than about 50% of Hispanics with a race of Other may indicate that interviewers have not received appropriate training on probing for the race of Hispanic respondents.  Fewer than about 20% of Hispanics with a race of Other may indicate that interviewers are imputing the race of Hispanics.  Cells in this table containing values that do not meet this standard are shown in a bold font.

Table II.5.  Geo-Stratum by month.  This table presents the number of interviews for each geographic stratum for each month.

III. Sample Management:  Magnitude and Consistency of Effort

     Is the Data Collector Calling Numbers Frequently Enough and According to the BRFSS             Callback Rules?	

Tables III.1 to III.4.  Date, Day of Week, and Final Disposition Code, by File Month and Minimum, Mean, and Maximum Number of Attempts for Completes and One-Plus Block Numbers Respectively.  A good rule-of-thumb is that about 85% of the completes and 75% of one-plus block numbers should received a final disposition in the first half of the interviewing days.  In Tables III.2 and III.4, a month row is printed in bold font if the cumulative  percent is less than 80% as of the fifteenth day of the month.  In Tables III.1, a date row is printed in bold font when the cumulative percent increases by more than 3% in the last four interviewing days of the month.  In Tables III.1 and III.3, any date row indicating interviewing beyond the source month is printed in bold font.  A bulge in the number of  dispositions after the beginning of the interviewing period could be an indicator that additional telephone numbers were released.  This should occur early enough in the interviewing period that there is time to fully call all the released numbers according to the BRFSS callback rules.  A relatively large number of dispositions per day well into the interviewing period or, worse, an increase in the number of dispositions per day at the end of the interviewing period is probably an indication that the data collector needs to devote more resources (for example, more hours of calling per month or more interviewers) to the BRFSS in order to fully call all the released numbers according to the BRFSS callback rules. The tables also indicate the day of the week on which numbers receive a final disposition.  The day of the week can be used to check on weekend calling.  During the last half, or at least the last several days, of the interviewing period, the number of final dispositions by day should be in the single digits.

In Tables III.3, after about the third interviewing day, the minimum number of attempts should go above 1 and keep going up until it levels off at close to 15 near the end of the interviewing period.  (There may be an occasional number with only a few call attempts, but the large majority of records dispositioned close to the end of the interviewing period should have close to 15 call attempts.)  Similarly, the mean number of attempts should be in the 6-8 range by the fifth interviewing day.  In Table III.3, any date row after the fifth interviewing day and prior to the tenth interviewing day in which the mean number of attempts is less than 7 is printed in bold.  

Table III.5.  Date and Day of Week of First and Last Dispositions in Replicate and File Month, Number of Days since First Disposition Date in File Month, Number of Days to Last Disposition Date in File Month, and Number of Days in Field for Replicate and File Month, by File Month for Replicates in Play 14 or Fewer Days.  

The frequencies of completes and incomplete households should be heavily concentrated in the first half of the interviewing days.  The frequencies of non-working and non-private residences should be heavily concentrated in the first two or three days of the interviewing period.  The frequencies of non-contacts should be concentrated toward the end but not at the very end of the interviewing period.  You should also look at the extent and pattern of weekend interviewing.  The hours for weekend interviewing are more limited than for weekdays and weeknights.  At the same time, more people tend to be at home during the weekend.  For these reasons, weekend calling should be approached strategically.  In particular, weekend calling should be heavy enough to call all available numbers during a weekend but it should not be wasted on numbers that have not been called before.  This table only prints if there is a replicate that has been in play fourteen or fewer days.

     How Consistent Is the Data Collection Effort from Month to Month?

Tables III.08 to III.13  Disposition Code.  The even numbered tables (III.08, III.10, and III.12) are Year-to-Date to give you baseline percentages for the year.  The odd numbered tables are by month to give you the monthly variability in the distributions.  You can use these disposition codes to measure consistency and, to a lesser extent, the quality of the data collection effort.  In Table III.09, cells are printed in bold when a disposition code category changes by 10 or more percentage points from one month to the next.  In Table III.11, cells are printed in bold when a disposition code category changes 5 or more percentage points from one month to the next.  Because the base of the percentages in Tables III.12 to III.13 is all records, a difference of even 2 percentage points from one month to the next could be meaningful.  In Table III.13, cells are printed in bold when a disposition code category changes by more than 2 percentage points from one month to the next.  

IV.  Proper Assignment of Disposition Codes

     Is the Data Collector Dispositioning Numbers According to Their Definitions?

Tables IV.1 to IV.5.  Household Roster Status or Number of Attempts.  The standards for these tables are indicated in the titles.  These tables are only generated when these standards have not been met.  Cells in these tables containing values that do not meet their standards are shown in a bold font.

V.  Unit Nonresponse

Table V.1.  Resolution, Screening Completion, Interview Completion, Cooperation, Overall Response, and CASRO Response Rates.  

The Resolution Rate is the proportion of all telephone numbers in the sample for which the status of the cases as households with working numbers has been resolved.  Cases for which household status remains unknown are excluded from the numerator.  The Screening Completion Rate is the proportion of all known households in which the presence of an eligible respondent has been determined.  Households in which the presence or absence of an adult is unknown are excluded.  The Interview Completion Rate is the proportion of contacted eligible respondents who successfully complete an interview.  This rate is a type of cooperation rate.  Minimal guidelines for these three completion rates are 75 percent for the Resolution Rate, 45 percent for the Screening Completion Rate, and 67 percent for the Interview Completion Rate.  Cells in this table containing values that do not meet these standards are shown in a bold font.

The CASRO Rate is a measure of respondent cooperation and is generally defined as the proportion of all eligible respondents in the sample for whom an interview has been completed.  This rate includes completed interviews and partial interviews in which at least 50 percent of the core questionnaire has been completed in the numerator.  The proportion of cases of unknown eligibility that are actually eligible is estimated to be the same as the proportion of cases of known eligibility that are eligible.  The Overall Response Rate is a conservative response rate that includes a higher percentage of all households in the denominator.

A CASRO Rate below 40% or an Overall Response Rate below 30 % should cause review of data collector practices that could impact it, especially sample management and interviewer recruitment, retention, training, supervision, and monitoring.  Cells in this table containing values that do not meet these standards are shown in a bold font.  Fluctuations of more than about 5 percentage points in these outcome rates or a downward trend for three or more months should occasion an inquiry into the data collector’s practices.

The Cooperation Rate is the proportion of all cases interviewed of all eligible units that were actually contacted.  Non-contacts are excluded from the denominator.  This rate is based on contacts with households containing an eligible respondent.  The denominator of the rate includes completed interviews plus the number of non-interviews that involve the identification of and contact with an eligible respondent.  The Refusal Rate is the percentage of all eligible respondents that refused to be interviewed or terminated an interview early in the core questionnaire.  A Cooperation Rate below 65 percent or a Refusal Rate at or below 35 percent indicates some problem with interviewing techniques.  Cells in this table containing values that do not meet these standards are shown in a bold font.

Table V.2.  Percent with Known Eligible Households. Cells in this table are printed in bold if less than 45% are determined to be eligible or if this percentage changes by 5 or more percentage points in successive months.

Table V.3.  Percent With Selected Respondent. This table tells you the percentage of households for which a respondent is selected once a household roster is begun.  This percentage should be at or very close to 100%.  Cells are printed in bold if less than 99% are selected.

Table V.4.  Percent Began Interview.  Cells are printed in bold if 60% or less began interviews.

Table V.5.  Percent Complete or Partial Complete.  Cells are printed in bold if less than 95% are complete.

Table V.6.  Percent Complete.  Cells are printed in bold if less than 92% complete or if 100% are complete.

VI.  Item Nonresponse

     Are the Interviewers Adequately Trained, Supervised, and Monitored?

Tables VI.1A. to VI.3. Income Missing Values.  
States should try to stay below 15% missing values on income for men.  States with a percentage of missing values above 15% should review their training and monitoring of interviewers with respect to income missing values.  States should try to stay below 20% missing values on income for women.  States with a percentage of missing values above 20% should review their training and monitoring of interviewers with respect to income missing values.  Cells in these tables containing values that do not meet these standards are shown in a bold font.

VII.  Household Rosters and Number of Phones

Table VII.1.  Household Roster Status, Records With Partial or Complete Household Rosters.  States should have well under 1% of records with a partial household status and no records with an inconsistent household status.  Cells in this table containing values that do not meet these standards are shown in a bold font.

Table VII.2.  Number of Adults, Records With Non-Missing Number of Adults.
Generally, between 25 and 40% of records should indicate one adult and between 45 and 60% two adults.  Records with six or more adults should be questioned.  Cells in this table are shown in a bold font if they contain values outside of these ranges.

Table VII.3.  Number of Phones.  Generally, about 85-90% of records should indicate one phone line.  Records with three or more phones should be questioned.  Cells in this table are shown in a bold font if they contain values outside of these ranges.

Table VII.4.  Number of Phones, By Number of Adults.  The percentage of records with one phone line should decrease as the number of adults increases.  Improbable combinations, for example, one adult and five phones, should be questioned.  Cells in this table are shown in a bold font if there are 4 or more telephone or 6 or more adults in a household.

VIII.  Recruitment, Retention, and Assignment of Interviewers

Tables VIII.1 and VIII.2.  Recruitment and Retention of Interviewers.  These tables will give you an idea of the stability of the interviewer workforce.  If the turnover seems excessive, you should inquire about the data collectors recruitment base and practices and about their working conditions.

Table VIII.3.  Minimum, Median, Mean, and Maximum Numbers of Completed Interviews per Interviewer.  This table will give you an indication of how much BRFSS interviewers are getting with the current year BRFSS survey.  Continuing small median and mean numbers of completed interviews per interviewer as the interviewer year progresses indicates that interviewers are not assigned enough to the survey to gain experience with it.  This could indicate either high turnover or an excessively large number of interviewers assigned to the BRFSS.

Table VIII.4.  Number of Completed Interviews by Interviewer.

IX.  Interviewer Outliers

     Are There Particular Interviewers Who Seem to Be Deviating from Sound Practices?

Tables IX.01 to IX.20.  These tables should be approached from a systemic and from an individual perspective.  From a systemic perspective, the question is whether or not management provides appropriate training, supervision, and monitoring of interviewers.  For several variables, one indication that this could be the case would be a poor showing on a global measure from an earlier section coupled with a flat distribution and a large standard deviation among interviewers. For example, a percent missing income of over 15% in Table VI.1A coupled with a standard deviation of over 5 percentage points in IX.9 should trigger questions about the training, supervision, and monitoring of interviewers in general with respect to asking and probing for income.   

From an individual perspective, the question is whether or not a particular interviewer is following protocol.  From this perspective, the data should be examined with a view toward finding interviewers who are outliers.  Tables IX.01, IX.02, and IX.03 are particularly useful in identifying outliers.  Interviewers who are outliers on several measures should be monitored especially carefully, even more especially if they consistently beat the norm.  “Appendix B: Explanations of Stem-and-Leaf Displays and Boxplots,” in this document, explains how to read these graphs shown in Table IX.06 to IX.20.

Appendix A:  List of Tables in the 2008 BRFSS Year-to-Date Data Quality Control Report

Note:  Each table title has three sections:  the name of the dependent variable (with categories in parentheses, if appropriate), either the phrase Year-to-Date or the name of a By-variable, and a description of the records in the table (the base).

I.  Sample Generation, Release, and Submission

Table I.1.  Density Status, By Assigned Month, Base = All Records

Table I.2.  Number of Records in Replicate, By Assigned Month, Base = All Records

Table I.3.  Interview Month By File Month, Base = Completes Only

Table I.4.  File Month By Assigned Month, Base = All Records

II. Bias

Table II.1.  Discrepancy in Sex Between 2008 Claritas Population Estimates and Unweighted BRFSS Data, Year-to-Date, Base = Completes Only

Table II.2.  Discrepancy in Age Between 2008 Claritas Population Estimates and Unweighted BRFSS Data, Year-to-Date, Base = Completes Only

Table II.3.  Discrepancy in Race/Ethnicity Between 2008 Claritas Population Estimates and Unweighted BRFSS Data, Year-to-Date, Base = Completes Only

Table II.4.  Race by Hispanic Origin, Year-to-Date, Base = Completes Only

Table II.5.  Geo-Stratum by Month, Year-to-Date, Base = Completes Only

III.  Magnitude and Consistency of Effort

Table III.1. Date, Day of Week, and Final Disposition Code, By File Month, Base = Records With One or More Attempts

Table III.2. Number and Percent of Completes in the First Fifteen Days of the Interview Month

Table III.3.  Date and Day of Week of Final Disposition and Minimum, Mean, and Maximum Number of Attempts,  By File Month, Base = One-Plus Block Numbers With One or More Attempts  

Table III.4.  Date and Day of Week of Final Disposition and Minimum, Mean, and Maximum Number of Attempts, By File Month, Base = Zero Block Numbers With One or More Attempts

Table III.5. Date and Day of Week of First and Last Dispositions in Replicate and File Month, Number of Days Since First Disposition Date in File Month, Number of Days to Last Disposition Date in File Month, and Number of Days in Field for Replicate and File Month, By File Month, Base = Replicates in Play Fourteen or Fewer Days

Table III.08.  Disposition Code (Categorized as Completed Interview; Eligible Household; Household or Probable Household, Eligibility Unknown; Household, No Eligible Respondent; Non-Contact; Non-Household), Year-to-Date, Base = Listed Records

Table III.09.  Disposition Code (Categorized as Completed Interview; Eligible Household; Household or Probable Household, Eligibility Unknown; Household, No Eligible Respondent; Non-Contact; Non-Household), By File Month, Base = Listed Records

Table III.10. Disposition Code, (Categorized as: Completed Interview: Eligible Household: Household or Probable Household, Eligibility Unknown: Household, No Eligible Respondent: Non-Contact: Non-Household), Year-to-Date, Base = All Records

Table III.11. Disposition Code, (Categorized as: Completed Interview: Eligible Household: Household or Probable Household, Eligibility Unknown: Household, No Eligible Respondent: Non-Contact: Non-Household), By File Month, Base = All Records

Table III.12.  Disposition Code (Categorized as Completed Interview; Eligible Household; Household or Probable Household, Eligibility Unknown; Household, No Eligible Respondent; Non-Contact; Non-Household), Year-to-Date, Base = All Records

Table III.13.  Disposition Code (Categorized as Completed Interview; Eligible Household; Household or Probable Household, Eligibility Unknown; Household, No Eligible Respondent; Non-Contact; Non-Household), By File Month, Base = All Records

IV.  Proper Assignment of Disposition Codes

Table IV.1. Disposition Code, (Categorized as: Completed Interview: Eligible Household: Household or Probable Household, Eligibility Unknown: Household, No Eligible Respondent), By Density Status, Base = Household or Probable Household Records

Table IV.2. Disposition Code, (Categorized as: Completed Interview: Eligible Household: Household or Probable Household, Eligibility Unknown: Household, No Eligible Respondent: Non-Contact: Non-Household), By Density Status, Base = All Records

Table IV.3. Household Roster Status, By Assigned Month, Base = Records With Final Disposition Codes of 110 to 260, (Should Have Only Consistent Household Roster)

Table IV.4. Number of Attempts, By Assigned Month, Base = Records With Final Disposition Codes of 315,332,325,340,345,350,360,365, (Should 15+ Attempts)

Table IV.5. Number of Attempts, By Assigned Month, Base = Records With Final Disposition Codes of 370, (Should 0 Attempts)

V.  Unit Non-response

Table V.1. Resolution, Screening Completion, Interview Completion, Cooperation, Refusal, Overall Response, and CASRO Response Rates, Year-to-Date and by Assigned Month, Base = All Records

Table V.2. Percent With Determined Household Eligibility, Base = Eligible Households and Households with Undetermined Eligibility, (Records with Final Disposition Codes of 110-355,370)

Table V.3.  Percent With Selected Respondent, By File Month, Base = Eligible Households (Records With Final Disposition Codes of 110-280)

Table V.4.  Percent Began Interview, By File Month, Base = Households With Selected Respondent (Records With Final Disposition Codes of 110-260)

Table V.5. Percent Complete or Partial Complete, By File Month, Base = Households with Begun Interview (Records with Final Disposition Codes of 110-210)

Table V.6. Percent Complete, By File Month, Base = Completes or Partial Completes, (Records with Final Disposition Codes of 110–120)

VI.  Item Non-response

Table VI.1A.  Income (77 and 99 Collapsed), Year-to-Date, Base = Completes Only

Table VI.2.  Income (77 and 99 Collapsed), By File Month, Base = Men Only

Table VI.3.  Income (77 and 99 Collapsed), By File Month, Base = Women Only

VII.  Household Rosters, Number of Adults, and Number of Phones

Table VII.1.  Household Roster Status, Year-to-Date, Base = Records with Partial or Complete Household Rosters

Table VII.2.  Number of Adults, Base = Records with Non-Missing Number of Adults

Table VII.3.  Number of Phones, Base = Completes Only

TableVII.4.  Number of Phones, By Number of Adults, Base = Completes Only
VIII.  Recruitment, Retention, and Assignment of Interviewers

Table VIII.1. Number of Interviewers by Number of Months Interviewer Working, Year-to-Date, Base = All Records

Table VIII.2. Recruitment and Retention of Interviewers, by Assigned Month, Base = All Records;

Table VIII.3. Minimum, Median, Mean, and Maximum Numbers of Completed Interviews per Interviewers, by Assigned Month, Base = Completes Only

Table VIII.4. Number of Completed Interviews by Interviewer ID, Year-to-Date, Base = Completes Only

IX.  Interviewer Outliers - A. Bias

Table IX.01.  Interviewer Outliers, Base=Completes Only

Table IX.02.  Interviewer Outliers, Base=Women Only

Table IX.03.  Interviewer Outliers, Base=Men Only

Table IX.04.  Distribution of Percent Female, By Interviewer ID, 
Base = Completes Only

Table IX.05.  Distribution of Percent Age 18–24, By Interviewer ID, Base = Completes Only

Table IX.06.  Distribution of Percent Age 65 Plus, By Interviewer ID, Base = Completes Only

Table IX.07.  Distribution of Percent White, By Interviewer ID, Base = Completes Only

Table IX.08.  Distribution of Percent Hispanic, By Interviewer ID, Base = Completes Only

Table IX.10.  Distribution of Percent White, Non-Hispanic, By Interviewer ID, Base = Completes Only

Table IX.11.  Distribution of Percent Other Race, By Interviewer ID, Base = Hispanics Only

Table IX.12.  Distribution of Percent White, By Interviewer ID, Base = Hispanics With Reported Race Only

IX.  Interviewer Outliers - B.  Item Non-response

Table IX.13.  Distribution of Percent Income Missing (77 and 99 Collapsed), By Interviewer ID, Base = Men Only

Table IX.14.  Distribution of Percent Income Less Than $10,000, By Interviewer ID, Base = Men Only

Table IX.15.  Distribution of Percent Income Greater Than $75,000, By Interviewer ID, Base = Men Only

Table IX.16.  Distribution of Percent Income Missing (77 and 99 Collapsed), By Interviewer ID, Base = Women Only

Table IX.17.  Distribution of Percent Income Less Than $10,000, By Interviewer ID, Base = Women Only

Table IX.18.  Distribution of Percent Income Greater Than $75,000, By Interviewer ID, Base = Women Only

Table IX.19.  Distribution of Percent Income Don’t Know/Not Sure (77), By Interviewer ID, Base = Men Only

Table IX.20.  Distribution of Percent Income Refused (99), By Interviewer ID, Base = Men Only

Appendix B:  Explanations of Stem-and-Leaf Displays and Boxplots

Stem-and-leaf displays and boxplots allow the examination of a distribution for the purpose of identifying extreme values, or outliers.  Stem-and-leaf displays and boxplots are found in the output of Section IX.  They also appear in the discussions of Tables VI.1A to VI.2B on pages 11 and 12 of this document.

In a stem-and-leaf display, the first column of numbers is the stem.  In these tables, the stem usually represents a whole percentage number.  The second sets of numbers are the leaves.  Each observation is represented by one digit (leaf).  In these tables, the leaves usually represent rounded tenths of a percent.  As a whole, a stem-and-leaf can be thought of as a vertical histogram in that the lengths of the leaves are proportional to the relative frequencies in an interval.  Immediately to the right of the stem-and-leaf display proper is a column showing the number of observations in each interval.  Observations toward the ends of a distribution that are separated from other observations by one or more blank intervals are candidates to be considered as outliers.

A box plot provides a more formal statistical approach to identifying outliers.  “The bottom and top edges of the box correspond to the sample 25th (Q1) and 75th (Q3) percentiles.  The box length is one interquartile range (Q3-Q1).  The center horizontal line with asterisk endpoints corresponds to the sample median.  The central plus sign (+) corresponds to the sample mean.  If the mean and median are equal, the plus sign falls on the line inside the box.  The vertical lines that project out from the box, called whiskers, extend as far as the data extend, up to a distance of 1.5 interquartile ranges.  Values farther away are potential outliers.  The procedure identifies the extreme values with a zero or an asterisk (*).  If zero appears, the value is between 1.5 and 3 interquartile ranges from the top or bottom edge of the box.  If an asterisk appears, the value is more extreme.” (SAS Institute Inc., SAS Procedures Guide, Version 8, Cary, NC: SAS Institute Inc., 1999.  1643 pp.  Page 1389.)

Appendix C:  BRFSS CASRO Response Rate Formula

Completes = Completed or Partially Completed Interviews 
Completes = (110+120+(210×.32))

Eligible=All respondents with known eligibility status categorized as eligible
Eligible = (110+120+210+220+230+240+250+260+270+280)

Ineligible= All respondents with known eligibility status categorized as ineligible
Ineligible=(405+410+420+430+435+440+450)

Unknown=All respondents with unknown eligibility status
Unknown=(305+310+315+320+325+330+332+335+340+345+350+355+360+365+370)

UNKNDNOM=Unknown respondents added to the denominator
UNKNDNOM = (Eligible/(Eligible + Ineligible)) × Unknown

CASRO = (Completes / (Eligible + UNKNDNOM))

Appendix D:  BRFSS Overall Response Rate Formula

Completes = Completed or Partially Completed Interviews 
Completes = (110+120+(210×.32))

Break-offs and Refusals = ((210×.68)+220)
Known Households = (230+240+250+260+270+280+305+310+315+335)
Ineligible Households = 410
All Likely Households= (345+350+320+325+330+332+340+370+355)

Households = (Known Households+Ineligible Households+Completes+Break-offs and Refusals+(.90×All Likely Households))
Eligible Households = (.98×Households)

Overall Response Rate = (Completes/Eligible Households)

Appendix E:  BRFSS Cooperation Rate Formula

Completes = Completed or Partially Completed Interviews 
Completes = (110+120+(210×.32))

Break-offs and Refusals = ((210×.68)+220)

Cooperation Rate = (Completes/(Completes+Break-offs and Refusals+250+260))

Appendix F:  Instructions for Assigning BRFSS Disposition Codes

I.	Introduction

This appendix presents a minimum set of final and interim codes for use with the BRFSS survey.  States may use more detailed interim and final disposition codes internally as long as the interim disposition codes used are subsets of those listed below and the interim and final disposition codes sent to the Behavioral Surveillance Branch (BSB) are those listed below.  The one major exception is that states using the WinCATI questionnaire supported by BSB can use only the interim and final codes specified below.  States may also treat the callback rules as minimal standards and adopt more strenuous efforts to complete interviews.

Section II discusses the criteria used to decide on final and interim disposition codes.  Section III describes the process for assigning final disposition codes that require taking into account the outcomes from more than one attempt.  Section IV presents the definitions of and callback rules for final disposition codes.  Section V presents the definitions of and callback rules for interim disposition codes.  Finally, Section VI describes the rules for assigning particular final disposition codes from specific patterns of interim disposition codes.

II.	Criteria for the Adoption of Final Disposition Codes

Final disposition codes serve at least four purposes for four usually distinct groups of people.  All four purposes relate to data quality but from different perspectives.

From a research perspective, the major concern is the potential for bias in the data.  Researchers are most interested in response rates which, conceptually, are the number of completes divided by an estimate of the number of eligible households in the sample.  From this perspective, telephone numbers for which it is unknown whether or not they ring into an eligible household should be assigned disposition codes that reflect as much information as is known about the telephone number.  These codes should distinguish as much as possible telephone numbers that have a large probability of ringing into an eligible household from those that have a small probability of ringing into an eligible household.

From a sponsor’s perspective, the major concern is adherence to protocol.  (A sponsor in this context is any organization with a financial stake in the BRFSS survey.)  A sponsor wants to know how good a job the data collector is doing.  Even conceptually, however, response rates should be affected by the characteristics of the target population in addition to the performance of the data collector.  (And in practice, response rates are also affected by sample design and characteristics of the telephone system.)  But the data collector has control only over its performance.  Protocol defines the standards that the data collector should meet and those standards have been designed to produce acceptable quality data.  From this perspective, disposition codes should distinguish between telephone numbers to which different protocols apply.  For example, the difference between a refusal from a selected respondent and a postponement from a selected respondent that never results in a completed interview is important from this perspective because different callback rules apply.  This same difference, however, is unimportant from a potential-for-bias perspective because they are both incompletes from an eligible household.

From a data collection perspective, the major concern is operational control.  Although adherence to protocol is part of operational control, efficiency and interviewer performance are more important than from other perspectives.  For example, the distinction between “Language problem after respondent selection” and “Physical or mental impairment after respondent selection” is unimportant from other perspectives but is important from an operational perspective.

From a weighting perspective, the major concern is to identify the amount of nonresponse that occurs at each point at which nonresponse is possible.  The collection of information on the point in the interview process at which a nonresponse occurs allows explicit adjustment for nonresponse at each point.  This, in turn, should lead to more accurate estimates of statistical parameters, such as means and regression coefficients.  From a sponsor’s perspective, how much nonresponse occurs each point at which it is possible is less important than the total amount of nonresponse.

The need to satisfy at least four distinct purposes that imply different criteria for the distinctions among final disposition codes, has resulted in the adoption of thirty-two final disposition codes.

III.	The Process for Assigning Final Disposition Codes from Interim Disposition Codes

When the BRFSS first began in 1984, most states conducted the survey on paper and there were only eleven final disposition codes.  In such a situation, it was reasonable to expect the interviewer to be aware of the call history of each piece of sample and to assign a final disposition accordingly.  With the adoption of CATI packages in all BRFSS states and advances in the capabilities of CATI packages, a different model for assigning disposition codes is now possible.  For many years, DOS Ci3 CATI and now WinCATI have assigned final disposition codes to records with fifteen or more attempts.  Over the past few years, BRFSS has been moving even further towards a model where the interviewer assigns a disposition code (which may be interim or final) after each attempt strictly based on what happened on that attempt only.  For example, in data year 2001, BRFSS protocol for the first time specified the assignment of final disposition codes based on patterns of previously assigned interim disposition codes.  It was envisioned that at least some states would program the assignment of final disposition codes and that interviewers would be assigning final disposition codes only in cases where the outcome of a single call attempt dictated a final disposition, for example a complete or a non-working number.

In 2002, further steps were taken towards the adoption of a model where the interviewer assigns a disposition code (which may be interim or final) after each attempt based strictly on what happened on that attempt only.  If an interim disposition code is assigned, then the CATI package reviews all the interim disposition codes for that telephone number to determine if a final disposition code is warranted.  For example, if a respondent (non-irately) refuses to continue, the interviewer would code the attempt as an interim refusal.  The CATI package then looks to see if the refusal was a second refusal and, if so, at what point during the interview process the refusal was made.  It then assigns the appropriate final disposition code and retires the number.  This process is run after each attempt that is given an interim disposition.

There are at least three advantages to such an approach:  (1) The interviewer needs to remember a smaller number of interim and final disposition codes than would otherwise be the case.  (2) The interviewer assigns only those final disposition codes that depend on what happened in a single attempt.  Thus, the interviewer does not need to be aware of the call history of a number to assign a final disposition code.  (Of course, the interviewer may still need to be aware of the call history of a number in order to try to complete an interview.)  (3) Human errors in the assignment of final disposition codes based on interim disposition codes are eliminated (assuming correct programming).  The disadvantages are that it will probably require additional programming to implement the approach and that not all CATI packages may be capable of implementing it.

Beginning with data year 2002, WinCATI was programmed to implement this approach.  Please note that WinCATI users of the BSB-prepared questionnaire are able to call a number more than 15 times only by scheduling an appointment on the fifteenth and each subsequent call attempt.

The interim disposition codes are the minimal set implied by the final disposition codes and two additional codes—Null attempt and Requires supervisor attention—that are useful for operational purposes.

IV.	Final BRFSS Disposition Codes

1.	Interview

110	Complete
Definition:  Selected respondent meets the criteria for a partial complete and has completed the interview through the last question.
Callback Rules:  Give final disposition upon completion of interview.

120	Partial Complete
Definition:  Sex and three or more questions from age, race, Hispanic origin, marital status, education, employment status, county code, and “Do you have more than one telephone number in your household?”  have been answered with a response other than Don’t know/Not sure (7, 77, or 777) or Refused (9, 99, or 999).
Callback Rules:  Make a second attempt to fully complete the interview after first refusal or termination.  Give final disposition if second attempt to fully complete the interview is unsuccessful.  Give final disposition on the fifteenth or subsequent call attempt even if there is only one occurrence of a refusal or termination.

2.	Non-Interview, Household With Eligible Respondent

210	Termination within questionnaire
Definition:  A hang-up or termination after the first question in the core has been asked and it or a subsequent question has received a response other than Don’t know/Not sure or Refused.  The selected respondent has not answered enough questions for the interview to quality as a 120 Partial complete.
Callback Rules:  Give final disposition after second refusal or termination or when a first-time refusal or termination will not be called a second time because of an irate respondent. Give final disposition on the fifteenth or subsequent call attempt even if there is only one occurrence of a refusal or termination.

220	Refusal after respondent selection
Definition:  A hang-up or termination after respondent selection but before respondent has given a response other than Don’t know/Not sure or Refused to one or more questions in the core.  The refusals can come from any adult in the household and the initial refusal could have come before respondent selection.  An automated message to not call the number again that is not in response specifically to that call attempt does not count as a refusal.  Such an outcome should receive an answering machine or technological barrier interim code.
Callback Rules:  Give final disposition after second refusal or when a first-time refusal will not be called a second time because of an irate respondent.  On the fifteenth or subsequent call attempt, give final disposition even if there is only one occurrence of a refusal.

230	Selected respondent never reached or was reached but did not begin interview during interviewing period
Definition:  Selected respondent was never spoken to or was spoken to and asked to be called again later one or more times.  Includes cases where the selected respondent was away from residence for part of the interviewing period.
Callback Rules:  Give final disposition only after (a) at least 5 calling occasions (each consisting of no more than 3 attempts at least one hour apart) for a minimum total of 15 call attempts, and (b) the 15 or more call attempts consist of at least 3 weekday calls, 3 weeknight calls, and 3 weekend calls.

240	Selected respondent away from residence during the entire interviewing period
Definition:  Selected respondent is expected to be away from residence during the entire interviewing period, for example, because of travel or a hospital stay.
Callback Rules:  Give final disposition when informed of absence.

250	Language problem after respondent selection
Definition:  After respondent selection, the selected or another respondent does not speak English or another language for which an interviewer and translated questionnaire are available well enough to be interviewed.
Callback Rules:  Give final disposition (1) the first time a selected respondent is contacted or is described by someone else as not speaking English or another language for which an interviewer and translated questionnaire are available well enough to be interviewed or (2) the second time a respondent who does not speak English well enough to answer the screening questions is contacted and there is not a translated questionnaire and interviewer available for the respective language.  If the first occurrence is on the fifteenth or subsequent call attempt, give final disposition.

260	Selected respondent physically or mentally unable to complete an interview during the entire interviewing period
Definition: After respondent selection, the selected or another respondent has a physical or mental condition that prevents the completion of an interview and that condition is expected to last through the entire interviewing period.  This includes a temporary condition, such as bereavement, that will last beyond the interviewing period.  (For conditions not expected to last through the entire interviewing period, schedule an appointment and keep trying.)
Callback Rules:  Give final disposition (1) the first time a selected respondent is contacted or is described by someone else as physically or mentally unable to complete an interview during the entire interviewing period or (2) the second time a respondent who is physically or mentally impaired is contacted.  If the first occurrence is on the fifteenth or subsequent call attempt, give final disposition.

270	Hang up or termination after number of adults recorded but before respondent selection
Definition:  Respondent hangs up or terminates call attempt after answering the number of adults question but before answering the number of men and number of women questions.  This differs from 280 in that the respondent explicitly refuses.
Callback Rules:  Give final disposition after second hang-up or termination or when a first-time hang-up or termination will not be called a second time because of an irate respondent.  If the first occurrence is on the fifteenth or subsequent call attempt, give final disposition.

280	Household contact after number of adults recorded but before respondent selection
Definition:  Respondent answered the number of adults question and asked to be called again later one or more times but the number of men and number of women were never determined.  On the surface, this is a postponement that was never re-started but may be an implicit refusal.  This differs from 270 in that the respondent never explicitly refuses.
Callback Rules:  Give final disposition only after (a) at least 5 calling occasions (each consisting of no more than 3 attempts at least one hour apart) for a minimum total of 15 call attempts, and (b) the 15 or more call attempts consist of at least 3 weekday calls, 3 weeknight calls, and 3 weekend calls.

3.	Non-Interview, Eligibility Undetermined

305	Household members away from residence during entire interviewing period
Definition:  A house sitter, house cleaner, or other non-member of a household states that all of the household members will be away from the residence during the entire interviewing period.
Callback Rules:  Give final disposition when informed.

310	Hang-up or termination, housing unit, unknown if eligible respondent
Definition:  A respondent hangs-up or terminates a call attempt before answering the number of adults question.  This differs from 315 in that the respondent explicitly refuses.
Callback Rules:  Give final disposition after second hang-up or termination or when a first-time hang-up or termination will not be called a second time because of an irate respondent.  If the first occurrence is on the fifteenth or subsequent call attempt, give final disposition.

315	Household contact, eligibility undetermined
Definition:  A respondent verified that the telephone number reaches a private residence and asked to be called again later one or more times but the number of adults in the household was never determined.  On the surface, this is a postponement that was never re-started but may be an implicit refusal.  This differs from 310 in that the respondent never explicitly refuses.
Callback Rules:  Give final disposition only after (a) at least 5 calling occasions (each consisting of no more than 3 attempts at least one hour apart) for a minimum total of 15 call attempts, and (b) the 15 or more call attempts consist of at least 3 weekday calls, 3 weeknight calls, and 3 weekend calls.

320	Language problem before respondent selection
Definition:  A respondent who does not speak English or another language for which an interviewer and translated questionnaire are available well enough to answer the screening questions answers the telephone twice before respondent selection.  Give final disposition even if other respondents who do not present a language problem have answered the telephone.
Callback Rules:  Give final disposition after second contact with a respondent who does not speak English well enough to answer the screening questions and there is not a translated questionnaire and interviewer available for the respective language.  If the first occurrence is on the fifteenth or subsequent call attempt, give final disposition.

325	Physical or mental impairment before respondent selection
Definition:  A respondent whose physical or mental impairment prevents him or her from completing the screening questions answers the phone twice before respondent selection.  Give final disposition even if other respondents without a physical or mental impairment have answered the telephone.
Callback Rules:  Give final disposition after second contact with a physically or mentally impaired respondent.  If the first occurrence is on the fifteenth or subsequent call attempt, give final disposition.

330	Hang-up or termination, unknown if private residence
Definition:  A respondent hangs-up or terminates a call attempt before confirming that the telephone number rings to a private residence.
Callback Rules:  Give final disposition after second hang-up or termination or when a first-time hang-up or termination will not be called a second time because of an irate respondent.  If the first occurrence is on the fifteenth or subsequent call attempt, give final disposition.

332	Contact, unknown if private residence
Definition:  A respondent did not verify that the telephone number reaches a private residence but asked to be called again later one or more times.  On the surface, this is a postponement that was never re-started but may be an implicit refusal.  This differs from 330 in that the respondent never explicitly refuses.
Callback Rules: Give final disposition only after (a) at least 5 calling occasions (each consisting of no more than 3 attempts at least one hour apart) for a minimum total of 15 call attempts, and (b) the 15 or more call attempts consist of at least 3 weekday calls, 3 weeknight calls, and 3 weekend calls.

335	Telephone answering device, message confirms private residential status
Definition:  One or more call attempts reached an answering machine but no person was ever spoken to.  The message confirms that the telephone number reaches a private residence by using the words, “home,” “house,” “family,” “residence” or a family name.
Callback Rules:  Give final disposition only after (a) at least 5 calling occasions (each consisting of no more than 3 attempts at least one hour apart) for a minimum total of 15 call attempts, and (b) the 15 or more call attempts consist of at least 3 weekday calls, 3 weeknight calls, and 3 weekend calls.

340	Telecommunication technological barrier, message confirms private residential status
Definition:  One or more call attempts reached a call blocking message, a message asking the caller to identify himself or herself, or other automated message, but no person was ever spoken to. A message confirms that the telephone number reaches a private residence by using the words, “home,” “house,” “family,” “residence” or a family name.
Callback Rules:  Give final disposition only after (a) at least 5 calling occasions (each consisting of no more than 3 attempts at least one hour apart) for a minimum total of 15 call attempts, and (b) the 15 or more call attempts consist of at least 3 weekday calls, 3 weeknight calls, and 3 weekend calls.

345	Telephone answering device, not sure if private residence
Definition:  One or more call attempts reached an answering machine but no person was ever spoken to.  The message leaves open the possibility that the telephone number is reaching a private residence but it does not explicitly state so.
Callback Rules:  Give final disposition only after (a) at least 5 calling occasions (each consisting of no more than 3 attempts at least one hour apart) for a minimum total of 15 call attempts, and (b) the 15 or more call attempts consist of at least 3 weekday calls, 3 weeknight calls, and 3 weekend calls.

350	Telecommunication technological barrier, not sure if private residence
Definition:  One or more call attempts reached a call blocking message, a message asking the caller to identify himself or herself, or other automated response, but no person was ever spoken to.  There is no message or a message leaves open the possibility that the telephone number is reaching a private residence but it does not explicitly state so.
Callback Rules:  Give final disposition only after (a) at least 5 calling occasions (each consisting of no more than 3 attempts at least one hour apart) for a minimum total of 15 call attempts, and (b) the 15 or more call attempts consist of at least 3 weekday calls, 3 weeknight calls, and 3 weekend calls.

355	Telephone number has changed status from household or possible household to non-working during the interviewing period
Definition:  On the second or subsequent call attempt, a telephone number is responding with a message indicating that the telephone number called is a non-working number or has been changed and there is at least one previous interim disposition of 505, 510, 515, 520, 525, 530, 535, or 540.  (If a “number changed” recording is encountered the first time that a telephone number is called, that number should received a final disposition of 450 Non-working/disconnected number.)
Callback Rules:  Give final disposition when notified.

360	No answer
Definition:  Among telelphone numbers which no person or device ever answered, half or more of the call attempts resulted in a normal telephone ring that no one answered.
Callback Rules:  Give final disposition only after (a) at least 5 calling occasions (each consisting of no more than 3 attempts at least one hour apart) for a minimum total of 15 call attempts, and (b) the 15 or more call attempts consist of at least 3 weekday calls, 3 weeknight calls, and 3 weekend calls.

365	Busy
Definition:  Among telephone numbers which no person or device ever answered, more than half of the call attempts resulted in a normal busy signal.
Callback Rules:  Give final disposition only after (a) at least 5 calling occasions (each consisting of no more than 3 attempts at least 10 minutes apart) for a minimum total of 15 call attempts, and (b) the 15 or more call attempts consist of at least 3 weekday calls, 3 weeknight calls, and 3 weekend calls.  If possible, contact the telephone company repair service to verify the number is in service.

370	On never call list
Definition:  To be assigned to (those few) telephone numbers that the BRFSS State Coordinator has determined, before calling begins, should not be called. 
Callback Rules:  This disposition should never be assigned to a telephone number with one or more call attempts.

4.	Not Eligible

405	Out-of-state
Definition:  The telephone number rings out-of-state. 
Callback Rules:  Give final disposition when informed.  This code should take priority over other possible final disposition codes.

410	Household, no eligible respondent
Definition:  No one 18 years of age or older uses the telephone.  To be assigned when no one in the household is 18 years of age or older or the telephone number is used by a teen under the age of 18 and the parents do not use that phone.
Callback Rules:  Give final disposition when informed.

420	Not a private residence
Definition:  The person answering the phone or an answering machine identifies the telephone number as a business, an institution (government office, educational facility, dormitory, nursing home, hospital, prison), a group home (fraternity or sorority, half-way house, shelter), a timeshare or vacation home at which no one is living for 30 days or more at the time of contact, Efax service, a pager, or a dedicated fax/data/modem line that s/he answered to identify as such.
Callback Rules:  Give final disposition when informed.  Also, this disposition code should be assigned to numbers identified with a pre-call status code (PRECALL variable) of 5 Business.  These telephone numbers need not be called.  Data collectors may wish to make one call attempt to these numbers to verify their pre-call status.

430	Dedicated fax/data/modem line with no human contact
Definition:  A telephone number used only as a fax, data, or modem line.
Callback Rules:  Give final disposition only after (a) at least 2 calling occasions (each consisting of no more than 3 attempts at least one hour apart) for a minimum total of 6 call attempts with at least one interim disposition code of 560 Fax/modem/data and all other disposition codes are 550 No answer, 555 Busy, 565 Fast busy, 570 Possible non-working number, or 575 Circuit busy.  (Attempts receiving interim disposition codes of 580 Null attempt or 585 Requires supervisor attention should not count as call attempts for this purpose.)

435	Cellular Telephone
Definition:  The telephone number rings to a cell phone. 
Callback Rules:  Give final disposition when informed.  This code should be assigned 
upon notification by the respondent that the conversation is taking place on a cellular or mobile phone.  This disposition will take priority over other possible final disposition codes.  Also, this disposition code should be assigned to numbers identified with a pre-call status code (PRECALL variable) of 4 Cell phone – PRO-T-S or 6 Cell phone – Interviewer.  These sample telephone numbers should not be called.

440	Fast busy
Definition:  A telephone number with at least one disposition of 565 Fast busy and all other dispositions are 550 No answer, 555 Busy, 570 Possible non-working number, or 575 Circuit busy.  (Attempts receiving interim disposition codes of 580 Null attempt or 585 Requires supervisor attention should not count as call attempts for this purpose.) 
Callback Rules:  Give final disposition only after (a) at least 2 calling occasions (each consisting of no more than 3 attempts at least one hour apart) for a minimum total of 6 call attempts with at least one interim disposition code of  565 Fast busy and all other disposition codes are 550 No answer, 555 Busy, 570 Possible non-working number, or 575 Circuit busy.  (Attempts receiving interim disposition codes of 580 Null attempt or 585 Requires supervisor attention should not count as call attempts for this purpose.)  

450	Non-working/disconnected number
Definition:  Usually recognized by a tritone, a recording indicating that the telephone number is non-working, a number that consistently rings to an incorrect number, or a number that cannot be verified by a respondent.  If a 450 is assigned on a second or subsequent attempt, the CATI system or a post-data collection program should check to see if there is at least one previous interim disposition of 505, 510, 515, 520, 525, 530, 535, or 540.  If there is, then the record should receive a final disposition of 355 Telephone number has changed status from household or possible household to non-working during the interviewing period.  If the current status of the telephone number is in doubt, give an interim disposition of 570 Possible non-working number or call operator or repair service.  After at least 15 call attempts, assign if all dispositions are 545 Phone number temporarily out of service, 570 Possible non-working number, or 575 Circuit busy.  (Attempts receiving interim disposition codes of 580 Null attempt or 585 Requires supervisor attention should not count as call attempts for this purpose.)
Callback Rules:  Give final disposition when the above criteria are met.  If 15 call attempts are required, give final disposition only after (a) at least 5 calling occasions (each consisting of no more than 3 attempts at least one hour apart) for a minimum total of 15 call attempts, and (b) the 15 or more call attempts consist of at least 3 weekday calls, 3 weeknight calls, and 3 weekend calls.  Also, this disposition code should be assigned to numbers identified with a pre-call status code (PRECALL variable) of 3 Non-working number.  These telephone numbers need not be called.  Data collectors may wish to make one call attempt to these numbers to verify their pre-call status.

V.	Interim Disposition Codes

Refusal:  hang-up or termination
Definition:  Respondent hangs up or refuses to continue the interview at any time during the call attempt from immediately after pick-up to any time before answering the last survey question.
Callback Rules:  Give interim disposition when this situation occurs.  Schedule callback for as long as practical up to two weeks after initial refusal.

510	Appointment
Definition:  Respondent asks for a callback at some other time or a child answers the phone and does not get an adult to speak to the interviewer.  
Callback Rules:  Schedule a callback for an appropriate time.

515	Language problem
Definition:  A respondent other than one known to be the selected respondent does not speak English or another language for which an interviewer and translated questionnaire are available well enough to respond appropriately.
Callback Rules:  Give interim disposition when this situation occurs.  Call back after an interval of at least one day.

520	Physical or mental impairment
Definition:  A respondent other than one known to be the selected respondent cannot respond appropriately because of a mental or physical impairment.
Callback Rules:  Give interim disposition when this situation occurs.  Call back after an interval of at least one day.

525	Answering machine, message confirms residential status
Definition:  An answering machine gives a message confirming that the telephone number rings to a private residence by using the words, “home,” “house,” “family,” “residence” or a family name in the message.
Callback Rules:  Give interim disposition when this situation occurs.  Call back after an interval of at least one hour.

530	Technological barrier other than answering machine, message confirms residential status
Definition:  A device other than an answering machine gives a message confirming that the telephone number rings to a private residence by using the words, “home,” “house,” “family,” “residence” or a family name in the message.
Callback Rules:  Give interim disposition when this situation occurs.  Call back after an interval of at least one hour.

535	Answering machine, not sure if private residence
Definition:  An answering machine gives a message that leaves open the possibility that the telephone number is reaching a private residence.
Callback Rules:  Give interim disposition when this situation occurs.  Call back after an interval of at least one hour.

540	Technological barrier other than answering machine, not sure if private residence
Definition:  A device other than an answering machine gives a message that leaves open the possibility that the telephone number is reaching a private residence.
Callback Rules:  Give interim disposition when this situation occurs.  Call back after an interval of at least one hour.

545	Phone number temporarily out of service
Definition:  A recorded message states that the telephone number is temporarily out of service
Callback Rules:  Give interim disposition when this situation occurs.  Call back after an interval of at least one day.

550	No answer
Definition:  A normal telephone ring that no one answers.
Callback Rules:  Give interim disposition when this situation occurs.  Call back after an interval of at least one hour.

555	Busy
Definition:  A normal busy signal.
Callback Rules:  Give interim disposition when this situation occurs.  Call back after an interval of at least ten minutes.

560	Fax/data/modem
Definition:  An electronic screeching sound.
Callback Rules:  Give interim disposition when this situation occurs.  Call back after an interval of at least one hour.

565	Fast busy
Definition:  A faster than normal busy signal.
Callback Rules:  Give interim disposition when this situation occurs.  Call back after an interval of at least one hour.

570	Possible non-working number
Definition:  A noise or other response that may indicate a non-working number.
Callback Rules:  Phone telephone company immediately or give interim disposition when this situation occurs and call back after an interval of at least one hour.

575	Circuit busy
Definition:  A recorded message states that “All circuits are busy.”
Callback Rules:  Give interim disposition when this situation occurs.  Call back after an interval of at least one hour.

580	Null attempt
Definition:  The number came up on an interviewer’s screen but is not attempted.
Callback Rules:  Give interim disposition when this situation occurs.  The supervisor should delete the call attempt information on a null attempt before the telephone number is attempted again.  Do not count as one of the up to fifteen attempts.  Call back as appropriate.

585	Requires supervisor attention
Definition:  An interviewer encounters an unusual situation that requires intervention by a supervisor.
Callback Rules:  Give interim disposition when this situation occurs.  The supervisor should replace the 585 interim disposition code with an appropriate interim or final disposition code before the telephone number is attempted again.  Call back as appropriate.

VI.	Assignment of Final Disposition Based on Interim Dispositions

In the process described earlier, after each call attempt where the interviewer has assigned an interim disposition code, the CATI package reviews all the interim disposition codes that the telephone number has received and, if appropriate, assigns a final disposition code to that telephone number.  The table below shows the appropriate final disposition that should be assigned in such cases.  States should contact their Project Officer about cases not covered by the table below.

States should ensure that every record whose last disposition in its call history is an interim disposition is assigned a final disposition according to the following rules.  The rules below are hierarchical.  The first rule should be applied first to all records with a last, interim disposition code, then the second rule to the remaining records, etc.  As noted above, beginning with data year 2002, WinCATI will be programmed to implement this approach.

A final disposition code of 450 Non-working/disconnected number may need to be converted to a final code of 355 Telephone number has changed status from household or possible household to non-working during the interviewing period.  Note that this conversion is programmed into the Ci3 questionnaire in WinCATI.  States using other CATI system should configure their CATI systems to make the change or they should make the change during their post-data collection processing.

When an interviewer assigns a final disposition code of 450 Non-working/disconnected number on a second or subsequent call attempt, the previous disposition codes should be reviewed.  If at least one previous code is 505, 510, 515, 520, 525, 530, 535, or 540, then the final disposition code should be changed to 355 Telephone number has changed status from household or possible household to non-working during the interviewing period.  This situation is included in the table below even though it involves changing a final (as opposed to interim) disposition code.

If There Is/Are	Assign Indicated Final Disposition Code If	
Two interim dispositions or one interim disposition on or after a fifteenth call attempt of 505 Refusal:  hang-up or termination	The questions answered meet the definition for a 120 Partial complete	
Two interim dispositions or one interim disposition on or after a fifteenth call attempt of 505 Refusal:  hang-up or termination	The questions answered meet the definition for a 210 Termination within questionnaire	
Two interim dispositions or one interim disposition on or after a fifteenth call attempt of 505 Refusal:  hang-up or termination	The questions answered meet the definition for a 220 Refusal after respondent selection	
Two interim dispositions or one interim disposition on or after a fifteenth call attempt of 505 Refusal:  hang-up or termination	The questions answered meet the definition for a 270 Hang up or termination after number of adults taken but before respondent selection	
Two interim dispositions or one interim disposition on or after a fifteenth call attempt of 505 Refusal:  hang-up or termination	The questions answered meet the definition for a 310 Hang-up or termination, housing unit, unknown if eligible respondent	
Two interim dispositions or one interim disposition on or after a fifteenth call attempt of 505 Refusal:  hang-up or termination	The questions answered meet the definition for a 330 Hang-up or termination, unknown if private residence	
At least one disposition of 510 Appointment	The questions answered meet the definition and callback rules for a 230 Selected respondent never contacted or was reached but did not begin interview during interviewing period	
At least one disposition of 510 Appointment	The questions answered meet the definition and callback rules for a 280 Household contact after number of adults taken but before respondent selection	
At least one disposition of 510 Appointment	The questions answered meet the definition and callback rules for a 315 Household contact, eligibility undetermined	
At least one disposition of 510 Appointment	The questions answered meet the definition and callback rules for a 332 Contact, unknown if private residence	
Two interim dispositions or one interim disposition on or after a fifteenth call attempt of 515 Language problem	(No further conditions.  Assign final disposition of 320 Language problem before respondent selection.)	
Two interim dispositions or one interim disposition on or after a fifteenth call attempt of 520 Physical or mental impairment	(No further conditions.  Assign final disposition of 325 Physical or mental impairment before respondent selection.)	
At least one disposition of 525 Answering machine, message confirms residential status	The call history meets the definition and callback rules for a 335 Telephone answering device, message confirms residential status	
At least one disposition of 530 Technological barrier other than answering machine, message confirms residential status	The call history meets the definition and callback rules for a 340 Telecommunication technological barrier	
At least one disposition of 535 Answering machine, not sure if private residence	The call history meets the definition and callback rules for a 345 Telephone answering device, not sure if private residence	
At least one disposition of 540 Technological barrier other than answering machine, not sure if private residence	The call history meets the definition and callback rules for a 350 Telecommunication technological barrier, not sure if private residence	
A last disposition of 450 Non-working/disconnected number and at least one previous disposition of 505, 510, 515, 520, 525, 530, 535, or 540  (See the two paragraphs preceding this table.)	The call history meets the definition and callback rules for a 355 Telephone number has changed status from possible household to non-working during the interviewing period	
At least one disposition of 560 Fax/data/modem and all other dispositions, not counting 580 Null attempt or 585 Requires supervisor attention, are 550 No answer, 555 Busy, 565 Fast busy, 570 Possible non-working number, or 575 Circuit busy	The call history meets the definition and callback rules for a 430 Dedicated fax/data/modem line	
At least one disposition of 565 Fast busy and all other dispositions, not counting 580 Null attempt or 585 Requires supervisor attention, are 550 No answer, 555 Busy, 570 Possible non-working number, or 575 Circuit busy.	The call history meets the definition and callback rules for a 440 Fast busy	
At least one disposition of 550 No answer	The call history meets the definition and callback rules for a 360 No answer	
At least one disposition of 555 Busy	The call history meets the definition and callback rules for a 365 Busy	
At least fifteen call attempts with  all dispositions, not counting 580 Null attempt or 585 Requires supervisor attention, 545 Phone number temporarily out of service, 570 Possible non-working number, or 575 Circuit busy	The call history meets the definition and callback rules for a 450 Non-working/disconnected number