This is the accessible text file for GAO report number GAO-08-1012 entitled '2010 Census: Census Bureau Needs Procedures for Estimating the Response Rate and Selecting for Testing Methods to Increase Response Rate' which was released on October 30, 2008. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Requesters: United States Government Accountability Office: GAO: September 2008: 2010 Census: Census Bureau Needs Procedures for Estimating the Response Rate and Selecting for Testing Methods to Increase Response Rate: 2010 Census: GAO-08-1012: GAO Highlights: Highlights of GAO-08-1012, a report to congressional requesters. Why GAO Did This Study: The U.S. Census Bureau (Bureau) estimates that it will spend at least $2 billion to enumerate households that did not return census forms during the 2010 Census. Increasing the response rate would reduce the number of households that Bureau field staff must visit. To address concerns about reducing the cost of enumerating these households, GAO (1) analyzed how the Bureau develops, supports, and updates the response rate estimate, and the extent to which the Bureau uses the estimate to inform its 2010 planning efforts; (2) described the methods the Bureau considered for increasing response in 2010 and how it tested these methods; and (3) assessed how the Bureau identifies and selects for testing methods to increase response rate, including considering other surveys’ methods. To meet these objectives, GAO analyzed the Bureau’s documentation for estimating the response rate and selecting for testing methods to increase response, and interviewed experts from other survey organizations. What GAO Found: The 2010 Census response rate estimate is not fully supported, systematically reevaluated, or clearly incorporated into the life cycle cost estimate and planning efforts for nonresponse follow-up—where census workers visit households that do not return their census forms. Specifically, the Bureau could not demonstrate support for one component underpinning the estimate—a general decline due to decreasing public participation in surveys—because it did not document its decisions or data sources when developing the estimate. The two other estimate components that affect responses are the short-form-only census and the replacement questionnaire. In 2001, the Bureau estimated the 2010 Census response rate to be 69 percent. However, from 2001 through 2008, the Bureau did not systematically reevaluate the estimate or consider test results from this decade to determine if the estimate should be updated. Although the Bureau revised the estimate to 64 percent after a major redesign of its nonresponse follow-up operation in 2008, the Bureau still lacks procedures for establishing when and how to reevaluate and, if necessary, update the estimate. To estimate costs and plan for nonresponse follow-up, the Bureau relies on response rate estimates for local census office types because these estimates reflect geographic differences. Officials said that the local estimates reflect components of the national estimate. However, only one of the three components from the national estimate—the replacement questionnaire—was clearly reflected in the local census office type estimates. Through various national and field tests and experiments, the Bureau tested nine methods to increase 2010 Census response and currently plans to implement two of these methods—the replacement questionnaire and two-column bilingual form. The Bureau also plans to use a communications campaign to increase response and plans to test campaign messages in 2009. In July 2006, the Bureau decided not to include an Internet response option in the 2010 Census. However, the Bureau recently announced that it is again considering including the Internet option in 2010, although it has not developed further plans for testing it. For 2010, the Bureau established test objectives and research questions to identify methods to test for increasing response. However, Bureau officials did not document the methods that they considered but decided not to test or the rationale behind those decisions. Although officials said that they considered cost, complexity of the test, and compatibility of experiments in their decisions, they did not specify how they weighed these factors to select and prioritize the nine methods they chose to test. Officials said that they consider the experiences of other survey organizations to identify potential methods to increase response, but they, along with some experts, noted that such methods may only be indirectly applicable to the decennial census. Nonetheless, testing modifications to methods the Bureau has previously considered or tested, such as testing a variety of telephone reminder messages, may yield additional opportunities for increasing response. What GAO Recommends: GAO recommends that the Secretary of Commerce direct the Bureau to establish procedures for developing, documenting, and reevaluating the response rate estimate and for selecting for testing methods to increase the response rate. In commenting on a draft of this report, Commerce generally agreed with GAO’s recommendations and committed to take action for the 2020 Census. To view the full product, including the scope and methodology, click on [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-1012]. For more information, contact Mathew J. Scirè at (202) 512-6806 or sciremj@gao.gov [End of section] Contents: Letter: Results in Brief: Background: The Response Rate Estimate Lacks Support, Is Not Systematically Reevaluated, and Is Not Clearly Incorporated into Planning Efforts: The Bureau Plans to Incorporate Additional Methods to Increase Mail Response in 2010: Bureau Lacks Procedures for Selecting for Testing Methods to Increase Response Rate: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Testing of Methods to Increase Response to 2010 Census: Appendix III: Comments from the Department of Commerce: Appendix IV: GAO Contacts and Staff Acknowledgments: Table: Table 1: Self-Response Methods Tested in 2010 Testing Cycle and Effect on Response Rate in Tests: Figures: Figure 1: Decennial Census Costs from 1970 through 2010 (Projected) in Constant 2010 Dollars: Figure 2: Excerpt from Advance Letter from Census 2000: Figure 3: Reminder Postcard from 2003 Census Test: Figure 4: Components of the Census Bureau's 2001 National Response Rate Estimate for the 2010 Census: Figure 5: Excerpt from Two-Column Bilingual Form from 2008 Dress Rehearsal: Figure 6: Text for Telephone Reminder Call from 2003 Census Test: Figure 7: Message on Flap to Distinguish Replacement Questionnaire from Initial Questionnaire from 2005 Census Test: Figure 8: Due Date on Envelope from 2006 Test: Figure 9: Example of a Census 2000 Communications Campaign Poster: United States Government Accountability Office: Washington, DC 20548: September 30, 2008: The Honorable Tom Davis: Ranking Member: Committee on Oversight and Government Reform: House of Representatives: The Honorable Michael R. Turner: Ranking Member: Subcommittee on Information Policy, Census and National Archives: Committee on Oversight and Government Reform: House of Representatives: The U.S. Census Bureau (Bureau) estimates that even after adjusting for inflation, the 2010 decennial census will be the most expensive census in our nation's history, costing from $13.7 billion to $14.5 billion.[Footnote 1] The Bureau estimates that more than $2 billion will be used to employ temporary field staff for nonresponse follow-up- -its largest field operation where enumerators interview households that did not return census forms. Increasing the response rate would reduce the number of households that Bureau field staff must visit during this nationwide operation.[Footnote 2] According to Bureau officials, a 1 percent increase in the response rate can save $75 million. The Bureau expects to hire over 700,000 temporary workers to conduct nonresponse follow-up with about 47 million households over the course of 10 weeks in 2010. The Bureau initially based the schedule, staffing, and funding it needed for nonresponse follow-up on an estimated national response rate of 69 percent. However, in February 2008, the Director of the Bureau initiated a replanning of the Field Data Collection Automation program--a major acquisition that includes systems; equipment, including handheld computers; and infrastructure for field staff to use in collecting data for the 2010 Census. After analyzing several options to revise the design of the 2010 Decennial Census, on April 3, 2008, the Secretary of Commerce announced that the Bureau would no longer use handheld computers in nonresponse follow-up and revised the estimated national response rate from 69 percent to 64 percent. The Bureau estimated that this option would result in a cost increase of $2.2 billion to $3 billion over the previously reported cost estimate of $11.5 billion. To address your concerns about reducing the cost of nonresponse follow- up operations, we reviewed the Bureau's estimated response rate and plans for increasing response. Specifically, we (1) analyzed how the Bureau develops, supports, and updates the response rate estimate, and the extent to which the Bureau uses the estimate to inform its 2010 planning efforts; (2) described the methods the Bureau considered for increasing response rates in 2010 and what it did to test these methods; and (3) assessed how the Bureau identifies and selects for testing methods to increase response rate, including considering other surveys' methods for increasing response. To meet these objectives, we reviewed documentation to support the components of the response rate estimate and research literature on methodologies to increase response to mail surveys and efforts to estimate survey response rate. We analyzed Bureau documents related to 2000 Census evaluations, the 2010 research and testing program, the 2010 Census life cycle cost estimate, and the communications campaign. We also interviewed Bureau officials in the Decennial Management Division and other divisions about methods to increase self-response. Further, we interviewed experts on survey methodology from Statistics Canada, the U.S. Department of Labor's Bureau of Labor Statistics, and various academic institutions, as well as former Census Bureau officials and researchers. We asked the experts a common set of questions in order to compare responses and identify recurring themes. Appendix I includes a list of experts we interviewed and additional information on our scope and methodology. We conducted our review from July 2007 through September 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Results in Brief: The 2010 Census response rate estimate is not fully supported, systematically reevaluated, or clearly incorporated into cost and planning efforts. The Bureau developed a national response rate estimate of 69 percent for the 2010 Census in 2001, but could not demonstrate support for one of the three components underpinning it-- the general decline due to decreasing public participation in surveys. (The other two components are the use of a short-form-only census and the introduction of a replacement mailing, both of which the Bureau expects to increase response.) The Bureau does not have procedures for developing the estimate, nor could the Bureau provide detailed documentation on how the response rate estimate was developed. Clear, detailed documentation would allow the Bureau and others to better assess the reliability of the estimate, such as determining whether the estimate's assumptions are supported and realistic. Although the Bureau reevaluated and developed a revised estimate of 64 percent after the Bureau announced a major redesign of the nonresponse follow-up operation in April 2008, the Bureau still does not have procedures for systematically reevaluating the estimate throughout the decade based on test results or other triggering events. For example, the Bureau did not reevaluate the estimate prior to April 2008 or use test results conducted during the decade to determine if the estimate should be updated. Having procedures for when and how to reevaluate the estimate would help the Bureau to ensure that the estimate is current and reliable. Finally, for estimating cost and planning for nonresponse follow-up, the Bureau relies on response rate estimates from four primary types of local census offices (LCO), which help to account for expected differences in response patterns across geographic settings. According to the Bureau, the LCO-type response rate estimates are, in part, based on components of the national estimate. However, the Bureau could not explain why it applied the replacement mailing component equally across all LCO-type estimates or how it incorporated the short-form-only component and the general decline component into any of the LCO-type estimates. Establishing a quantitative basis for applying the components to the LCO-type estimates would better inform the Bureau's cost estimate and planning efforts for operations that are based on response, such as hiring enumerators for nonresponse follow-up. Overall, the Bureau lacks procedures for developing, documenting, and reevaluating the response rate estimate. Nonetheless, Bureau officials stated that they tried to conservatively estimate the response rate to ensure that they are adequately prepared during nonresponse follow-up and to avoid repeating what happened in 1990, when the Bureau overestimated the response rate, requiring a supplemental appropriation of $110 million and forcing it to extend nonresponse follow-up by up to 8 weeks for some areas. Conservatively estimating the response rate may be a reasonable approach. However, having clear documentation to support the estimate and establishing procedures for when or how to reevaluate the estimate to ensure that it reflects current information from testing would enable the Bureau and others to assess whether the estimate is reliable. An unreliable response rate estimate can produce an inaccurate cost estimate and can increase risk and uncertainty to operational plans that are based on the response rate estimate. Through various national and field tests and experiments, the Bureau tested nine methods to increase self-response to the 2010 Census and plans to implement two of these methods--the replacement questionnaire and two-column bilingual form. An additional method the Bureau plans to include in the 2010 Census to increase response, the Integrated Communications Campaign, has not been tested, although a similar communications campaign was part of the 2000 Census. The Bureau does not plan to include three methods it tested to increase mail response- -a telephone reminder call, messaging to distinguish initial and replacement questionnaires, and a due date on the initial mailing package--because they did not significantly increase response or required further testing. Bureau officials said that they will be conducting additional testing on the use of a due date as part of the 2010 Census Program for Evaluations and Experiments. The National Academy of Sciences also recommended such analysis. At the time of this report, the Bureau had not yet completed its plans for 2010 evaluations and experiments. The Bureau also does not plan to include four methods it tested to increase self-response through electronic systems--interactive voice response; a letter encouraging responding via the Internet, instead of sending a replacement questionnaire; computer-assisted telephone interviewing; and the Internet--because they did not increase response or were too costly. In July 2006, the Bureau decided not to include the Internet option in the design of the 2010 Census largely because it had underestimated contract costs for developing the system that included the Internet option, but also because test results indicated that the Internet did not always increase the response rate and that security over respondent data was a concern. However, the Bureau recently announced that it is again considering including the Internet option in 2010, even though it has not developed further plans for testing it. Finally, the Bureau has not yet developed detailed testing or evaluation plans for the communications campaign, though it plans to test campaign messages in 2009. To identify methods for increasing response in 2010 that it planned to test, the Bureau developed a research strategy that included establishing test objectives and research questions. However, the Bureau did not document the methods it considered but decided not to test during the 2010 testing cycle or the rationale behind those decisions. Further, for methods that the Bureau decided to test, officials could not provide support for how they selected or prioritized these methods. Although officials said that they considered cost, complexity of the test, and compatibility of experiments in their decisions, they did not specify how they defined or weighed these factors to select and prioritize the nine methods they chose to test. Documenting decisions about methods that were not selected for testing would help the Bureau more effectively build capacity and institutional knowledge about changes in methods to consider in the future. Bureau officials also said that they consider the experiences of other survey organizations to identify potential methods to increase response rate. However, both Bureau officials and some survey experts acknowledged that methods used to increase response in other surveys may only be indirectly applicable to the decennial census. Nonetheless, testing modifications to methods the Bureau has previously considered or tested, such as testing a variety of telephone reminder messages, may yield additional opportunities for increasing response. To enhance credibility of the response rate for determining cost and planning for future decennial censuses and to inform assumptions underlying the 2020 response rate estimate, we recommend that the Secretary of Commerce direct the Bureau to establish procedures for documenting the process for developing the response rate estimate, including analyzing 2010 data to assess the reasonableness of assumptions used in applying the national estimate's components to the LCO-type estimates, as well as establishing when and how to reevaluate the response rate estimate. Further, to improve the planning and transparency of the Bureau's research and testing for future censuses, we recommend that the Bureau develop procedures for selecting methods to test for increasing response. On September 22, 2008, the Secretary of Commerce provided written comments on a draft of this report (see app. III). Commerce generally agreed with our conclusions and recommendations and provided suggestions where additional context or clarification was needed. Where appropriate, we incorporated these changes. Background: In 1970, the Bureau moved away from conducting the census door-to-door and began mailing census questionnaires to households to be filled out and returned by mail. Since the 1970 Census, the Bureau has used mail as the primary method for collecting census data. For Census 2000, the mailout/mailback method was used for more than 80 percent of the population.[Footnote 3] Households that fail to mail back the census questionnaires are included in nonresponse follow-up workload, where enumerators follow up with door-to-door visits and telephone calls or solicit census data from knowledgeable people, such as neighbors. The census response rate declined dramatically from 1970 through 1990. The 1970 response rate was 78 percent. The rate decreased to 75 percent for the 1980 Census and then decreased again to 65 percent for the 1990 Census.[Footnote 4] Although the Bureau estimated that the 2000 Census response rate would continue to decline to 61 percent, actual response exceeded the estimate, reaching 65 percent for the mailout/mailback universe prior to nonresponse follow-up.[Footnote 5] The response rate is defined as the percentage of census forms completed and returned for all housing units that were on the Bureau's address file eligible to receive a census questionnaire delivered by mail or by a census enumerator. The denominator used in calculating the response rate includes vacant housing units and other addresses that were determined to be undeliverable or deleted through other census operations.[Footnote 6] Since 1970, per household census costs have increased (in constant 2010 dollars) from about $14 in 1970 to an estimated $88 in 2010--a figure that does not include the recent, major redesign of the Field Data Collection Automation program. According to the Bureau, factors contributing to the increased costs include an effort to accommodate more complex households, busier lifestyles, more languages and greater cultural diversity, and increased privacy concerns. In addition, the number of housing units--and hence, the Bureau's workload--has continued to increase. The Bureau estimated that the number of housing units for the 2010 Census will increase by almost 14 percent over 2000 Census levels (from 117.5 million to 133.8 million housing units). As a result of multiple factors, the total inflation-adjusted life cycle cost for the decennial census has increased more than tenfold since 1970, as shown in figure 1. Figure 1: Decennial Census Costs from 1970 through 2010 (Projected) in Constant 2010 Dollars: This figure is a bar graph showing decennial census costs from 1970 through 2010 (projected) in constant 2010 dollars. The X axis represents the census year, and the Y axis represents the cost in billion of dollars. Census year: 1970; Cost in billions of dollars: 1.0. Census year: 1980; Cost in billions of dollars: 2.6. Census year: 1990; Cost in billions of dollars: 4.1. Census year: 2000; Cost in billions of dollars: 8.2. Census year: 2010 (projected); Cost in billions of dollars: 11.8. [See PDF for image] Source: GAO analysis of U.S. Census Bureau figures. Note: This figure does not reflect the Bureau's estimated increase in the 2010 Census life cycle cost estimate ranging from $13.7 billion to $14.5 billion, which the Bureau announced on April 3, 2008, with the replan of the Field Data Collection Automation program. [End of figure] The Bureau conducts its testing, evaluation, and experimentation program for the decennial census primarily through the Decennial Statistical Studies Division. The division develops and coordinates the application of statistical techniques in the design and conduct of decennial census programs. The Bureau conducts various tests throughout the decade to determine the effect and feasibility of form or operational changes. For example, in the 1990s, the Bureau tested whether adding a mandatory response message and implementing a multiple mailing strategy, such as sending an advance letter and reminder postcard, as shown in figures 2 and 3, would increase the response rate. Based on positive test results, these methods were added to the Census 2000 design. Figure 2: Excerpt from Advance Letter from Census 2000: This figure is an excerpt from advance letter from census 2000. United States Department Of Commerce: Bureau of the Census: Washington, DC 20233-2000: Office Of The Director: March 6, 2000: About one week from now, you will receive a U.S. Census 2000 form in the mail. When you receive your form, please fill it out and mail it in promptly. Your response is very important. The United States Constitution requires a census of the United States every 10 years. Everyone living in the United States on April 1, 2000, must be counted. By completing your census form, you will make sure that you and members of your household are included in the official census count. Official census counts are used to distribute government funds to communities and states for highways, schools, health facilities, and many other programs you and your neighbors need. Without a complete, accurate census, your community may not receive its fair share. You can help in another way too. We are now hiring temporary workers throughout the United States to help complete the census. Call the Local Census Office near you for more information. The phone number is available from directory assistance or the Internet at [hyperlink, http://www.census.gov/jobs2000]. With your help, the census can count everyone. Please do your part. Thank you. Sincerely, Signed by: Kenneth Prewitt Director: Bureau of the Census: Enclosure: United States Census 2000: [See PDF for image] Source: U.S. Census Bureau. [End of figure] Figure 3: Reminder Postcard from 2003 Census Test: United States Department Of Commerce: Economics and Statistics Administration: U.S. Census Bureau: Washington, DC 20233-0001: Office Of The Director: February 7, 2008: A few days ago, you should have received a request for your participation in the 2003 National Census Test. The test is being conducted by the U.S. Census Bureau to help develop new methods for the next census in 2010. If you have already mailed back your completed census form, please accept our sincere thanks. If you have not responded, please mail it back as soon as possible. Taking an accurate census is important for all communities throughout the United States to get their fair share of federal funding. Your participation in this test is important to the success of the next census. Sincerely, Signed by: Charles Louis Kincannon: Director, U.S. Census Bureau: [See PDF for image] Source: U.S. Census Bureau. [End of figure] The Decennial Statistical Studies Division also conducts evaluations of census operations typically using data collected as part of the decennial process to determine whether individual steps in the census operated as expected. It is also responsible for conducting experiments during the census that may be instructive for future censuses. These experiments usually involve using alternative processes for a subset of the population. The Bureau is working with internal and external stakeholders on defining and developing its program for experiments and evaluations for the 2010 Census, known as Census Program for Evaluations and Experiments, with design work and implementation starting in 2008 and continuing through 2011. The final set of activities would include analysis, documentation, and presentation of the research, and these activities would start in 2009 and be completed by the end of fiscal year 2013. The Response Rate Estimate Lacks Support, Is Not Systematically Reevaluated, and Is Not Clearly Incorporated into Planning Efforts: 2010 Census Response Rate Estimate Is Not Fully Supported: The Bureau developed the response rate estimate for the 2010 Census in 2001 but could not demonstrate support for one of the three components underpinning the estimate. To establish the 2010 estimate, Bureau officials told us that they used Census 2000 mail response data as the baseline and then incorporated three other components to arrive at a response rate estimate of 69 percent: an increase in response related to eliminating the long form and moving to a short-form-only census, an increase in response related to sending a replacement questionnaire to nonresponding households, and a general decline in mail response due to decreasing public participation in surveys. The components of the estimate are outlined in figure 4, and a detailed explanation of the baseline and each component follows. Figure 4: Components of the Census Bureau's 2001 National Response Rate Estimate for the 2010 Census: This figure is a chart showing the components of the census bureau's 2001 national response rate estimate for the 2010 census. [See PDF for image] Source: GAO analysis of U.S. Census Bureau information. [End of figure] Baseline Rate: Included in the baseline national response rate of 65 percent[Footnote 7] are the effects of methods, such as the multiple mailing strategy and the communications campaign (called the Partnership and Marketing Program) that were implemented in Census 2000 and are planned to be implemented in the 2010 Census as well. Bureau officials stated that the communications campaign, which included paid advertising, had an impact on the response rate achieved in 2000, but were unable to quantify that effect and did not project the campaign's effect for 2010. Short-Form-Only Census: The Bureau estimated an increase of 1 percentage point from eliminating the long form and moving to a short-form-only census. The Bureau conducted an analysis comparing Census 2000 response rates for the short form and the long form. Even though the difference in response rates for the two form types was rather large--the short form had a response rate that was more than 10 percentage points higher than the long form--the overall effect on the response rate estimate was small because the long form was sent to only approximately 17 percent of housing units in 2000. Replacement Mailing: In 2001, the Bureau expected that sending a replacement questionnaire to households that had not responded by a certain date would increase the response rate by 7 percentage points.[Footnote 8] The magnitude of this component was based on test results from the 1990s, which showed an increase in the response rate related to the replacement mailing of at least 7 percentage points, and possibly 10 percentage points or more. Bureau officials stated that they estimated this effect conservatively because they assumed that a higher response rate from a replacement questionnaire is more likely to occur in tests than in the decennial census. Also, testing showed that not all the respondents who return replacement questionnaires would be removed from the nonresponse follow-up workload prior to being enumerated because of the timing of the enumerator's visit. General Decline: The Bureau estimated a decrease of 4 percentage points from the baseline level due to what it believes is a decline over time in public survey participation. However, this assumption is not supported by quantitative analysis or research studies but rather based on the opinion of subject matter experts and senior officials. Further, the Bureau could not demonstrate who was consulted, when they were consulted, or how they decided on the amount of the general decline. Best practices for cost estimation from our Cost Assessment Guide[Footnote 9] call for documenting assumptions and data sources used to develop cost estimates, such as the general decline component of the response rate estimate. In addition, several experts we interviewed agreed that the Bureau should have used quantitative analysis of mail response data from other surveys to support the general decline component when it developed the estimate in 2001. To help support the general decline component, we suggested that the Bureau use changes in participation in the American Community Survey, which uses a similar mailing strategy to the decennial census and for which response is also required by law, for comparison purposes. In May 2008, the Bureau completed an analysis of American Community Survey response rate data from 2000 to 2007, which demonstrated a decline of 6.6 percentage points in cooperation rates[Footnote 10] to the initial questionnaire. Although the decline in response to the American Community Survey may not be directly comparable to response behavior for the decennial census, it provides some support for the Bureau's assumption that mail survey participation may be declining. The Bureau does not have procedures for developing the response rate estimate. Specifically, the Bureau has no established policies for documenting deliberations related to or data sources used in developing the estimate, including no 2010 Census decision memorandum to document the original estimate prepared in 2001. Several experts also suggested that the Bureau should develop a model for developing the estimate and incorporate response rate data and demographic data from other surveys.[Footnote 11] Bureau officials, however, noted that they have not determined how to apply data from other surveys to estimate the census response rate because the census differs from other surveys, nor do they plan to develop a model of the response rate estimate. Instead, Bureau officials stated that they tried to conservatively estimate the response rate to ensure that they are adequately prepared during nonresponse follow-up and to avoid repeating what happened in 1990, when the Bureau overestimated the response rate, requiring a supplemental appropriation of $110 million and forcing it to extend nonresponse follow-up by up to 8 weeks for some areas. In contrast, in 2000 the response rate exceeded the Bureau's estimate. Conservatively estimating the response rate may be a reasonable approach. However, having clear, detailed documentation about decisions related to or data sources used in developing the estimate would enable the Bureau and others to better assess whether the estimate is reliable, such as whether its assumptions are supported and realistic. An unreliable response rate estimate can produce an inaccurate cost estimate and can increase risk and uncertainty to operational plans that are based on the response rate estimate. The Bureau Does Not Systematically Reevaluate the Response Rate Estimate: Although the Bureau updated the response rate estimate as a result of a major redesign of census operations undertaken in April 2008, the Bureau still lacks procedures for establishing when and how to reevaluate the response rate estimate. From 2001 through April 2008, the Bureau did not reevaluate the estimate to determine whether it should be updated, even though, after establishing the initial estimate in 2001, the Bureau completed several evaluations from Census 2000 and conducted several census tests that could have informed the estimate. During the first few years of the decade, the Bureau completed 12 evaluations that discussed aspects of the Census 2000 response rate. In addition, the Bureau conducted five tests from 2003 through 2007 that were designed, in part, to test methods to increase response. For example, in 2005 and 2007, the Bureau tested a two-column bilingual form, which includes identical questions and response options in English and Spanish. The 2005 test demonstrated a significant positive effect on mail response rates of 2.2 percentage points overall, and a 3.2 percentage point increase in areas with high concentrations of Hispanic and non-White populations, and the 2007 test revealed a similar impact. However, Bureau officials stated that they were concerned that the two-column bilingual form has not been tested in a census environment. In addition, they noted that the bilingual form will be sent to only a portion of the country--approximately 10 percent of census tracts--with high concentrations of Spanish-speaking populations, based on American Community Survey data. Best practices state that assumptions, such as the response rate estimate, should be revised as new information becomes available. Our Cost Assessment Guide recommends that preliminary information and assumptions be monitored to determine relevance and accuracy and be updated as changes occur to reflect the best information available. Further, according to several experts, the Bureau could have updated the estimate based on response rate data and demographic data from other surveys, and another expert suggested that the Bureau use triggering events, such as after tests, for reviewing the response rate estimate. Based on this expert's experience, the Bureau could establish a change control board chaired by senior Bureau officials to determine whether the response rate estimate should be revised. However, Bureau officials explained that they do not update the estimate based on results from tests because the tests cannot replicate the decennial census environment. Overall, the Bureau has not specified when or how it is to reevaluate its response rate estimate. Establishing these procedures would help the Bureau ensure that the estimate is current and reliable in order to better inform planning efforts. The Bureau revised the estimate down to 64 percent after announcing that nonresponse follow-up would be changed from an automated to a paper-based operation. Prior to the redesign, the Bureau had planned to reduce the nonresponse follow-up workload by using handheld computers to remove households that returned their forms late--including many replacement mailing returns--from the enumerator assignments on a daily basis. According to Bureau officials, they revised the response rate estimate based on the timing of mail returns in 2000 and replacement questionnaire returns in the 2003 test. However, the revised estimate does not fully reflect recently designed procedures to remove late mail returns in 2010. Specifically, for 2010 the Bureau now plans to mail the replacement questionnaire earlier, send a blanket replacement questionnaire to some areas and a targeted replacement questionnaire to others,[Footnote 12] and conduct three clerical removals of late mail returns immediately prior to and during the nonresponse follow-up operation. These operational plans are still being finalized pending further analysis. Bureau officials said that although they hope that these revised operations will increase response, they did not update the response rate estimate to reflect these current operational plans because they have not tested this approach under decennial census conditions and therefore have no basis for estimating the potential effect of these operational changes on response. Local Response Rate Estimates Used in Planning Efforts Lack Support: To estimate costs and to plan for activities such as nonresponse follow- up, questionnaire printing, and postage needs, the Bureau uses a life cycle cost model that relies on response rate estimates for four primary LCO types[Footnote 13] and not the national estimate. Using LCO- type estimates helps to account for expected differences in response patterns across geographic settings. For example, response rates for inner-city areas are estimated to be lower than in suburban areas. These estimates range from 55 percent to 72 percent, but lack support for how they were developed. In determining the 2010 response rate estimates for the four LCO types, the Bureau said that it equally applied two components from the national estimate--the replacement mailing component and the short-form-only component--to the LCO types in the cost model. However, the Bureau did not conduct quantitative analysis to determine whether the replacement mailing component in the national estimate should be applied equally for each LCO type. By applying an equal percentage point increase across all LCO types for the replacement mailing, the Bureau has, in effect, assumed that the LCO type with the lowest baseline--48 percent--would experience a higher relative increase in response--15 percent--due to the replacement mailing than the LCO type with the highest baseline--65 percent--which would be expected to experience an 11 percent increase in response. The Bureau has not demonstrated support for assuming that the LCO types will experience these different relative increases in response. Further, the Bureau could not demonstrate whether or how the short-form-only component and the general decline component of the national estimate were reflected in the life cycle cost model. Best practices for cost estimation from our Cost Assessment Guide call for assumptions to be thoroughly documented, data to be normalized, and each cost element to be supported by auditable and traceable data sources. Following these best practices could allow the Bureau to enhance its use of the response rate estimate in planning for the decennial census and better inform stakeholders about the reliability of the estimate. It is unclear why officials could only explain how one component of the national response rate estimate was applied to LCO- type estimates in the life cycle cost model. However, according to the Bureau, it is currently documenting how these estimates are used to calculate the costs for operations and how the estimates have changed over time. Having support for the LCO-type estimates would better inform the Bureau's planning efforts for operations that directly rely on response, such as determining workload and hiring enumerators for nonresponse follow-up. The Bureau Plans to Incorporate Additional Methods to Increase Mail Response in 2010: Two of Nine Methods Tested to Increase Response Rate Are Planned to Be Included in the 2010 Census: We determined from reviewing Bureau testing documents that through various national and field tests and experiments, the Bureau tested nine methods to increase self-response to the 2010 Census, as shown in table 1. The Bureau currently plans to implement two of these methods- -the replacement questionnaire and two-column bilingual form (see fig. 5).[Footnote 14] Three other methods for increasing mail response and four methods for increasing response through an electronic data collection system were tested but are not planned for implementation in 2010. An additional method the Bureau plans to include in the 2010 Census to increase response, the Integrated Communications Campaign, has not been tested, although the Bureau conducted a similar campaign in 2000 and plans to test campaign messages with audiences in early 2009. Additional details regarding the testing of these methods can be found in appendix II. Table 1: Self-Response Methods Tested in 2010 Testing Cycle and Effect on Response Rate in Tests: Methods planned for implementation in 2010: 1. Replacement questionnaire; Year(s) tested: 2003, 2006; Effect on response rate in tests (percentages): 8.8 to 10.3. Methods planned for implementation in 2010: 2. Two-column bilingual form; Year(s) tested: 2005, 2007; Effect on response rate in tests (percentages): 1.4 to 2.2. Methods not planned for implementation in 2010: Methods not planned for implementation in 2010; Year(s) tested: [Empty]; Effect on response rate in tests (percentages): [Empty]. Methods not planned for implementation in 2010: 3. Telephone reminder call; Year(s) tested: 2003; Effect on response rate in tests (percentages): 0. Methods not planned for implementation in 2010: 4. Due date on initial questionnaire mailing package; Year(s) tested: 2003, 2006; Effect on response rate in tests (percentages): 0 to 2.0[A]. Methods not planned for implementation in 2010: 5. Messaging to distinguish the replacement from the initial questionnaire; Year(s) tested: 2005; Effect on response rate in tests (percentages): -1.2. Electronic response options: 6. Interactive voice response; Year(s) tested: 2000, 2003; Effect on response rate in tests (percentages): -4.9 to 0[B]. Electronic response options: 7. Letter encouraging internet response instead of sending replacement questionnaire; Year(s) tested: 2005; Effect on response rate in tests (percentages): -3.7. Electronic response options: 8. Computer-assisted telephone interviewing; Year(s) tested: 2000; Effect on response rate in tests (percentages): 2.1. Electronic response options: 9. Internet; Year(s) tested: 2000, 2003; Effect on response rate in tests (percentages): 0 to 2.5. Source: GAO analysis of U.S. Census Bureau information. Notes: We included only those years in which the tests focused primarily on determining the method's effect on response, rather than on data quality, operational feasibility, or other aspects of survey design. A zero effect indicates that the method tested had no statistically significant effect on response rate compared to the control group. Test response rates may not be fully comparable to response rates or return rates calculated for decennial censuses, in part because ineligible housing units, such as those that are vacant or undeliverable as addressed, are treated differently in the denominators. [A] The 2 percentage point increase in response rate was obtained when the due date was tested in conjunction with a compressed mailing schedule, which shortened the time period between the mailing of the questionnaire and Census Day. [B] The Bureau notes that the 2000 test result showing no statistically significant effect on response is difficult to interpret because a portion of the sample in the interactive voice response panel either received the census form late or did not receive it at all. [End of table] Figure 5: Excerpt from Two-Column Bilingual Form from 2008 Dress Rehearsal: This figure is a copy of an excerpt from two-column bilingual form from 2008 rehearsal. [See PDF for image] Source: U.S. Census Bureau. [End of figure] For the 2010 Census, the Bureau does not plan to include three methods it tested to increase mail response, as shown in figures 6, 7, and 8. Two of these methods--a telephone reminder call and messaging to distinguish the initial and replacement questionnaires--were not found to significantly improve response and therefore will not be implemented as part of the 2010 Census. The third method--including a due date on the initial mailing package--was found to generate faster responses when tested in 2003, and increase overall response by 2 percentage points when tested in conjunction with a compressed mailing schedule in 2006.[Footnote 15] However, the Bureau also believes that the use of a due date alone could cause a lower response rate because some people may not send back the census form after the due date. According to Bureau officials, they are not including this method in the decennial census design because they would like to further test it--both with and without a compressed mailing schedule--in a census environment. It will be important for the Bureau to optimize this testing opportunity by designing the test to determine the extent to which the faster and higher response is due to a compressed schedule versus a due date, as well as exploring other test treatments the Bureau has recommended in the past, such as including the due date on multiple mailing pieces. Bureau officials said that additional testing on the use of a due date will be conducted as part of the 2010 Census Program for Evaluations and Experiments, which the National Academy of Sciences recommended in its December 2007 interim report.[Footnote 16] The Bureau did not provide further details on plans for the Census Program for Evaluations and Experiments, but officials have said that they will consider costs and staffing needs in deciding what to evaluate. The academy's final report is due in September 2009.[Footnote 17] Figure 6: Text for Telephone Reminder Call from 2003 Census Test: Hello. A few days ago the U.S. Census Bureau sent a request to your address, asking for your participation in the 2003 National Census Test. If you’ve already responded to this request, please accept our sincere thanks. If you haven’t, please take a few minutes to complete your census form and return it by mail. Your response will help us develop the best procedures for counting the U.S. population. Thank you. [See PDF for image] Source: U.S. Census Bureau. [End of figure] Figure 7: Message on Flap to Distinguish Replacement Questionnaire from Initial Questionnaire from 2005 Census Test: This figure is a copy of a message on flap to distinguish replacement questionnaire from initial questionnaire from 2005 census test. [See PDF for image] Source: U.S. Census Bureau. [End of figure] Figure 8: Due Date on Envelope from 2006 Test: [See PDF for image] Source: U.S. Census Bureau. [End of figure] In addition to testing methods to increase mail response, the Bureau also tested four methods to increase self-response through electronic data collection systems: (1) providing for response via an interactive voice response system;( 2) sending a letter to encourage either responding via Internet or returning the initial questionnaire, instead of sending a replacement questionnaire; (3) using computer-assisted telephone interviewing; and (4) providing capability for responding via the Internet. The Bureau found that the first two methods did not increase response in tests and does not plan to include them in the 2010 Census. Computer-assisted telephone interviewing allows respondents to use the telephone to connect with operators who record interview responses electronically. The Bureau found that this method increased the overall response rate in a Census 2000 experiment. However, in a March 2004 report, the Bureau also stated that the method would likely be too costly in terms of hardware, software, and staffing resources, compared to the increase in response it might generate. Computer-assisted telephone interviewing was not tested after the 2000 Census. Although the Internet option was found to increase overall response during the Census 2000 experiment, it did not increase overall response when tested again in 2003, when a replacement mailing was also tested. According to a July 2006 Bureau decision memo titled, "Rationale for the Decision to Eliminate the Internet Option from the DRIS Contract," the Bureau decided not to include the Internet option in the design of the 2010 Census largely because it had underestimated the costs of the contract that included developing the Internet response option. In responding to a draft of this report, the Bureau stated that they made the decision because test results showed that offering the Internet option did not increase overall response and would not offer any cost savings. According to the 2006 memo, the Bureau also determined that the operational risks and costs to develop this option outweighed the potential benefits. In terms of benefits, the Bureau found improvements in data completeness and data timeliness from Internet responses in tests conducted in 2000/2001 for the American Community Survey and in 2003 and 2005 for the National Census Tests. The Bureau noted that these benefits could translate into reduced costs since less follow-up is required to improve data accuracy, and earlier receipt of responses could result in fewer replacement questionnaires that need to be mailed and fewer households that need to be enumerated during nonresponse follow-up. However, with only 6 to 7 percent of the test population using the Internet option, the Bureau concluded that no cost savings could be realized from reducing the number or size of data capture centers (facilities that process returned questionnaires) planned for 2010. Finally, the Bureau stated that the inability to fully test the Internet option and growing concerns about Internet security made it unfeasible for the Bureau to implement the Internet as a response option for 2010. Despite its July 2006 decision, the Bureau recently announced that it is considering including the Internet option as part of the 2010 Census design; however, the Bureau has not developed plans for further testing this option. Testing and Evaluation Plans for the 2010 Communications Campaign Are Not Yet Fully Developed: Through its contractor and subcontractors, the Bureau has taken a number of steps to inform the planning of the communications campaign but has not yet fully developed the campaign's testing and evaluation plans.[Footnote 18] From late 2006 through early 2008, focus groups were conducted with various ethnic groups and hard-to-count populations, such as unattached singles, to identify potential barriers and motivators to participation, to better understand methods of communication that work for different groups, and to develop the campaign's overall creative expression. The Bureau also developed an audience segmentation model using Census 2000 response data, updated through 2006 using American Community Survey data, to provide a more detailed understanding of the characteristics, such as home ownership and unemployment level, of those more or less likely to respond, as well as where they live, in order to better target communications to encourage census participation. In addition, in 2008, a national phone and in-person survey is being conducted to further explore barriers and motivators to response, particularly in hard-to-count populations, which would inform campaign message development. According to agency officials, beginning in early 2009, campaign messages are to be tested by showing storyboards to audiences that will use electronic devices to vote on the messages. These messages will be tested in 14 languages and for other populations, such as Spanish speakers in Puerto Rico. According to the draft plan, testing of events, partnership toolkits, promotional items, and public relations ideas will also be conducted. However, the Bureau has not yet developed detailed plans for this testing because, according to one official, the Bureau intends to further develop testing plans as future fiscal year funding amounts become available. An example of a Census 2000 communications campaign poster is shown in figure 9. Figure 9: Example of a Census 2000 Communications Campaign Poster: This figure is a copy of a census 20000 communications campaign poster. [See PDF for image] Source: U.S. Census Bureau. [End of figure] In addition, although the Bureau expects to award a contract by the end of fiscal year 2008 for an independent evaluation measuring the campaign's performance against its goals of increasing mail response, improving accuracy, and reducing the differential undercount, and improving cooperation with enumerators, it has not yet done so. In the past, the Bureau has said that although evaluations have shown that the Census 2000 communications campaign increased awareness for that census, it was difficult to link increased awareness to changes in respondent behavior. Bureau officials said that they have attempted to analyze Census 2000 data to identify factors that influence behavior, but their research results were inconclusive. Going forward, it will be important for the Bureau to determine its plans for evaluating the 2010 communications campaign so that it does not miss opportunities to collect data in the census environment to inform future campaigns. Bureau Lacks Procedures for Selecting for Testing Methods to Increase Response Rate: For 2010, the Bureau developed a strategy to identify various methods to test for increasing response. Specifically, the Bureau established test objectives and research questions, such as identifying the optimal mix of response options for the public to respond to the 2010 Census and determining security and confidentiality issues surrounding technology. The Bureau also developed the 2010 Census Integrated Program Plan and 2010 Census Operations and Systems Plan to better document its planning. However, Bureau officials did not document the methods that they considered but decided not to test in the 2010 testing cycle or the rationale behind those decisions. Further, for methods that the Bureau decided to test, officials could not provide support for how they selected or prioritized these methods. Although officials said that they considered cost, complexity of the test, and compatibility of experiments in their decisions, they did not specify how they defined or weighed these factors to select and prioritize the nine methods they chose to test. To ensure thorough and comprehensive planning of the decennial census, our past work on lessons learned from Census 2000 highlighted the importance of documentation to support research, testing, and evaluation, and a comprehensive and prioritized plan of goals, objectives, and projects.[Footnote 19] While the Bureau has developed the 2010 Census Integrated Program Plan and 2010 Census Operations and Systems Plan to better document its planning, these and other planning documents we reviewed did not provide support for how the Bureau selected and prioritized methods to test. It is unclear why the Bureau lacks procedures for documenting decisions concerning how it selected for testing methods to increase response. Documenting decisions about methods that were not selected for testing would help the Bureau more effectively build capacity and institutional knowledge about changes in methods to consider in the future. According to Bureau officials, they consider the experience of other survey organizations when identifying methods for increasing response rate. For example, they said that they attend research conferences to learn about experiences of other organizations that conduct national surveys to identify potential methods to increase response rate. Both Bureau officials and some survey experts noted that methods used to increase response in other surveys may only be indirectly applicable to the decennial census. For example, for the Economic Census, officials said that the Bureau sends nonresponding businesses multiple reminder letters; the final letter to large businesses informs them that the Department of Justice will be pursuing nonrespondents.[Footnote 20] Bureau officials said that these methods are less feasible for the decennial census to implement because of the shorter time frame for obtaining responses and concerns about being respondent-friendly to households. In addition, one survey expert noted that methods applicable to small-scale surveys, such as personalizing a cover letter, may be less feasible to implement for the decennial census. Further, survey experts we interviewed generally said that they were unaware of additional methods from censuses undertaken by other countries or private sector surveys that the Bureau could consider to increase mail response. Some experts noted differences between how the United States and other countries conduct their censuses, which may make it difficult to directly transfer other countries' practices to the U.S. census. For example, some European censuses use a population register to collect names, unlike the United States, which builds its survey frame from phone and address lists. In addition, past research has provided evidence that government-sponsored self-administered surveys, such as the decennial census, tend to achieve higher response rates than nongovernmental surveys. Nonetheless, testing modifications to methods that the Bureau has previously considered or tested in earlier studies may yield additional opportunities for increasing response. For example, the Bureau could test various telephone reminder messages stating that response is mandatory by law or providing instructions for obtaining a new census form. Although Bureau officials said that they have previously used the American Community Survey to inform the census design, analyzing respondent behavior to the American Community Survey, because of its similar mailing strategy to the decennial census, could help the Bureau regularly refine its survey methodology for increasing census response. Although the survey forms are different, many concepts, such as targeting the second mailing, modifying the appearance of the mailing, and varying telephone and Internet messages to prompt nonrespondents, could be tested with reasonable inference to the census. Conclusions: Nonresponse follow-up is the largest field operation, and the Bureau estimates that it will cost more the $2 billion. To control the cost of nonresponse follow-up, it will be important for the Bureau to devise a strategy for getting people to return their census forms. A reliable response rate estimate is a critical element necessary for determining the resources needed to carry out nonresponse follow-up. The Bureau did not have support for one of the components of the 2010 response rate estimate--a general decline in responsiveness--and the Bureau does not have procedures for reviewing the estimate after testing to determine whether it should be revised. Establishing procedures for developing the response rate estimate, including documenting data sources and decisions, would enable the Bureau and others to better assess the estimate's reliability. Also, establishing procedures for when, such as after tests or other triggering events, and how to reevaluate the estimate would help the Bureau ensure that it is providing the most current response rate estimate for planning nonresponse follow-up and other activities, such as questionnaire printing. The Bureau's strategy of estimating the response rate conservatively may be prudent given past difficulties with conducting nonresponse follow-up after overestimating the response rate in 1990. Nonetheless, establishing and following procedures for developing, documenting, and reevaluating the estimate are important steps for understanding differences between the estimate and the actual response rate for 2010 and for evaluating the components and underlying assumptions when developing the estimate for the next census. Successful enumeration depends on early research, testing, and evaluation of census methods to increase response. Establishing procedures for selecting and prioritizing the testing of methods--such as the Internet or reminder telephone call--including documenting methods considered but not tested, would help the Bureau demonstrate that it has chosen an optimal research strategy for the decennial census, more effectively build capacity and institutional knowledge about changes in methods to consider in the future, and enable it to more efficiently begin testing for the next census. Recommendations for Executive Action: To enhance credibility of the response rate for determining cost and planning for future census activities, to inform assumptions underlying the 2020 response rate estimate, and to improve the planning and transparency of the Bureau's research and testing, we are recommending that the Secretary of Commerce direct the Bureau to take the following three actions: * Establish procedures for developing the 2020 response rate estimate, including documenting the data sources supporting the estimate's components and decisions that are made in establishing the components and analyzing 2010 data to assess the reasonableness of assumptions used in applying the national estimate's components to the LCO-type estimates. * Establish procedures for reevaluating and updating the 2020 estimate, including identifying events or changes in related operations that should trigger a review and documenting the results of such reviews. * Establish procedures for selecting methods for increasing response rate that will be the subject of research and testing, including requirements for documenting how the Bureau defines and weighs factors used to select methods and documentation on methods considered but not tested. Agency Comments and Our Evaluation: The Secretary of Commerce provided written comments on a draft of this report on September 22, 2008. The comments are reprinted in appendix III. Commerce generally agreed with our conclusions and recommendations and stated that the Bureau is committed to developing and implementing a documented, systematic methodology for establishing response rate estimates for future censuses and reevaluating the estimates throughout the decade. Because the recommendations in our draft report focused on costs and planning for the 2010 Census, we revised our recommendations to reflect actions to be taken to support future census planning, including analyzing 2010 data to assess the reasonableness of assumptions used in applying the national estimate's components to the LCO-type estimates. Commerce also provided technical corrections, which we incorporated as appropriate. In its comments, Commerce disagreed with our statement that the Internet response option increased response in the Census 2000 experiment. Commerce cited a summary statement from a Bureau report that concluded that the use of multiple response modes tested in 2000 does not increase response. However, Bureau analyses from October 25, 2002, and March 2004 on the Census 2000 experiment stated that the overall response rate increased when the Internet was offered as an alternative response mode. We therefore made no changes in the report. Commerce strongly disagreed with our statement about the reason why the Bureau decided not to include the Internet option in the design of the 2010 Census. In the report, we state that underestimating contract costs was the primary reason the Bureau eliminated the Internet option, which we attribute to a July 19, 2006, memo documenting the Bureau's rationale for eliminating the Internet response option from the Decennial Response Integration System (DRIS) contract. This memo states that the decision "was due largely to the fact that the Census Bureau underestimated the FY 2006-2008 contractor costs proposed to develop DRIS." We therefore left this statement unchanged. However, we added the Bureau's explanation provided in its agency comment letter that the decision was based on test results, which showed that offering this option did not increase overall response and would not offer any cost savings. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days after its issue date. At that time, we will send copies of this report to the Secretary of Commerce, the Department of Commerce's Inspector General, the Director of the U.S. Census Bureau, and interested congressional committees. We will make copies available to others upon request. This report will also be available at no charge on GAO's Web site at [hyperlink, http://www.gao.gov]. If you or your staff have any questions concerning this report, please contact Mathew J. Scirè at (202) 512-6806 or sciremj@gao.gov or Ronald S. Fecso at (202) 512-2700 or fecsor@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV. Signed by: Mathew J. Scirè: Director, Strategic Issues: Signed by: Ronald S. Fecso: Chief Statistician: [End of section] Appendix I: Objectives, Scope, and Methodology: The objectives of this report were to (1) assess how the U.S. Census Bureau (Bureau) develops, supports, and updates the response rate estimate, and the extent to which the Bureau uses the response rate estimate to inform its 2010 planning efforts; (2) describe the methods the Bureau considered for increasing response rates in 2010 and what it did to test these methods; and (3) assess how the Bureau identifies and selects for testing methods to increase response rates, including considering other surveys' methods for increasing response. The scope of our review was limited to the census mailout/mailback universe, which covered more than 80 percent of households in Census 2000. The majority of households in this category have standard city-style addresses, allowing them to receive their census questionnaires in the mail in addition to being expected to return their questionnaires by mail. We excluded from our review operations and methods aimed at enumerating those not included in the mailout/mailback universe, such as the Be Counted initiative and Enumeration at Transitory Locations; those initiatives related to, but not primarily focused on, increasing self-response, such as improving the quality of the Master Address File and providing assistance through Questionnaire Assistance Centers; methods tested prior to 2000 and already implemented in previous censuses; and those primarily intended to improve operational efficiency, such as postal tracking. To determine how the Bureau develops and uses the response rate estimate, we reviewed documentation to support the components of the response rate estimate and research literature on efforts to estimate survey response rate. We interviewed Bureau officials in the Decennial Management Division about the process used to develop and update the estimate, the assumptions on which the estimate is based, and how the estimate is used. We also interviewed experts on survey methodology from Statistics Canada, the U.S. Department of Labor's Bureau of Labor Statistics, and various academic institutions, as well as former Census Bureau officials and survey methodologists to obtain their views on the strengths and weaknesses of the Bureau's process for developing and updating the estimate. We compared responses to a common set of questions we asked the experts in order to identify themes. We also reviewed Bureau strategic planning documents--in particular, the 2010 Census life cycle cost model with supporting documentation, as well as the 2010 Census Integrated Program Plan and the 2010 Census Operations and Systems Plan--to understand the Bureau's use of the estimate, and evaluated the Bureau's practices for using the response rate estimate for generating the life cycle cost estimate against best practices criteria in our Cost Assessment Guide and other relevant GAO products. To describe the methods the Bureau considered for increasing response rates in 2010 and how the Bureau tested these methods, we reviewed Bureau analyses and evaluations of tests on methods to increase mail response and various Bureau planning documents, such as the 2010 Census Operations and Systems Plan. In addition, we reviewed documentation on the Bureau's communications campaign, such as evaluations of the 2000 Partnership and Marketing Program and the 2010 Census Integrated Communications Campaign Plan. We interviewed Bureau officials in the Decennial Management Division to obtain additional context about the methods they considered and tests they conducted. To assess how the Bureau identifies and selects methods to test for increasing response rates, we reviewed various Bureau planning documents, such as research and development planning group action plans, for factors that the Bureau considers in selecting methods to test. We interviewed Bureau officials in the Decennial Management Division and other divisions about the process for identifying and selecting methods for increasing response. We also reviewed our past work on lessons learned from Census 2000 on the importance of documentation and planning in order to evaluate the Bureau's process for selecting and prioritizing methods to test. We interviewed experts on survey methodology from Statistics Canada, the U.S. Department of Labor's Bureau of Labor Statistics, and various academic institutions, as well as former Census Bureau officials and researchers, about additional methods for increasing response that the Bureau could consider and to obtain their perspectives on the Bureau's process for identifying, testing, and implementing methods. We compared responses to a common set of questions we asked the experts in order to identify recurring themes. We also reviewed research literature on methodologies to increase response to mail surveys to identify additional methods that the Bureau could consider. We interviewed the following experts in survey methodology: * Anil Arora, Director General, Census Program Branch, Statistics Canada: * Paul Biemer, Distinguished Fellow in statistics, Research Triangle Institute: * Don A. Dillman, Regents Professor, Thomas S. Foley Distinguished Professor of Government and Public Policy, and Deputy Director for Research and Development in the Social and Economic Sciences Research Center, Washington State University, and former Senior Survey Methodologist, U.S. Census Bureau: * Elizabeth Martin, former Senior Survey Methodologist, U.S. Census Bureau: * Colm O'Muircheartaigh, Vice President for Statistics and Methodology, National Opinion Research Center and Professor, Irving B. Harris Graduate School of Public Policy Studies, University of Chicago: * Kenneth Prewitt, Carnegie Professor of Public Affairs, School of International and Public Affairs, Columbia University and former Director, U.S. Census Bureau: * Clyde Tucker, Senior Survey Methodologist, Bureau of Labor Statistics: We conducted our review from July 2007 through September 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Testing of Methods to Increase Response to 2010 Census: Replacement mailing. The Bureau plans to include the replacement mailing in the 2010 Census design. The replacement mailing involves sending a new questionnaire to households after households have received the initial census questionnaire in order to increase the likelihood that households will respond. The Bureau tested a targeted replacement mailing--where households that have not returned their initial questionnaires by a cutoff date receive the replacement form-- in the early 1990s and found that this method resulted in a 10 to 12 percent increase in the mail response rate. A blanket replacement mailing--where all households received a replacement questionnaire, including those that had already responded--was planned for Census 2000, but the Bureau dropped it from the design because of operational concerns that became apparent in the 1998 dress rehearsal. A targeted replacement mailing was incorporated into the 2010 design, based on the test results from the 1990s and the Bureau's plans to move to a short- form-only census. In 2003, the Bureau tested the targeted replacement mailing and was able to confirm the results of the tests from the early 1990s. Subsequent testing of the replacement mailing focused on ensuring that the Bureau could implement it successfully, including processing and removing the returns from the nonresponse follow-up workload using the handheld computers.[Footnote 21] The decision to eliminate the handheld computers from nonresponse follow-up and make nonresponse follow-up a paper-based operation diminished the Bureau's ability to remove late mail returns (including replacement questionnaires that are returned) from the nonresponse follow-up workload. The Bureau then examined options to get a better effect from the replacement mailing and decided to try to get the replacement mailings out more quickly. In order to do this, the Bureau decided that it would send a blanket replacement mailing to census tracts identified as low response areas, based on Census 2000 response data and updated demographic data from the American Community Survey. The replacement mailing packets for these tracts can be printed and labeled in advance--approximately 25 million total. Households not in these areas will receive a replacement questionnaire on a targeted basis--that is, only if they are in census tracts identified as medium response areas and if their initial questionnaires have not been received by the cutoff date. The Bureau estimates that it can print and label 15 million of these targeted replacement mailing packages in 4 days. Under this schedule, the Bureau will finish mailing out the replacement questionnaires on April 8, whereas under the original plan the replacement questionnaire mail out would not be completed until April 19. As a result, more households should be able to return the replacement questionnaires in time to be removed from the nonresponse follow-up workload. Two-column bilingual form. The Bureau plans to use a two-column bilingual form in certain locations in 2010. The two-column bilingual form provides two response columns, one in English and one in Spanish. Each column contains the same questions and response options, and respondents are instructed to choose the language that is most comfortable for them. The Bureau tested the bilingual form in 2005 to determine whether it would (1) increase overall response to the census and (2) lower item nonresponse (when a household returns the form but does not respond to a particular question) when compared to the standard English form. In the 2005 test, the two-column bilingual form panel demonstrated a significant positive effect on mail response rates of 2.2 percentage points overall, and a 3.2 percentage point increase in areas with high concentrations of Hispanic and non-White populations, but did not achieve lower rates of item nonresponse. The Bureau conducted another test of the two-column bilingual form in 2007, which produced response rate results similar to the 2005 results. In 2010, the Bureau plans to mail the two-column bilingual form to households in communities with heavy concentrations of Spanish speakers and areas with low English proficiency. Telephone reminder call. The Bureau does not plan to use a telephone reminder call in 2010. The multiple contact strategy that the Bureau used in 2000 included a reminder postcard mailed 1 week after the initial questionnaire packets were mailed. As part of the 2003 National Census Test, the Bureau tested the effect of an automated telephone reminder call in place of a reminder postcard on increasing the mail response rate. Out of a sample of 20,000 households, the Bureau was able to obtain phone numbers for 6,208 (31 percent), and these households received telephone calls to remind them to return their questionnaires if they had not already done so. The initial results indicated a significantly higher cooperation rate for the telephone reminder call panel when compared to the control panel, which received reminder postcards. The Bureau later conducted a supplementary analysis to determine if the higher cooperation rate was related to an underlying higher propensity to cooperate among households with listed telephone numbers. This analysis compared cooperation rates of two groups of households: those for which telephone numbers were available and those for which telephone numbers were not available. Among households for which a telephone number was available, the Bureau observed no significant difference in cooperation rates for housing units that received a reminder postcard compared to those that received a reminder telephone call. The Bureau does not plan to use a telephone reminder call in place of a reminder postcard in the 2010 Census. Due date on initial questionnaire mailing package. The Bureau will not place a due date on the initial questionnaire mailing package as part of the 2010 Census design. The value of placing a due date on the mailing package with the initial questionnaire is that it might evoke a sense of urgency or importance in respondents, leading to increased response to the census. In 2003, when the Bureau tested this method, it found that the inclusion of a due date did not significantly affect the overall response rate. However, it did result in a significantly higher response rate at an earlier point in the test, which can decrease the size of the replacement questionnaire mailing. The Bureau tested a due date again in 2006, this time combined with a "compressed schedule," in which questionnaires were mailed 14 days (as opposed to 21 days, as in the control group) before Census Day. A due date for mailing the questionnaire back was given in the advance letter, on the outgoing envelope and cover letter for the questionnaire mailing, and on the reminder postcard. The final report of the test concluded that the inclusion of the due date and compressed schedule resulted in a significantly higher mail response rate, by 2.0 percentage points. In addition, several measures of data completeness and coverage showed significant improvement. Bureau officials decided not to include the use of a due date in the 2010 Census, noting that they would like to conduct further testing to better understand the effects of a due date versus a compressed schedule. The Bureau is considering whether to test both the use of a due date and a compressed schedule in its 2010 Census Program for Evaluations and Experiments. Messaging to distinguish the replacement from the initial questionnaire. The Bureau does not plan to use a message to distinguish the replacement questionnaire from the initial questionnaire in 2010. The inclusion of a message on the replacement questionnaire informing respondents that they did not need to return the replacement questionnaire if they had already provided a response from the previous mailing was thought to be a way to reduce the number of multiple returns, and possibly improve overall response. The Bureau tested this method in 2005, and while the results showed a significant decline in the submission of multiple returns, they also showed a significant decrease in the response rate of 1.2 percentage points. Including a message to distinguish the replacement from the initial questionnaire was not recommended for, and will not be included in, the 2010 Census. Methods to Increase Self-Response Using Electronic Data Collection Systems: Interactive voice response. The Bureau does not plan to use interactive voice response in 2010. Interactive voice response allows respondents to use the telephone to respond verbally to digitized voice files that contain census questions and instructions. Speech recognition software is used to determine and record responses. Interactive voice response was tested in 2000, with households given the choice of providing their census data via interactive voice response or a paper questionnaire.[Footnote 22] The response rate for the interactive voice response panel was not statistically different from that of the control (mail only) group. However, the results are difficult to interpret because a portion of the sample in the interactive voice response panel either received the census form late or did not receive it at all. In 2003, the Bureau tested two different strategies for getting respondents to provide their census data using interactive voice response. One strategy, known as a push strategy, did not include a paper questionnaire in the initial mailing but rather pushed respondents to reply using interactive voice response. The other strategy, known as a choice strategy, included an initial paper questionnaire with the interactive voice response information on it, allowing respondents to choose their response mode. All nonrespondents received a paper replacement questionnaire. Households that had a choice of responding via paper or interactive voice response had overall response rates similar to households that only received paper, with about 7 percent of respondents given the choice using interactive voice response. Households in the push strategy had a significantly lower response rate--4.9 percentage points lower than households that only received a paper questionnaire. The Bureau does not plan to give respondents the option of providing their census data using interactive voice response in 2010. Letter encouraging Internet response or returning initial questionnaire instead of sending replacement questionnaire. The Bureau will not send a letter encouraging Internet response or returning the initial questionnaire instead of sending a replacement questionnaire in 2010. Instead of mailing a replacement paper questionnaire to nonrespondents, a letter could be sent to households encouraging them either to complete the paper questionnaire that they previously received or to use the Internet to submit their responses. The Bureau tested this method in 2005, sending a replacement mailing that contained a letter with an Internet address and an identification number needed to access the Internet questionnaire in place of a paper replacement questionnaire. Compared to sending out a paper replacement questionnaire, this method resulted in significantly fewer responses overall, decreasing the response rate by 3.7 percentage points. The Bureau will not include a letter encouraging the use of the Internet instead of a replacement questionnaire in 2010, in part because the Internet option was dropped from the 2010 Census design. Computer-assisted telephone interviewing. The Bureau will not use computer-assisted telephone interviewing for increasing response in 2010. Computer-assisted telephone interviewing allows respondents to use the telephone to connect with operators, who conduct interviews and record responses electronically. In a 2000 experiment, households were given the option of returning a paper form or providing their census data via computer-assisted telephone interviewing.[Footnote 23] When compared to households that were only given paper questionnaires, computer-assisted telephone interviewing brought about a 2.06 percentage point improvement in the overall response rate and also had a low item nonresponse rate. However, it entailed substantial costs for hardware, software, and programmer and interviewer time. The Bureau has not tested computer-assisted telephone interviewing since 2000 and does not plan to use this option in 2010. Internet. The Internet response option allows respondents to use an Internet-based questionnaire--with screens designed to resemble the paper questionnaire--to respond to the census. Respondents answer multiple-choice questions by clicking the appropriate buttons and checkboxes and text-entry questions by typing their answers into response fields. The Bureau gave respondents the option to respond by Internet in a 2000 experiment. Some households in the test were given the choice of providing their census data via Internet or a paper questionnaire.[Footnote 24] The experiment found that the Internet response option resulted in a 2.46 percentage point increase in response. In 2003, the Bureau again tested allowing respondents the option of providing their answers via the Internet by sending a paper questionnaire along with instructions for responding via the Internet in the initial mailing.[Footnote 25] Households that had a choice to respond by paper or the Internet had a similar overall response rate to households that were provided only paper, with about 10 percent of respondents choosing to respond by Internet. The Bureau decided not to include the Internet in the 2010 Census, despite including it in the scope of the contract awarded in 2005 for the Decennial Response Integration System. The Bureau noted that this decision was based on a number of factors, including the Bureau's underestimation of the contractor costs for the first 3 years of the contract, as well as test results that indicated that the Internet would not increase the response rate and concerns about the security of respondents' data prior to the Bureau receiving it. Communications campaign. Census 2000 included a greatly expanded outreach and promotion campaign--including, for the first time, paid advertising--in an attempt to increase public awareness of and promote positive attitudes about the census. This program, called the Partnership and Marketing Program, was considered a success after the Bureau reversed the trend of declining mail response rates, and the Bureau made plans to continue the program for 2010. Bureau officials stated that the 2010 campaign consolidates all census communications under a single communications contract. This Integrated Communications Campaign aims to motivate the entire populations of the 50 states, the District of Columbia, Puerto Rico, and other U.S. territories to participate in the census through partnerships (with community groups, businesses, colleges, faith-based organizations, and other targeted groups); public relations; events; Census in Schools; and paid advertisements in broadcast, print, and online media. Bureau officials noted that the advertising campaign would not be included in the dress rehearsal because the Bureau's experience including the advertising campaign in the 1998 Dress Rehearsal did not provide the feedback needed to revise the creative aspects of the campaign. To refine the communications campaign, the Bureau conducted an audience segmentation analysis to identify how to best reach people with the paid advertising campaign. In addition, the Bureau conducted focus groups in 2006 and 2007 to provide information on what motivates individuals to respond. The communications campaign is scheduled to run from mid-2008 through June 2010. In 2008 and 2009, most of the activities are focused on preparing and mobilizing partnerships. Further development of specific messages for various audiences and communication channels will take place from November 2008 through April 2009. Starting in mid-2009, the partnership program will begin outreach to certain hard-to-count populations. Activities and events targeting all audiences will begin in January 2010 and peak in March 2010. [End of section] Appendix III: Comments from the Department of Commerce: The Secretary Of Commerce: Washington, D.C. 20230: September 19, 2008: Mr. Mathew J. Scire: Director: Strategic Issues: United States Government Accountability Office: Washington, DC 20548: Dear Mr. Scire: I enclose the U.S. Department of Commerce's comments in response to recommendations contained in the United States Government Accountability Office's Draft Report, 2010 Census: Census Bureau Needs Procedures for Estimating the Response Rate and Selecting for Testing Methods to Increase Response Rate (GAO-08-1012). Sincerely, Signed by: Carlos M. Gutierrez: Enclosure: Comments in Response to the United States Government Accountability Office Draft Report Entitled: 2010 Census: Census Bureau Needs Procedures for Estimating the Response Rate and Selecting for Testing Methods to Increase Response Rate: September 30, 2008: General Comments: The Census Bureau appreciates the opportunity to review the GAO's draft report on this subject. We are in agreement with GAO's central fording—that the 2010 Census response rate estimate is not fully supported by a documented, systematic methodology, and that it has not been reevaluated since it was first established in 2001. While this is unfortunate, we believe that at this late date we cannot perform additional tests or substantive research that would allow us to establish a revised estimate. While it is possible that another review of available information might lead us to revise the projected response rate, such a revision would involve interpretation; it would not be based on significant test data. Moreover, we feel strongly that despite declining response rates for censuses and surveys throughout the decade, the decennial census environment is unique. For example, all people, households, and governments are involved, there is extensive outreach, partnership, and promotion, and it is a mandatory activity required by the U.S. Constitution. Accordingly, we remain persuaded that the response rate of 64% arrived at through examination of Census 2000 data is viable for planning purposes. We recognize that the actual response rate in 2010 may be different than we project. From our perspective, the major risk comes from overestimating the rate. If the actual rate is significantly lower than we project (as happened in 1990), we will be faced with some major challenges that will be difficult to overcome in the middle of census operations. We have very little leeway in our schedule at that point; the dates for producing apportionment and redistricting data are fixed by law. Thus, we would need to utilize contingency funding—or if necessary, request additional funding from Congress— in order to quickly hire and train more field staff so as to stay on schedule. The Census Bureau is committed to developing and implementing a documented, systematic methodology for establishing response rate estimates for future decennial censuses, beginning with an analysis of 2010 data in preparation for the 2020 Census. We also commit to reevaluating our estimates throughout the decade. Our specific comments on the report follow. Detailed Comments: On page 3, toward the middle of the page, the report states that we have not reevaluated the response rate change to 64% (which we made after the decision to drop use of handheld computers for nonresponse follow-up). Last month we briefed auditors on some of the changes made to the response rate estimate, and our preliminary revisions to the use of replacement mailings. At that time, we also shared with them the Census 2000 data and our analysis of response patterns that led to our decision to lower the estimated rate to 64%. At the top of page 5, the report states that our test results of Internet response "did not always increase the response rate." In our tests in Census 2000 and during this decade, the option never increased the overall response rate, so we suggest the sentence be reworded to accurately reflect our test results. At the bottom of page 5, the GAO recommends "that the Secretary of Commerce direct the Bureau to establish and implement procedures for documenting the process for developing the response rate estimate, as well as establishing when and how to reevaluate the response rate estimate." As we state in our summary comments, the Census Bureau is committed to doing this in preparation for the 2020 Census. On page 6, the report states: "Households that fail to mail back the census questionnaires are included in nonresponse follow-up workload, where enumerators follow up with telephone calls and door-to-door visits, or solicit census data from knowledgeable people, such as neighbors." This sentence should be adjusted to accurately capture the sequence of events that occurs during the census. First, one or more door-to-door visits is made. If no one is home, or if the respondent is busy, contact information is left, after which it may be possible to conduct the enumeration interview over the phone. As a last resort, when no data has been collected and no time is left to make additional visits, census data may be solicited from knowledgeable people, such as neighbors. On pages 7-8, footnote 6 states: "The Bureau also measures the percentage of surveys completed and returned for occupied housing units with deliverable addresses, called the return rate. Although the return rate is considered the more precise measure of public cooperation with the census, it is less useful than response rate for measuring nonresponse follow-up workload " We believe this footnote should be clarified to note that the mail return rate cannot be calculated until after nonresponse follow-up (NRFU) has been completed. The mailout universe includes all the addresses that will turn out to be vacant, and some that will be deleted as nonexistent or non-residential, during the NRFU operation. At the bottom of page 12, the discussion about replacement mailing effects states that our estimate of the effect was based on research during the 1990s. Replacement questionnaires were also tested in the 2003 National Census Survey, the 2005 National Census Test, the 2006 Census Test, and in the 2008 Census Dress Rehearsal. Our estimates are based on these more recent results. Also, we believe we get a larger impact from mailing replacement questionnaires in a test than in a census because during a census, we get a higher response rate in the first place. That is, a higher proportion of households will reply to the initial mailing during the actual census (as a result of paid advertising and partnership efforts), so this reduces the potential number of households that need to respond to the second mailing. Some of the testing that has occurred this decade deals with the timing. of printing and sending replacement questionnaires compared to the enumeration activities of NRFU. We were largely successful in removing very late mail returns from the NRFU workload in an automated environment. Due to the return to a paper-based NRFU operation, our revised plans for 2010 involve printing and distributing the replacement questionnaires earlier in hopes of gaining some of the same benefit. However, given the late decision to move to a paper-based NRFU operation, these plans have not been tested. In the first paragraph on page 16, the last portion needs to be revised to clarify that the operational plans to implement our revised replacement mailing approach are still being finalized. In that regard, the additional analysis that we are conducting does not pertain to fine- tuning the estimated effects; it pertains to the specifics of how to operationalize the approach by Type of Enumeration Area. Finally, because we have never tested these revisions under true decennial census conditions, we have no basis for precisely estimating how much response rate improvement they will yield. At the bottom of page 22, the report states that the Internet response option increased overall response during the Census 2000 experiment. We disagree with such a broad conclusion. During Census 2000, we conducted an experiment to examine both the use of incentives and the use of other response modes (Internet, interactive voice response, and an operator telephone interview). While there were variations in response rates for these different modes depending on whether incentives also were offered, the overall conclusion was: "The use of alternative response modes does not increase overall response rates to the census. Rather, it shifts households who would respond via the paper census to the other mode." (See Synthesis of Results from the Response Mode and Incentive Experiment for Census 2000 (RMIE), March 2003.) On page 23, we strongly disagree with the report's summary of why we decided not to pursue an Internet response option for the 2010 Census. On multiple occasions, the Census Bureau has stated that the decision was made because our tests had shown that offering this option did not increase overall response rate, and thus would not offer any cost savings, much less offset the actual costs that would be incurred to offer the option. While it is true we also have expressed—and continue to have—concerns about Internet security, that was not the primary reason we decided not to pursue this for the 2010 Census. On page 35, with regard to the discussion of the bilingual form, we wish to clarify that the results for item non-response for the 2005 National Census Test were thought to be attributed to a form design problem. The form was tested again in a survey in 2007 with changes to the form design, which corrected to some extent the problems with item non- response. Also, it should be noted that the determination of when to use the bilingual form is based not just on whether an area has a heavy concentration of Spanish speakers, but also on whether an area has low English proficiency. On page 36, the second paragraph regarding telephone reminder calls should say that the initial results indicated a significantly higher cooperation rate. It is also worth mentioning in this paragraph that there are costs to obtaining the phone numbers and making the calls, that there was a low percentage of phone numbers found in 2003, and that there is increasing use of individual cell phones over household landlines. In deciding whether to pursue this option, all of these considerations have to be weighed. [End of section] Appendix IV: GAO Contacts and Staff Acknowledgments: GAO Contacts: Mathew J. Scirè, (202) 512-6806 or sciremj@gao.gov: Ronald S. Fecso, (202) 512-2700 or fecsor@gao.gov: Acknowledgments: In addition to contacts named above, Lisa Pearson, Assistant Director; David Bobruff; Don Brown; and Elizabeth Fan made key contributions to this report. Tom Beall, Susan Etzel, Andrea Levine, Donna Miller, Ellen Rominger, and Elizabeth Wood provided significant technical support. [End of section] Footnotes: [1] The $13.7 billion to $14.5 billion estimate is expressed by the Bureau in nominal year dollars. [2] In this report, we use the term response rate to refer to the national mail response rate the Bureau uses to estimate nonresponse follow-up workload, unless otherwise noted. [3] The Bureau uses the U.S. Postal Service to deliver questionnaires to housing units with city-style addresses (house number and street name). However, in areas where housing units do not receive mail at a city- style address, the Bureau delivers questionnaires through an update/ leave process, in which enumerators deliver a census questionnaire to each housing unit. The household is asked to complete and return the questionnaire by mail. [4] According to the Bureau, these rates reflect mail response received as of the cutoff dates for determining nonresponse follow-up workload, and because these rates were computed differently, caution should be used in comparing the rates. [5] The Census 2000 response rate includes responses received by Internet, which other years do not include. [6] The Bureau also measures the percentage of surveys completed and returned for occupied housing units with deliverable addresses, called the return rate, which is calculated after nonresponse follow-up has been completed. Although the return rate is considered the more precise measure of public cooperation with the census, it is less useful than response rate for measuring nonresponse follow-up workload. [7] The 65 percent rate used as the baseline for the 2010 estimate is the rate that was achieved for the mailout/mailback universe on April 18, 2000, when the nonresponse follow-up universe was identified. [8] The Bureau later described the ability to remove all late mail returns from the nonresponse follow-up workload--both initial questionnaires and replacement questionnaires--as included in the replacement mailing effect. [9] GAO, Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs: Exposure Draft, GAO-07-1134SP (Washington, D.C.: July 2007). [10] The response rates in the American Community Survey analysis are comparable to the Bureau's definition of mail return rates for the decennial census in that vacant and nonexistent housing units are excluded from the denominator in the calculation. The Bureau's mail response rate, as defined for the decennial census, includes vacant and nonexistent housing units in the denominator. [11] The Bureau has used some of these data for identifying hard-to- count populations as part of the planning for the 2010 communications campaign, but these data are not incorporated into the mail response rate estimate. [12] In a blanket replacement mailing, all households in a certain area would receive a replacement questionnaire, regardless of whether they had already responded. In a targeted replacement mailing, only households that have not returned their initial questionnaire by a cutoff date would receive a replacement questionnaire. The original plan for the 2010 Census included using a targeted replacement mailing for nonresponding households in mailout/mailback areas. The revised plan uses a blanket replacement mailing for some areas, targeted replacement mailing for other areas, and no replacement mailing for the remaining areas. [13] In 2000, Type A LCOs were located in inner-city and urban areas. Type B offices were located in urban and metropolitan areas. Type C offices were located in suburban areas, small and medium-size cities, towns, and some rural areas. Type D offices were located in more rural areas. [14] The replacement questionnaire and two-column bilingual English/ Spanish census form are planned to be implemented for the first time in the 2010 Census. [15] Under the compressed mailing schedule, questionnaires were mailed 14 days as opposed to 21 days before Census Day. [16] National Academy of Sciences, Experimentation and Evaluation Plans for the 2010 Census: Interim Report (Washington, D.C.: Dec. 7, 2007). [17] The National Academy of Sciences has convened a panel of experts to review the Bureau's program of research, evaluation, and experimentation for the 2008 Dress Rehearsal and the 2010 Census. The panel will consider priorities for evaluation in the 2010 Census. The panel will conduct its work over a 3-year period, from September 2006 to September 2009. [18] In September 2007, the Bureau awarded a contract to DraftFCB, Inc., a communications agency, to create, produce, and implement an integrated marketing and communications campaign in support of the 2010 Census. The contract, with an estimated value of around $200 million to $300 million, was structured as a series of task orders for specific pieces of the campaign, to be issued over 4 years. [19] GAO, 2000 Census: Lessons Learned for Planning a More Cost- Effective 2010 Census, GAO-03-40 (Washington, D.C.: Oct. 31, 2002). [20] Conducted every 5 years, the Economic Census surveys U.S. businesses and provides official measures of output for industries and geographic areas and key source data for the gross domestic product and other indicators of economic performance. [21] In addition to using the handheld computers to enumerate households during nonresponse follow-up, the Bureau planned to use them for managing enumerators' workloads during nonresponse follow-up. Late mail returns, including replacement questionnaires, would be removed automatically from the nonresponse follow-up workload as soon as they were checked in, so long as the households had not yet been visited by an enumerator. In this way, the response rate estimate would include responses received even after the beginning of the nonresponse follow- up operation. [22] Another interactive voice response test panel was offered an incentive--a telephone calling card good for 30 minutes of domestic calls--for using interactive voice response instead of the paper questionnaire. The incentive did not increase the overall response rate compared to that of the mail-only control panel. [23] Another computer-assisted telephone interviewing test panel was offered an incentive--a telephone calling card good for 30 minutes of domestic calls--for using computer-assisted telephone interviewing instead of the paper questionnaire. The overall response rate for this incentive panel (71.81 percent) was similar to the overall response rate for the paper-only control panel (71.44 percent), though more respondents from the incentive panel chose to use computer-assisted telephone interviewing than the panel where no incentive was offered. [24] Another Internet test panel was offered an incentive--a telephone calling card good for 30 minutes of domestic calls--for using the Internet instead of the paper questionnaire. The overall response rate for this incentive panel (71.50 percent) was similar to the overall response rate for the paper-only control panel (71.44 percent), though more respondents from the incentive panel chose to use the Internet than the panel where no incentive was offered. [25] Another panel received a paper questionnaire along with instructions for responding via the Internet or interactive voice response. This panel had a similar response rate as the paper-only panel, with about 7 percent of respondents using the Internet. A third panel received instructions for responding via the Internet or interactive voice response but no initial paper questionnaire. This panel had a significantly lower response rate--by 5.7 percentage points--than households provided only the paper questionnaire. GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: