This is the accessible text file for GAO report number GAO-08-414 entitled '2010 Census: Bureau Needs to Specify How It Will Assess Coverage Follow-up Techniques and When It Will Produce Coverage Measurement Results' which was released on May 17, 2008. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to the Chairman, Committee on Oversight and Government Reform, House of Representatives: United States Government Accountability Office: GAO: April 2008: 2010 Census: Bureau Needs to Specify How It Will Assess Coverage Follow-up Techniques and When It Will Produce Coverage Measurement Results: Improving and Measuring Census Coverage: GAO-08-414: GAO Highlights: Highlights of GAO-08-414, a report to the Chairman, Committee on Oversight and Government Reform, House of Representatives. Why GAO Did This Study: An accurate decennial census relies on finding and counting people—only once—in the right place and collecting complete and correct information. GAO reviewed the Department of Commerce’s U.S. Census Bureau’s (Bureau) plans to (1) improve the census’s accuracy—or coverage—with its coverage follow-up operation, (2) assess the accuracy of the census through its census coverage measurement program, and (3) evaluate these efforts after 2010. GAO reviewed documentation and interviewed Bureau officials about the Bureau’s coverage follow-up and measurement plans. GAO compared plans for measuring coverage in 2000 and 2010 and interviewed 15 experts about changes to the plans. What GAO Found: The Bureau plans to expand its coverage follow-up operation to improve the accuracy of enumeration data collected in 2010. As in Census 2000, coverage follow-up is intended to resolve count discrepancies (when the number of household members reported on a completed questionnaire fails to match the number of persons for whom information is collected) and complete the enumeration of large households (more than six persons living in a housing unit). For 2010, the Bureau researched new techniques to increase the number of ways a person or household could be selected for coverage follow-up. These include (1) using coverage probes on the census form to identify cases with potential under- or overcount problems, (2) using administrative records to identify households that include persons who may not have been counted, and (3) using computer matching to determine potential duplicate persons. However, it is not clear on what basis the Bureau will select or prioritize cases from this potentially expanded universe. The Bureau plans to further assess the techniques during the 2008 Dress Rehearsal, leaving little time to consider and implement improvements for the 2010 coverage follow-up operation. Because of the potential for the coverage follow-up workload to increase, it will be important for the Bureau to be clear on how it will decide which techniques to use and how it will select coverage follow-up cases to ensure that the operation may best improve coverage. Although the Bureau’s overall design for measurement is similar in many respects to the design for 2000, its measurement plan differs in important ways. The Bureau is planning to use a sample design similar to that used in 2000 and still intends to produce net coverage error estimates. However, the Bureau does not plan to use measurement results to adjust the 2010 count of the nation’s population and is planning to make four significant changes: (1) conduct the measurement interview 4 months later than it did in 2000, (2) use a different method—logistic regression—to estimate net error, (3) use estimates of the components of coverage error in an effort to improve future decennials, and (4) expand computer matching and clerical matching to match persons in its postenumeration survey to the entire census. Still, the Bureau has not specified when it will produce estimates of net coverage error and the components of coverage error for the 2010 Census. The Bureau also has not clearly described how it will associate the components of coverage error to census operations in order to improve future decennials. Because of the Bureau’s changes to its coverage measurement program, it is important for the Bureau to solidify its plans and share them with stakeholders to ensure that there is an agreement on what the program will produce and when the Bureau will produce its results. The Bureau is in the early stages of planning how it will evaluate coverage follow-up and measurement after the 2010 Census. As part of this effort, the Bureau is considering recommendations of the National Academy of Sciences for the Bureau’s program of evaluations and experiments for Census 2010. The Bureau has not yet developed a timeline for completing its evaluation plans. What GAO Recommends: GAO recommends that the Secretary of Commerce direct the Bureau to specify (1) criteria it will use to assess techniques for identifying coverage follow-up cases; (2) when it will provide coverage measurement estimates and how it will relate components of coverage error to specific census operations to improve future decennials; and (3) key decision points and plans for evaluating aspects of the 2010 coverage follow-up and measurement efforts. In commenting on a draft of this report, Commerce had no substantive disagreements with our recommendations and will develop action plans for each one; and provided technical comments that we incorporated as appropriate. To view the full product, including the scope and methodology, click on [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-414]. For more information, contact Mathew J. Scirè at (202) 512-6806 or sciremj@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: Bureau Plans to Expand Coverage Follow-up to Improve Census 2010 but Has Not Finalized Plans for Identifying Follow-up Cases: Bureau Has Made Changes to Coverage Measurement for 2010 Intended to Benefit the 2020 Census, but Key Aspects of Program Plan Remain Unclear: Bureau Plans for Evaluating the 2010 Coverage Follow-up Operation and Census Coverage Measurement Program Are in Early Stages: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Comments from the Department of Commerce: Appendix III: GAO Contact and Staff Acknowledgments: Figures: Figure 1: Decennial Census Population Net Undercount Rates from the Bureau's Demographic Analysis in Percentages, 1940-2000: Figure 2: Operation Flowchart for the Bureau's 2008 Dress Rehearsal Coverage Follow-up Operation: Figure 3: Undercount and Overcount Probes on Census Questionnaire for the 2008 Dress Rehearsal: Figure 4: Timeline for Coverage Follow-up Interviews and Census Coverage Measurement Interviews Using Actual Dates for 2000 and Estimated Dates for 2010: United States Government Accountability Office: Washington, DC 20548: April 15, 2008: The Honorable Henry A. Waxman: Chairman: Committee on Oversight and Government Reform: House of Representatives: Dear Mr. Chairman: An accurate decennial census relies on finding and counting people-- only once--in the right place and collecting complete and correct information. The Department of Commerce's U.S. Census Bureau (Bureau) faces the daunting challenge of cost effectively counting a population that is growing steadily larger, more diverse, and according to the Bureau, increasingly difficult to find and reluctant to participate in the census. These challenges are made even more difficult by the Bureau's April 3, 2008 decision to redesign certain aspects of the decennial census. Two basic errors can affect the count of the population: omission of persons who should have been counted and erroneous enumerations of persons who should not have been counted, such as persons counted more than once or persons counted in the wrong geographic place. To ensure accuracy, the Bureau, among other operations, will conduct a coverage follow-up operation in 2010 to determine if additional persons might have been missed (undercounted) from already enumerated households, or if persons might have been counted in error (overcounted). Further, apart from the enumeration process, the Bureau will conduct a census coverage measurement program to assess the accuracy of the 2010 Census. The census is a critical national effort mandated by the Constitution. Census data are used to apportion seats in the Congress, redraw congressional districts, allocate billions of dollars in federal assistance to state and local governments, and for numerous other public and private sector purposes. The Bureau estimates that it has generally undercounted the total population in prior censuses. For the first time, in 2000, although estimates showed that it undercounted certain population groups, the Bureau estimates that it somewhat overcounted the total population. For 2010, the Bureau plans to improve the accuracy of its population count by, among other things, expanding its coverage follow-up operation. Further, the Bureau and others questioned the accuracy of the data obtained from the Bureau's coverage measurement program in 2000. As a result, for 2010, the Bureau plans to revise its methodology for determining coverage error--the extent to which persons are under-or overcounted in the enumeration process. As agreed, we determined (1) how the Bureau plans to improve coverage through its coverage follow-up operation; (2) how the Bureau's census coverage measurement plans for assessing the accuracy of the 2010 Census compare with efforts in 2000 and what the potential impact will be of those changes, including the Bureau's plans for using its measurement program results to adjust population counts; and (3) how the Bureau plans to evaluate the coverage follow-up and measurement efforts after the 2010 Census. To meet these objectives, we reviewed information about the Bureau's coverage follow-up and coverage measurement plans and interviewed Bureau officials. We reviewed relevant evaluations from Census 2000 and studies conducted by the Bureau and others, including the National Academy of Sciences. Further, we interviewed 15 experts about the Bureau's coverage measurement methodology and plans. We identified these experts based on published reports as well as recommendations from the Bureau and the National Academy of Sciences. (For a list of experts we interviewed, see app. I.) We conducted this performance audit from January 2007 to March 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Results in Brief: The Bureau plans to improve coverage by expanding the techniques it uses to identify and select cases for follow-up. However, it is not clear on what basis the Bureau will decide which techniques to use to identify potential cases, and which of these cases it will select or prioritize for coverage follow-up. As in Census 2000, coverage follow- up is intended to resolve count discrepancies (when the number of household members reported on a completed questionnaire fails to match the number of persons for whom information is collected) and complete the enumeration of large households (more than six persons living in a housing unit). The Bureau plans to include all such count discrepancy cases and all large households in its coverage follow-up universe in 2010. To improve coverage for 2010, the Bureau researched new ways to identify a more comprehensive set of cases for follow-up. Specifically, to increase the number of ways a person or household could be selected for follow-up, the Bureau tested three new coverage follow-up techniques: (1) using coverage probes on the census form mailed to households and during the nonresponse follow-up interviews to identify instances where persons may be under-or overcounted; (2) using administrative records to identify households that include persons who may not have been counted; and (3) using computer matching to determine potential duplicate persons. The Bureau is uncertain how many cases for follow-up are likely to be found based on these new techniques. Nonetheless, the number of follow-up cases could be very large and very costly to resolve. Based on its experience with the 2008 Dress Rehearsal, the Bureau expects to decide what techniques it will use for identifying and selecting cases during the 2010 Census coverage follow- up operation and how many cases it can afford to select for follow-up. Bureau officials explained that they intend to consider cost and the percentage of cases resulting in a change relative to the cost--known as yield--but the Bureau has not fully specified how it will make decisions about expanding the coverage follow-up universe or how it will select or prioritize cases from those identified by the new techniques. Using results from the 2006 Census Test, the Bureau has already decided that coverage follow-up will only be conducted by telephone rather than personal visits because the yield was higher for telephone follow-up. This decision, however, does not necessarily reflect trade-offs in the limitations of each technique used to select cases for follow-up. For example, coverage may not be improved for groups for which the Bureau does not have telephone numbers--the very wealthy or the very poor. Given that the Bureau expects to use the dress rehearsal to learn more about the expanded selection techniques, and that the dress rehearsal has been delayed by a month, the Bureau will have little time to complete its final plans for coverage follow- up for the 2010 Census. Because of the potential for the coverage follow-up workload to expand, it will be important for the Bureau to be clear on how it will decide which techniques to use and cases to select to ensure that coverage follow-up may best improve coverage. Although the Bureau's plan for coverage measurement in 2010 is similar in many ways to its measurement design for 2000 and previous coverage evaluation programs, its plan differs in important ways, including its overall objectives, timing, statistical techniques used, and use of computer matching. Nonetheless, much of what the Bureau will do is yet to be decided, including most notably, when the Bureau will produce estimates of net coverage error and components of coverage error, and how it will meet one of its objectives for coverage measurement in 2010--improving future census operations. One important aspect of the Bureau's plans for 2010--specification of the logistic regression model--cannot be known until after the Bureau collects the 2010 coverage measurement data. Similarities between 2000 and 2010 include the Bureau's sample design for coverage measurement[Footnote 1] and that the Bureau once again intends to use a dual systems approach to produce net coverage error estimates--or the difference between the coverage measurement program's estimate of the total population and the decennial count of the total population.[Footnote 2] However, several of the Bureau's objectives for the 2010 coverage measurement program differ. For 2010, the Bureau places greater focus on improving future decennials. In contrast, for 2000, the Bureau designed coverage measurement to consider adjusting the count of the nation's population. The Bureau does not plan to use coverage measurement results to adjust the 2010 count of the nation's population because it believes that with a sufficiently correct count, adjustment would introduce as much or more error than it was designed to correct. The Bureau's measurement program in 2010 differs in other significant ways from that in 2000, for example, (1) starting the coverage measurement interview about 4 months later, to permit expansion of the Bureau's efforts to conduct the coverage follow-up operation;[Footnote 3] (2) estimating net coverage error using logistic regression, which is a standard statistical modeling technique; (3) estimating the components of net coverage error to better understand the details of erroneous enumerations and omissions; and (4) expanding computer matching and clerical matching to match persons in the coverage measurement program's postenumeration survey to the entire census to resolve or identify duplicates.[Footnote 4] Bureau officials said that the coverage measurement interview will begin later because they expect to expand the coverage follow-up operation, and that to maintain independence and minimize possible contamination between enumeration and measurement, the Bureau did not want to overlap coverage follow-up and coverage measurement. Experts we interviewed were generally concerned about the contamination issue but also viewed the time delay as a very significant challenge because respondent recall[Footnote 5] is vital to conducting the measurement program. The Bureau has simulated the use of its new method of estimating net error--logistic regression--by using data from 2000 and found it to work. The Bureau will be unable to fully specify the model it will ultimately use until after the data collection phase of its coverage measurement program. One aspect of its plans that the Bureau should be able to specify now is when it will produce net coverage error estimates and components of coverage error estimates for 2010. Further, though the Bureau has preliminary plans--using data table shells--to associate components of coverage error to census operations, the Bureau has not clearly explained how it plans to meet its objective of improving future decennials. For the most part, experts we interviewed, after they reviewed our summary of the Bureau's coverage measurement plans, stated that the Bureau had improved its research model and had taken significant and constructive steps in attempting to address key issues for coverage error, but several experts noted that more detailed planning is needed. Because of the Bureau's changes to its coverage measurement program, it is important for the Bureau to make its plans clear and share them with stakeholders to ensure that there is an agreement on what the program will produce and when the Bureau will produce its measurement results. The Bureau's plans for evaluating its coverage follow-up operation and census coverage measurement program for the 2010 Census are in early stages of development. In terms of evaluation, the Bureau suggested research on various aspects of coverage follow-up and measurement and contracted with the National Academy of Sciences to recommend proposed research for the 2010 Census. The academy's final report is due by September 2009, but an interim report was issued December 2007. The interim report contained recommendations for the Bureau's program of evaluations and experiments for the 2010 Census. Bureau officials have no timeline yet for making decisions about the evaluations of the Bureau's coverage follow-up operation or census coverage measurement program following the 2010 Census, and they noted that to determine which recommendations to implement, the Bureau will consider costs and staffing needs. We are recommending that the Secretary of Commerce direct the Bureau to provide Congress and other census stakeholders with more specific plans on conducting coverage follow-up and census coverage measurement and evaluating the results of those programs for the 2010 Census. To improve program operations, we recommend that the Bureau specify the criteria it will use to assess the techniques for identifying cases for coverage follow-up in the 2008 Dress Rehearsal and the timeline it plans to follow that will ensure that the results of the assessment may be used in planning for 2010. To enhance transparency and oversight for improving coverage measurement for the 2010 Census, we recommend that the Bureau describe when it will provide estimates of net coverage error and components of coverage error and how it plans to relate components of coverage error to census operations in order to improve future decennials. Finally, we recommend that the Bureau provide key decision points and plans for evaluating aspects of the 2010 Census coverage follow-up operation and census coverage measurement program. The Secretary of Commerce provided written comments on a draft of this report. The comments are reprinted in their entirety in appendix II and we have incorporated in the report, as appropriate, the Department of Commerce's technical comments. Commerce had no substantive disagreements with our recommendations and stated that it will develop formal action plans for each recommendation. While Commerce believes a timeline for producing estimates of net coverage error and components of coverage error is certainly a reasonable expectation, Commerce noted that it is unclear as to how it can at this time provide any specifics on how coverage results might improve future censuses before the coverage data are obtained and analyzed. We recognize that the Bureau has developed table shells that associate components of coverage error to census operations, but these table shells and plans for how the Bureau will conduct its analyses have not been shared with the Congress and other stakeholders. In addition, it is not clear from current Bureau plans how it will go about obtaining the data required to complete the tables. Background: Historically, the census has been affected by undercounts--those who were missed in the decennial count. Further, certain groups may be undercounted more than others--such as babies, minorities, and renters who move often. Seeking to obtain an accurate count has been a concern since the first census in 1790. Concern about undercounting the population continued through the decades. In the 1940s, demographers began to obtain a more thorough understanding of the scope and nature of the undercount. For example, the selective service registration of October 1940 showed 2.8 percent more men than the census count. According to the Bureau, operations and programs designed to improve coverage have resulted in the total undercount declining in all but one decade since the 1940s. These measures of coverage are based on demographic analysis, which compares the census count to birth and death certificates and other administrative data (see fig. 1). Figure 1: Decennial Census Population Net Undercount Rates from the Bureau's Demographic Analysis in Percentages, 1940-2000: This figure is a versicle bar graph showing the decennial census population net undercount rates from the bureau's demographic analysis in percentages, 1940-2000. The X axis is the year, and the Y axis is the percentage. Year: 1940; Percentage: 5.4%. Year: 1950; Percentage: 4.1%. Year: 1960; Percentage: 3.1%. Year: 1970; Percentage: 2.7%. Year: 1980; Percentage: 1.2%. Year: 1990; Percentage: 1.6%. Year: 2000; Percentage: 0.1%. [See PDF for image] Source: U.S. Census Bureau. [End of figure] In contrast to the long-standing use of demographic analysis to estimate census undercount, modern coverage measurement began with the 1980 Census, when the Bureau compared decennial figures to the results of an independent sample survey of the population. In using statistical methods such as these, the Bureau began to generate detailed measures of the differences among undercounts of particular ethnic, racial, and other groups. In 1990, the Bureau relied on its Post-Enumeration Survey to verify the data it collected through the 1990 Census. For this effort, the Bureau interviewed a sample of households several months after the 1990 Census, and compared the results to census questionnaires to determine if each sampled person was correctly counted, missed, or double counted in the census. Using the definition of those missed minus those double counted, the Bureau estimated whether there was a net undercount or net overcount. For the 1990 Census, the Bureau estimated a net undercount of about 4 million people. During Census 2000, for the first time, the Bureau's coverage measurement program measured a net population overcount. To estimate the accuracy of the 2000 Census, the Bureau conducted its Accuracy and Coverage Evaluation, expecting it could be used to adjust the Census 2000 results for all nonapportionment purposes if it improved the census data. The original March 2001 estimates of the Accuracy and Coverage Evaluation, which was an independent sample survey designed to estimate the number of people who were under-and overcounted in the census, a problem that the Bureau refers to as coverage error, became available in time to correct census files. However, in a revised March 2003 evaluation, the Bureau was unable to conclude that the adjusted data were more accurate and determined that the 2000 Census tabulations would not be adjusted for any purpose. The decision not to adjust was consistent with decisions made during the 1990 decennial, when the Department of Commerce decided, because of other problems, not to use the postenumeration survey to adjust the decennial count. During this decade, the Bureau has conducted a testing and evaluation program to assess certain activities that it believes can improve the accuracy of the census. In preparing for the 2010 Census, the Bureau has conducted two field tests--in 2004, the Bureau tested in the Queens Borough of New York City and southwest Georgia and, in 2006, tested at the Cheyenne River Reservation in South Dakota and parts of Austin, Texas, in Travis County. In 2003 and 2005, the Bureau also conducted national surveys to test various aspects of changes planned for 2010. We reported earlier about the importance of these earlier assessments leading to a design that would be sufficiently mature so that the dress rehearsal for the 2010 Census demonstrates the feasibility of the various operations and technologies planned for the decennial under conditions that are as close as possible to the actual census.[Footnote 6] The Bureau is currently conducting its dress rehearsal, one of the final planning activities for the decennial. Census Day for the 2008 Dress Rehearsal activities in sites located in North Carolina and California will be May 1, 2008, a delay of 1 month from the Bureau's initial plan. The first 2010 Census operation, building an accurate address list, overlaps with dress rehearsal activities and has also already begun.[Footnote 7] For the 2010 Census, the majority of the housing units in the country will receive paper census questionnaires delivered either by mail or by census field workers before April 1, 2010. Those households that do not return their questionnaires will be contacted by census field workers during the nonresponse follow-up operation to determine the number of people living in the housing unit on Census Day and other information. The Bureau also uses other techniques for counting persons. For example, people in group quarters (such as college dormitories, homeless shelters, and nursing homes) will be enumerated in separate operations at the facilities by enumerators who will then leave forms listing the names of the people living or staying there and then return to pick up the completed forms and, if necessary, conduct follow-up interviews. For the 2010 Census, the Bureau has tested and planned revisions to the census form with the goal of improving the census' coverage. The revisions are intended to improve methods for ensuring that persons are counted in the correct place and only once.[Footnote 8] These changes include revising residency rules, simplifying the instructions provided to respondents on residency, and including questions--called coverage probes--about persons who may be missing or incorrectly included in the response. For Census 2000, the Bureau relied on 31 residence rules to instruct respondents on who to include on the mail-back form. The Bureau revised these 31 rules to 1 rule with 13 broad living situations for 2010. For 2010, the Bureau is also planning on using two coverage probes, one asking about potentially missed people (undercounted) and one asking about potentially duplicated people (overcounted), on the census form. Although the Bureau used coverage probes in the nonresponse follow-up operation for 2000, it did not include probes in the 2000 Census mailed questionnaires as is now planned for 2010. The Bureau intends to use answers to these coverage probes and other techniques to determine if an enumeration is likely to be incorrect or incomplete and, therefore, if cases should be selected for follow-up. The Bureau uses the results of the coverage follow-up operation to correct the results of the initial enumeration. Coverage follow-up interviews with respondents determine if changes should be made to their household rosters as reported on their initial census returns. Specifically, these results lead to a correction of census information and counts, as coverage follow-up results are combined with the initial enumeration and are used as the official enumeration of the household. Bureau Plans to Expand Coverage Follow-up to Improve Census 2010 but Has Not Finalized Plans for Identifying Follow-up Cases: For Census 2010, the Bureau is planning to improve coverage by expanding its coverage follow-up operation to obtain more accurate enumeration data. To prepare for this, the Bureau has conducted tests of an expanded follow-up operation. As in Census 2000, coverage follow- up is intended to resolve count discrepancies and complete the enumeration of large households. For 2010, the Bureau also plans to use additional techniques to increase the number of ways a person or household could be selected for follow-up. However, the Bureau has not finalized plans for how to identify and select cases from the expanded follow-up universe. The expansion of selection criteria for coverage follow-up has the potential to dramatically increase the workload for this operation and, as a result, increase costs. While the Bureau plans to budget for coverage follow-up in order to achieve the best results or highest yield of changes to household rosters, it is unclear on what basis the Bureau will assess the additional techniques for identifying potential cases or the basis for selecting cases for follow-up using the expanded techniques. The Bureau plans to assess its experiences with coverage follow-up, including important interfaces within the Bureau and between the Bureau and its contractor, during the 2008 Dress Rehearsal in making further decisions on follow-up for 2010. For example, three separate groups within the Bureau have responsibility for selecting follow-up cases, but one of those groups eliminates overlap in cases before sending them to the contractor for telephone follow-up. Recent decisions to delay the dress rehearsal, including follow-up activities, increase the risk that the results of additional testing of these techniques in 2008 will not be sufficiently timely or complete to improve operations for 2010, and the Bureau has not stated when it will complete its assessment of coverage follow-up for the 2008 Dress Rehearsal. Bureau Plans to Expand Techniques for Identifying Coverage Follow-up Cases: The Bureau is planning to improve coverage by expanding its coverage follow-up operation. To meet this objective, the Bureau expects to rely on previously used techniques as well as techniques it is now developing to identify cases for follow-up. As in Census 2000, coverage follow-up for 2010 will include: * resolving count discrepancies, when the number of household members reported on a completed questionnaire fails to match the number of persons for whom information is collected, and: * completing the enumeration of large households, in which there are more than six persons living in a housing unit. For 2010, the Bureau also plans to use additional techniques for selecting a person or household for follow-up. The techniques planned include: * using questions--called coverage probes--on the census form itself and in the nonresponse follow-up interviews to identify where persons may be omitted or counted more than once, * using administrative records to identify persons who may be omitted, and: * using computer matching from the universe of census returns to identify persons who may have been counted more than once. Although the Bureau has not finalized coverage follow-up plans for the 2010 Census, in June 2007, the Bureau developed a work flow and narrative for the 2008 Dress Rehearsal coverage follow-up operation. This work flow shows the process for collecting enumeration data, selecting coverage follow-up cases, and conducting follow-up (see fig. 2). Once the Bureau selects a case for follow-up, it plans to work through its Decennial Response Integration System (DRIS) contractor, which will conduct coverage follow-up interviews by telephone. The telephone interviews determine if changes should be made to the household rosters as reported on the initial census returns. The questions in the follow-up interview are designed to identify if persons were omitted or counted in error because they should have been counted at a different address or not at all--for example, because a person had been born after or died before Census Day. Figure 2: Operation Flowchart for the Bureau's 2008 Dress Rehearsal Coverage Follow-up Operation: This figure is a an operation flowchart showing the bureau's 2008 dress rehearsal coverage follow-up operation. [See PDF for image] Source: GAO analysis of U.S. Census Bureau data. [End of figure] As shown in figure 2, enumeration data will be captured by DRIS. Also, in selecting cases for follow-up, the Bureau will identify any overlap in cases represented by large households, count discrepancies, coverage probes, administrative records review, and computer matching. While the Bureau has tested new techniques for coverage follow-up, it has not yet tested these steps with the new processing interfaces within the Bureau--Decennial Systems and Processing Office, Decennial Statistical Studies Division, and Data Integration Division--and between the Bureau and DRIS. During the 2008 Dress Rehearsal, from May 27 to August 9, 2008, the Bureau will not only be assessing various aspects of the expanded follow-up, but it will also be testing for the first time the DRIS integration of data as well as the Bureau's process for identifying overlap cases. After the Bureau selects cases for follow- up, the DRIS contractor assigns an identification number to each case and oversees telephone interviews. Each case may receive up to seven follow-up telephone calls, and the DRIS contract includes such aspects as on-demand automated call distribution, call routing, reporting, and quality assurance monitoring. Resolving Count Discrepancies: For 2010, as in Census 2000, all count discrepancy cases identified on mailed paper census forms will be part of the coverage follow-up operation. The Bureau will check completed questionnaires for discrepancies between the number of persons reported as members of a household and the number of persons for whom census information was provided on the census forms. For example, questionnaires that fail edit checks might include cases where the respondent may indicate that five people were living in the housing unit on Census Day, but there is information for only two people. Enumerating Large Households: As in 2000, the Bureau is also aiming to improve coverage of people in housing units by completing the enumeration of large households, meaning those that indicate on their census forms that there are more than six persons in the housing unit and for which there was not enough room on the mailed questionnaire form for self-response. The questionnaire has space for only six household members to be fully listed. The nonresponse follow-up interview is not limited in the number of people that can be reported and all household members can be enumerated during nonresponse follow-up; therefore, no large household cases will need to be enumerated during the coverage follow-up operation as a result of the nonresponse follow-up interview. Using Coverage Probes to Identify Potential Under-or Overcounting: The Bureau plans to expand its universe for coverage follow-up in 2010 by including cases resulting from the use of coverage probes. For the dress rehearsal, the Bureau plans to contact anyone who affirmatively answers the undercount question (which is at the household level) and the overcount question (which is at the person level) (see fig. 3). If either or both coverage probes prove unproductive in the dress rehearsal, Bureau officials indicated that they might be eliminated from the follow-up operation. The coverage probes are included on the mail-back questionnaire and among the questions asked during the nonresponse follow-up. During nonresponse follow-up, for example, if a person responds in the affirmative to the coverage undercount probe, the interviewer collects the name and demographic information for the added person. All names of individuals added to the household roster[Footnote 9] through the undercount probe during nonresponse follow-up will be sent to the coverage follow-up operation for additional follow-up. Figure 3: Undercount and Overcount Probes on Census Questionnaire for the 2008 Dress Rehearsal: This figure is a copy of undercount and overcount probes on census questionnaire for the 2008 dress rehearsal. [See PDF for image] Source: GAO presentation of U.S. Census Bureau information. [End of figure] As shown in figure 3, for the 2008 Dress Rehearsal, the undercount probe asks if there are any additional people staying at the residence on Census Day that were not included earlier--children, such as newborn babies or foster children; relatives, such as adult children, cousins, or in-laws; nonrelatives, such as roommates or live-in baby sitters; or people staying at the residence temporarily. The overcount probe for the 2008 Dress Rehearsal asks if persons sometimes lived or stayed elsewhere--such as while attending college, while in the military, while staying at a seasonal or second residence, during a child custody arrangement, while in jail or prison, while in a nursing home, or for another reason. The overcount probe is asked of the first six household members on the mail-back questionnaire and of all persons in the nonresponse follow-up interview. Evaluations of the coverage follow-up operation for the Bureau's 2004, 2005, and 2006 tests examined the performance of both the undercount and overcount probes. In the Bureau's 2004 test, the overcount probe was found to be more effective in finding households with erroneous enumerations than the undercount probe was in finding persons omitted from the count. In this test several problems were identified with the undercount probe, most notably a low rate of added residents to the household roster. Results from the test showed that of the 3,919 respondents at the Georgia test site who answered the undercount probe affirmatively, only 53 persons were added to the household roster. The Bureau noted that the low rate of added persons was "perplexing." Other problems included difficulties with the interviewer reading the probe as worded and indications that use of the probe may have actually increased the number of count discrepancy cases. In 2005, the Bureau tested two different versions of both the undercount and overcount probes, and changes to both probes were adopted for the 2006 Census Test. During the 2006 test, the Bureau found that the overcount probe "performed as expected" and generated a higher rate of deleted persons from the census test roster for both the mail-back questionnaire and nonresponse follow-up operation than other new techniques tested to identify coverage follow-up cases. Although the Bureau designed the undercount probe to add persons, the undercount probe resulted in more deleted persons than added persons.[Footnote 10] The Bureau noted that over half of the deletions stemmed from cases in which the respondent either answered "yes" to both coverage probes or the respondent answered "yes" to the undercount probe and had a count discrepancy. Indeed, in a separate analysis that examined the rate of overlap among coverage follow-up cases--meaning households were flagged for coverage follow-up through more than one technique--the Bureau found that the highest significant rate of overlap was among cases where the respondent answered "yes" to both coverage probes. Additionally, there were some operational difficulties using the undercount probe during the nonresponse follow-up operation in the 2006 Census Test.[Footnote 11] For example, the Bureau found that although persons answered the undercount probe affirmatively (1,234 households) during nonresponse follow-up, very few persons (8) were added to the coverage follow-up roster.[Footnote 12] The Bureau referred to this number as "surprisingly low," because if a household replies "yes" to the undercount probe, the Bureau expected to see a person added to the roster for coverage follow-up. In its evaluation of coverage follow-up during the 2006 Census Test, the Bureau speculated that there could have been operational error during nonresponse follow-up that led to this discrepancy. Furthermore, of those 8 persons who were added and appeared in coverage follow-up, only 3 could be confirmed as residents during follow-up, leading the Bureau to conclude that the undercount question added persons incorrectly more than half of the time after nonresponse follow-up. Although the Bureau has made some slight modifications to the response options for the overcount probe it plans to use in the dress rehearsal, it plans to use the same undercount probe for the dress rehearsal as it did for the 2006 test. Using Administrative Records to Identify Potentially Omitted Persons: For the dress rehearsal, the Bureau plans one wave of matching with administrative records to identify duplicate or suspected incorrect data. For administrative matching, the Bureau will use the Enhanced Statistical Administrative Records System, which is a Bureau-wide resource that collects certain data from the Internal Revenue Service, the Social Security Administration,[Footnote 13] the Selective Service, the Department of Housing and Urban Development, as well as other federal agencies. The Bureau will match census returns against records from these sources using various algorithms to associate persons with housing units to identify households with potential undercounting.[Footnote 14] The Bureau plans to match names on census questionnaires with names in the administrative database--if a household is found to have more persons in the administrative records than on the census form, it would be a candidate for coverage follow- up. The Bureau first tested selecting cases for follow-up through comparison with administrative records in 2005. For the 2006 Census Test, all of the cases with matches from the administrative records search had personal visits; however, Bureau officials noted that personal visits were conducted because staff had sufficient time, given the smaller geographic area of a test site. Personal visits will not be part of the national 2010 coverage follow-up operation. The 2008 Dress Rehearsal marks the first time that cases selected through administrative record matching will be tested in the coverage follow-up telephone operation. Using Computer Matching to Identify Persons Who May Have Been Duplicated: For the dress rehearsal, the Bureau will conduct several waves of computer matching for its unduplication process. In 2000, the Bureau had difficulty with duplicate enumerations and did not have a planned operation for dealing with the problem; officials explained that in 2000 there was an "ad hoc" plan that they developed "on the fly" for identifying duplicates. However, Bureau officials noted that one of the lessons learned from 2000 was that the Bureau needed to expand the search for duplicates by computer matching. Thus, to improve the 2010 Census, along with the overcount probe specifically designed to try and find duplicates, the Bureau will also match census returns against the universe of all other census returns to identify duplicates nationwide; for the dress rehearsal, the matching will be conducted within the test site, rather than on a national scale. While the Bureau has been evaluating and testing this procedure, the limited likelihood of duplication of individuals that will occur within census test sites of limited geographic size, as compared to the actual national decennial, limits the Bureau's ability to forecast how the nationwide matching and unduplication system will affect workload for coverage follow- up.[Footnote 15] Although the Bureau has not finalized its coverage follow-up techniques, it plans to apply a computer-based record linkage system to the data collection files during the 2010 Census to match files against themselves to find potential duplicates based on housing unit and person information. Those households would then be selected for follow- up. For example, erroneous enumerations are created when duplicated people are counted correctly in one housing unit and incorrectly in another housing unit. Duplicated housing units, which occur when multiple addresses of the same housing unit are identified during census operations, are removed. Further, in contrast to Census 2000, the addresses that could be included in the census in both the housing unit and group quarter enumerations can no longer cause duplication because, according to the Department of Commerce, the Bureau now has one list for both housing units and group quarters. Consequently, according to Bureau officials, the Bureau faces the challenge of identifying the types of duplicates that can be most easily identified and addressed, which will result in the most cost-effective yield. The Bureau's 2006 test of computer matching found that potential duplicates generally were not deleted. Specifically, in its evaluation of computer matching during the 2006 Census Test, the Bureau found that 93 percent of respondents marked as possible duplicates of persons counted outside the census block were neither deleted nor linked to a deleted match. The Bureau considered this result disappointing. It noted that the coverage follow-up interviews did not resolve the cases very well because follow-up interviews were not completed for 24 percent of the cases and, in 61 percent of the cases, respondents did not confirm during the follow-up interview that they lived or stayed somewhere else. The interviewer cannot provide information about the suspected duplicate during the follow-up interview because of concerns related to confidentiality; the interview is limited in its potential for success in that the respondent has to volunteer information about the suspected duplicate case. Consequently, the Bureau is not only challenged in identifying suspected duplicate cases, but in actually resolving the cases as well. Bureau officials explain that for the unduplication process, the Bureau will assign a match score and identify links among suspected duplicates. The Bureau plans to develop a match score threshold that will vary by level of geographic distance between matches. The Bureau will then assess the quality of the match based on the match score and conduct further research on the cases exceeding this threshold. For example, once the Bureau identifies potential duplicates, it determines whether the potential duplication is a potential housing-level problem (duplicated persons identified within a census block) or person-level problem (more long distance-identified duplicates nationally). If it is a housing-level problem, then the Bureau would consider using personal visits by field staff from another of its operations, the field verification operation, to verify housing unit additions. Further, according to the Department of Commerce, field verification may be undertaken to determine if housing units should be deleted from the list. If it is a person-level problem, the Bureau would consider a coverage follow-up operation telephone interview to determine if the person is counted twice. Bureau Will Rely on Assessment of Dress Rehearsal to Complete Coverage Follow-up Plans for the 2010 Census: The Bureau is developing a number of techniques for selecting cases for follow-up and plans to expand the coverage follow-up operation overall. However, it has not determined which techniques it will use, what thresholds it will use in selecting cases for follow-up, or the resources it expects to devote to the coverage follow-up operation. The most pressing question remaining is how the Bureau will prioritize its expanded universe of follow-up cases identified through coverage probes, administrative record matching, and the unduplication process. Currently, households that display count discrepancies, are large, reply in the affirmative to coverage probes, are flagged through comparisons with administrative records, or are flagged as potential duplicates are all eligible for follow-up. Although there may be a large number of eligible cases for coverage follow-up, the Bureau is unsure how many cases it can financially afford for follow-up, and discussions are still under way within the Bureau about what units will be sent to the coverage follow-up operation. The Bureau is reviewing data from tests--such as the add and delete rates within each type of case--to aid in determining what types of techniques will be most productive in terms of identifying overcounted and missed persons. The Bureau plans to rely on additional data from its assessment of the coverage follow-up operation in the 2008 Dress Rehearsal in making decisions about which techniques, including coverage probes, administrative record matching, and the unduplication process, it plans to use for identifying and selecting cases for coverage follow-up in 2010. Dress rehearsal plans for coverage follow-up should mirror the plans intended for the decennial because the 2008 Dress Rehearsal will be the Bureau's last opportunity to assess the various procedures and systems for 2010 under as close to census-like conditions as possible. However, the expanded coverage follow-up operation conducted during the dress rehearsal will not necessarily be what the Bureau implements in 2010 because the Bureau plans to use the dress rehearsal to assess key aspects of a likely coverage follow-up operation, including (1) the potential coverage follow-up workload and how it may be distributed, (2) the extent to which the coverage follow-up operation improves census coverage through such factors as the percentage of added residents and deleted residents, and (3) problems administering the follow-up interview and managing work. The assessment will also address problems with the operational work flow and interfaces and provide recommendations to fix or improve these problems. While the Bureau's expectation is that it will have the necessary data from the dress rehearsal to inform the final plans for the 2010 coverage follow-up operation, Bureau officials have noted that the delayed dress rehearsal (a delay in the Census Day from April 1 to May 1, 2008) may make it more difficult to incorporate the findings from the dress rehearsal into the Bureau's plans for the 2010 Census. Bureau officials have stated in several forums that resource constraints are a factor in the final decision on the scope of follow- up in 2010. The expansion of selection criteria has the potential to dramatically increase the workload for coverage follow-up in 2010 and, as a result, increase costs. Bureau officials have cautioned that it is not necessarily cost-effective for the Bureau to follow up on the entire universe eligible to be included in the coverage follow-up operation. Thus, the Bureau will be using results--such as the percentage of cases resulting in a change relative to the cost or yield--from the dress rehearsal to decide which techniques to use and how many cases it can afford to include for follow-up. Evaluations of the impact of the criteria the Bureau will use in judging what techniques to use for identifying cases for follow-up or the thresholds for selecting specific cases for follow-up will be critical to determining the final plan for the coverage follow-up operation. The Bureau has made one judgment about how it will proceed for 2010 as a result of its 2006 test. Specifically, in its evaluation of the 2006 Census Test, the Bureau calculated that the cost was $155 per roster change for coverage follow-up cases done by telephone versus $766 per roster change for cases done by personal visits. Consequently, the Bureau concluded that it should conduct follow-up interviews by telephone. This decision, based partly on the percentage of cases with a change and partly on cost, represents two criteria. Effective program management, including assessment of risks,[Footnote 16] requires that agencies identify information that is critical to achieving important program objectives.[Footnote 17] In this case, the Bureau should clearly state the basis for deciding which techniques to use or the number of cases it will include for coverage follow-up. Because the Bureau has not specified these criteria, it is not clear what criteria--other than roster change and cost--the Bureau will consider. For example, the Bureau noted in its decision to conduct coverage follow-up by telephone that not adding a personal visit component to the operation would limit its ability to reduce coverage errors for households it could not reach by telephone. Bureau research indicated that households for which it did not have telephone numbers tended to be very wealthy or very poor. Thus, coverage may not be improved for these demographic groups through telephone follow-ups alone. Bureau Has Made Changes to Coverage Measurement for 2010 Intended to Benefit the 2020 Census, but Key Aspects of Program Plan Remain Unclear: The Bureau's overall design of its census coverage measurement program for the 2010 Census is similar in many respects to its measurement design for 2000 but differs in some important ways. The objectives, timing, and some techniques to be used for 2010 differ from those used in 2000; however, the Bureau has not made clear when it will produce estimates of net coverage error and components of coverage error, nor does it make clear how it will meet one of its objectives for coverage measurement in 2010--improving future census operations. In 2000, the Bureau focused on estimating net coverage error with the possibility of producing adjusted numbers at the census block level. In contrast, for 2010, while still intending to produce net coverage error estimates at higher levels of geography--the national and state levels--the Bureau has refocused its efforts in order to use its measurement program to help improve enumeration methodologies for the future. Importantly, the Bureau does not plan for the measurement program to prepare adjusted numbers for the 2010 Census. In addition to starting coverage measurement interviews 4 months later than it did in 2000, the Bureau has also made several other important changes to its measurement program. For example, to better estimate net coverage error, the Bureau plans to use logistic regression for the first time. The Bureau also plans to assess the association of components of coverage error to various census operations. Finally, to better measure coverage, the Bureau plans to better identify duplicates by expanding its computer and clerical matching. Expert statisticians and methodologists we interviewed generally agreed that the Bureau's changes for its 2010 measurement program are an improvement from the 2000 coverage measurement program; however, several experts noted that more detailed planning is needed. For the 2010 coverage measurement program, the Bureau expects to include multiple operations. First, the Bureau selects its sample of census block clusters to be included in the coverage measurement program. Using the sample, the Bureau collects information on the housing units in the census blocks to develop an address list independent from that used in the census enumeration. The Bureau then uses this address list to draw a sample of housing units to include in the person interview operation. During this operation, Bureau field staff interview--in person--residents of these housing units and collect demographic, Interview Day and Census Day residency information of all persons living at the housing unit. According to the Department of Commerce, additional information will be collected in 2010 to identify alternate addresses where persons on the coverage sample roster may have been counted on Census Day and this information will be used in person-based computer and clerical matching operations. Next, the Bureau compares data collected during the person interviews with all census enumerations. Where the Bureau is unable to make a match, the Bureau follows up by sending interviewers back to the housing unit to reinterview the person or persons in question. During the final operation, the Bureau produces estimates of coverage error for persons and housing units. For 2010, the Bureau plans to provide estimates of net coverage error at the national level by race/ethnicity, age/sex, and tenure--the status of an occupied housing unit as either owner occupied or renter occupied. The Bureau will also produce estimates of net coverage error for states. 2010 Sample Design for Census Coverage Measurement Is Similar to That for 2000: The Bureau's 2010 coverage measurement program sample design is similar to that used for measurement in 2000. As in 2000, the Bureau plans to use a 300,000 national housing unit sample. The methods are basically consistent with the 2000 coverage measurement program and are expected to yield similar levels of reliability, although slightly different methods are being used to determine state sampling allocation. The objective for 2010 is to have the sample allocated proportionally across states and the District of Columbia, with a minimum of 1,800 housing units in each state. Moreover, the Bureau is using a differential within state sample allocation in order to allow for oversampling in areas as needed. Notably, according to Bureau officials, it is possible that that the sampling numbers could still change depending on budget or operational constraints. Most experts we spoke with thought that the Bureau's sample design was appropriate for its objectives. Census Coverage Measurement Interviews Will Begin Later, and Estimates Will Take Longer to Complete in 2010: In 2000, the timeline for coverage measurement overlapped with the Bureau's coverage follow-up efforts, though it was conducted independently from the census enumeration; however, measurement and coverage follow-up are planned to be conducted sequentially for the 2010 Census. Specifically, in 2010, the census coverage measurement interview will begin after the coverage follow-up operation is completed (see fig. 4). According to the Bureau's 2010 Census Operations and Systems Plan, dated August 31, 2007, the Bureau expects to conduct the person interviews for coverage measurement from August 20 through October 16, 2010. In 2000, the Bureau conducted the coverage measurement person interviews from April 24 through September 11, 2000. Figure 4: Timeline for Coverage Follow-up Interviews and Census Coverage Measurement Interviews Using Actual Dates for 2000 and Estimated Dates for 2010: This figure is a timeline for coverage follow-up interviews and census coverage measurement interviews using actual dates for 2000 and estimated dates for 2010. [See PDF for image] Source: GAO analysis of U.S. Census Bureau data. [End of figure] According to the Bureau, the expansion of the coverage follow-up operation contributed to delaying the measurement program in order to maintain data independence and reduce the likelihood of data contamination resulting from the overlap of follow-up and measurement field operations.[Footnote 18] In past coverage measurement programs, the person interview phase overlapped with coverage follow-up conducted as part of the census enumeration. In 2010, the Bureau plans to delay the start of the person interview phase for coverage measurement to minimize the risk of data contamination. While many experts agreed that delaying the coverage measurement interviews will reduce the risk of data contamination, some experts also highlighted that the delay may also negatively affect the data quality because it could contribute to recall bias. Recall bias, as it pertains to coverage measurement, means that a person incorrectly recalls where he or she or other household members resided on Census Day and thus biases the coverage measurement data. Two experts also noted that delaying coverage measurement interviews could also affect quality of data on movers. Generally, experts said that decreasing the amount of time between the coverage follow-up interview and the coverage measurement interview would benefit the coverage measurement data. However, a few experts who were also familiar with the issue stated that they were unsure of the magnitude of contamination and, therefore, did not know if the Bureau should decrease the time. Several of these experts argued that the Bureau could make a better case by using results from its contamination studies to determine if the delay is warranted. In 2006, the Bureau conducted a contamination study using the results of the 2006 Census Test to determine if the coverage measurement person interviews caused a difference in responses to the coverage follow-up interviews.[Footnote 19] Although the Bureau stated that the number of sample cases involved in the study was small and that the result should be viewed with caution, it concluded that there might be some small impact on coverage follow-up results if the measurement interview is conducted at the same time as the coverage follow-up interview. For 2010, the Bureau also expects to complete its coverage error estimates later than was the case for the Bureau's initial 2000 estimates. In 2000, the Bureau produced initial estimates in March 2001; however, after evaluating the 2001 estimates, the Bureau released revised estimates in March 2003. According to Bureau officials, the Bureau has preliminary plans to complete coverage measurement estimation[Footnote 20] for the 2010 Census sometime after October 2011[Footnote 21]--6 months after the Bureau provides population counts for redistricting. Bureau officials have stated that the decision to delay measurement estimation will allow for more time to adequately evaluate the 2010 measurement data and estimates, noting that the Bureau did not have sufficient time for analysis in 2000. Objectives for the 2010 Census Coverage Measurement Differ from Those for 2000, but How the Bureau Will Meet New Objectives Is Unclear: The Bureau's 2010 coverage measurement objectives have changed from those of Census 2000. In 2000, the Bureau focused on estimating net coverage error with the possibility of adjusting the census count at the census block level of geography. Though the Bureau still intends to produce net error estimates for 2010--at the state level and higher--it has refocused the coverage measurement program on improving future enumeration methodologies; consequently, it does not plan to use the results of its coverage measurement program to improve or adjust the count of the nation's population in 2010. For 2010, the Bureau's goals for measuring coverage accuracy include the following aspects: * The Bureau plans to use a dual systems approach to estimate net coverage error by using logistic regression instead of the poststratification technique used in 2000. The Bureau believes its move to logistic regression will also allow more flexibility because it can incorporate a greater number and type of factors or predictors in the model. * The Bureau plans to improve future census operations by associating estimates of components of coverage error, including erroneous enumerations and omissions, to census operations. * To better resolve and identify duplicate enumerations, a major source of error in previous decennials and coverage measurement programs, the Bureau plans to expand computer and clerical matching to match persons in its postenumeration survey to the entire census from the more limited matching done in 2000. The Bureau plans to conduct a census coverage measurement program for the 2010 Census, but Bureau officials said that they did not know when the Bureau would produce detailed documentation describing its plans. The Bureau has developed preliminary plans for associating components of coverage error to census operations, but these plans also are not finalized. Bureau Does Not Plan to Adjust the 2010 Census in Time for Redistricting: According to the Bureau, it has no plans to use the results of its coverage measurement program to improve or adjust the 2010 Census count. In response to a May 12, 2003 letter from Representative William Lacy Clay asking about decennial plans, the Bureau explained that with a sufficiently correct count, adjustment would introduce as much or more error than it was designed to correct. If the Bureau were asked by the Congress to provide adjusted numbers, Bureau officials commented that it would be unable to quickly or easily do so because the current coverage measurement program's timeline produces estimates after October 2011, and the Bureau has not designed the software to produce estimates at lower geographic levels. Consequently, according to the Bureau, it cannot prepare adjusted numbers in time to meet the legal deadline for producing count data for redistricting, which is April 1, 2011. Furthermore, the Bureau is not planning to provide net error estimates below the state level; such estimates are necessary for redistricting. Some of the interviewed experts agreed that producing adjusted numbers would be challenging; however, several also said that the Bureau could still provide some adjusted numbers if asked to do so. Regarding the challenges, two experts stated that adjusted numbers could be produced at a higher geographic level, the national level, but not accurately at lower geographic levels, the state and local levels, without appropriate changes to the current design and methodology. For example, one expert explained that the Bureau would need to use poststratification to produce estimates at the state and local levels. Additionally, several experts noted that the 2010 coverage measurement timeline poses a challenge for producing adjusted numbers by the April 1, 2011, redistricting deadline. Bureau Plans to Use a Different Statistical Method for Estimating Net Error but Has Not Fully Specified Its Plans: The Bureau plans to use logistic regression to improve its estimation of net coverage error. Of the experts familiar with logistic regression, the majority thinks that the methodology is an improvement; however, a few experts question how the Bureau will explain a more complex method to its stakeholders. Previously, for the 2000 coverage measurement, the Bureau used a poststratification[Footnote 22] method. However, Bureau officials said that poststratification was limited because of the small sample sizes of many of the strata. They explained that logistic regression permits greater flexibility in dealing with small sample sizes because it allows for a greater number and type of factors or predictors in the model.[Footnote 23] By definition, poststratification is limited because it only allows for a small number of factors with defined categories to be included. In addition to greater flexibility, logistic regression can potentially reduce errors in estimation. According to Bureau officials, to verify estimates from logistic regression, the Bureau is also considering using some poststratification. Logistic regression provides greater flexibility in dealing with small samples, but it may be more difficult methodologically to use logistic regression to produce state-level estimates by race/ethnicity, age/sex, and tenure or to estimate lower than the state level. Some experts argued for continuing to produce estimates using poststratification to enable direct comparisons of error estimates to prior decennials and to verify that logistic regression is working. To enhance transparency and oversight for improving coverage measurement for the 2010 Census, the Bureau should provide the Congress with timely and accurate information to allow monitoring of agency activities--specifically, timely and accurate information about its plans for coverage measurement.[Footnote 24] The Bureau plans to refine its estimation methods for 2010 by using logistic regression to estimate net coverage error, but it does not have an operational or working plan that outlines what resources--such as the amount of time that it will take to run the data set(s)--will be used to complete this operation. Further, despite conducting some exploratory logistic regression work using the 2006 test data, as well as simulating its estimation model for logistic regression using decennial data from 2000 and finding workable models, the Bureau has not documented what approaches or processes it intends to apply in the further development of the regression model or models it plans to use. In addition, the Bureau does not have a written plan detailing how it will explain to stakeholders and Bureau data users the rationale for moving from poststratification to logistic regression and what effect this change may have on data interpretation--particularly as it relates to comparisons with past census data. Experts we interviewed also noted some challenges, including communicating the new method to users of census data and stakeholders, having an adequate number of staff to complete the analysis, and designing the best models--such as choosing the variables to include. Bureau's New Focus on Estimating Components of Coverage Error Is Intended to Benefit Future Decennials, but Plans Are Incomplete: The Bureau plans to estimate the components of coverage error for 2010 in order to associate the errors to census operations to improve future decennials; in 2000, the coverage measurement program focused on net coverage error, not components of coverage error. Although the Bureau will once again estimate net coverage error, Bureau documentation shows that estimating the components of coverage error is now a higher priority for the Bureau. Coverage error comprises two types of errors: errors of omission and errors from a person or housing unit mistakenly enumerated (erroneous enumerations). The impact of omissions and erroneous enumerations are combined to determine the net error. During Census 2000, the Bureau estimated the net coverage error of the census. However, net error can disguise significant count problems--the Bureau could have a large overcount in some areas and large undercount elsewhere, and it might balance out to a small net error nationwide. In addition, the coverage measurement definitions of erroneous enumerations used in 2000 were not as precise as those that will be used in 2010. For example, the 2000 coverage measurement program did not require Bureau officials to ascertain where a person should have been counted. Instead, some enumerations were treated as omissions or erroneous enumerations in an area where they should have been counted. According to the Department of Commerce, because of the Bureau's new goal to measure the components of coverage error, the Bureau will use a definition for component error estimation that is not as strict as that used for net error. For 2010, erroneous enumerations for the coverage measurement program are defined to include the following: duplicate enumerations, persons born after or who die before Census Day, or persons who are not residents of a housing unit in the United States on Census Day. Though the Bureau recognizes the potential benefits of estimating the components of coverage error to improve future decennials, it has not fully developed its plans for doing so in 2010. Providing the Congress and other key stakeholders with information about how the Bureau plans to use the 2010 coverage measurement program to improve future decennials would enhance transparency and oversight of coverage measurement.[Footnote 25] For Census 2000, the Bureau did not associate the estimates of erroneous enumerations and omissions to specific census operations. For the 2010 Census, the Bureau has not finalized plans to evaluate the quality of its estimates of components of coverage error (i.e., estimates of variance) and has only recently determined the level of geography for which estimates will be made. Moreover, although the Bureau has developed table shells that associate components of coverage error to census operations, the Bureau has not clearly explained how it plans to implement operations and data collection strategies that will support the analyses and lead to the completion of the tables and convert those analyses into plans to improve the census operation for 2020. Importantly, Bureau officials stated that it will be difficult to link estimates of the components of coverage error to specific census operations. Instead, they characterize the process of associating the components of error to census operations as an exploratory effort. A Bureau official explained that associating errors of omission to operations will be most challenging because omitting persons from the census count could be caused by errors in multiple operations. For example, a person could have been omitted because of a bad address or an incorrect enumeration. It may not be possible to determine which operation contributed to the omission. In contrast, she said that it may be more feasible to associate errors of erroneous enumerations to individual census operations because there is more that they know about the enumerations. In associating errors to census operations, a Bureau official stated that the Bureau may use a modeling approach to identify those factors with the highest probability and greatest significance in predicting specific erroneous enumeration errors. Additionally, Bureau officials noted that how the Bureau associates components of coverage error to census operations cannot be determined without first determining the final models for estimating the components of coverage error for the 2010 Census. These models cannot be specified without the actual 2010 Census data and therefore will be specified after data collection. Finally, as recommended by the National Academy of Sciences, Bureau officials acknowledged that there is ongoing experimental research involving the use of a master trace sample database in associating the components of coverage error to census operations. This database is intended to help the Bureau trace response and operational data through various stages of the 2010 Census--such as address listing and data collection. The National Academy of Sciences notes that such a database should allow the Bureau to retain a full history of relevant census operations for a designated sample of households in census blocks. The Bureau should then be able to use the database to investigate the source of a variety of potential deficiencies that can arise in a complicated undertaking like the decennial census. More specifically, the database could be used to determine what percentage of census omissions are from partially enumerated households and what percentage of omissions are from the merged administrative records database. According to a Bureau official, the Bureau plans to begin developing the database in 2008. However, the official noted several limitations with such a database. For example, the database consists of a sample of the U.S. population, as opposed to the whole census population. Still, Bureau officials noted that improving the master trace sample database in 2010 is important, citing the example of using the database for complex cluster analyses that link to coverage measurement. While most experts we contacted agreed that estimating the components of coverage error will likely help to improve future decennials, some experts questioned the Bureau's ability to link the actual errors to specific census operations. For example, one expert said that it "will be practically impossible to project errors to operations." Additionally, some experts discussed other challenges that the Bureau may face in accurately estimating the components of coverage error, including distinguishing between different types of erroneous enumerations and omissions as well as resolving duplicates and accurately matching the data. Still, some experts did say that it may be possible for the Bureau to link certain erroneous enumerations to actual operations. Because of the Bureau's new objective to estimate components of coverage error and relate them to census operations, it is important for the Bureau to solidify its plans and share them with stakeholders to ensure that there is an understanding on what the program will produce in order to improve future decennials. Bureau Plans Expanded Matching for Resolving and Identifying Duplicate Enumerations: Similar to the matching planned to expand the coverage follow-up operation, the Bureau's plans for its 2010 coverage measurement program include expanding the matching process in order to better resolve and identify duplicate enumerations--a major source of error in the 2000 Census and coverage measurement program. Some of the experts we contacted are confident that the plans will be an improvement. Shortly after the census enumeration is completed, a postenumeration survey will be conducted in a sample of block clusters. This re-enumeration sample--known as the P-sample--is used with the E-sample (the actual census enumeration in those sample areas) to derive a corrected population estimate.[Footnote 26] For 2010, the Bureau will also use computer matching to compare the P-sample with all census enumerations, with the objective of identifying and resolving duplicate enumerations. This is an expansion from 2000, when the P-sample was matched against enumerations from sample blocks and surrounding blocks only, although housing units are matched to census housing in the search area. As in 2000, the Bureau will also use clerical matching to do additional matching, but will also do nationwide matching.[Footnote 27] Despite the Bureau's effort to improve its matching process and minimize the number of duplicates in 2010, the Bureau does not yet have complete plans for the 2008 Dress Rehearsal or for the 2010 Census to help accomplish this goal. Bureau officials noted that materials are being developed for certain matching procedures, such as clerical matching. Further, Bureau officials said that budgetary constraints may also affect the matching operations and changes may occur before or following the 2008 Dress Rehearsal. For example, the Bureau may not conduct person matching during the dress rehearsal. Though a few experts cited the challenges the Bureau will continue to face in resolving duplicate enumerations, several also noted that the expanded matching process should help to reduce the number of duplicates considerably. An expert explained that by expanding the geography covered in the Bureau's matching operations, it will be easier to identify duplicates as well as households that may have been overlooked in previous coverage measurement matching operations because the search area was limited to respective census blocks and a few nearby surrounding geographic areas. Still, he also said that the expanded matching will be an improvement only if implemented well. Bureau Plans for Evaluating the 2010 Coverage Follow-up Operation and Census Coverage Measurement Program Are in Early Stages: The Bureau is in the early stages of developing plans for evaluating its 2010 Census coverage follow-up operation and census coverage measurement program. To assist in this effort, the Bureau has contracted with the National Academy of Sciences to recommend proposed research for the 2010 Census. The academy's final report is due in September 2009.[Footnote 28] In its December 2007 interim report, the academy recommended that the Bureau conduct certain experiments and evaluations, such as assessing the use of administrative records and coverage probes for identifying cases for follow-up.[Footnote 29] Bureau officials have also suggested research on various aspects of coverage follow-up and measurement for 2010. Officials said that the Bureau will consider costs and staffing needs in deciding what it will evaluate. However, the Bureau has not yet developed a timetable for making decisions about evaluating the 2010 coverage follow-up operation or census coverage measurement program. For the most recent decennial censuses, the Bureau has carried out experiments and evaluations. A census experiment usually involves field data collection during the census in which alternatives to current census processes are used and assessed for a subset of the population. An evaluation is usually an analysis conducted on data collected as part of the decennial process to determine whether individual steps in the census operated as expected. The Bureau plans to work with internal and external stakeholders on defining and developing its program for experiments and evaluations for the 2010 Census, with design work and implementation starting in 2008 and continuing through 2011. The final set of activities would include analysis, documentation, and presentation of the research, and these activities would start in 2009 and be completed by the end of the fiscal year 2013. According to the National Academy of Sciences' December 2007 interim report, its purpose was to reduce the possible subjects for census research from a list of over 50 research topics. The academy reviewed the Bureau's initial list of research topics to help reduce the possible subjects for census experimentation, and offered broad advice in its interim report on plans for evaluations of the 2010 Census. For this smaller number of research topics, the academy recommended several experiments and evaluations relating to coverage improvement and measurement. For example, the academy recommended that the Bureau assess the use of administrative records in the event that the Bureau opts to review administrative records to identify potential omissions. It also recommended that the Bureau include one or more alternate questionnaire experiments, such as an examination of the usefulness of the residence rules and concepts on the census questionnaire. Further, the academy recommended a possible experiment on comparing telephone interviews to personal interviews for the coverage follow-up operation because a telephone-only operation would follow up for only those households that provide a contact telephone number. However, at this point, the Bureau has not yet made definitive decisions about what it will evaluate and when it will make these decisions. In its interim report, the National Academy of Sciences suggested that the Bureau design a master trace sample database--a tool for investigating the source of potential deficiencies that can arise in such a complicated undertaking as the decennial census--that would help assess the benefits of the coverage follow-up interview. This database could link person, household, and housing unit characteristics; census processes; and the presence or absence of census component coverage error. The academy suggested that the use of this database would better identify the sources of high rates of census component coverage error, including erroneous enumerations, enumerations in the wrong place, duplications, and omissions. The academy also suggested that the Bureau use a reverse record check for coverage measurement. It noted that the reverse record check is an alternative method for estimating the completeness of census coverage of the population, which may have advantages over the methods of dual systems estimation and demographic analysis that have been used for this purpose to date. This procedure has not been used previously to evaluate census data, mainly because of the 10-year period between censuses, which complicates the need to trace people's addresses from one census to the next. However, the Bureau now uses the American Community Survey, a monthly survey of 250,000 housing units to collect and provide long-form data on an annual basis. A reverse check would facilitate comparing decennial enumerations to information collected more frequently during the American Community Survey. The National Academy of Sciences believes that it will be very important for the Bureau to finalize specific designs of its experiments by summer 2008 to meet the planning needs of the census. Because the data needed to support census evaluations are typically output files from the census itself, the exact structure of individual evaluations is not yet as time-sensitive as the experiments. However, the academy notes that some early planning for evaluations is critical so that the necessary data extracts can be prepared and retained, especially because much of the data collection in 2010 will be carried out by contractors, and thus data retention requirements need to be arranged with contractors as early as possible. It also suggests that the Bureau increase its in-house expertise in experimental design for the census. Some experts we interviewed about the Bureau's plans for coverage measurement also noted that the Bureau should plan its evaluations with more urgency. An expert noted that in 1990 and 2000, when the Bureau faced challenges with its adjustment process, there were fewer staff members available to work on evaluations; he said that perhaps this would not be a problem for the 2010 Census because the Bureau did not plan to adjust the census count. Further, a few experts suggested that the Bureau should ask outside researchers, in addition to the National Academy of Sciences, for assistance and feedback in its evaluation planning and development. A few experts also suggested that the Bureau should consider developing more qualitative evaluations, such as studies to look deeper into why certain populations, such as the homeless, are omitted from the census at greater rates. Conclusions: With only 2 years remaining before Census Day, the Bureau faces formidable challenges in successfully implementing and completing the 2010 Census. These challenges are even more formidable given the Bureau's recent decision to redesign certain aspects of the 2010 Census, and make more critical the need for the Bureau to have in place specific and concrete plans for ensuring and measuring the accuracy of the decennial census. To accurately count everyone residing in the United States, the Bureau must overcome significant challenges due to the nation's increasing diversity, as well as, according to the Bureau, a population increasingly reluctant to participate in surveys. In response to these challenges, the Bureau plans to expand its coverage follow-up operation--a final quality assurance method intended to help accurately enumerate the population--by focusing on resolving count discrepancies, enumerating large households, and following up on households with potential enumeration problems identified through techniques the Bureau plans to further test during its dress rehearsal. As in prior decennials, the Bureau plans to measure the accuracy of the 2010 decennial through a coverage measurement program. The Bureau intends to focus its coverage measurement efforts in 2010 on learning more about the components of census error in order to improve operations during the next decennial--2020--and future decennials. The Bureau does not plan to use its 2010 coverage measurement program to adjust the count of the nation's population. The Bureau has initiated efforts to improve the accuracy of the census count by expanding coverage follow-up and improving measurement; however, some areas bear enhanced monitoring, oversight, and management as the Bureau moves from its 2008 Dress Rehearsal to final plans for the 2010 Census. Providing internal and external stakeholders a full description of its plans for coverage follow-up and measurement will help the Bureau ensure that the plans for and expected outcomes of these important operations are understood not only by the Bureau but also by stakeholders. Importantly, many of the operational decisions for the 2010 Census depend on the upcoming 2008 Dress Rehearsal. For example, the Bureau has not finalized its plans for 2010 regarding the scope of its coverage follow-up operation and will base final decisions on how to select cases for follow-up in part on its experiences in the dress rehearsal. Coverage follow-up improvements have been tested on a site-level scale but have not yet been tested with new processing interfaces between the Bureau and its contractor for collecting and disseminating information. The Bureau plans to expand coverage follow- up but has not clearly stated the basis it will use for deciding which techniques to use or the number of cases it will include for follow-up. The Bureau will rely on the dress rehearsal to assess the effectiveness of certain coverage follow-up techniques, such as the undercount probe. Existing analyses suggest that this probe is of limited use. Overall, given the uncertainty surrounding key aspects of the Bureau's plans for coverage follow-up for 2010, as well as the delayed dress rehearsal, the content and magnitude of the 2010 coverage follow-up bear continued scrutiny. Operational plans for coverage measurement lack specificity, and the 2010 Census will be the first time that the Bureau can implement its expanded computer matching for both coverage follow-up and measurement on a national level. Importantly, although an announced purpose for revising the census coverage measurement program was to provide improvements for the 2020 Census and the Bureau is working toward associating components of error to specific operations, it has neither clearly explained how it will do this nor the expected dates for when its plans for this analysis will be complete. All of these unknown aspects increase the level(s) of risk of not meeting these important goals. By having plans and timelines in place, the Bureau increases the likelihood of completing its coverage measurement analyses in time to inform the planning for the 2020 decennial. Because the Bureau plans significant changes to its coverage follow-up and measurement activities, it should be planning well in advance of the decennial how it will evaluate and measure their success. Timely and informed oversight, monitoring, and management can promote the success of these evaluative activities. The Bureau's program for evaluations and experiments has great potential to help improve the next decennial. The National Academy of Sciences has made interim recommendations on various ways that the Bureau could assess its decennial data collection activities. Among other things, the academy recommends using a master trace sample because such a sample would improve the Bureau's efforts to relate census errors to census operations. We believe that such a master trace sample has the potential to help the Bureau better understand sources of error. Nonetheless, designing such a sample is a daunting task. The Bureau has begun to evaluate how to implement the academy's recommendation. The academy's interim report also suggested that the Bureau soon finalize specific designs of the experiments to meet its planning needs so that necessary data retention requirements could be arranged with contractors as early as possible. Recommendations for Executive Action: We are recommending that the Secretary of Commerce direct the Bureau to provide the Congress and other census stakeholders with more specific plans on conducting coverage follow-up, implementing coverage measurement, and evaluating the results of those programs for the 2010 Census. Specifically, we recommend that the Bureau provide: * the criteria it will use to assess the techniques for identifying cases for the coverage follow-up operation in the 2008 Dress Rehearsal and the timeline for completing this assessment to permit planning for 2010--the timeline should specify dates for preliminary assessments, final assessments, and when operational changes might be made to the 2010 Census; * its plans for conducting coverage measurement for the 2010 Census, which should include a description of when it will provide estimates of net coverage error and components of coverage error and how it plans to relate components of coverage error to census operations in order to improve future decennials; and: * key decision points and plans for evaluating aspects of the 2010 Census coverage follow-up operation and census coverage measurement program. Agency Comments and Our Evaluation: We provided a draft of this report to the Secretary of Commerce for his review and comment. Commerce had no substantive disagreements with our recommendations and stated that it will develop formal action plans for each recommendation. While Commerce believes a timeline for producing estimates of net coverage error and components of coverage error is certainly a reasonable expectation, Commerce noted that it is unclear as to how it can at this time provide any specifics on how coverage results might improve future censuses before the coverage data are obtained and analyzed. We recognize that the Bureau has developed table shells that associate components of coverage error to census operations, but these table shells and plans for how the Bureau will conduct its analyses have not been shared with the Congress and other stakeholders. In addition, it is not clear from current Bureau plans how it will go about obtaining the data required to complete the tables. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days after the report date. We will then send copies of this report to the Secretary of Commerce, the Department of Commerce's Inspector General, the Director of the U.S. Census Bureau, and interested congressional committees. We will make copies available to others upon request. This report will also be available at no charge on our Web site at [hyperlink, http://www.gao.gov]. If you or your staff have any questions concerning this report, please contact me at (202) 512-6806 or sciremj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. Sincerely yours, Mathew J. Scirè: Director, Strategic Issues: [End of section] Appendix I: Objectives, Scope, and Methodology: To determine (1) how the U.S. Census Bureau (Bureau) plans to improve coverage through its coverage follow-up operation, (2) how the Bureau's census coverage measurement plans for assessing the accuracy of the 2010 Census compare with efforts in 2000, and (3) how the Bureau plans to evaluate the coverage follow-up and measurement efforts after the 2010 Census, we requested and obtained source documents from the Bureau's headquarters in Suitland, Maryland. We also interviewed Bureau officials in several of its divisions, including the Decennial Management Division and the Decennial Statistical Studies Division. Further, we reviewed evaluations regarding coverage follow-up and coverage measurement. In addition, for the second objective, related to the Bureau's census coverage measurement program, we interviewed 15 experts about the Bureau's coverage measurement methodology and plans. Prior to our interviews, we summarized the Bureau's plans for coverage measurement and met with Bureau officials to obtain confirmation of our summary. We then identified experts based on published reports as well as recommendations from the Bureau and the National Academy of Sciences and contacted them about reviewing our summary of the Bureau's measurement plans and interviewing them about the Bureau's plans, including the extent to which the Bureau's plans addressed concerns and limitations of previous coverage evaluation programs, the extent to which using logistic regression instead of poststratification would improve the estimation of net coverage error, the main challenges in estimating components of coverage error, and the implications of delaying the coverage measurement interviews. Fifteen experts responded affirmatively to our request for interviews and we subsequently interviewed the following persons: Dr. Barbara Bailar: Statistical Consultant: Washington, D.C.: Dr. Thomas Belin: Professor of Biostatistics, Psychiatry, and Biobehavioral Sciences: Department of Biostatistics: School of Public Health: University of California, Los Angeles: Los Angeles, California: Dr. Michael Brick: Senior Statistician: Vice President, Director of Survey Methods: Associate Director of the Statistical Staff at Westat: Rockville, Maryland: Dr. James Brown: Senior Lecturer in Official Statistics: School of the Social Sciences: University of Southampton Southampton, United Kingdom: Mr. Dave Dolson: Director: Social Survey Methods Division: Statistics Canada: Ottawa, Canada: Dr. Stephen Fienberg: Maurice Falk: University Professor of Statistics and Social Science: Department of Statistics: Carnegie Mellon University: Pittsburgh, Pennsylvania: Dr. Tim Johnson: Director: Survey Research Laboratory: University of Illinois at Chicago: Chicago, Illinois: Mr. Don Royce: Director General: Methodology Branch: Statistics Canada: Ottawa, Canada: Dr. Duane Steffey: Director: Statistical and Data Sciences: Exponet, Inc.: Menlo Park, California: Dr. John Thompson: Executive Vice President of Survey Operations: National Opinion Research Center at the University of Chicago: Chicago, Illinois: Dr. Robert Tortora: Chief Methodologist: Gallup Washington: Washington, D.C.: Dr. Roger Tourangeau: Director: Joint Program in Survey Methodology: University of Maryland: College Park, Maryland: and: Acting Director: Program in Survey Methodology: University of Michigan: Ann Arbor, Michigan: Dr. Kenneth Wachter: Professor of Demography and Statistics: University of California at Berkeley: Berkeley, California: Dr. Martin Wells: Director of Research: Charles A. Alexander: Professor of Statistical Sciences: Cornell University: Ithaca, New York: Dr. Kirk Wolter: Head of the Center for Excellence in Survey Research: Professor in the Department of Statistics: Senior Fellow at the National Opinion Research Center: University of Chicago Chicago, Illinois: We conducted this performance audit from January 2007 to March 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Comments from the Department of Commerce: Note: GAO comments supplementing those in the report text appear at the end of this appendix. The Secretary Of Commerce: Washington, D.C. 20230: April 3, 2008: Mr. Mathew J. Scire: Director: Strategic Issues: United States Government Accountability Office: Washington, DC 20548 Dear Mr. Scire: The U.S. Department of Commerce appreciates the opportunity to comment on the United States Government Accountability Office's Draft Report Entitled Census 2010: Bureau Needs to Specify How It Will Assess Coverage Follow-up Techniques and When It Will Produce Coverage Measurement Results, (GAO-08-414). I enclose the Department's comments on this report. Sincerely, Signed by: Carlos M. Gutierrez: Enclosure: U.S. Department of Commerce Comments on the United States Government Accountability Office Draft Report Entitled Census 2010: Bureau Needs to Specify How It Will Assess Coverage Follow-up Techniques and When It Will Produce Coverage Measurement Results, (GAO-08-414) March 2008: The U.S. Department of Commerce appreciates the United States Government Accountability Office's (GAO) efforts to review our coverage improvement and coverage measurement processes planned for the 2010 Census and this opportunity to review the draft report. We have no substantive disagreements with these recommendations. Once the report is issued in final form, we will develop formal action plans for each recommendation. While we believe a timeline for producing the estimates is certainly a reasonable expectation (Recommendation 2), we are unclear as to how we could at this time provide any specifics on how our coverage results might improve future censuses before the coverage data are obtained and analyzed by Census Bureau and other experts. Specific Comments on the Draft Report: Page - Highlights: Item 4 in the second paragraph reads (4) expand computer matching to match housing units and persons in its post enumeration survey to the entire census. (See comment 1.): Comment: The item should be revised to read (4) expand computer and clerical matching to match persons in its post enumeration survey to the entire census. The nationwide matching is not done for housing units. Page 4: The error above regarding expanding computer matching also appears on page 4 of your report and should be revised. (See comment 2.): Page 4: For clarification, there is important information that needs to be included about the coverage measurement interviewing starting four months later than in 2000. The telephone phase was dropped due to the increased complexity of the person interview instrument, and loss of time in the conversion from a telephone phase to personal-visit implementation. Part of the delay in starting the 2010 Person Interview operation is due to eliminating the telephone phase. (For Census 2000, the person field visit started on 6/1/00, while the telephone phase started on 4/24/00). Although the interviewing will start four months later compared to the Census 2000 phone phase, person interviewing ends only one and a half months later than in 2000 (9/1/2000 vs. 10/16/2010). This should reduce the concerns related to recall bias. (See comment 3.): Page 4, footnote 2: The footnote reads, The dual systems approach . estimates of the remaining error in the count after omissions and erroneous enumerations offset one another. Comment: The second sentence may give a wrong impression that a net error estimate requires estimates of omissions and erroneous enumerations. To be more accurate, we usually say "reflects" rather than "estimates." We would suggest correcting the sentence to read, The net error reflects the error remaining after omissions and erroneous enumeration offset one another. (See comment 4.): Page 4, footnote 3: The footnote reads, "Before computer matching is completed, the Bureau conducts . to estimate errors in the census. such as nonexistent persons or duplicates." (See comment 5. Now footnote 4, p4..): Comment: We offer several clarifications to this footnote. A post enumeration survey consists of two samples: the P and E samples. P- sample persons are matched not only to the E sample, but also to the entire census. E-sample persons are matched to the entire census to identify and resolve duplicates. E-sample housing units are matched to census enumerations in the block cluster and surrounding ring of blocks. Page 16: For clarification, as written, the third paragraph implies that when the nonresponse follow-up (NRFU) undercount probe tells us there may be an additional person who was not included on the NRFU roster, the coverage follow-up (CFU) interview will clarify the residency situation so that a person who should be included (but was not on the NRFU roster) will be added in CFU. The report describes this as "included in the CFU roster after NRFU," which is not accurate. We do not add these persons and collect these data during NRFU. If CFU roster clarifications indicate someone should be added, this happens during CFU. (See comment 6. Now footnotes 11, p. 16..): Page 18: For clarification of the first paragraph, the limitation with identifying duplicates in site tests is not so much the limited sample size of the test sites, but rather the limited likelihood of duplication of individuals that will occur within a site of limited geographic size. (See comment 7.): Page 18: We suggest two clarifications to the second paragraph: (1) duplicated housing units occur when multiple addresses or representations of an address for the same housing unit appear on the address list. The ones that are "identified during census operations" are removed from the list for subsequent operations and (2) the next sentence about units being included on both the housing unit list and group quarters list is no longer accurate, which represents a big change from Census 2000. We now have one list for both, with units identified as either housing units or group quarters. Therefore, in general, we can no longer have the situation described in the document that "addresses can be included in the census in both the housing unit and group quarters enumerations." (See comment 8. Now on p. 19..): Page 19: We suggest two clarifications to the second paragraph: (I) the description of the unduplication process is a bit reversed. During the automated matching process, a match score is assigned to all links based on the level of match. The Census Bureau assesses the quality of the match based on the match score and (2) the Field Verification operation verifies housing unit status of the units selected for follow- up in the operation. Units that are invalid are marked as deletes or duplicates, and valid units are marked as verified. There are no additions during the Field Verification. In some cases, the Field Verification operation verifies the existence of units added in other operations, in particular, the Be Counted program. For the housing unit unduplication cases, the units are already on the list, and Field Verification is undertaken in order to determine if housing units should be deleted from the list. Page 20: For clarification of the second paragraph, the Census Bureau plans to use all our CFU test results to make final decisions about what to implement in 2010. not just the 2008 results. In particular, we have results from the 2005 National Content Test and 2006 Census Test. The 2008 implementation was intended to be an effort to collect additional data. (See comment 10.): Page 22: Your statement reads, "Finally, to better measure coverage, the Bureau plans to better identify duplicates by expanding its computer matching." (See comment 11.): Comment: While true, we also will expand clerical matching. The following revision is suggested: "Finally, to better measure coverage, the Bureau plans to better identify duplicates by expanding its computer and clerical matching." Page 23: The first sentence reads, "During this operation, Bureau field staff interview— in person—residents of these housing units and collect demographic and Census Day residency information of all persons living at the housing unit." Comment: The sentence should be revised to read as follows: "During this operation, Census Bureau field staff interview—in person—residents of these housing units and collect demographics, Interview Day, and Census Day residency information of all persons living at the housing unit. Additional information will be collected in 2010 to identify alternate addresses where the coverage sample roster people may have been counted on Census Day. This information will be used in person- based computer and clerical matching operations." (See comment 12.): Page 23: The fourth sentence in the first paragraph reads, "Additionally, the Bureau conducts computer matching to further resolve cases." (See comment 13.): Comment: This statement is inaccurate. A computer match is not done after we conduct the follow-up. Page 23: The last sentence in the first paragraph reads, "However, unlike for Census 2000, the Bureau will not produce state-level estimates by race/ethnicity, age/sex, and tenure or estimates lower than the state level." (See comment 14.): Comment: This statement is incorrect. The Census Bureau did not produce any state- level estimates broken down by any characteristics in Census 2000. Page 26: The third bullet also reflects the same error as occurred on the Highlights page and page 4. Refer to the comment made on the Highlights page above. (See comment 15.): Page 28, first full paragraph: For clarification, the Census Bureau has not "redefined what constitutes an erroneous enumeration" because of any shortcomings. Instead, because of the new goal to measure the components of coverage error, it became important to use (for component error estimation) a definition that is not as strict as that used for net error. But the strict definition will continue to be used for net error in 2010. (See comment 16. Now p. 29..): Page 31, second paragraph: For clarification, technically speaking, the P-sample is not used to estimate the correct population, but is used to estimate the match rate, while the E-sample is used to estimate the correct enumeration rate. Both of these samples are needed to estimate the true population. Matching is performed to the census, not the E- sample, although housing units are only matched to census housing in the search area. (See comment 17. Now footnote 26, p. 31): Page 31, footnote 25: Your footnote reads, "Bureau officials have noted that the clerical matching procedures being planned for 2010 are comparable to those used for coverage measurement in the 2000 Census." (See comment 18. Now footnote 27, p. 32..): Comment: The clerical matching procedures for 2010 are much more complex than in 2000. We have to account for the duplicate searching at alternate respondent-provided addresses collected in the person interviews and person follow-up interviews. So they are not strictly comparable. Pages 31-32, last sentence: Your statement reads, "Although the Bureau will not include the coverage measurement person follow-up interviews in the dress rehearsal, for 2010 the Bureau plans to use information collected from those interviews as well as initial person interviews in its computer-matching operation." (See comment 19.): Comment: This statement is inaccurate. Information collected from Person Follow-up is not used in computer matching, but only in clerical matching. Page 33: The last paragraph references 52 research topics covering all aspects of its decennial operations that the Bureau provided for the National Academy of Sciences to review. Comment: The paragraph is misleading and should be clarified as follows: The Census Bureau provided a list of 54 research questions sorted among 11 research topics to the National Academy of Sciences for their review. For the most part, the research questions were not identified as either evaluations or experiments. The Academy did not reduce the list of subjects for census experimentation to about six. Rather, the Academy's interim report presents its ideas for 2010 Census experiments, which in some cases are similar to the Census Bureau's proposed research. (See comment 20.): The following are our comments on the Department of Commerce's letter dated April 3, 2008. GAO Comments: We clarified our recommendation on relating components of coverage error to census operations in order to improve future decennials. Specifically, we recognize that the Bureau has developed table shells that associate components of coverage error to census operations, but these table shells and plans for how the Bureau will conduct its analyses should be shared with the Congress and other stakeholders. In addition, it is not clear from current Bureau plans how it will go about obtaining the data required to complete the tables. 1. We have revised the Highlights page to reflect the Department of Commerce's clarification that for the census coverage measurement program, the Bureau will expand its computer and clerical matching to match persons in its postenumeration survey. 2. We have revised the report to reflect the Department of Commerce's clarification that the Bureau will expand its computer and clerical matching to match persons in its postenumeration survey. 3. In footnote 3, we have revised the report to clarify that the although the coverage measurement interview will start 4 months later compared to the Census 2000 telephone phase, person interviewing for the 2010 Census ends 1-1/2 months later than in 2000. 4. In footnote 2, we have revised the sentence to indicate that the net error reflects the error remaining after omissions and erroneous enumerations offset one another. 5. In footnote 4, we have revised the definition of the P-sample and E- sample based on the Department of Commerce's clarification. 6. On page 16, we have clarified information about using the undercount probe during nonresponse follow-up and coverage follow-up. 7. On page 18, we have clarified the information about the limited likelihood of duplication of individuals that will occur within a site of limited geographic size. 8. On page 19, we have incorporated the clarifying information provided by the Department of Commerce. 9. On page 19, we have clarified information about the matching process. 10. On page 20, we have revised the report to reflect that the 2008 Dress Rehearsal is intended to provide additional data to make final decisions about the Bureau's implementation of coverage follow-up in 2010. 11. On page 22, we have revised the report to reflect that the Bureau will expand its computer and clerical matching to match persons in its postenumeration survey. 12. On page 23, we have incorporated the Department of Commerce's revisions on information that is collected during the census coverage measurement program. 13. On page 23, we have deleted the incorrect sentence. 14. On page 23, we have deleted the incorrect sentence. 15. On page 26, we have revised the sentence to reflect that the Bureau will expand its computer and clerical matching to match persons in its postenumeration survey. 16. On page 29, we revised the sentence to clarify the reason for refining the Bureau's definition for erroneous enumeration. 17. On page 31, we clarified the definition of the P-sample and E- sample as suggested. 18. On page 31, for footnote 26, we acknowledged the change in clerical matching for the 2010 Census. 19. On page 31, we deleted the incorrect sentence. 20. On page 33, we provided attribution for the information to the National Academy of Sciences' December 2007 interim report. [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: Mathew J. Scirè (202) 512-6806 or sciremj@gao.gov: Acknowledgments: In addition to the contact name above, Ron Fecso, Chief Statistician; Ernie Hazera, Assistant Director; Jenna Aurand; Betty Clark; and Jennifer Edwards made key contributions to this report. Tom Beall, Andrea Levine, Donna Miller, Lisa Mirel, and Elizabeth Wood provided significant technical support. [End of section] Footnotes: [1] The Bureau is planning to use a 300,000 national housing unit sample. [2] The dual systems approach uses a postenumeration survey to produce an estimate of the true population, which is then used to produce an estimate of the net coverage error. The net error reflects the error remaining after omissions and erroneous enumerations offset one another. See GAO, 1990 Census Adjustment: Estimating Census Accuracy-- A Complex Task, GAO/GGD-91-42 (Washington, D.C.: Mar. 11, 1991). [3] Although the interviewing will start 4 months later compared to the Census 2000 telephone phase, person interviewing for the 2010 Census ends 1-1/2 months later than in 2000. [4] The Bureau conducts a postenumeration survey, independent of the census, in a sample of block clusters. This sample is known as the P- sample and estimates the number of persons missed by the census. A postenumeration survey consists of two samples: the P-sample and the E- sample. P-sample persons are matched not only to the E-sample, but also to the entire census. E-sample persons are matched to the entire census to identify and resolve duplicates. E-sample housing units are matched to census enumerations in the block cluster and surrounding ring of blocks. [5] The coverage measurement interview depends upon the respondents' ability to recall the composition of their households on Census Day. The longer the period of time between Census Day and the day that the measurement interview is conducted, the harder it is for respondents to recall membership of their households on Census Day. [6] For example, see GAO, 2010 Census: Cost and Design Issues Need to Be Addressed Soon, GAO-04-37 (Washington, D.C.: Jan. 15, 2004); 2010 Census: Basic Design Has Potential, but Remaining Challenges Need Prompt Resolution, GAO-05-9 (Washington, D.C.: Jan. 12, 2005); and 2010 Census: Census Bureau Should Refine Recruiting and Hiring Efforts and Enhance Training of Temporary Field Staff, GAO-07-361 (Washington, D.C.: Apr. 27, 2007). [7] Starting in January 2007, the Bureau notified state and local governments that it would seek their help in developing a complete address file through the Local Update of Census Addresses program. Address canvassing--a field operation to build a complete and accurate address list in which census field workers go door to door verifying and correcting addresses for all households and street features contained on decennial maps--will begin in April 2009. For more information, see GAO, 2010 Census: Census Bureau Has Improved the Local Update of Census Addresses Program, but Challenges Remain, GAO-07-736 (Washington, D.C.: June 14, 2007). [8] See also National Academy of Sciences, Once, and Only Once, and in the Right Place: Residence Rules in the Decennial Census (Washington, D.C.: 2006). [9] A household that completes a coverage follow-up interview could either finish the interview with the same roster of household members listed at the beginning of the interview or make some change to the roster. A roster change occurs when a household adds at least one person to the original roster, deletes at least one person from the original roster, or both. [10] Of the 4,191 completed interviews, at least one person was added to 360 households while at least one person was deleted from 531 households. [11] During the nonresponse follow-up operation, if persons reply affirmatively to the undercount probe, the census field workers collect their names and demographic information. Then, these persons appear on the coverage follow-up roster and the coverage follow-up interview confirms the Census Day residency for these persons. [12] While only 8 persons were included in the coverage follow-up roster after nonresponse follow-up, the coverage follow-up added an additional 69 persons from households that were initially enumerated through nonresponse follow-up. [13] Bureau officials said that its administrative records system uses a unique identifier in place of the Social Security number. Social Security numbers are only used for the purpose of validating the Bureau's administrative records system. [14] If the records do not match, the Bureau does not inform the other agencies because of issues related to census confidentiality. [15] In the 2004 Census Test, the Bureau was successful in identifying duplicates using person matching, but it discovered operational complications with trying to resolve person and housing situations within the same operation. Basically, the Bureau concluded that with such a complicated interview, it was too much to ask of the interviewers that they understand the distinction between person and housing duplication. A major component of the 2006 Census Test was to find a way to determine before follow-up whether person duplication was resulting from housing-level issues or person-level issues. [16] See GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1 (Washington, D.C.: Nov. 1999). [17] See GAO, Internal Control Management and Evaluation Tool, GAO-01- 1008G (Washington, D.C.: Aug. 2001). [18] Data contamination occurs when a person's inclusion or exclusion from one census operation affects the probability of his or her inclusion in a second independent census operation. [19] The experts were not aware of these results because the documents were for internal Bureau use only. [20] According to Bureau officials, the dress rehearsal timeline is the best current indicator of the timeline for the 2010 Census. However, Bureau officials have told us that coverage measurement estimation will not be done for the 2008 Dress Rehearsal. In addition, the Bureau will not include its coverage measurement housing unit field operation, housing matching operation, or person follow-up. The Bureau is also considering dropping person matching from the dress rehearsal. [21] The exact date is yet to be determined. [22] Poststratification entailed calculating dual systems estimates separately for different demographic and geographic groups. Dual systems estimation requires that a postenumeration survey be conducted. It produces an estimate of the true population, which is then used to produce an estimate of the net coverage error. In the 2000 coverage measurement program, over 400 strata or groups were examined. [23] For instance, logistic regression allows for the use of continuous variables like age, instead of only the categorical variables used in poststratification. Additionally, unlike poststratification, logistic regression is able to include or omit interactions between variables. [24] See GAO-01-1008G. [25] See GAO-01-1008G. [26] According to the Department of Commerce, the P-sample is used to estimate the match rate, while the E-sample is used to estimate the correct enumeration rate. Both of these samples are needed to derive a corrected population estimate. Matching is performed to the census, not the E-sample. [27] Bureau officials have noted that clerical matching procedures being planned for 2010 are largely comparable to those used for coverage measurement in the 2000 Census although, with additional information being collected in 2010, the Bureau will have to account for duplicate searching at alternate respondent-provided addresses collected in the person interviews and person follow-up interviews. [28] The National Academy of Sciences has convened a panel of experts to review the Bureau's program of research, evaluation, and experimentation for the 2008 Dress Rehearsal and the 2010 Census. The panel will consider priorities for evaluation in the 2010 Census. The panel will conduct its work over a 3-year period, from September 2006 to September 2009. [29] National Academy of Sciences, Experimentation and Evaluation Plans for the 2010 Census: Interim Report (Washington, D.C.: Dec. 7, 2007). GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: