The Census 2000 Testing, Experimentation, and Evaluation Program

Part A: The Census 2000 Testing, Experimentation, and Evaluation Program Overview

In their final report on the design of Census 2000, the Commerce Secretary's 2000 Census Advisory Committee concluded: "What everyone wants is as simple as A-B-C... A Better Census."

But how will we know if we achieve a better census in 2000, and how will we build a better one for 2010? An important source for answering these questions will be the Census 2000 Testing, Experimentation, and Evaluation Program. Besides being used to assess Census 2000, this program will help design testing for early 2010 Census planning and provide information for the American Community Survey, Master Address File Updating System, and other Census Bureau censuses and surveys. As other countries look to the U.S. Census Bureau as a leader in techniques and methodologies, the results of the Census 2000 Testing, Experimentation, and Evaluation Program may also help them in making more informed decisions in designing their censuses.

Important factors affecting the next decade include:

• The implementation of the American Community Survey in lieu of a long form census data collection;
• The ability to dramatically change the 2010 Census when collecting only short form census data;
• A continually maintained housing unit address frame;
• The modernization of the Master Address File Updating System;
• The role of administrative records in the 2010 Census, the American Community Survey, and address list development;
• The changing community role in the census as manifested through partnerships, governmental activities, and constituent groups;
• The impact of a rapidly changing technological environment on census data collection, capture, processing, and dissemination;
• Difficulty in eliciting public response to censuses and surveys; and
• The ability to limit the potential for duplicate responses when alternative ways of responding to the census are offered.

The Census 2000 Evaluation Program will measure the effectiveness and impact on data quality of the Census 2000 design, operations, systems, and processes. It will provide measures of the success of Census 2000 and its operations which are of interest to internal and external stakeholders. For example, it will inform data users and stakeholders about data quality and limitations of the data, help explain the quality of census data, and provide information needed for historical comparability of census methods and procedures. This also will inform planning and development of the 2010 Census, the American Community Survey, and the Master Address File Updating System. It will help determine what simplifications can be made to the overall 2010 Census design, assist in operational planning, and inform questionnaire development and alternative data collection methodologies. Over 100 ⁽¹⁾ studies are planned in the following areas:

Response Rates & Behavior Analysis Content & Data Quality Data Products Partnership and Marketing Special Places and Group Quarters Address List Development Field Recruiting & Management Field Operations Coverage Improvement Ethnographic Studies Data Capture Processing Systems

Quality Assurance Evaluations Accuracy & Coverage Evaluation Survey Operations Coverage Evaluations of the Census and of the Accuracy and Coverage Evaluation Survey Accuracy & Coverage Evaluation Survey Statistical Design & Estimation Organization, Budget, and Management Information System Automation of Census Processes

Many of the issues we are trying to understand with these evaluation studies are described below. In some cases, we will be able to reach firm conclusions, while in others it will be more difficult to disentangle effects of the census procedures from the external environment.

• The effectiveness of the Partnership and Marketing Program's paid advertising in changing awareness and mail response behavior of various groups and hard-to-count populations;
• Whether national and regional objectives of the expanded Partnership Program were accomplished;

• The effectiveness of operations used to build, update, and assign geographic codes to the Census 2000 address list. This will involve studies of the Master Address File, the Census Bureau's geographic database, the Postal Service's Delivery Sequence File, field operations, and partnership operations such as the Local Update of Census Addresses;
• Coverage rates for various demographic groups and areas, as measured by the Accuracy and Coverage Evaluation Survey and by demographic analysis;
• The effectiveness of the various Accuracy and Coverage Evaluation Survey operations in measuring errors in the census;

• The relative effectiveness of various operations designed to improve overall coverage or reduce differential coverage errors for hard-to-enumerate groups and areas;
• The use, effectiveness, and data quality of various modes available for responding to the census (Mail, Nonresponse Followup, Internet, Telephone Questionnaire Assistance, Be Counted forms);
• The coverage, content, comparability, and sources of information used to construct the group quarters frame for the decennial census (and American Community Survey);

• The use and effectiveness of language assistance guides and non-English language questionnaires;

• The success of the Data Capture System, including the Optical Mark Recognition, Optical Character Recognition, and operational problems;
• The ability of various field and processing operations to identify and unduplicate multiple responses for the same household or individual;
• The effectiveness of recruiting, training, and pay strategies in obtaining the workforce needed to conduct field operations;
• The completeness and accuracy of data, as measured by item imputation rates, proxy rates, and comparisons to external benchmarks, for both mail returns and enumerator completed questionnaires;
• The effects of the new race and Hispanic origin questions on the content and quality of data, particularly in comparison to data based on different questions in previous censuses;
• The reliability, functionality, maintenance, and security needs of many of the major automated systems designed to support Census 2000; and
• The effectiveness of the quality assurance strategy used for Census 2000.

Census 2000 Alternative Questionnaire Experiment
Administrative Records Census Experiment
Social Security Number, Privacy Attitudes, and Notification Experiment
Response Mode and Incentive Experiment
Census 2000 Supplementary Survey
Use of Employee Reliability Inventory File for Nonresponse Followup Enumerators
Ethnographic Studies

Key things we will learn from these studies include:

• An assessment of different questionnaire design and content on coverage and data quality, including the effects of the amount and presentation of residence rules, instrument design, and a comparison of the 1990 race question with that used in 2000;
• An assessment, under decennial conditions, of the use of various types of administrative records as a primary data collection tool - two major approaches will be studied;
• Public response and the effects on mail and item response to a request for Social Security Numbers on the census short form, and to two variations of a notification about the Census Bureau's proposed use of administrative records obtained from other government agencies;
• Public response to alternative modes of response such as Computer-Assisted Telephone Interviews, interactive voice response, and the Internet;
• The effects of offering alternative self-administered data collection modes, as well as offering an incentive to respondents who use these modes;
• The operational and technical feasibility of collecting long form data using the methods of the American Community Survey, a key element in validating the plan to eliminate the long form from the 2010 Census;
• The validity and feasibility of using a noncognitive test of personality based competencies to select interviewers with better interpersonal skills, thereby reducing turnover and improving work performance; and
• Qualitative data about response behavior for hard-to-enumerate subgroups of the population.

Results from the Census 2000 Testing, Experimentation, and Evaluation Program will help inform the Census Bureau's efforts to achieve the following objectives for the 2010 Census and other programs:

• Improve coverage of the population and reduce the differential undercount;
• Improve the accuracy of responses and locating people geographically;
• Increase mail response rates and reduce field activities;
• Maintain and refine an open process with all stakeholders throughout the decade while increasing the confidence of our customers; and
• Spread the cost of data collection and updating the address list more evenly throughout the decade to reduce risk, simplify logistics, and improve manageability.

We will use a three pronged strategy to achieve these goals:

• Enhance the Master Address File and geographic database through modernization initiatives such as a web-based system, global positioning system, and an on-going Local Update of Census Address (LUCA) program. This will:

- enhance LUCA;

- directly attack the issues of a complete and unduplicated address list; and

- facilitate automation and electronic collection.

• Through the American Community Survey, collect and tabulate long form data every year. This will:

- expand our ability to target;

- simplify the 2010 process allowing us to focus on coverage; and

- provide long form data on a flow basis.

• Reengineer the 2010 Census process through early planning, taking into account opportunities afforded by no long form and an enhanced Master Address File. Using technology and a short-form only census will:

- establish a flexible cost effective infrastructure that will facilitate coverage improvement;

- set up a data flow design to allow for efficiencies to the process; and

- establish a foundation upon which the "perfect census" can be built.

The Census 2000 Testing, Experimentation, and Evaluation Program will help us to address key planning questions for this decade:

• Do we need to lengthen or shorten time periods for census operations for quality or operational reasons?
• Are there any unforeseen operational difficulties when collecting long form data using the methods of the American Community Survey?
• What is the overall effect of a continuously maintained address file?
• How can we be most effective with partnerships, promotion, and advertising?
• What is the potential impact of using administrative records?
• How accurate are our sample design and procedures for estimating total and differential undercount?
• What is the impact on field activities and infrastructure of hiring and training many more enumerators than are needed for decennial operations in order to compensate for expected turnover?
• Which response options are most effective?

Conclusion

The design of Census 2000 is by far the most ambitious decennial census in history, particularly in its use of an open planning process, promotion, partnerships, new technologies, statistical methodology, and alternative methods for hard-to-count populations and areas. Yet as our nation continues to grow and the need for rapid and accurate data continues, all of these approaches need to be further refined and developed to meet the challenges of providing data in the 21^st Century - more data needs at lower levels of geography on a more timely basis.

The Census 2000 Testing, Experimentation, and Evaluation Program will assist the Census Bureau in evaluating Census 2000 and in exploring new survey procedures in a census environment. It builds the foundation for making early and informed decisions about the role and scope of the 2010 Census in the federal statistical system and its interaction with the American Community Survey and the Master Address File Updating System. This work provides critical analysis and information for Census Bureau planning and implementation of decisions for the 2010 Census and the American Community Survey.

1. Refer to "Part C: The Census 2000 Evaluation Program - Introduction" for more information on the planned evaluations and recent changes in the program.