Appendix A

Approach to Performance Measurement

CDC and partners are concerned with a spectrum of health issues, including infectious diseases, chronic conditions, adverse reproductive outcomes, environmentally related conditions, occupationally related health events, and injuries. This array of health conditions and outcomes requires a variety of intervention strategies for populations, in addition to clinical preventive services for individuals. CDC engages in extensive dialogue with partners, communities, and the public to identify and implement intervention strategies that address the specific needs of diverse populations. Examples include the provision of prophylactic measures (e.g., vaccination, post-exposure prophylaxis), educational services (e.g., dissemination of public health messages, counseling), inspection of food establishments, and control of disease outbreaks. For these activities, the rational development of public health policy depends on public health information.

A variety of CDC data systems provide the science base for identifying health problems, designing interventions, and monitoring program performance (See Appendix B). These data systems face considerable challenges in addressing each of these three areas. For the most part, data systems that were designed to support scientific objectives are now becoming important for the monitoring of performance. Challenges in obtaining data to monitor performance under GPRA include the following:

1.	As GPRA measures are refined over time, data systems to produce data with a frequency that corresponds to the periods during which performance is measured.

2.	As health system changes, historical data series may not continue to produce needed data. For example, the move toward managed care may make medical information increasingly proprietary and impede access to data for research and statistical purposes. Similarly, changes in relationships among healthcare providers and laboratories may make public health surveillance based on case reports more difficult. At the same time, these changes present opportunities for new data-system partnerships.

3.	Data systems will need to produce information of sufficient quality and precision to detect relatively small changes in performance indicators. This may require investments in larger sample sizes for surveys and new technologies for improving data quality. Continuing research will be required to establish the data systems and underlying evaluation approaches to assess causes (program interventions) and effects (outcomes) for performance monitoring.

4.	Many national data systems are the source of GPRA measures for CDC and other health programs. These systems must be assessed and upgraded to remain current with the public health infrastructure. Resources to ensure the maintenance and strengthening of these data systems are included in the FY2002 CDC budget request and need to be continued.

5.	Because many CDC and DHHS programs are implemented at state and local levels, it will be increasingly important to obtain reliable, systematic data at these levels for monitoring of program implementation, performance, and outcomes.

Ascertaining what information is needed and how to collect it is a complex issue. Information for action must be useful to public health programs at local, state, and national levels. CDC and partners use at least seven categories of information to understand and address disease, injury, and disability using the public health model. These categories of information include:

•	Reports of health events affecting individuals

•	Vital statistics on the entire population

•	Information on the health status, risk behaviors, and experiences of populations

•	Information on potential exposures to environmental agents

•	Information on public health programs

•	Information useful to public health but obtained by organizations not directly involved in public health practice

•	Information on the healthcare system and its impact on health

Reports of health events: Reports of cases of diseases of public health importance form the basis for many CDC programs. The National Notifiable Disease Surveillance System (NNDSS) seeks reports on all cases of >40 conditions in the United States. To minimize the burden placed on those who report the data, CDC limits the amount of information collected for each case. NNDSS data are used to monitor disease trends, evaluate public health programs, and identify unusual occurrences of conditions that may require further epidemiologic investigation at the local level.

For some public health purposes, effective action requires additional details on each case. Supplemental data collection systems have therefore been developed for some of the diseases reported to NNDSS. These systems may be less comprehensive in terms of populations represented but provide more detailed information on characteristics of the occurrence of disease. For example, cases of hepatitis are reported weekly to NNDSS for publication in the Morbidity and Mortality Weekly Report (MMWR). In addition, the Viral Hepatitis Surveillance Project collects data on risk factors for different types of viral hepatitis in selected geographic areas. These data have been used to document the importance of behaviors associated with sexual activity and drug use as risk factors for transmitting hepatitis B virus and to target education and vaccination programs.

Control of some conditions requires more detailed information than can be obtained feasiblely from a large group of clinicians or institutions. Networks of selected healthcare providers have therefore been organized to meet these targeted information needs. For example, CDC's Sentinel Event Notification System for Occupational Risks (SENSOR) targets groups of healthcare providers as a component of a comprehensive approach for obtaining data on which to base efforts to prevent workplace-related morbidity. The National Nosocomial Infections Surveillance System (NNIS) receives reports from a selected group of hospitals on the incidence and characteristics of hospital-acquired infections. Data from this system have been instrumental in alerting health authorities to the emergence of antibiotic-
resistant strains of bacteria, which in turn has led to the development of recommendations for the appropriate use of antibiotics.

Vital statistics: Vital records (e.g., births, deaths) are the primary source of some of the most fundamental public health information. Data on teen births, access to prenatal care, maternal risk factors, infant mortality, causes of death, and life expectancy are among the staples of public health information provided by vital statistics. Vital statistics are often the most complete and continuous information available to public health officials at the national, state, and local levels; the timely availability of these data is critically important.

In the United States, the legal authority for vital registration rests with the states and territories. CDC's National Center for Health Statistics (NCHS) produces national vital statistics by collecting data from the vital records of the states. NCHS works with the states to ensure a uniform national data base through the promotion of standard data collection forms and data preparation and processing procedures and also provides partial financial support for state systems.

Health status, risk factors, and experiences of populations: Since the determinants of many health problems are behavioral, environmental, or genetic, health agencies need information that is not readily available from medical records on the prevalence of various types of behavior and on access to care. Thus, regularly conducted surveys of the general population are needed for public health. These surveys range from large-scale assessments of the general population to assessments targeted at high-risk (i.e., particularly vulnerable) populations. This need is particularly acute at the state and local levels. Surveys provide information on: 1) baseline health status, 2) morbidity, 3) prevalence of behavioral risk factors, 4) use of healthcare services and identification of underserved populations, and 5) potential for exposure to toxic agents. Information generated from the surveys is used in developing prevention and control programs and in ensuring adequate delivery of health services.

Potential exposure to environmental agents: Information on exposures to environmental agents can be used in evaluating the risks to health from noninfectious diseases, injuries, and certain infectious diseases. For example, measurement of airborne particulates is useful in assessing risks related to pulmonary disorders such as asthma and lung cancer. Information on vectors that may carry agents of infectious disease is important in evaluating the risk for acquiring such infections.

Program information: Data needed to operate public health programs include the number of clients served and the costs of services rendered. These data are useful to public health officials in assessing the effectiveness of public health programs, comparing programs, documenting the need for continuing a particular program, and maintaining accountability for tax dollars spent.

Information from other organizations: Data useful for public health are currently or potentially available from organizations whose functions may not be related to those of CDC and state and local health departments. Data from the Bureau of the Census, for example, are needed for both the reliable computation of rates and the proper adjustment of rates for comparison over time or in different geographic areas. The Environmental Protection Agency (EPA) compiles environmental air-monitoring data to assess compliance with standards for air pollutants established by the Clean Air Act. Data collected through this system are used by public health officials for hazard alerts when pollutants exceed federal standards and in studies of the effects of air pollutants on morbidity associated with respiratory diseases. The Occupational Safety and Health Administration (OSHA) and the Bureau of Labor Statistics compile data on the occurrence of work-related injuries and illnesses and exposure to hazards in the workplace, which can be used for surveillance and research. The Department of Transportation operates the Fatal Accident Reporting System, used in public health to assess risk factors for motor-
vehicle-related injuries and deaths. Crime statistics gathered by the Federal Bureau of Investigation (FBI) assist in evaluating the public health impact of intentional injuries, and the Consumer Product Safety Commission collects data on injuries related to consumer products.

Information on the healthcare system: Information is also needed on the healthcare system and the health impact resulting from changes in the system. CDC provides a great deal of information to monitor the capacity of the healthcare system, utilization of the system, and access to health insurance and services by the American people. These data include: inventories of healthcare providers; patterns of utilization of health services such as hospitalization rates and uptake of new technologies; and access to health care and barriers (both financial and non-financial) to access.

Appendix B

Data Verification and Validation

Data verification and validation help to ensure that the data CDC uses to assess performance is of sufficient quality. The following data systems have been referenced in the CDC Performance Plan as sources for data used in assessing program implementation and effectiveness.

Behavioral Risk Factor Surveillance System

In 1984, CDC initiated the Behavioral Risk Factor Surveillance System (BRFSS), a unique, state-based surveillance system designed to collect prevalence data on behavioral risks and conditions that affect health. States conduct monthly telephone surveys using a standardized questionnaire to determine the distribution of behavioral risk factors. Survey responses are forwarded to CDC, where the data are aggregated and published at year's end. The BRFSS provides flexible, timely, and ongoing data collection that allows for state-to-state and state-to-nation comparisons. Participating states use data derived from the BRFSS to identify demographic variations in health-related behaviors, target services, address emerging and critical health issues, propose legislation for health initiatives, and measure progress toward state and national health objectives. The system's broad network of information gathering also enables states to evaluate their disease prevention and health promotion efforts.

The BRFSS survey instrument is a three-part questionnaire developed jointly by CDC and the states:

•	Core component: The fixed core is a standard set of questions asked by all states on demographic characteristics and behaviors that affect health (e.g., tobacco use, alcohol consumption). The rotating core includes two sets of questions, each asked in alternating years by all states, that address different topics. The emerging core consists of up to five questions that typically focus on late-breaking issues. These questions are added to the core for one year and evaluated at year's end to determine their potential value in future surveys.

•	Optional CDC modules: These are sets of questions on specific topics (e.g., smokeless tobacco use, arthritis) that states can opt to include in their questionnaires.

•	State-added questions: These questions are developed or acquired by participating states and added to their questionnaires.

Each year, states and CDC agree on the content of the core components and optional modules. For ease of comparability and use, many of the questions are taken from established national surveys. More than 30 validity and reliability studies attest to the quality and validity of data derived from the BRFSS.

Clinical Laboratory Improvements Act of 1988 (CLIA)

The Clinical Laboratory Improvements Act of 1988 (CLIA) is designed to ensure the sound and scientific development of new laboratory methods. CLIA includes standards that must be met before certification of a laboratory method. These standards include an exacting series of internal and external evaluations. Among the internal checks is the development of a detailed procedures manual for each method. Manuals must be verified and approved by senior laboratory personnel who were not directly involved in the development of the method. CLIA also provides detailed specifications for quality control and calibration of laboratory equipment. Further internal control is provided through regular review from a designated Quality Assurance Officer tasked with ensuring that generally accepted international scientific standards are being followed in the development of the method. External evaluation and control are provided through regular on-site inspections by statutorily approved, independent inspection teams. Inspectors review the internal procedures established by the organization to ensure compliance with CLIA standards. To date, CDC has passed all on-site CLIA inspections.

Group B Streptococcal Disease Surveillance, part of the Active Bacterial Core Surveillance (ABCs)

In 1989, CDC initiated active surveillance for group B streptococcal (GBS) disease as part of the Active Bacterial Core Surveillance (ABCs) system, an active surveillance system for several pathogens that cause invasive disease. Surveillance was conducted in five geographic areas that were awarded contracts after a competitive request for proposals. In 1994, active surveillance for GBS disease was included as a core activity of the newly established Emerging Infections Program (EIP) network, a cooperative agreement program that addresses important public health issues related to infectious diseases. In 1999, the EIP network comprised eight states; all participated in ABCs and conducted active surveillance for invasive GBS disease.

Specific objectives for GBS disease surveillance are to: 1) assess the impact of CDC prevention guidelines published in May 1996, 2) determine the extent to which continuing cases of early-onset GBS disease are preventable through current prevention strategies, 3) identify serotypes responsible for disease to guide vaccine development, 4) evaluate progress in the elimination of serotype b disease, 5) detect possible emergence of disease due to other capsular types, and 6) determine possible preventable reservoirs of the bacteria. Data collection focuses on disease occurrence. State surveillance officers contact personnel in all microbiology laboratories that process bacterial cultures from sterile sites to find cases of GBS. Laboratory audits are also conducted semi-annually to detect possible underreporting. Data are transmitted electronically from the EIPs to CDC's ABCs team on a monthly basis. Annual surveillance reports are made available on the Internet at the ABCs website. Laboratory testing of isolates collected as part of surveillance is performed in reference laboratories. Electronic files containing results of laboratory testing of each state's isolates are fed back to that state on a monthly basis.

Routine laboratory audits to ensure the completeness of data collection represent a tremendous strength of the system. Each month, CDC staff review data and transmit potential errors to state personnel for evaluation. Performance standards for active surveillance have been established in each site to permit aggregation of data collected via somewhat different approaches. Detailed instructions for completion of case report forms ensure consistency across sites. State surveillance officers and CDC's ABCs team hold monthly conference calls to address logistical and technical aspects of the system and meet annually to review and update protocols, present special studies, and discuss innovations. Site visits are currently conducted on an as-needed basis, but annual site visits are planned.

Easy access to the data is provided through a website that includes the basic protocol and one-page yearly surveillance reports for each pathogen. Additional information on GBS is available on a website focused on that infection, with many materials targeted to pregnant women or healthcare providers and public health workers concerned with pregnant women.

The principal limitation of GBS disease surveillance through the ABCs is that it is not conducted throughout the United States. Substantial geographic variation in the incidence of invasive GBS disease has been noted, and it is unclear whether states outside ABCs areas have experienced changes in the incidence of GBS disease that are comparable to those noted in the surveillance areas. One way of addressing this limitation is to increase the availability of ABCs methods and tools. Through the website and frequent publications, CDC is attempting to provide other state health departments with information that can help them assess whether the efforts involved in conducting invasive GBS disease surveillance, particularly for early-onset disease in infants <7 days, are feasible in their locales.

Integrated Resources Information System

CDC's Integrated Resources Information System (IRIS) is a collection of applications to assist management in budget, staffing, and project planning, tracking, and reporting. The IRIS budget application provides detailed budget information by CDC component. It allows managers to view budget reports grouped by a variety of options. IRIS staffing is a view-only application designed to allow users to quickly access personnel data reports and project employee salaries for a specified time period. The projects application allows managers to plan, track, and manage various types of projects. This application provides access to project data, resources, and administrative functions. All information for a project must be maintained in the IRIS projects component to ensure consistency and reliability of data. The IRIS reports application is the data. retrieval, and reporting component.

National Health and Nutrition Examination Survey

The National Health and Nutrition Examination Survey (NHANES) is a program of studies to assess the health and nutritional status of adults and children in the United States. Started in the early 1960s, NHANES is the only national source of objectively measured health data capable of providing accurate estimates of both diagnosed and undiagnosed medical conditions in the population. Findings from the survey are essential for determining rates of major diseases and health conditions and for developing public health policies and prevention interventions. The survey screens 15,000 households per year and selects 3,500. From this sample, 5,000 persons are interviewed and examined annually. Samples are recruited from 15 counties or clusters of counties each year. Samples comprise sufficient numbers to provide reliable estimates by gender and age group for non-Hispanic whites, Mexican Americans, and African Americans.

Data are collected via health interview, physical examination, and clinical and laboratory tests. Interviews are conducted in respondents' homes. Physical examinations are performed in specially designed mobile examination centers that travel to survey locations throughout the country. These centers allow for the collection of data on chronic conditions, nutritional status, medical risk factors, dental health, vision, illicit drug use, blood lead levels, food safety, and other factors that are not possible to assess by use of interviews alone. The medical team consists of a physician, dentist, medical and health technicians, and dietary and health interviewers; trained bilingual staff conduct the household interviews.

An advanced computer system using high-end servers, desktop PCs, and wide-area networking is used to collect and process all NHANES data, nearly eliminating the need for paper forms and manual coding operations. Household interviewers use notebook computers with electronic pens for data collection in the field. Data collected in the mobile examination centers are automatically transmitted via a frame relay network into central databases. Survey information is available to CDC within 24 hours of collection.

Information from NHANES is disseminated through an extensive series of publications and articles in scientific and technical journals. Survey data are also available on CD-ROM and computer diskettes. In previous years, data were available for analysis approximately 31 months after collection. A goal is to improve the timeliness of data dissemination. The computerized system has already substantially improved access to the data from the field.

A comprehensive quality assurance program is instituted before data collection begins, with appropriate training that requires significant practice time for the health examiners and interviewers. Training focuses on hands-on experience rather than didactic methods. During data collection, health examiners and survey staff meet regularly to discuss operations, updates, and problems. Staff are retrained as needed.

NHANES relies on both passive and active monitoring systems for operational and content-related quality control. Passive quality control uses automated computer procedures for detecting data anomalies. After careful analysis, appropriate activities can be undertaken to resolve any data collection issues. Active quality control relies on examiner feedback to identify and evaluate problems and select remedies. NHANES primarily relies on physical measurements from well-established biomedical procedures. In most instances, these measurements represent the gold standard data against which self-reported data might be validated for other subjective data collection modalities. New technologies under consideration are evaluated to determine if they provide valid estimates of the condition, risk factor, or measurement for which they are being used. The evaluation might include a scientific literature review, expert workshop, or validity study.

National Health Interview Survey

The National Health Interview Survey (NHIS) is the principal source of information on the health of the civilian, non-institutionalized population of the United States. The purpose of the NHIS is to monitor the health of the U.S. population through the collection and analysis of data on a broad range of health topics. A strength of the survey is the ability to display these health characteristics by many demographic and socioeconomic factors. NHIS data are used widely throughout DHHS to monitor trends in illness and disability and to track progress toward achieving national health objectives. The data are also used by the public health research community for epidemiologic and policy analysis.

The NHIS is a cross-sectional household interview survey. Sampling and interviewing are continuous throughout each year. Households chosen for interviews are a probability sample representative of the target population. NHIS data are collected annually from approximately 43,000 households including about 106,000 persons. Survey participation is voluntary, and the confidentiality of responses is ensured. The annual response rate is >90% of eligible households in the sample.

The NHIS has three modules:

•	The basic module remains largely unchanged from year to year and allows for trend analysis. Data from more than one year can also be pooled to increase the sample size for analytic purposes. The basic module contains a family core, a sample adult core, and a child core through which data are collected on the family unit and from one randomly selected adult and child.

•	Periodic modules collect more detailed information on some of the topics included in the basic module.

•	Topical modules respond to new data needs as they arise.

Data are collected through a personal household interview conducted by staff employed and trained by the U.S. Bureau of the Census according to procedures delineated by CDC. Data are reviewed and analyzed extensively to ensure their validity and reliability. The survey sample is designed to yield estimates that are representative and that have acceptably small variations. Before the actual survey, cognitive testing is performed by CDC's Questionnaire Design Research laboratory, and pretests are conducted in the field. Once collected, data are carefully edited, checked, and compared to data from earlier surveys and/or independent sources. Staff members calculate descriptive statistics and perform in-depth analyses, which result in feedback on the analytic usefulness of the data.

In the past, it has taken approximately 26 months for the survey data to be released for a given year. Improving the timeliness of NHIS data is a GPRA performance measure.

National Hospital Discharge Survey

The National Hospital Discharge Survey (NHDS), conducted annually since 1965, is a national probability survey designed to meet the need for information on characteristics of inpatients discharged from non-federal, short-stay hospitals in the United States. The NHDS collects data from a sample of approximately 300,000 inpatient records acquired from a national sample of about 500 hospitals. The NHDS provides national and regional estimates of U.S. inpatient hospital utilization by the demographic characteristics of patients discharged, conditions diagnosed, and surgical and non-surgical procedures performed. Approximately 95% of eligible sample hospitals respond to the survey.

The NHDS uses two data collection methods: 1) a manual system in which hospital staff or staff of the U.S. Bureau of the Census abstract data from medical records, and 2) an automated system in which CDC purchases machine-readable medical record data from commercial organizations, state data systems, hospitals, or hospital associations. Approximately 40% of hospitals provide data through the automated system. Data are generally available about 17 months after collection. Timeliness is being addressed as part of the GPRA effort.

An ongoing quality control program helps to ensure the accuracy of NHDS data. NHDS data have been found to be a good reflection of information found in medical records. What is not known is the degree to which medical record information reflects actual performance.

National Immunization Survey

The Childhood Immunization Initiative (CII) is one of many federal, state, and local programs mounted to raise vaccination levels in young children. The CII established a 1996 goal of increasing vaccination levels for 2-year-old children to at least 90% for measles-mumps-rubella, diphtheria and tetanus toxoids and pertussis vaccine, oral poliovirus vaccine, and Haemophilus influenzae type b vaccine. In addition, the CII established a goal for 1996 to increase vaccination levels for 2-year-old children to at least 70% for three or more doses of hepatitis B vaccine.

The National Immunization Survey (NIS) is used to assess progress towards these goals. NIS data provide current, population-based, state and local estimates of vaccination coverage produced by a standard methodology. Quarterly data are collected via household interviews in 50 states, the District of Columbia, and 27 urban areas. Interviews are conducted by telephone with randomly selected households. . Each quarter, CDC calculates estimates of vaccination coverage levels and makes valid comparisons of state efforts to deliver vaccination services. CDC uses NIS data to evaluate progress towards national vaccination goals and to identify states with the highest and lowest immunization rates.

To ensure the accuracy and precision of coverage estimates, immunization data for surveyed children are also collected through a mail survey of their pediatricians, family physicians, and other healthcare providers. The parents and guardians of NIS-eligible children are asked during the telephone interview for consent to contact childrens' medical providers. Types of immunizations, dates of administration, and additional data about facility characteristics are requested from immunization providers identified during the telephone survey of households. NIS estimates of vaccination coverage therefore reflect a comparison of information provided by both immunization providers and households.

National Vital Statistics System

Vital statistics are often the most complete and continuous information available to public health officials at the national, state, and local levels. The National Vital Statistics System is responsible for the nation's official vital statistics. The registration of vital events – births, deaths, marriages, divorces, fetal deaths – is a state function, and vital statistics are provided through state-based registration systems. Since 1902, the federal government has obtained use of the records for statistical purposes through cooperative arrangements with the responsible agencies in each state. Standard forms for the collection of data and model procedures for the uniform registration of events are developed and recommended for state use through cooperative activities of the states and CDC. CDC also provides training and instructional materials to the states as part of ongoing technical assistance.

The purpose of collecting the data is to monitor trends over time through vital life events. Vital records and reports originate with private citizens, such as the family affected by the events, physicians, or funeral directors. By law, birth registration is the direct responsibility of the hospital of birth or the attendant at the birth. In the absence of an attendant, the parents of the child are responsible for registering the birth. Although procedures vary from hospital to hospital, personal information is usually obtained from the mother; medical information may be obtained from the chart or from a worksheet completed by the birth attendant. Reporting requirements vary from state to state; in general, the completed certificate must be filed with the state or local registrar within 10 days of birth. Published data represent all counties and places of 10,000 or more population. Electronic files include data for states, counties, large cities (population of 100,000 or more), and metropolitan statistical areas.

By law, death registration is the direct responsibility of the funeral director or person acting as such. The funeral director obtains the data required, other than the cause of death, from the decedent's family or other informant. The attending physician provides a best medical opinion about the cause and manner of death; later this information is coded by the state or CDC according to uniform codes. Demographic information is also recorded. If no physician was in attendance or if the death was due to other than natural causes, the medical examiner or coroner investigates the death and provides the cause and manner. Reporting requirements for death vary, but in general the completed certificate must be filed within 3 to 5 days of the death. Published data include all counties and places of 10,000 or more population. Electronic files include data for states, counties, large cities (population of 100,000 or more), and metropolitan statistical areas.

Fetal deaths are also reported through the National Vital Statistics System. All fetal deaths of 20 weeks or more gestation that occur in the United States are recorded. A linked birth/infant death file allows for the analysis of demographic and health characteristics from certificates of live births in combination with causes of death and other data from death certificates of infants who died before their first year of life. The linked file set includes information on all the infants who died in the United States each year, as well as information on all live births. An additional file includes information on death records not linked to birth certificates. The match rate is about 97%-98%. Data are organized by calendar year.

Provisional and final estimates of the number of marriages and divorces are obtained from each state able to provide these figures. Since data are not available from all states, national divorce rates are not produced. Detailed characteristics of marriages and divorces have not been available since 1996.

Vital statistics data are collected using uniform procedures and are accurate and consistent. The data are reported as soon as they are analyzed by CDC staff. Monthly provisional numbers and rates are published in the National Vital Statistics Reports. These figures are based on approximate counts of the number of events that occurred in a given state; an estimation procedure is used to convert these occurrence estimates into state-specific estimates of the number and rate of resident events. Preliminary data collected through the National Vital Statistics System are made available to the public approximately 10 months after the end of the collection year. Data are presented for a 12-month period and are published semi-annually in the National Vital Statistics Reports. Final data are released about 18 months after collection via National Vital Statistics Reports, public use data tapes, CD-ROM, Series Reports, the Internet, and journal articles. Use of electronic products have greatly increased the accessibility of the data and reduced the costs to researchers and other users.

The data collected through the National Vital Statistics System represent all registered vital events in the United States and adequately represent the true rates of events. To more accurately record birth and death information, new birth and death certificates are being designed through a collaborative effort with states, researchers, and other interested parties. The revised certificates reflect changing data needs and emerging public health applications; they will be implemented in 2003.

Sentinel Surveillance for Chronic Hepatitis C

Although a large number of persons in the United States are chronically infected with HCV and many will develop chronic liver disease, the burden of disease has not been well characterized. There is no ongoing surveillance, and few population-based studies have been conducted from which to determine the incidence and prevalence of chronic liver disease and the relative proportion of cases attributable to viral hepatitis and other etiologies. To begin to collect this information, CDC established a pilot surveillance system for chronic liver disease in 1998. The data-collection system has three components:

•	A standard interview questionnaire, developed by CDC, is used by all sites to ensure comparability of data and facilitate aggregation of data as appropriate. The instrument includes questions from other established surveillance systems and from previous studies of chronic liver disease. Questions focus on demographic characteristics, clinical information, quality of life issues, and exposures and risk factors.

•	A standard form is used to abstract clinical and laboratory information from the patient's clinical chart. This information, collected consistently across sites, includes data needed to determine disease etiology, treatment history, medication use, and other relevant clinical information.

•	A serum sample is collected and sent to CDC to identify serologic markers for viral hepatitis.

An important characteristic of the pilot is its comprehensiveness. For the first time, all patients with chronic liver disease in several geographic areas are being identified using a common methodology, with consistent information collected in all sites. The goal is to expand the use of the methodology and data collection instruments to other sites throughout the United States to develop a comprehensive picture of the occurrence and characteristics of chronic liver disease and to monitor trends.

Although quality assurance and quality control instruments are still under development, several validation studies have been conducted. To assess the completeness of reporting, CDC conducted a survey of primary care practitioners and a review of all first-time liver biopsies. These studies indicated that overall surveillance was comprehensive and was successful in identifying the vast majority of patients in the target population. A review of a randomly selected subset of charts failed to reveal any significant errors in chart abstraction. To assess the overall validity of the study, early preliminary results have been compared to the few existing relevant data. This evaluation, demonstrating that the incidence of newly diagnosed chronic liver disease has increased in recent years, is already contributing to CDC's efforts to more accurately estimate the burden of illness from chronic liver disease.

U.S. Sentinel Physician Surveillance for Influenza

Established in 1982, the U.S. Sentinel Physician Surveillance for Influenza is one of four primary sources of influenza surveillance data. The sentinel physician surveillance system is an active system of surveillance conducted from October through May. Each week during that period, several hundred volunteer physicians around the country report the total number of patients seen and the number of those patients with influenza-like illness by age group.

During the 1997-98 influenza season, 27 states and the District of Columbia elected to participate in a pilot program to upgrade the sentinel physician surveillance system. The pilot merged CDC's national sentinel surveillance system and state-based systems into one integrated system based on common methodologies and standards. During the 1998-99 influenza season, the enhanced sentinel physician surveillance system was expanded to include 40 states and the District of Columbia, and an Internet reporting system was developed. States are responsible for establishing, recruiting, and maintaining state-based sentinel physician groups and for ensuring that data are collected and transmitted regularly to a central data repository at CDC, which is updated daily. CDC is responsible for coordinating the system nationally, maintaining the reporting systems, processing and analyzing the data, and maintaining the Internet site. Efforts to improve the system are continuous.

Sentinel physicians can report data via any of three methods: 1) Internet reporting, 2) touchtone phone reporting, or 3) facsimile transmission with manual entry of data. A program developed by CDC integrates the three sources of data and uploads the data to the Internet site. Data are available daily to each state coordinator. A summary of influenza activity is available to the general public each week.

CDC has undertaken a continuous process to simplify use of the system, clarify case definitions, and offer multiple options for input and access. With daily updates and weekly summaries, the information is extremely timely and pertinent for decision making. CDC epidemiologists analyze the data for outlying information and perform routine checks for coherence. State coordinators routinely check the timeliness of reporting and troubleshoot problems at the local level. Guidelines are provided to sentinel physicians for optimal timing of specimen collection for virologic testing on certain patients. There is no way to ascertain that the data on influenza-like illness is free of error, but, as the number of participating sentinel physicians increases, the potential consequences of errors decrease. Given that sentinel surveillance provides an index of current influenza activity, consistent reporting by a stable group of physicians is imperative for data reliability. Increasing sentinel physician sites and sentinel physician participation in each state would greatly increase the validity of the data.

Youth Risk Behavior Surveillance System

CDC established the Youth Risk Behavior Surveillance System (YRBSS) in 1990. One of the components is a national school-based survey that was first conducted in 1990 and has been repeated biennially since 1991. The national Youth Risk Behavior Survey (YRBS) measures six categories of priority health risk behaviors that contribute to the leading causes of mortality and morbidity among youth and adults in the United States: 1) behaviors that contribute to unintentional and intentional injuries; 2) tobacco use; 3) alcohol and other drug use; 4) sexual behaviors that contribute to HIV infection, other sexually transmitted diseases and unintended pregnancy; 5) dietary behaviors; and 6) physical activity.

The YRBS is administered in the spring to nationally representative samples of students in grades 9-12 attending both public and private schools. Professional data collectors, trained specifically for the YRBS, are used as field staff to ensure standard administration procedures. The YRBSS uses a three-stage cluster sample to select schools and classes of students within schools. African-American and Hispanic students are oversampled to provide accurate estimates for these subgroups in each survey cycle. By combining data from multiple survey cycles it is also possible to obtain accurate estimates for Asian and Native American youth. The sample size totals approximately 12,000 students per survey. School response rates average 76%; student response rates average 88%.

The YRBS questionnaire is designed for self-administration by use of a computer-scannable booklet. The questionnaire has been modified as needed to address emerging public health problems. A reliability study of the questionnaire conducted in 1993 demonstrated that students reported health risk behaviors reliably over time. Psychometric work has demonstrated that the questionnaire yields accurate and high-quality data. Standardized data editing and cleaning procedures improve data accuracy and consistency. Data are released within 12 months of data collection and are made available to the public via the Internet. A new psychometric study of the questionnaire is planned for Spring 2000.

Appendix C

Key Improvements in the CDC FY 2003 Performance Plan

As of December 31, 2001, CDC has achieved or exceeded targets set for 145 of the 217 performance measures in CDC's FY 2001 Performance Report. Only 20 targets were not met, and data is outstanding for 52 of the performance measures contained in the plan. Measures with outstanding data will be reported on as soon as results become available. We anticipate that we will have data available for 45 measures in CY 02 and 6 measures in CY 03; one measure will not be available until CY 04. However, at this point, CDC has achieved or exceeded 88% of its targets for which data is available.

Numbers tell only part of CDC's performance story. In an on-going effort to improve our performance plan and report, we have recently extensively revised our plan. Each section of the plan now addresses each of CDC's identity themes in greater detail. These themes are: protecting the health and safety of Americans, providing credible information to enhance health decisions, and promoting health through strong partnerships. In addition, more complete descriptions about CDC's programs, their intended results, and on-going activities have been provided.

In FY 2001, CDC achieved or exceeded a variety of goals in each of the identity theme areas.

Protecting the health and safety of Americans:

•	Protecting Americans from infectious diseases begins with well-staffed and well-equipped state and local health departments. In FY 2001 57 sites including 50 state, six local, and one territorial health department were funded for the Epidemiology and Laboratory Capacity (ELC) program to strengthen their capacity to identify and respond to public health threats. Because of outbreaks such as West Nile virus and the threat of other emerging infectious diseases, expanding this program to all eligible state and local health departments became a key priority.

•	In March 2000 a panel of experts reviewed extensive information from a variety of sources and concluded that measles is no longer an epidemic in the United States. The elimination of endemic measles from the US is an historic public health achievement and the fulfillment of a goal expressed by public health experts even before the first vaccine was licensed in 1963. In 2000 there were only 86 cases of indigenous measles reported in the US.

•	The US has seen dramatic reductions in perinatal HIV transmission rates in the past decade. In 1999, an estimated 300 to 400 babies were born with HIV infection, compared to 1,000 to 2,000 US infants born with HIV infection during the early 1990s. These declines reflect the success of widespread implementation of PHS recommendations for routine counseling and voluntary HIV testing of pregnant women and the use of zidovudine (AZT) by infected women during pregnancy and delivery and for treatment of the infant after birth. Revised guidelines were published in 2001.

•	The diabetes control programs (DCP) – funded in all 50 states, the District of Columbia, and eight territories – identify high-risk populations, improve the quality of care, involve communities in controlling diabetes, and increase access to care with measurable success. For example of a 2-year period the New York DCP reduced hospitalization rates by 35% and decreased lower extremity amputations rates by 39%. In Michigan a long-standing DCP has produced a 45% lower rate of hospitalizations, a 31% lower rate of lower-extremity amputations, and a 27% lower death rate for participants.

•	Because most fire-related deaths and injuries occur while residents are asleep effective detection and alerting systems are essential. Indeed a working smoke alarm can reduce the risk of death by about 50%. In 2.5 years CDC's 14-state smoke alarm installation/education program has installed over 100,000 smoke alarms in homes. This program has been credited with saving 150 lives.

Providing credible information to enhance health decisions:

•	In FY 2001 CDC continued to expand the information on the amount and types of environmental chemicals that affect people's health. CDC can now measure the presence of more than 200 such substances including metals, pesticides, dioxins, and others in blood and urine. To communicate these findings to the public, CDC produced a National Report on Human Exposure to Environmental Chemicals in FY 2001. This report provides the public an assessment of the US population's exposure to environmental chemicals that may cause cancer, birth defects, and respiratory diseases, and other illnesses. Information from this report will also aid in monitoring the effectiveness of programs designed to reduce exposures.

•	The health of America's communities hinges on the expertise of the public health workforce – they are our first line of defense against virtually all health problems in our communities. Training is an important aspect of improving public health performance. CDC continues to implement new systems to broaden training options nationally and internationally. CDC designed the Life-Long Public Health Learning System and increased training opportunities through distance-based leaning technology. CDC exceeded the distance-leaning performance target by expanding the range of programs offered and increasing the number of participants. The 135,000 distance-learning participants in FY 1999 surpassed the number targeted through FY 2002. In FY 2000 CDC increased the number of participants to 148,000.

•	CDC translates occupational research finding into various media for workers, employers, policy makers, and practitioners. In addition to expanding information on its website, the National Institute for Occupational Safety and Health (NIOSH) operates a toll-free telephone service to answer public inquiries – often providing life-saving advice to callers. Since 1995 calls to this line have increased by 60%. In FY 2000, inquiries on health care topics increased 100% over 1999.

Promoting health through strong partnerships:

•	Through the Global AIDS Program, CDC is working with experts from US and international agencies such as HRSA, USAID, CAREC, UNAIDS, WHO, and UNICEF to help ministries of health in Africa, Asia, and Latin America address the devastating impact of HIV/AIDS. In FY 2001 CDC expanded programs to support improve national surveillance programs to 22 countries and to prevent perinatal transmission of HIV to 13 countries.

•	CDC's Bioterrorism Preparedness and Response program is a crosscutting, collaborative effort among multiple CDC offices, ATSDR, FBI, FEMA, DOJ, APHL, FDA, USAMRIID, NACCHO, and VA. Through this program CDC provides funding to 50 states, four localities, and one US territory to enhance some or all of their areas of bioterrorism preparedness. CDC achieved or exceeded all targets for FY 01 bioterrorism measures.

•	As long as polio transmission occurs anywhere in the world, it remains a threat to American children. CDC continues to collaborate with many partners including WHO, Rotary International, USAID, the Task Force for Child Survival and Development, UNICEF, and other international agencies to bolster polio eradication efforts by providing scientific assistance and financial support. This collaboration is unique among public health initiatives for the unprecedented level of partnership. This global initiative is on target for certification of polio eradication by 2005. Global polio incidence has declined more than 99% from 1988 to 2000, about 250,000 lives have been saved and 4 million cases of childhood paralysis have been avoided, and the number of polio-endemic countries dropped from 125 to only 20 at the end of 2000. In 2000 the American Region of WHO completed its ninth year without a reported case of polio. CDC did not meet the target for purchase of polio vaccines in FY 2001 largely because the price of the vaccine increased by 27% thus decreasing the amount of vaccine CDC was able to purchase.

Appendix D

Performance Measurement Linkages with Budget, Cost Accounting,

Information Technology Planning, Capital Planning,

and Program Evaluation

Clinger-Cohen Act

CDC is currently implementing the requirements under the Clinger-Cohen Act of 1996 (CCA) for information technology (IT) capital investment planning, monitoring, and performance measurement. The Information Technology Investment Review Board (ITIRB) process has been established and was released CDC-wide on January 5, 1999, via the CDC Intranet. CCA compliance became a component of the CDC budget planning process for the FY2001 budget. Major IT investments associated with budget initiatives required responding to the "Raines Rules" as part of the submission.

Also in compliance with CCA, CDC has developed several components of the agency's information technology architecture, such as certain health data standards, networking and telecommunications architecture, information security, and the majority of the agency's administrative procedures. More extensive work on other core business processes, information flows, process and data models is ongoing.

In addition to efforts in the implementation of CCA, CDC has a well integrated GPRA and IRM Strategic Plan that aligns IT products and services with CDC's ever-changing mission needs and directions. The IRM strategic goals, strategies and performance measures support the mission, mission goals, and CDC's GPRA performance plan.