Detailed Information on the
Agency for Toxic Substances and Disease Registry Assessment

Explicitly tie budget requests to the accomplishment of annual and long-term goals, and present resource needs in a complete and transparent manner.

Tracking and reporting progress on programmatic and administrative cost-efficiencies

Measure: Reduce the average cost per site to deliver public health findings and recommendations to the public.

Explanation:In the event of a known or suspected public health threat, the timeliness with which critical information is delivered to the public may greatly influence the speed with which the American people can take protective actions. Toward this end, ATSDR is working to provide critical public health findings and recommendations to the public in the most expedient manner. Historical data demonstrate that ATSDR's health consults (HCs) can be conducted in a fraction of the time (and therefore at less cost) required to conduct public health assessments (PHAs). In many cases, HCs are sufficient to provide the public with the information they need, therefore ATSDR is working to increase the proportion of sites that are addressed with HCs rather than PHAs, where appropriate. Through ATSDR's efforts to address more sites with health consults, ATSDR has achieved a total cost savings of $2.7 million for FY 2005 and FY 2006 as compared to the FY 2004 baseline value. The reduction in man-hours allows ATSDR to use this available manpower to either increase the number of sites addressed or spend more time on complex sites that may require more thorough investigation and analyses. ATSDR expects to achieve a 30% decrease in the average cost per site to deliver relevant public health findings and/or recommendations by 2012 through a continued emphasis on conducting HCs rather than PHAs.

Measure: Advance understanding of the relationship between human exposures to hazardous substances and adverse health effects by completing toxicological profiles for substances hazardous to human health.

Explanation:A significant part of ATSDR's work is determining the relationship between human exposures to hazardous substances and health effects. As required by law, ATSDR prepares toxicological profiles (ToxProfiles) for hazardous substances found at the National Priorities List (NPL) sites and upon request from the scientific community. This "Priority List of Hazardous Substances" is a catalog of the hazardous substances most commonly found at facilities and those that pose significant potential threat to human health. Hazardous substances may be added or deleted from the NPL annually, therefore each year there may be substances for which ToxProfiles must be developed. Each profile provides a summary and comprehensive evaluation, and an interpretation of available scientific information on a substance. Because ToxProfiles are intended to be comprehensive in nature, when there are insufficient data to provide a complete picture of the health effects of a toxic substance, ATSDR identifies what data are needed, and works to collect needed information to complete the profile. This measure tracks the number of identified data gaps that are resolved annually.

Measure: Reduce exposures to toxic substances and mitigate the likelihood of future toxic exposures by increasing EPA's, state regulatory agencies', or private industries' acceptance of ATSDR's recommendations at sites with documented exposures.

Explanation:ATSDR responds to toxic substance releases when they occur or as they are discovered. One of the agency's primary responsibilities during these events is to provide information and recommended actions, from a public health perspective, to the agency or industry responsible for cleaning up the released toxins in the environment and/or mitigating the likelihood of future releases. This serves as a proxy outcome measure, as ATSDR serves in an advisory capacity, with no regulatory or enforcement authority. The protection of the public's health from toxic substance releases is dependent on the extent to which 1) ATSDR's recommendations are adopted by those entities that do have enforcement authority, such as EPA and state regulatory agencies, and 2) private industries adhere to relevant recommendations and regulations. This measure reports the percentage of ATSDR's public health and safety recommendations that are accepted by EPA, state regulatory agencies, and private organizations. The annual results may fluctuate as decisions are made regarding pending adoption of ATSDR recommendations.

Measure: Protect human health by preventing or mitigating human exposures to toxic substances or related health effects at sites with documented exposures.

Explanation:ATSDR's Goal 3 outcome measure captures the impact of the agency on human health in communities where actual or potential exposures exist. The long-term measure tracks the percentage of sites where human health risks or effects have been mitigated. The measure compares documented human health risks or effects at the time of the initial site assessment to those after intervention, thus measuring the reduction in people's actual or potential exposures. Depending on the toxic substance(s) and route(s) of exposure, the impact of interventions on human health can be measured through the following: ?? Morbidity/Mortality rates that measure, for example, the reduction in childhood cancer or birth defects rates. ?? Biomarkers, which signal the presence of toxic substances in the body, are used in cases where reliable and affordable tests are available. ?? Environmental monitoring that measures reduction in environmental contaminants to below levels of human health concern. ?? Behavioral change that documents changes in behavior that prevent future exposures.

Is the program purpose clear?

Explanation: The Agency for Toxic Substances and Disease Registry (ATSDR) has a clear mission. ATSDR's work falls into four functional areas: (1) protecting the public from toxic exposures; (2) building the scientific knowledge base about toxic substances; (3) delivering health education about toxic chemicals; and (4) maintaining health registries. ATSDR's mission is aligned with its principal legislative mandates. Since the creation of ATSDR, hazardous sites have been identified around the country. ATSDR also helps protect people from acute toxic exposures caused by hazardous leaks and spills, environment-related poisonings, and natural and terrorism-related disasters.

Evidence: The program has a clear mission statement: "To serve the public by using the best science, taking responsive public health actions, and providing trustworthy health information to prevent harmful exposures and disease related to toxic substances." ATSDR's mission is aligned with its principal legislative mandates, the Comprehensive Environmental Response, Compensation and Liability Act of 1980 (CERCLA, also known as the Superfund law) and the Superfund Amendments and Reauthorization Act of 1986 (SARA). CERCLA legislation designates ATSDR as the lead public health agency responsible for implementing its health-related provisions. SARA further expanded ATSDR's responsibilities in environmental public health, such as conducting public health assessments, establishing and maintaining toxicologic databases, disseminating information, and providing medical education.

Does the program address a specific and existing problem, interest, or need?

Explanation: The program addresses specific existing and potential problems and needs related to the impact of hazardous substances on human health. ATSDR is the lead public health agency responsible for responding to hazardous waste sites and emergency releases of toxic chemicals. During FY 2006, ATSDR and its state partners were active in over 500 sites annually throughout the United States and served over 3.2 million people in 346 communities. This number represents the number of people identified as having been potentially or actually exposed to contamination.

Evidence: The Comprehensive Environmental Response, Compensation, and Liability Act (CERCLA) requires the program, in conjunction with the Environmental Protection Agency, to revise and update a priority list of hazardous substances every two years. These substances are most commonly found at sites that pose the most significant potential to human health because of their known or suspected toxicity and their potential for human exposure.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The program has a unique role as the nation's public health agency for chemical safety. ATSDR was established through the Superfund legislation, which designates ATSDR as the lead public health agency responsible for implementing the health-related provisions of CERCLA. Although it shares common concerns with other federal agencies, the program has a unique focus. At the state and local levels, ATSDR provides a unique, national-level coordination nexus of state and local environmental public health efforts through technical assistance and funding to its 31-member Cooperative Agreement Program.

Evidence: In the area of toxic substances, other federal agencies??such as the Environmental Protection Agency (EPA), the National Institute of Occupational Safety (NIOSH), the Chemical Safety and Hazard Investigation Board (CSHIB), and the National Institute for Environmental Health Sciences (NIEHS)??address substances in the environment and/or the workplace. ATSDR concentrates almost exclusively on the human health effects of substances in the environment. In addition, while many state and local agencies have environmental programs, their staffs typically do not have ATSDR's level of scientific expertise in assessing environmental public health issues.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program is free of design flaws that could prevent it from achieving goals and objectives. ATSDR's organizational structure addresses the major requirements of the Agency as outlined under Superfund. The program's offices and divisions work together to provide a full spectrum of site-related services and products, as well to provide technical support to the Agency's Cooperative Agreement Program. In addition, ATSDR's regional staff members are co-located with EPA in 10 regions to facilitate coordination of site-related activities. ATSDR employs staff with expertise in toxicology, epidemiology, health education, and environmental health science.

Evidence: No known evidence demonstrates that another approach or mechanism would be more efficient or effective to achieve the intended program purpose. The program achieves its goals and long-term measures using a cascading performance measurement system, which includes the alignment of employee objectives to agency goals.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The program is designed to support efficient and effective use of resources, with resources targeted to areas of highest public health risk. The program dedicates its site-related resources to the Superfund sites on the National Priorities List (NPL), petition sites, and other sites where ATSDR has identified a public health need. ATSDR funds its partners according to need as well as merit, and can demonstrate that funding is being directed to areas of greatest concern. Without ATSDR, these activities would likely not be conducted.

Evidence: The program has selected the 31 Cooperative Agreement partners because they account for 80 percent of the toxic waste sites in the country. ATSDR's structure reflects the disciplines needed to provide site-related support. That structure includes a regional component of staff dedicated to and located within 10 EPA regional offices as a means of providing readily available support to constituents and state and local agencies. ATSDR distributes information and findings by publishing public health assessments, health consultations, and other materials. These materials are distributed directly to communities (e.g., copies at public libraries) as well as via the media and the web. The agency also conducts medical grand rounds and other training for physicians in communities exposed to toxic chemicals.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has established three long-term performance measures that address the range of agency operations. ATSDR initiated a strategic planning process in FY 2001 to define its strategic direction. During ATSDR's FY 2005 PART audit, the Agency re-focused its goals and measures on public health outcomes. These long-term measures have five year time frames, and measure public health outcomes, not outputs.

Evidence: Complete information on long-term performance measures can be found in the measures tab of this PART assessment. In summary, the performance measures evaluate the program's performance in preventing exposure to toxic substances, increasing understanding of human health effects related to exposures to toxic substances, and mitigating the risk of human health from toxic exposures.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has specific quantifiable targets for the performance measures described in Question 2.1. Baselines have been established for all three long-term measures. The targets are long-term, with an end target of 2012. The targets assume improved performance on an annual basis.

Evidence: Complete information on the targets and timeframes established for achievement of the long-term performance goals can be found in the measures tab attached to this PART assessment. Quantitative performance targets and baseline levels are included within the measures tab.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has developed a set of annual performance measures to assess progress toward achieving its long-term outcome measures. Each of the long-term outcome measures is accompanied by a corresponding annual measure that tracks progress towards achievement of the long-term outcome measure. These annual measures are focused on public health outcomes, not outputs.

Evidence: Complete information of the annual performance goals can be found in the measures tab accompanying this PART analysis.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has established baselines and targets for its annual performance measures. Clear baselines exist from which to project targets and assess changes in performance. The baselines are based on historical and trend data which also assist in the setting of targets. The annual performance targets assume improvement each year over the prior year. Note that the annual target for goal two appears to be flat, but this is due to five-year research cycles. For the five years being measured, performance will be 64 percent higher than the most recent five-year research cycle available.

Evidence: Complete information on the baselines and annual targets for measures can be found in the measures tab accompanying this PART assessment.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: State grantees, voluntary research partners, and other government partners commit to and work toward the program's annual and/or long-term goals. Since the implementation of the program's performance measures, program partners have actively participated in reporting performance results. Program staff are provided to serve as a national liaison for communications and partnership-building to: a) develop focused strategies to jointly address emerging environmental health concerns; b) develop a vision to inform forward strategic planning, to include the evaluation of potential proposals for regulatory or statutory change; and c) promote cooperation and alignment in community activities across the country conducted by EPA, NCEH/ASTDR and other federal, state, and local government partners.

Evidence: The program supports funding to its partners through a number of grants and cooperative agreements. Notices of Funding Availability are published in the Federal Register requiring each project proposal to incorporate measurable outcomes that are specifically tied to at least one of ATSDR's long-term goals. Grantee proposal applications and Annual Plans of Work (APOWs) are aligned with ATSDR's long-term goals, demonstrating a commitment on the part of grantees to work towards accomplishing ATSDR's performance goals. The program has also taken steps to formalize evaluation with its partners through outcome evaluation guidance.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The program has undergone a number of independent evaluations and peer reviews of its programs. These evaluations are of high quality and sufficient scope. Beginning in 2003, ATSDR instituted a formal process whereby every program in ATSDR (as well as the Centers for Disease Control and Prevention) undergoes an external peer review at least once every five years by the members of the Agency's Board of Scientific Counselors (BSC). The BSC is an independent group of scientists appointed by the Secretary for Health and Human Services. Participants on each workgroup are selected on the basis of the particular expertise needed to evaluate the program under review. These reviews document the central roles of the program, describe program contributions and challenges, and provide recommendations on how programs can improve. Since 2002, programs in three of four ATSDR divisions have undergone BSC peer review. In addition, the program has received additional evaluations of several key program components.

Evidence: Evidence includes a BSC peer review of the Hazardous Substances Emergency Events Surveillance Program, conducted as part of the program's regularly scheduled peer review process. In addition, as part of its review and evaluation of HHS performance at Department of Energy nuclear facilities, the National Research Council evaluated ATSDR contributions under the MOU between HHS and DOE. Peer reviews were also conducted on every ATSDR and ATSDR-funded study and research result as required by CERCLA section 104 (i) (13), and amended by Superfund Amendments and Reauthorization Act of 1986 (SARA). In 2005, an expert panel evaluated ATSDR's water-modeling activities and made suggestions for improvement.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: While the program does an admirable job at strategic planning and displaying performance information in budget documents, the budget documents do not include the information needed to achieve a Yes answer for Question 2.7. Budget documents do not make a clear link between resources and the achievement of performance goals. Budget documents do not clearly indicate the full costs of achieving performance goals.

Evidence: The program's performance budget to Congress detailing programmatic activities, funding, and performance information. The full document can be accessed from www.cdc.gov.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Despite the program's solid record in strategic planning, the program continues to seek out opportunities for improvement. ATSDR has dedicated methodologies, personnel, processes and systems in place to improve its strategic and performance planning processes.

Evidence: Evidence includes designated "Champions" within each office and division, the creation of a strategic planning team within the Office of Policy Planning, and Evaluation (OPPE), and a performance planning and evaluation process that is embedded into the culture of the organization. Office and Division Strategic Planning Champions lead the efforts to develop and monitor annual performance plans that are developed for the three-year federal budget window. ATSDR's process to support integration and improvements relies on evaluation and action tracking. The agency conducts formal mid-year and year-end evaluation meetings with Champions and office/division directors to discuss performance. Meetings are able to identify and highlight issues for the Assistant Administrator. Accomplishments, issues, and recommendations from the year-end evaluations are documented and tracked to ensure that the recommendations are addressed.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program has a formal process for collecting performance information and using the information to adjust priorities, improve management operations, and make budget decisions. ATSDR's performance-planning process includes the collection of both internal and external performance information. For each of ATSDR's offices and divisions, as well as its cooperative-agreement partners, ATSDR tracks performance on a regular basis and uses this information to make budget and resource allocation decisions, policy and/or program improvement changes, as well as grants-funding and award decisions. Cooperative agreement RFAs specify performance expectations. Regular communication with the grantees helps the vast majority of programs on track toward meeting mutual goals with the Cooperative Agreement Program. The program will occasionally encounter difficulties with grantees that require more training, more frequent discussions, face-to-face meetings, and negative evaluations. If none of these actions spur a grantee to improve performance, funding may be discontinued. The program solicits external input and guidance from its partners and external stakeholders to its strategic plan, and it involves many partners and stakeholders in program and product evaluations and reviews. In addition, ATSDR may make changes to its policies, programs, and/or procedures as a result of these inputs.

Evidence: The program produces annual performance reports that include accomplishments and recommendations. The final performance report is designed to be used to assist in the budget process for the following fiscal year. Technical reviewers provide regularly-scheduled, detailed feedback to agency grantees in performance evaluations that specify recommended actions and areas of needed improvement. These reviews assess partner performance, respond to the program's performance reporting requirements, and monitor program progress in achieving goals. Additional evidence includes funding letters to grantees that were cut from the program's cooperative agreements due to poor performance.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Program managers and strategic planning "champions" are responsible for cost and schedule outcomes and performance results. Program partners are held accountable for cost, schedule, and performance results. Agency divisions are responsible for identifying near-, mid-, and long-term targets by specific program areas. In addition, Grants Administrators must adhere to the Procurement and Grants Office (PGO) guidance on post-award monitoring and ensure that grantees are in compliance with their specific grant agreements and applicable laws and regulations. The program evaluates progress of its partners through progress reports and program evaluations. The program may take and has taken various measures to correct poor performance, including revising work plans or reducing funding.

Evidence: Evidence includes sample "cascading" work plans for senior program managers. Evidence also includes an agency-wide formal process for advertising, awarding, and monitoring grants to ensure that program partners are held accountable for achievement of program results. Notices of Funding Availability, published in the Federal Register, require that each project proposal incorporate measurable outcomes tied to at least one of the program's long-term outcome goals.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Funds are obligated in a timely manner and spent for the intended purposes. In FY 2006, the program obligated 99.82 percent of its funds. CDC's Financial Management Office (FMO) ensures that appropriated funds are properly obligated in a timely manner and that mechanisms are in place to ensure that funds are spent for the purpose for which they are intended. The program has mechanisms in place to ensure that program partners spend their funding for the intended purpose. The program defines measurable objectives and specific activities that will result in progress toward achieving public health impact. Aligned to these efforts will be the execution spending plans. The spending plans provide CDC with a detailed sketch of CDC estimated resources needed for the fiscal year by quarter. Each plan is then used to control the incurrence of obligations and is subject to strict fund control procedures. Reviews indicate that the agency successfully prevents erroneous payments.

Evidence: Evidence includes standard operating procedures of the budget execution branch at CDC, which explains efforts to ensure that spending plans are executes properly and support agency goals. Spending plans developed at the program level also serve as evidence. The spending plans are used to certify and monitor the status of funds at the program and agency level. Status of funds reports display the funds allotted to the program, and list obligations, commitments, and unobligated balances. CDC uses this information to monitor obligation rates and potential variances. Grantees develop budget justifications and spending plans that clearly state uses of funding to ensure that resources are obligated consistent with program goals. Risk assessments were completed to determine whether the program was susceptible to improper payments exceeding $10 million and a 2.5 percent error rate and required to estimate improper payments under the Improper Payments Information Act of 2002 (IPIA) and the related OMB Guidance.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program also has established a new efficiency measure with baselines and targets. The program has regular procedures in place to achieve efficiencies and cost effectiveness. The program has conducted several competitive sourcing actions, and also established a High Performing Organization within the CDC IT services office.

Evidence: The program has established a new efficiency measure, with baselines are targets. This evidence is included in the measures tab of this PART assessment. The measure will aim to improve the speed with which the program delivers findings and makes public health recommendations available to the public. Other major efforts to improve program efficiency include competitive sourcing studies to meet requirements of the President's Management Agenda and review and reorganization of organizational and reporting structures across the agency. Since beginning the A-76 program (OMB Circular, No. A-76, Performance of Commercial Activities) in FY 2003, CDC has completed 14 competitions. Thus far, A-76 studies have resulted in the contracting out of only the Printing Services functional area. CDC completed five studies of its Facilities Planning and Management Office in FY 2003 and prevailed in all five studies, indicating that through rigorous and complex analysis of work, CDC was performing at a cost to the taxpayers less than that of comparable service providers. In FY 2004, six A-76 competitions were conducted including: Animal Care; Laboratory, Glassware and Associated Laundry services; Office Automation; Printing; Materials Management; and Library Services. In FY 2005, three new standard competitions were announced for Computer Clerk Support, Statistical Support, and Writer and Editor Services. In addition to the standard competitions, the CDC Information Technology Services Office Restructuring Initiative was submitted to OMB as a High Performing Organization (HPO) and was the first HPO approved by OMB as a restructuring alternative to A-76.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program collaborates effectively with related Federal programs, and with relevant State and local government organizations, including EPA, the DoD, and DOE. Through MOUs and IAGs with a variety of organizations, ATSDR avoids duplication by delineating the services it will perform during cooperative efforts. ATSDR also leverages its resources by collaborating with many federal, state, and local agencies. The Agency offers technical and financial support to its partners at the state and local level through on-going cooperative agreements.

Evidence: Evidence includes MOUs and contracts with other relevant Federal agencies to outline and guide collaborations. Evidence also includes quarterly reports produced to detail activities with DoD and DOE. The program has also established and agreement with EPA to permit an ATSDR staff member to occupy a position at EPA's emergency response center, should an emergency occur.

Does the program use strong financial management practices?

Explanation: While the most recent independent audit for the Department of Health and Human Services identified two material weaknesses, these material weaknesses do not relate to the specific activities of this program. The program's financial management practices are strong enough to warrant a "Yes" response for this question. A new financial management system will provide timely and accurate financial statements in support of operations. Procedures are in place to minimize improper payments.

Evidence: In 2005, CDC implemented the new Unified Financial Management System (UFMS). UFMS is an integrated, Department-wide financial system that produces financial information to support decision-making and cost-effective business operations at all levels throughout the Department. UFMS replaced the legacy mainframe-based financial system, which was over 15 years old. Some of the direct benefits of implementing UFMS include providing programs with more real time data, streamlining processes to free up program resources to do more value-added work, and producing accurate, timely, and reliable financial reports, enabling CDC to make fact-based operational decisions. Program staff, contract and grant management personnel, and financial management personnel perform activities on a continuous basis to identify and manage improper payments. These activities include grant and contract oversight functions, site visits, prepayment reviews, voucher reviews, and quality control reviews. Furthermore, system controls were designed to reduce the risk of impropriety. System controls include limited access to data files and programs, required authorizations, and reconciliation checks. In addition to these on-going activities, CDC has performed a risk assessment of significant programs as required by the Improper Payment Information Act of 2002. Based on results of the FY 2006 risk assessment, the agency concluded that none of the programs reviewed were susceptible to significant improper payments.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program is continuing to make improvements to financial management processes, including restructuring its budget and financial accounting system. CDC and ATSDR initiated changes in core accounting practices, professional staff recruitment, financial systems training, and customer service. CDC commissioned a business case for timelines, cost estimates, and functional and technical solutions. CDC was integral in instituting HHS' Unified Financial Management System, which automates financial accounting processes. Several additional changes were triggered by the recent consolidation of the National Center for Environmental Health and ATSDR. For example, a number of positions were reclassified from administrative to front-line public health. Other positions were changed from supervisory to non-supervisory to eliminate smaller organizational units as part of a de-layering effort and to bring agency practice into line with HHS directives for supervisor-to-employee ratio.

Evidence: Evidence includes end-of-year performance review follow-up plans that include accomplishments and recommendations. Evidence also includes "Strike Teams" created by the program to improve the timeliness of the program's response at some toxic chemical sites. In addition, CDC has worked to improve accountability of Federal managers for program performance through the DHHS' Senior Executive Service (SES) and Organizational Performance Management System and the Program Management Appraisal Program (PMAP). Together, these performance management systems connect expectations to mission and link performance ratings with measurable outcomes.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: In accordance with CDC Procurement and Grants Office (PGO) regulations, 100 percent of the program's grants and cooperative agreements are competitively awarded. The evaluation of applications assesses not only the program's merit, but also the applicant's existing need. The evaluation process involves an objective review of all applications prior to selection. The program operates a fair and open competition for all grants, and conducts outreach to encourage the participation of new grantees through program announcements.

Evidence: Evidence includes documents detailing the formal process used by the program to evaluate grant applications. Evidence also includes documentation of the CDC-wide process that all programs must adhere to when advertising, awarding, and monitoring grants. Cooperative agreement awards are reviewed for technical merit by an Objective Review Committee.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program follows CDC's Procurement and Grants Office practices for overseeing and evaluating grants. The program requires semi-annual reporting on programmatic and financial performance from its grantees. Within the cooperative agreement program, grantees are required to report quarterly progress. Quality-assurance procedures are part of this process. Results of these reports are reviewed during monthly calls and are addressed with grantees during routine visits to the State or territory. At the end of each year, a formal evaluation of each program is conducted by ATSDR.

Evidence: Evidence includes sample site visit reports prepared by program staff evaluating and assessing grantee performance. Evidence also includes guidance on post-award monitoring to ensure grantee compliance with applicable laws and regulations. Grantees are required to submit reports.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program collects and compiles grantee performance data through its annual Cooperative Agreement Program evaluation. The evaluation assesses performance of Cooperative Agreement partners against the requirements of the program announcement, highlighting strengths and opportunities for improvement. The program provides grantees with guidance on documenting and evaluating the results of programmatic activities and collects/compiles outcome reporting forms from grantees. The program also conducts annual productivity tracking for each grantee and compiles the information by state across a number of factors. The program disseminates performance data through a number of mechanisms. For example, grantee performance is highlighted in the ATSDR Congressional Justification and in related performance reporting required for GPRA. The program also provides results of grantee performance on its web site. Health assessments and consultations completed by grantees are among the most visible products of grantee performance and these are available on the program's web site through an easy-to-access state map interface. In addition, the web site contains state fact sheets that provide summaries of grantee funding and accomplishments.

Evidence: Evidence includes the program's annual Congressional Justification of the agency's budget request, which can be found online at http://www.atsdr.cdc.gov/performanceplan/FY_2008_ATSDR_CJ.pdf. Evidence also includes public health assessments made available on the internet. These can be found at http://www.atsdr.cdc.gov/hac/pha/index.asp. State fact sheets providing budget and performance data can be found online at http://www.atsdr.cdc.gov/factsheets.html.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program has three long-term outcome goals to measure performance. The program is on track to meet all three of the long-term performance goals. Since the establishment of baselines, the program has made progress towards each of the long-term goals on an annual basis. For each of the goals, ambitious long-term performance targets have been established.

Evidence: Evidence supporting Question 4.1 can be found in the measures tab accompanying this PART assessment.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program has established annual outcome goals to support measurement of progress towards the achievement of the program's three long-term outcome goals. For each of the three annual goals, the program has achieved or exceeded expectations for the past several years.

Evidence: Evidence supporting Question 4.2 can be found in the measures tab accompanying this PART assessment.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program has established a new efficiency measure, with baselines and targets developed. In addition to the establishment of an efficiency measure, the program has implemented many administrative improvements that have increased efficiency and effectiveness.

Evidence: Evidence supporting Question 4.3 can be found in the measures tab accompanying this PART assessment. The program's efficiency measure tracks the amount of time it takes for the program to make public health information available to the public. From 2002 to January 2006, the program more than doubled its supervisory ratio from 1:5.5 to 1:12.6. The program has completed several competitive sourcing studies from 2003 through 2005, with CDC/ATSDR winning 13 of 14 studies. Multiple business services have been consolidated in recent years. In October 2004, the budget execution function was consolidated, resulting in a 20% reduction in staff. In FY 2005, graphics services were consolidated, resulting in staff savings of 38 FTEs and 6 contractors, in addition to a 32% cost savings. As a result of human capital and other CDC/ATSDR business services improvements, the agency has reduced its number of administrative staff by approximately 900 as of the end of 2005. This reduction of will allow the redirection of administrative staff positions to front line public health efforts.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: No other agencies exist that have a similar purpose or goals as ATSDR. Two key pieces of legislation make ATSDR's purpose clear. The first, the Comprehensive Environmental Response, Compensation and Liability Act of 1980 (also known as the Superfund law), designates ATSDR as the lead public health agency responsible for implementing its health-related provisions. ATSDR is charged with assessing health hazards at specific hazardous waste sites, with helping to prevent or reduce exposure and the illnesses that result, and with building the scientific knowledge and understanding of the health effects that may result from exposure to hazardous substances. The second, the Superfund Amendments and Reauthorization Act of 1986, broadened ATSDR's responsibilities in the areas of conducting public health assessments, establishing and maintaining toxicologic databases, disseminating information, and providing medical education.

Evidence: N/A

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: : Since 2003, the program has had a formal process in place whereby every activity within ATSDR (as well as the rest of CDC) is subject to an external peer review at least once every five years. These peer reviews are conducted by members of the agency's Board of Scientific Counselors, an independent group of scientists and academics. Of the three ATSDR programs that have been evaluated to date, two have received generally favorable results. Some evaluations have identified weaknesses, but in general, evaluation results are positive.

Evidence: A peer review of external scientists and academics found the Hazardous Substance Emergency Events Surveillance System (HSEES) program to have "excellent" performance and made "significant, positive, public health impacts attributable to the program." The workgroup expressed strong support for HSEES as it currently operates, but did recommend the development of strategic and evaluation plans, better coordination with other federal agencies, and helping states to expand collaborations between public health and environmental/occupational departments. In response, the HSEES program has implemented most of the peer review workgroup's recommendations. An additional peer review of the Division of Toxicology and Environmental Medicine (DTEM) found that the program "is performing at a generally high level" and is meeting the performance measures it tracks. The peer review report does point out, however, that the output measures in use by DTEM are insufficient for determining program impact. A similar independent evaluation of the National Exposure Registry (NER) in the Division of Health Studies found that the program was not performing effectively and the program needed an extensive restructuring. On the basis of the workgroup's findings, NER has been closed. An evaluation of ATSDR products/activities as part of a broader evaluation of an HHS program, the Worker and Public Health Program, which operates in collaboration with the Department of Energy (DOE), reported that ATSDR is generally meeting the needs of the DOE and the communities that live around Superfund sites. This evaluation was completed by the National research Council. Of the detailed evaluation of four ATSDR products??public health assessments (PHAs), health studies, toxicological profiles, and health education and promotion??toxicological profiles and health education and promotion received generally positive reviews. PHAs and health studies were also found to be useful, but areas for improvement were noted. The evaluation concluded that ATSDR products are "generally of high quality" and "have value" for the DOE program.