Detailed Information on the
Human Health Risk Assessment Program Assessment

Expand efficiency measure to include all major work products.

Implement new IRIS review process.

Investigate alternative approaches for measuring progress related to providing timely, high quality scientific assessments.

Reassess meaningfulness of current efficiency measure in light of recent National Academy of Sciences (NAS) report on efficiency measurement.

Implement regular, independent evaluations that assess the program's effectiveness specifically related to its influence on key risk management decisions made by the Agency's environmental media offices.

Measure: Utility of ORD's priority health hazard assessments for Agency, state and local risk assessors.

Explanation:This measure captures the assessment by an independent expert review panel of the relevance, quality, and use of the program's research under Long-Term Goal 1 (LTG1). Using a well-defined, consistent methodology developed through an OMB/ORD/Board of Scientific Counselors (BOSC) workgroup, the BOSC provides a qualitative rating and summary narrative regarding the performance of each Long-Term Goal. Rating categories include: Exceptional, Exceeds Expectations, Meets Expectations, and Not Satisfactory. Full ratings are expected approximately every 4 years, though the BOSC will provide progress ratings at the mid-point between full program reviews. Targets for this measure are set using the previous BOSC rating??and BOSC recommendations??as a guide. The program outlines an action plan in response to BOSC recommendations; completion of the actions in this plan demonstrates progress from the baseline. Long Term Goal 1 - Integrated Risk Information System (IRIS) and other priority health hazard assessments: Agency, state and local risk assessors are provided state-of-the-science health hazard assessment information on priority substances included in planned Agency decisions and actions.

Measure: Percentage of regulatory decisions in which decision-makers used HHRA peer-reviewed health assessments.

Explanation:The measure calculates the percent of Agency regulatory decisions for which clients use HHRA peer-reviewed health assessments. The measure is calculated by reviewing regulatory and ROD decisions made by EPA program offices in recent years, determining how many quantitative health assessment values were used in these EPA program decisions, and what percentage of these values had been developed by the HHRA Program. An additional stipulation is that the HHRA peer-reviewed health assessment values must not be out-of-date. Out-of-date is operationally defined as greater than ten years old and where new scientific information has been identified through the literature verification process that may change the health assessment value.

Measure: Utility of ORD's state-of-the-science risk assessment models, methods and guidance for EPA programs, states, and other risk assessors.

Explanation:This measure captures the assessment by an independent expert review panel of the relevance, quality, and use of the program's research under Long-Term Goal 2 (LTG2). Using a well-defined, consistent methodology developed through an OMB/ORD/Board of Scientific Counselors (BOSC) workgroup, the BOSC provides a qualitative rating and summary narrative regarding the performance of each Long-Term Goal. Rating categories include: Exceptional, Exceeds Expectations, Meets Expectations, and Not Satisfactory. Full ratings are expected approximately every 4 years, though the BOSC will provide progress ratings at the mid-point between full program reviews. Targets for this measure are set using the previous BOSC rating??and BOSC recommendations??as a guide. The program outlines an action plan in response to BOSC recommendations; completion of the actions in this plan demonstrates progress from the baseline. Long Term Goal 2- State-of-the-science risk assessment models, methods and guidance: EPA programs, states and other risk assessors use the risk assessment guidance, models, and methods provided to enhance the quality and objectivity of assessments through the incorporation of contemporary scientific advancements for use in decision-making.

Measure: Utility of ORD ISAs for providing best available scientific information on identifiable effects resulting from exposure to criteria pollutants.

Explanation:This measure captures the assessment by an independent expert review panel of the relevance, quality, and use of the program's research under Long-Term Goal 3 (LTG3). Using a well-defined, consistent methodology developed through an OMB/ORD/Board of Scientific Counselors (BOSC) workgroup, the BOSC provides a qualitative rating and summary narrative regarding the performance of each Long-Term Goal. Rating categories include: Exceptional, Exceeds Expectations, Meets Expectations, and Not Satisfactory. Full ratings are expected approximately every 4 years, though the BOSC will provide progress ratings at the mid-point between full program reviews. Targets for this measure are set using the previous BOSC rating??and BOSC recommendations??as a guide. The program outlines an action plan in response to BOSC recommendations; completion of the actions in this plan demonstrates progress from the baseline. Long Term Goal 3 - Integrated Science Assessments (ISA; formerly known as Air Quality Criteria Documents): ISA's are updated to reflect the best available scientific information on identifiable effects on public health and the environment outcomes from exposure to the criteria pollutants.

Measure: Percentage of planned outputs delivered in support of HHRA Technical Support Documents

Explanation:At the end of the fiscal year, the program reports on its success in meeting its planned annual outputs (detailed in the program's Multi-Year Plan). The program strives to complete 100% of its planned outputs each year so that it can best meet EPA and other partners' needs. To ensure the ambitiousness of its annual output measures, ORD has better formalized the process for developing and modifying program outputs, including requiring that ORD programs engage partners when making modifications. Involving partners in this process helps to ensure the ambitiousness of outputs on the basis of partner utility. In addition, EPA's Board of Scientific Counselors (BOSC) periodically reviews programs' goals and outputs and determines whether they are appropriate and ambitious.

Measure: Average cost to produce assessment documents.

Explanation:The HHRA Program's efficiency measure tracks the average cost to produce both IRIS assessments and Integrated Science Assessments (ISAs). This measure needs further development in light of the continuing evolution of the IRIS process.

Is the program purpose clear?

Explanation: The purpose of EPA's Human Health Risk Assessment (HHRA) Program is to provide timely, peer-reviewed health assessments of priority environmental contaminants to support science-based decision-making in EPA's regulatory and cleanup programs.

Evidence: The Agency's Strategic Plan (2006-2011) highlights the program and its purpose as a Means and Strategy supporting EPA's Goal 4: Healthy Communities and Ecosystems (see http://www.epa.gov/ocfo/plan/plan.htm). HHRA operates under EPA's Office of Research and Development (ORD) and in conformance with ORD's HHRA Multi-Year Plan that serves to direct HHRA's activities and to communicate the program's objective, goals and annual performance measures within ORD and with others.

Does the program address a specific and existing problem, interest, or need?

Explanation: Release of chemicals to environmental media (air, land, water) continues to pose potentially unacceptable risks to human health and the environment. EPA is responsible for mitigating these risks through regulatory and remedial risk management actions. Risk assessments, along with other relevant information, serve as the basis for regulatory decisions and are a critical component of risk-management decision-making. The HHRA program provides a common scientific foundation for human health risk assessment.

Evidence: Current needs are evidenced by analysis of Superfund Records of Decision from 2003 - 2006 that indicate that no contemporary risk value was available for approximately 54% of EPA's decision-making needs. HHRA's peer-reviewed health assessments are integral to EPA's review of the adequacy of current air standards and to the evaluation of the health risks posed to Americans living in areas designated as non-attainment for at least one of the criteria air pollutants. EPA, using health assessments coordinated between the HHRA program and the EPA Office of Water, is currently working to establish additional risk-based standards for microbial contaminants, disinfectants, disinfection by-products, and microbial pathogens or other contaminants found in drinking water distributional systems.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The HHRA program is the only federal program that provides qualitative and quantitative assessments of both cancer and non-cancer risks. The HHRA program does not excessively overlap other federal or non-federal efforts. Coordination is maintained with the Department of Health and Human Services' Agency for Toxic Substances and Disease Registry (ATSDR) regarding public health issues at Superfund sites. The Program's Integrated Risk Information System (IRIS) was established to mitigate intra-agency overlap by coordinating the development and review of these Agency-wide assessment needs for multimedia substances.

Evidence: A potential overlap between ATSDR and EPA is limited to developing the subset of non-cancer toxicity values for major pollutants at Superfund sites. This overlap is being addressed through an ATSDR/EPA Memorandum of Understanding (MoU), in place since 2004. MoU activities currently focus on shared literature searches, the exchange of scientific information, and coordination of document reviews, with continuing discussion on further means to coordinate outputs.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The Board of Scientific Counselors (BOSC), a committee established by EPA's Office of Research and Development (ORD) under the Federal Advisory Committee Act (FACA), provides independent, unbiased recommendations regarding program design and structure. Agency management acted on BOSC recommendations in 2005 to redesign the HHRA program to focus explicitly on providing peer-reviewed health assessments. This objective is achieved through (1) the preparation of hazard identification and dose-response assessments under the Integrated Risk Information System (IRIS) program, (2) the preparation of air quality criteria documents (AQCDs) and (3) methods development and guidance to ensure that EPA risk assessment products are consistent with the state-of-the-science and information quality objectives.

Evidence: The program design is described in the HHRA Multi-Year Plan (MYP), which has been peer-reviewed by HHRA partners. The 2005 MYP reflects recent changes in program design in response to two independent reviews by the BOSC.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The HHRA Program establishes its IRIS priorities based on declarations of need from EPA programs. An annual call is made for nominations of high priority chemicals for assessment or reassessment under the Agency's IRIS Program. The IRIS staff compile Agency nominations and work with appropriate EPA Program Offices and Regions to determine commitments for the coming and future fiscal years. Annual operating plans and monthly review of funds assure that resources are invested as planned. To assure the outputs reach the intended beneficiaries, completed IRIS assessments are disseminated on the internet. The schedule of AQCD assessments is based on statutory requirements. AQCD progress including project plans and all public drafts are posted on the EPA National Center for Environmental Assessment (NCEA) Web site.

Evidence: This IRIS priority-setting process culminates in publication of the annual IRIS agenda in the Federal Register. An example notice [FRL-7880-9] can be found at: http://www.epa.gov/ord/orma/part2/Q_1.5/EVIDENCE_2a_2005_IRIS_FRN.pdf. IRIS assessments can be found at: IRIS Web site http://www.epa.gov/iris. AQCD progress and deliverables are posted at: NCEA Web site http://www.epa.gov/ncea.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The HHRA Program has two specific long-term performance measures that reflect the program purpose of providing timely, peer-reviewed health assessments of priority environmental contaminants. The program is measuring (1) the percentage of Agency regulatory decisions in which clients use HHRA's peer-reviewed health assessments, and (2) the timeliness of critical-path documents required by EPA's Office of Air and Radiation to meet statutory deadlines.

Evidence: The Agency has committed to including these measures in forthcoming GPRA documents, including the 2007 Performance and Accountability Report. These output measures are accepted by OMB because they directly support the program's long-term goal of identifying and synthesizing the best available scientific information to support Agency guidance and policy decisions related to the health of people, communities and ecosystems. The Agency and OMB have agreed on a methodology and data sources for implementation of these measures.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has ambitious targets and has defined a baseline for both of its long-term measures. The target for the use of health assessments in decision-making will require the program to address complex scientific issues in a timeframe that is useful to their clients within the Agency. The timing target for completion of AQCD peer review is ambitious given the complexity of the issues, the difficulty of reaching consensus across multiple partners, and the requirement to routinely update assessments.

Evidence: The HHRA program's target for increased use of its peer-reviewed health assessments in Agency regulatory decisions is 48% by 2008 and 52% by 2010, over the baseline of 44% use. The program's timing target for AQCDs (complete peer review at least 60 days prior to publication of the associated draft Staff Paper) was developed jointly with the EPA office that produces the Staff Paper.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The Human Health Risk Assessment program has three annual performance measures to track annual completion of milestones that contribute to the long-term outcome of EPA programs using the program's peer-reviewed health assessments in decision-making. These annual measures tie directly to the program's research milestones that are determined through the multi-year planning process and are published in the program's Multi-Year Plan. Progress against these measures is monitored quarterly and reported annually.

Evidence: The Agency has committed to including these measures in forthcoming GPRA documents, including the 2007 Performance and Accountability Report. The Agency and OMB have agreed on a methodology and data sources for implementation of these measures.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has defined targets and baselines for each of its annual performance measures. The targets for the annual measures are milestones created through the multi-year planning process. The Multi-Year Plan is updated periodically with new outputs put into place for out-years (3-4 years in advance). The current-year milestones listed in the plan at the beginning of the fiscal year form the targets for the annual measures. The measure is calculated in a given year as percent of milestones completed relative to these targets. Targets are ambitious because changing conditions (e.g., loss of key personnel or Agency reprogramming of resources) present challenges to program management. Additionally, significant coordination across several organizational elements is required to meet all outputs on schedule.

Evidence: The Human Health Risk Assessment Multi-Year Plan, available at http://www.epa.gov/osp/myp/HHRA.pdf, lists the annual performance measures that constitute the target milestones. The Agency has committed to including these measures in forthcoming GPRA documents, including the FY 2007 Performance and Accountability Report.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The program's principal activities are carried out by EPA staff at the National Center for Environmental Assessment. Some contract support is used to perform key data analyses. The program's goals are incorporated in statements of work and contractor partners are required to submit regular performance reports. All contracts are reviewed/audited both internally and externally on a frequent basis.

Evidence: The program provided examples of contracts for data analysis and peer review activities that include text requiring a commitment on the part of those contractors to support the program's goals. Example reports for monitoring contractor commitments include progress reports and voucher checklists.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The program is assessed every 4-5 years by the Board of Scientific Counselors (BOSC). The BOSC comprises a distinguished body of scientists and engineers who may be drawn from academia, industry, non-EPA government or state agencies, and the environmental community. The BOSC was established under the Federal Advisory Committee Act (FACA) to provide unbiased and independent counsel to EPA's Office of Research and Development on the management and operation of its programs. Two BOSC reviews have been conducted to date, with a third scheduled for late 2006. These reviews address not only the scientific relevance and quality of the program but also evaluate program performance and improvements made since the previous review. The HHRA program uses feedback from the BOSC review and report to improve its program design, management, and performance.

Evidence: The BOSC Charter, review schedule and past reports are available online at: http://www.epa.gov/osp/bosc/charter.htm.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The Human Health Risk Assessment (HHRA) program's budget and performance goals are clearly presented in a stand-alone Program-Project in EPA's budget execution system. The program is housed within a single organizational unit; the National Center for Environmental Assessment in EPA's Office of Research and Development. Annual budget requests to Congress demonstrate the linkage between resources requested and annual and long-term performance goals. The program does not generate any costs or requirements that must be absorbed by other programs.

Evidence: The Agency accounts for all direct and indirect costs, including work year costs, for this program in the annual operating plan and in the annual budget materials provided to OMB. EPA's annual Congressional Justification includes a stand-alone description of the program which describes how program resources will be used to achieve program goals. The program's operating plan links its resources to its annual performance targets and the Agency's long-term goals. The program demonstrated the relationship between funding and performance levels through examples of unit cost estimates (including FTE) for its key products (i.e., various types of risk assessments).

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Based on the recommendations from internal and external management reviews, EPA's National Center for Environmental Assessment (NCEA) consolidated its risk assessment activities in 2003 to address strategic and operational deficiencies. The Human Health Risk Assessment Program (HHRA) was formed as part of that reorganization to foster a more integrated approach to resource allocation, prioritization and accountability for risk assessment. The HHRA Multi-Year Plan for research was recently developed to serve as the strategic plan for implementing the new annual and long-term performance goals of the program. Continued monitoring of the HHRA Program's strategic direction is accomplished through weekly planners' meetings by the NCEA management, coordination with the EPA Office of Research and Development planning process, and periodic independent reviews.

Evidence: The HHRA Multi-Year Plan, describing the rationale for consolidation and the new program structure is available at: http://www.epa.gov/osp/myp/HHRA.pdf.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: The HHRA Program does not engage in technology demonstrations or other industry-relevant research.

Evidence:

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The program follows an internal prioritization process to guide budget requests and funding decisions. External advisory bodies such as the EPA Science Advisory Board and the Board of Scientific Counselors are consulted and their recommendations are considered in priority-setting. The Multi-Year Plan, which guides programmatic research and resource allocation, explicitly states the program's goals and priorities. The program's priorities are strongly influenced by client needs. Nominations for Integrated Risk Information System (IRIS) assessments originate from the other EPA programs and regional offices. These nominations are then prioritized using an IRIS selection process that is documented in the Federal Register. Assessments for Air Quality Criteria Documents are prioritized based on EPA's need to meet the Clean Air Act legal mandates.

Evidence: The program is able to identify a set of current priorities that is a subset of activities covered in their Multi-Year Plan. An example of the Federal Register Notice describing the IRIS chemical selection process can be found at: ENVIRONMENTAL PROTECTION AGENCY, [FRL-7880-9] Integrated Risk Information System (IRIS); Announcement of 2005 Program; Request for Information. The program maintains a performance planning and tracking chart for Air Quality Criteria Document (AQCD) deadlines that guides their work on AQCD assessments.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Human Health risk Assessment (HHRA) program collects performance information monthly from support contractors and quarterly from EPA's program and regional offices and the HHRA managers. The HHRA Program compiles performance information in the Integrated Resource Management System (IRMS). Performance information on HHRA's health assessments is also routinely tracked and reported to Agency headquarters and the public. Milestone status information is used to inform the annual planning process and to update the Multi-Year Plan. In creating the HHRA Program, this information was used to adjust resources and reconfigure its health assessment activities.

Evidence: The IRMS system enables program managers to pull real-time reports that show the status of all multi-year plan milestones which are the basis for the program's annual performance measures. IRIS Track, available at: http://cfpub.epa.gov/iristrac/index.cfm, allows the public, as well as Agency managers, to see the program's progress. Baseline data, including the performance of program partners, is collected (e.g., IRIS Track) and used to set baselines and meaningful, ambitious performance goals and to revise the process to improve partner performance.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The HHRA Program managers and partners are held accountable for cost, schedule and performance results by inclusion of performance evaluation criteria in annual performance reviews. The Director of the National Center for Environmental Assessment (NCEA) is the manager responsible for program results. The Center Director is held accountable for specific performance standards related to program goals through mid-year and end-of-year performance reviews conducted by the Deputy Assistant Administrator for Management in EPA's Office of Research and Development. For program partners operating under contracts and assistance agreements, statements of work, deliverables, costs, and schedules are written into award terms. Contractors are explicitly held accountable for deliverables, costs, and schedules in evaluation criteria. The HHRA Program monitors its contract and assistance agreement partners through progress reports. Project officers and work assignment managers use checklists to assist in monitoring invoices. ORD project officers, work assignment managers, and project managers are responsible for seeing that agreements are awarded and managed according to government regulations. All staff involved in management or technical oversight of extramural vehicles are required to take the appropriate contract and assistance agreement training and to keep certifications current. ORD conducts periodic reviews of NCEA's management of contracts and assistance agreements.

Evidence: Sample performance agreements for managers show clearly defined or quantifiable standards for cost, scheduling, and performance results. Specific performance standards for program partner (contractors) are evidenced in contract documents which include performance criteria for contractor evaluations and acceptance of work.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: The Human Health Risk Assessment (HHRA) Program has consistently obligated its funds according to its operating plan with minimal unobligated balances remaining at year-end. HHRA management ensures that funds are spent for their intended purpose by allocating resources to the subordinate line organizations at the activity and object class level. During budget execution, HHRA reports and tracks obligations and expenditures in the Agency's Integrated Financial Management System (IFMS) against the Operating Plan. HHRA's partners are support contractors and collaborators. Assurance that funds are spent timely and for intended purposes is provided by project officer review of invoices to confirm that they meet contract or IAG requirements.

Evidence: Program appropriations are available for two-year periods. The percentage of available funds obligated for the last three two-year periods was 98.78% for 2002/2003, 99.13% for 2003/2004, and 98.47% for 2004/2005. Project Officers use the EASYLite Electronic Approval System (http://oasint.rtpnc.epa.gov/fmc2_prv/easylite.welcome) that allows them to approve invoices on line. This system gives instant validation of account totals, ensures account balances can not be exceeded, and indicates the amount actually paid by the Treasury. Project officers review and maintain copies of the invoices and confirmation of payment (via emails) from the finance center and documentation of payments are maintained on line in the EASYLite system.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The Human Health Risk Assessment (HHRA) program routinely uses several strategies to improve cost-effectiveness in program execution including use of the internet to disseminate its products, use of Standard Operating Procedures (SOPs) for each type of assessment, and regular analysis of administrative operations in order to identify and implement efficiencies (e.g., competitive sourcing). The HHRA program has an efficiency measure (with baselines and targets) in place that tracks the average cost to produce Air Quality Criteria Documents (AQCDs), also called Science Assessments. AQCDs are one of the two major types of assessments produced by the program. The program is also working to develop a similar measure for its Integrated Risk Information System (IRIS) assessments.

Evidence: HHRA products are disseminated to EPA client offices and the public via internet websites (e.g., http://www.epa.gov/IRIS). The program provided examples of IRIS SOPs and the Work Plan for Air Quality Criteria Science Assessments that guide consistent production of these products. Regular review of administrative process efficiency is evidenced by minutes from a meeting of the Information Management WorkGroup for the Office of Research and Development (September 14, 2005) moving to adoption of an electronic system for peer review plans and clearance tracking. The Agency has committed to including its efficiency measure in forthcoming GPRA documents, including the FY 2008 Performance and Accountability Report.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The Human Health Risk Assessment (HHRA) program is coordinating operations with ATSDR under a Memorandum of Understanding (MoU) which calls for the two programs to share the results of literature searches and analyses of chemicals of interest to both. Interagency working groups have sometimes been established to coordinate inputs to major health assessments, including methanol, formaldehyde, and perchlorate. However, lack of transparency and communication with stakeholders has sometimes resulted in costly and time-consuming re-work of the assessment. Expert recommendations (reported in GAO's recent examination of EPA's risk assessment process) were that increased involvement with a broad range of stakeholders early in the process would help identify alternative methods and models and obtain stakeholder concurrence with EPA's approach. The HHRA program is currently working with other federal agencies to address these recommendations, but a plan is not yet in place.

Evidence: GAO's report GAO-06-595 (May 2006) documented the recommendations of independent experts and interviews with EPA's own risk assessors on the question of how the risk assessment process at EPA could be improved. Some examples of ongoing collaborations include working with the Boron Institute and CIIT to develop data for the health assessments of boron and formaldehyde, respectively. HHRA also has extensive international collaborations, including with WHO, IPCS, IARC, and OECD. These allow HHRA to take advantage of data gathered and insights developed by scientists around the world.

Does the program use strong financial management practices?

Explanation: The Human Health Risk Assessment (HHRA) program follows EPA's financial management guidelines for committing, obligating, reprogramming, and reconciling appropriated funds. Agency officials have a system of controls and accountability (EPA's Resources Management Directives System), based on GAO, Treasury and OMB guidance as well as generally accepted accounting practices (GAAP), to minimize improper payments. The program is served by up to seven Funds Control Officers (FCOs) that have documented experience and/or training in EPA's budget execution and financial management systems. The program has no material weaknesses as reported by the Office of the Inspector General (OIG) and has procedures in place to minimize erroneous payments.

Evidence: EPA's Annual Reports and Financial Statements, including audit opinions, are available at: http://www.epa.gov/ocfo/finstatement/finstatement.htm. In their latest FMFIA report (FY 2005), the National Center for Environmental Assessment (the office that houses the HHRA program) certified that management controls were adequate and reported no material weaknesses. The program provided training records for Fund Control Officers to demonstrate their competence in financial management.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The Human Health Risk Assessment (HHRA) program is evaluated under the Federal Managers Financial Integrity Act (FMFIA) and other internal annual reviews of various management functions. Key management elements of the HHRA program have received independent, external review by a Board of Scientific Counselors (BOSC), an independent review panel convened regularly by EPA's Office of Research and Development.

Evidence: In 2004-2005, the program was reorganized to create a new management structure to oversee the development of health assessments, establishing several focus areas for its methods development support, and disinvesting in areas that did not directly support health assessments. Key management initiatives are documented in the Management Multi-Year Plan for EPA's Office of Research and Development. A BOSC review of management issues, as well as performance issues, is scheduled for late 2006.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The HHRA Program does not allocate funds to extramural parties for key research. The quality of the individual health assessment outputs of the HHRA program is ensured and maintained by rigorous peer review. Individual scientists conducting intramural work for the program have specific performance standards related to their assessments and are rated on the quality and timeliness of their work through EPA's Performance Appraisal and Recognition System.

Evidence:

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Human Health Risk Assessment program has two new long-term measures with baselines established by 2005. The first ambitious targets were established for 2008. The program has performed extensive analysis of the data available since 2005 and the results indicate that the program is on a path to meet its 2008 targets.

Evidence: The program submitted to OMB the "Metrics and Measures Support Document" (September 2006) that compiles the data collected to date for the program's measure of the percentage of its risk assessment products used by decisionmakers. The program also provided the peer review schedule of its past and upcoming Science Assessment reports. This schedule is the basis for future targets assigned to their second long-term measure.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The Human Health Risk Assessment program has three new annual measures with baselines established by 2004 and targets first established in 2007. Although there are no targets for comparison before 2007, actual results for 2005 and 2006 show improvement or optimal performance over time.

Evidence: In 2006, the program completed 100% of its planned outputs in support of its goals for Human Health Assessments and Air Quality Criteria/Science Assessments (two of its three performance measures). Under the third performance measure, the program completed 81% of its planned outputs for technical support documents, up from 44% in 2005.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Human Health Risk Assessment (HHRA) program has one new efficiency measure with a baseline established by 2004 and targets first established in 2007. Only a portion of the program's work is captured by this measure, but the program is working to broaden the measure as one of its followup actions. For the measure in place, there are no targets for comparison before 2007, but actual results for 2005 and 2006 show improvement over time. In terms of process efficiency, the HHRA Program has completed two cost-saving administrative improvements and a third is in progress.

Evidence: In 2006, the program nearly halved its average cost to produce Air Quality Criteria/Science assessment documents in 2005. Administratively, HHRA stopped managing its own IT support after 2004 and began using a centralized service that provides consolidated computer infrastructure and maintenance. After analysis in FY2005, HHRA conducted a limited competitive sourcing competition, resulting in savings to the government. HHRA management is now analyzing the benefits to consolidating the administrative support functions in two of its organizational units.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: In 2001, the Government Accountability Office (GAO) conducted a comparative review of human health risk assessment procedures in EPA, FDA, OSHA and DOT. The review focused on process and did not compare the effectiveness of each agency's risk assessment program with respect to successful support of the Agency's mission. The review provided only general comment on the importance of transparency and concerns regarding methods to handle uncertainty.

Evidence: GAO report GAO-01-810 (August 2001) investigated risk assessment procedures and process in four federal agencies (EPA, FDA, OSHA and DOT).

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: In response to recommendations by the independent Board of Scientific Counselors (BOSC) convened by EPA's Office of Research and Development, EPA undertook a major realignment of its human health risk assessment program in 2004, consolidating activities that were previously conducted under many different programs. There has not yet been a BOSC review of the newly structured program, although one is planned for 2006/2007. Positive indications that major program elements are achieving results are documented in earlier BOSC reports, however, these results represent program outputs (e.g., use of Integrated Risk Information System by risk assessors). A key product of the program, Air Quality Criteria Documents (AQCDs), is evaluated by the independent Clean Air Scientific Advisory Committee (CASAC). CASAC reports have made statements supporting the suitability of the AQCDs as the scientific basis for health-based standards in air.

Evidence: The 2003 BOSC report indicated that the National Center for Environmental Assessment (NCEA), which houses the HHRA program, had made several key advancements since the 1998 BOSC report. Example letter reports from CASAC to EPA's Administrator document the results of their review of AQCDs.