Detailed Information on the
Public Water System Supervision Grant Program Assessment

Implement recommendations from the second triennial drinking water data quality review which are designed to improve the overall quality of the data in EPA??s drinking water compliance reporting system.

Develop an efficiency measure that is more useful and meaningful for tracking annual programmatic efficiency.

Develop a new long-term outcome performance measure to assess the impact of drinking water compliance improvements on public health.

Measure: Percent of population served by community water systems that will receive drinking water that meets all applicable health-based drinking water standards through approaches including effective treatment and source water protection.

Explanation:This measure tracks the percentage of population served by community water systems that meet all applicable health-based drinking water standards. If systems meet such standards, the population's exposure to contaminants is reduced.

Measure: Percent of community water systems that meet all applicable health-based standards through approaches that include effective treatment and source water protection.

Explanation:This measure tracks the percentage of population served by community water systems that meet all applicable health-based drinking water standards. If systems meet such standards, the population's exposure to contaminants is reduced.

Measure: People receiving drinking water that meets all applicable health-based standards per million dollars spent to manage the national drinking water program

Explanation:This measure will be changed for PWSS, UIC, and DWSRF and will include EPA funds, UIC and PWSS grants (with associated state match) , DWSRF grant set-aside funds and DWSRF funds provided for assistance (including state match, bond proceeds, repayments and interest earnings).

Measure: Percent of community water systems that have undergone a sanitary survey within the past three years (five years for outstanding performers).

Explanation:Each year, all States are must be in compliance with the requirement to conduct sanitary surveys at community water systems once every three years, as documented by file audits of a random selection of water systems. Change in measure due to input from states. Better to measure CWS.

Is the program purpose clear?

Explanation: The Public Water System Supervision (PWSS) Grant funds are used by States to assist them with implementation and enforcement of their State PWSS Programs. The State PWSS programs implement and enforce the National Primary Drinking Water Regulations which are intended to assure that the public is provided an adequate quality of safe drinking water.

Evidence: SDWA §1443(a) ' authorizes EPA to make grants to States to carry out their PWSS programs SDWA §1413 ' outlines the primary activities of a State program. 40CFR142 subpart B ' outlines "Primary Enforcement Responsibility."

Does the program address a specific and existing problem, interest or need?

Explanation: Compliance with drinking water standards is critical to ensure that public drinking water supplies are safe and of adequate quality. The grant program transfers resources to States that take primary responsibility for implementing and enforcing the federally-mandated standards, a responsibility that otherwise falls to the federal government. The Association of State Drinking Water Administrators (ASDWA) report highlights the value and need for both State programs and financial assistance.

Evidence: SDWA §1413 allows EPA to delegate responsibility for enforcement of those standards to States. SDWA §1443 provides for Grants to States that have been delegated enforcement responsibility. ASDWA report: "Public Health Protection Threatened by Inadequate Resources for State Drinking Water Programs."

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: While there are other private/non-profit organizations (NRWA, RCAC) that partner in providing some technical assistance to water systems, only EPA and the States oversee and enforce the drinking water regulations, and the PWSS Grant Program is the "primary" source of federal funds for the implementation of the State PWSS Programs that accomplish this mission. While the federal PWSS Grant does not provide the only source of funding for State program costs, GAO reports provide evidence that there are insufficient resources available from other public and private programs to meet the need.

Evidence: GAO Report GAO/RCED-92-184 'Drinking Water: Widening Gap Between Needs and Available Resources Threatens Vital EPA Program' dated July 1992 GAO Report GAO/RCED-93-144 'Drinking Water Program: States Face Increased Difficulties in Meeting Basic Requirements' dated June 1993 GAO Report GAO/RCED-00-199 'Drinking Water: Spending Constraints Could Affect States' Ability to Implement Increasing Program Requirements' dated August 2000

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The authorizing statute allows States fairly wide latitude in structuring a State PWSS program. This flexibility is a more effective and efficient method of achieving compliance than imposing a rigid program structure because provision of safe drinking water is ultimately a local responsibility. However, the delegation of enforcement responsibility to States is fairly rigorous, so if States cannot or do not effectively implement the program, EPA can use its authorities to ensure compliance. There is no evidence that a purely federal program would be more effective or efficient.

Evidence: SDWA §1412 SDWA §1413 SDWA §1443(a) Geographic proximity to the regulated entities, variability in local needs, and experience in overseeing water systems that pre-dates the federal program suggest that state-centered programs are likely to be more efficient and effective than a centralized federal program.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: EPA uses an allotment formula that distributes the PWSS grant funds among States according to anticipated relative workload. Grant funds are annually allocated to States based on numbers of public water systems, population, and geographical area (all SDWA based). There is also a minimum amount guaranteed to each State regardless of the credit they receive through the factors and weightings. This minimum recognizes that there is a baseline funding need, to establish and maintain a PWSS program, in even the smallest State.

Evidence: SDWA §1443 EPA's grant regulations 40CFR35 form the basis for the allotment of funds. Primacy requirements mandate that States implement regulations for all system types. Additionally, State enforcement discretion allows States to prioritize enforcement resources toward higher risk public health problems in accord with the program's purpose.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has an outcome-based, long-term perfomance measure that supports the goals of "Water Safe to Drink" by reducing exposure to contaminants. The program implemented a new outcome efficiency measure that is included in the 2006 GPRA strategic plan. The program has committed to develop a high level outcome measure (based on GAO's Hierarchy of Indicators) to assess the reduction in waterborne disease outbreaks due to unsafe drinking water.

Evidence: The performance measure tracks the percent population served by community water systems that receives drinking water in compliance with health-based standards. The measure is listed in EPA's 2003 Strategic Plan as a strategic target to track progress on the proposed goal of "Water Safe to Drink." For the efficiency measures (see measures tab) a "Measures Implementation Plan" has been submitted to OMB.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The targets and timeframes for the outcome-based performance measure and efficiency measure are ambitious given the existing external factors that limit EPA control and measurement of program progress. The measures emphasizes the importance of sustaining compliance as well as returning systems to compliance.

Evidence: The baselines and targets provided in the measures tab for the long-term outcome measure are published in EPA's Strategic Plan, and the program's more detailed Subobjective Implementation Plan.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The long-term efficiency measures serve as an adequate measure of outcome efficiency for the program. The program has an outcome-based annual measure that tracks the rate of compliance of the nation's 53,000 community water systems with drinking water standards. If systems are in compliance, the population's exposure to contaminants is reduced. A new output measure is the percentage of community water systems where sanitary surveys are completed at least once every three years. Sanitary surveys, which are inspections of systems conducted by the States, aid in the process of evaluating a public water system's progress in complying with Federal and State regulations, and identify areas which require the improvement of the capabilities of the system to provide safe drinking water.

Evidence: The PWSS Grants program contributes to the performance measure proposed in EPA's 2003 Strategic Plan: Increase the percent of community water systems that meet Pre-2001 and Post-2001 drinking water standards.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The targets and timeframes for the outcome-based annual measure are ambitious given the the existing external factors that limit collection of reliable data. For example, a percentage of the systems do not submit compliance reports. In future years, EPA must reduce external factors to continue to receive a yes answer. Baselines, targets and timeframes for the new output measure have been developed and are ambitious.

Evidence: The baselines and targets provided in the measures tab for the annual outcome measure are published in EPA's Strategic Plan, and the program's more detailed Subobjective Implementation Plan.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Many Performance Patnership Agreements (PPAs) between EPA and Primacy States include commitments by the States to ensure that progress is achieved toward environmental goals in addition to supporting EPA's databases for tracking progress on performance goals and good stewardship of funds. The program has developed new guidance to encourage all State partners to incorporate these commitments in their PPAs. In future evaluations, the program will need to show progress in documenting commitments for all Primacy States and Tribes.

Evidence: Example PPAs from States/Tribes.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Independent evaluations are not conducted on a regular basis and those within the last ten years have focused on program management rather than a determination of whether program implementation is achieving outcome goals. For example, GAO reports have commented on States' implementation of sanitary survey activities, but not evaluated whether these activities resulted in improved compliance rates. Additionally, audit-based approaches do not meet the quality criteria for determing the impact of the program.

Evidence:

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The Agency estimates and budgets for the full annual costs of operating its programs, taking into consideration any changes in funding, policy and legislative changes. All spending categories and the resource levels and activities associated with them are included in the annual Congressional Justification. Presentation to Congress of the Agency's budget, including resources for the PWSS Grant program, include alignment to its Strategic Plan goals. A State resource needs model to estimate program implementation costs is under development, but currently the Agency is unable to show how changes in funding relate to changes in performance.

Evidence: Annual Congressional Justification. Budget Automation System (BAS) reports.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: EPA has taken steps to strengthen its oversight of States' management and use of the PWSS grant funds. The Agency has developed national guidance for the FY2005 grants directing that recipients ensure that grant funds are targeted toward achieving the Agency's performance goals.

Evidence: FY2005 Public Water System Supervision (PWSS) State Program Grants - Guidance and Tentative Allotments, dated April 13, 2004

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Data for outcome performance measures are compiled from the Safe Drinking Water Information System (SDWIS). EPA's Inspector General recently reported data quality problems with this reporting system and concluded that EPA is not accurately reporting its performance. No evidence is provided regarding use of these or other performance data to manage the PWSS Grant program.

Evidence: OIG, March 5, 2004. EPA Claims to Meet Drinking Water Goals Despite Persistent Data Quality Shortcomings, No. 2004-P-0008.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: EPA has designated PWSS program managers in all regional offices and at the national level. The PWSS staff and managers, at headquarters and in the regions, have responsibilities that are specified under performance standards through personnel performance agreement and related appraisals. Additionally, Federal regional grant project officers are held accountable for ensuring that all policies and procedures of the EPA Grants Administration Division are followed. Grantees are accountable through grant agreements with EPA for program costs. EPA Regions' annual review of state performance under the grants and audit results can be used to adjust grant conditions. State primacy can be revoked if its program does not meet the performance requirements of SDWA 1413.

Evidence: 40 CFR 35: EPA Grant Regulations EPA Grants Management Plan Performance Appraisal forms for Office Director, Branch Chief and Grant Officer.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Each Region's program office, and grant's management office, conducts post-award monitoring of assistance agreements that tracks the draw-down of funds and enables them to assess the timeliness of resource obligations and commitments. State grants are issued under Performance Partnership Grants or Performance Partnership Agreements that are subject to audit under the Single Audit Act. EPA's OIG reviews the audit findings and there are procedures in place to take administrative actions as needed.

Evidence: For FY2003, EPA reports that of the $92,674,100 PWSS grants allocated, $87,312,200 (94%) were obligated by the end of the fiscal year. Single Audit Act reports.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: Insufficient evidence that the program has efficiency measures or incentives in place that focus on process efficiencies or cost effectiveness. For example, although unobligated balances are low, the program does not appear look for ways to perform grants management with fewer resources.

Evidence:

Does the program collaborate and coordinate effectively with related programs?

Explanation: Coordination of the PWSS Grant programs in States with State Drinking Water State Revolving Fund programs, NGOs providing technical assistance to drinking water systems and federal enforcement entities is coordinated by specifying roles and responsibilities in State workplans, grant agreements with NGOs and PPAs. PWSS coordinates with related EPA programs (Drinking Water State Revolving Fund and Underground Injection Control programs) to work toward shared performance goals.

Evidence: Example PPAs from States/Tribes. EPA's 2003 Strategic Plan on the proposed goal of "Water Safe to Drink."

Does the program use strong financial management practices?

Explanation: The program follows EPA's financial management guidelines for committing, obligating, reprogramming, and reconciling appropriated funds. Agency officials have a system of controls and accountability, based on GAO and other principles, to ensure that improper payments are not made. At each step in the process, the propriety of the payment is reviewed. EPA trains individuals to ensure that they understand their roles and responsibilities for invoice review and for carrying out the financial aspects of program objectives. EPA received an unqualified audit opinion on its FY02 financial statements and had no material weaknesses associated with the audit. The PWSS Grant program has no material weaknesses as reported by the Office of the Inspector General (OIG) and has procedures in place to minimize erroneous payments.

Evidence: Annual Congressional Justification. Budget Automation System (BAS) reports. Unqualified audit opinion on EPA FY02 financial statements. Fiscal Year 2002 Advice of Allowance Letter. 2002 Integrity Act Report.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: EPA is developing an action plan to address data quality issues in SDWIS/FED. EPA has taken a number of steps to strengthen the management and oversight of its assistance agreements, focusing on improving competition in the awarding of grants and enhancing compliance review and monitoring of grants. To build upon these efforts, EPA has developed a Grants Management Plan, which brings ongoing planning efforts into one comprehensive document, establishing a clear and strong direction for the Agency.

Evidence: EPA, March 2004. Drinking Water Data Reliability Analysis and Action Plan (2003). Agency's Grants Management Plan www.epa.gov/ogd/EO/finalreport.pdf

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The PWSS Program, under the authority of Section 1443 of SDWA and codified in 40 CFR 31.40(b) requires grantees to submit information on the various activities they have undertaken over that year. In addition, Regions collect and review annual reports and conduct annual performance reviews of primacy state programs (40 CFR 142.17).

Evidence: Example End of Year Reports and Mid-Year Status reports.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Compliance data that is reported to SDWIS/FED is available to the public through EPA's Envirofacts Data Warehouse that can be accessed through the internet. EPA also posts the most recently completed National Compliance Report (which provides summary data on compliance results on a Stateby- State basis) on its own website.

Evidence: Envirofacts website: www.epa.gov/enviro/html/sdwis/sdwis_ov.htmlEPA website on drinking water data and databases: www.epa.gov/OGWDW/databases.htmlEPA FY2003 "Annual Report": www.epa.gov/ocfopage/finstatement/2003ar/2003ar.htmEPA FY2002 'Annual Report': www.epa.gov/ocfopage/finstatement/2002ar/2002ar.htm

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program is implementing a long-term outcome efficiency measure and is committed to a strategy and workplan. The DWSRF is progressing toward its long-term performance target although data quality issues suggest the numbers may under-report actual performance. It also met targets that measure relevant outputs that link to the long-term goal

Evidence: Summarized in measures tab.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The PWSS Grant program appears to have met its annual performance targets although data quality issues suggest the numbers may under-report actual performance. There are no data yet for the new annual output measures.

Evidence: Summarized in measures tab.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Compliance rates have improved slightly during the recent period when grant funds have been effectively level, even while State program demands have increased. However, there is no evidence to link this observation to improved program efficiencies. The program does not appear to have incentives and measures for cost efficiency or effectiveness.

Evidence:

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There are no other federal agencies or nongovernmental programs engaged in implementation and enforcement of drinking water regulations. The State programs are supported in part from the grants issued by this program, so a comparison of State and federal programs would not be meaningful.

Evidence:

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: No evidence that the program has been comprehensively assessed by an independent entity using methodology that meets the criteria for quality.

Evidence: