Detailed Information on the
Water Quality Research Assessment

Finalize ambitious long-term outcome performance measures, which assess the utility of the program's research products and services with respect to the outcome goals of its clients.

Develop and implement a protocol for more frequent review and use of financial and performance tracking data to improve budget and performance integration.

Reassess meaningfulness of current efficiency measure in light of recent National Academy of Sciences (NAS) report on efficiency measurement.

Identify appropriate targets for bibliometric analysis measures by benchmarking with other agencies.

Improve the collection of partner performance information to more clearly link to programmatic goals so managers can take appropriate actions to improve overall program performance.

Measure: Percentage of planned outputs delivered in support of the protection of human health and ecosystems as related to designated uses for aquatic systems and the beneficial use of biosolids long-term goal.

Explanation:At the end of the fiscal year, the program reports on its success in meeting its planned annual outputs (detailed in the program's Multi-Year Plan). The program strives to complete 100% of its planned outputs each year so that it can best meet EPA and other partners' needs. To ensure the ambitiousness of its annual output measures, ORD has better formalized the process for developing and modifying program outputs, including requiring that ORD programs engage partners when making modifications. Involving partners in this process helps to ensure the ambitiousness of outputs on the basis of partner utility. In addition, EPA's Board of Scientific Counselors (BOSC) periodically reviews programs' goals and outputs and determines whether they are appropriate and ambitious.

Measure: Percentage of planned outputs delivered in support of the diagnostics and forecasting techniques for the protection of human health and ecosystems as related to designated uses for aquatic systems and the beneficial use of biosolids long-term goal.

Explanation:At the end of the fiscal year, the program reports on its success in meeting its planned annual outputs (detailed in the program's Multi-Year Plan). The program strives to complete 100% of its planned outputs each year so that it can best meet EPA and other partners' needs. To ensure the ambitiousness of its annual output measures, ORD has better formalized the process for developing and modifying program outputs, including requiring that ORD programs engage partners when making modifications. Involving partners in this process helps to ensure the ambitiousness of outputs on the basis of partner utility. In addition, EPA's Board of Scientific Counselors (BOSC) periodically reviews programs' goals and outputs and determines whether they are appropriate and ambitious.

Measure: Percentage of planned outputs delivered in support of the 1) restore impaired aquatic systems, 2) protect unimpaired systems, 3) provide human health risk and treatment process information on the beneficial use of biosolids, and 4) forecast the ecologic, economic, and human health benefits of alternative approaches to attaining water quality standards long-term goal."

Explanation:At the end of the fiscal year, the program reports on its success in meeting its planned annual outputs (detailed in the program's Multi-Year Plan). The program strives to complete 100% of its planned outputs each year so that it can best meet EPA and other partners' needs. To ensure the ambitiousness of its annual output measures, ORD has better formalized the process for developing and modifying program outputs, including requiring that ORD programs engage partners when making modifications. Involving partners in this process helps to ensure the ambitiousness of outputs on the basis of partner utility. In addition, EPA's Board of Scientific Counselors (BOSC) periodically reviews programs' goals and outputs and determines whether they are appropriate and ambitious.

Measure: Percentage of WQRP publications rated as highly cited publications.

Explanation:This metric provides a systematic way of quantifying research performance and impact by counting the number of times an article is cited within other publications. The "highly cited" data are based on the percentage of all program publications that are cited in the top 10% of their field, as determined by "Thomson's Essential Science Indicator" (ESI). Each analysis evaluates the publications from the last ten year period, and is timed to match the cycle for independent expert program reviews by the Board of Scientific Counselors (BOSC). This "highly cited" metric provides information on the quality of the program's research, as well as the degree to which that research is impacting the science community. As such, it is an instructive tool both for the program and for independent panels??such as the BOSC?? in their program reviews. To best establish ambitious and appropriate targets in the future, ORD will collect benchmarking information by conducting an analysis of bibliometric measures used in R&D programs outside of EPA.

Measure: Percentage of WQRP publications in 'high impact' journals.

Explanation:This measure provides a systematic way of quantifying research quality and impact by counting those articles that are published in prestigious journals. The "high impact" data are based on the percentage of all program articles that are published in prestigious journals, as determined by "Thomson's Journal Citation Reports" (JCR). Each analysis evaluates the publications from the last ten year period, and is timed to match the cycle for independent expert program reviews by the Board of Scientific Counselors (BOSC). This "high impact" metric provides information on the quality of the program's research, as well as the degree to which that research is impacting the science community. As such, it is an instructive tool both for the program and for independent panels??such as the BOSC?? in their program reviews. To best establish ambitious and appropriate targets in the future, ORD will collect benchmarking information by conducting an analysis of bibliometric measures used in R&D programs outside of EPA.

Measure: Number of peer reviewed publications per FTE.

Explanation:The universe of peer-reviewed publications includes 1) journal articles, 2) books and book chapters, and 3) EPA reports, where at least one EPA author is listed or where the publication is the result of an EPA grant. If a publication includes more than one EPA author, that publication is counted only once. Materials submitted for publication but not yet published are not included. FTE are actual program full time equivalents. The program is also submitting data on extramural vs. intramural costs to support the measure. Data and targets are based on a three year moving average.

Is the program purpose clear?

Explanation: The purpose of the Water Quality Research Program (WQRP) is to provide timely, cutting edge water quality research products and information to the Office of Water, EPA Regions and States for use in support of regulatory and non-regulatory activities. Ultimately, the WQRP strives to provide sound science that leads to decisions resulting in environmental and/or human health benefits. Federal legislation authorizes this research. (1) The program is structured to conduct research in three sequential areas(2): (1) to support water quality criteria development; (2) to develop diagnostic tools that aid in establishing causal relationships between pollution and water quality impairments; and (3) to provide information that supports sustainable watershed management practices through the demonstration of technologies, the application of decision tools and for forecasting restoration and benefits of management practices. Research under these three rubrics is designed to lead to the promulgation of protective standards, the identification of contaminant contributions to impaired waters, and the tools needed to restore and protect the nation's waters (3,4)

Evidence: (1) Federal Water Pollution Control Act as amended, TITLE I--RESEARCH AND RELATED PROGRAMS, especially Sec. 104 (http://www.epa.gov/region5/water/pdf/ecwa_t1.pdf) (2) Water Quality 2003 MYP http://www.epa.gov/osp/myp.htm (3) WQRP Program Design Model (4) Water Quality Program mission statement (5) The Board of Scientific Counselors (BOSC) Review of the Office of Research and Development's Water Quality Research Program at the US Environmental Protection Agency - Final Report, May 3, 2006 indicated that the program purpose is clear.

Does the program address a specific and existing problem, interest, or need?

Explanation: The public relies on the nation's waterways for a variety of uses from a means to shipping cargo to a source for drinking water. Consequently, there is a public need to restore and maintain the chemical, physical, and biological integrity of the nation's waters, as addressed in the Federal Clean Water Act. (1) Likewise, the public has a need for sound science leading to effective water quality decision making, which the Water Quality Research program is addressing. The WQRP addresses specific regulatory needs of its primary clients in the EPA Office of Water. (2,3,4,5,6,7,8,9) The WQ BOSC concluded that the program "appropriately addresses EPA's Goal 2 (Clean and Safe Water) by creating the tools necessary for the Office of Water to establish water quality criteria and respond when those criteria are not being met. (10)

Evidence: (1) Federal Water Pollution Control Act, as amended (http://www.epa.gov/region5/water/pdf/ecwa_t1.pdf) (2) Water Quality 2003 MYP http://www.epa.gov/osp/myp.htm (3) OWOW Top Ten Research Needs and NRSC Short Term Science Needs (4) Action Plan for Beaches and Recreational Waters, EPA 600-R-98-07 (http://www.epa.gov/ordntrnt/ORD/htm/documents/600r98079.pdf) (5) Beaches Environmental Assessment and Coastal Health Act of 2000, SEC. 3. ( http://www.epa.gov/waterscience/beaches/files/beachbill.pdf) (6) A National Assessment of Landscape Change Impacts to Aquatic Resources: A 10 Year Research Strategy for the Landscape Sciences Program, EPA/600/R-00/001, www.epa.gov/esd/land-sci/pdf/157leb00.pdf (7) WQRP Program Design Model (8) Aquatic Stressors :Framework and Implementation Plan for Effects Research, EPA 600/R-02/074, September 2002 (9) Twenty Needs Report, http://www.epa.gov/owow/tmdl/20needsreport_8-02.pdf (10) Board of Scientific Counselors (BOSC) Review of the Office of Research and Development's Water Quality Research Program at the US Environmental Protection Agency - Final Report, May 3, 2006

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The WQRP minimizes redundancies and duplication with other programs by seeking formalized input to support its multi-year research plan revisions. (1,2,3) The WQRP coordinates with key stakeholders such as the EPA Office of Water (OW), the National Regional Science Counsel (NRSC), EPA Regions, States, and non-governmental organizations. (1,2,3,4,5,6) In addition to seeking stakeholder input in planning WQRP research, the program also provides input into other research organizations such as the Global Water Research Coalition and the Water Reuse Foundation. (6,7,8)

Evidence: (1) OW/ORD Strategic Planning and Research Coordination (SPRC) Action Plan for Aquatic Ecology and Water Quality, Oct., 31, 2002) (2) Client research needs lists, BOSC Materials Tab D, slides 22-26 (3) Envisioning the Agenda for Water Resources Research in the Twenty-First Century http://books.nap.edu/catalog/10140.html (4) State/USEPA Office of Research and Development Water Quality Partnership Project Cooperative Agreement April 1, 2005 (5) Board of Scientific Counselors (BOSC) Review of the Office of Research and Development's Water Quality Research Program at the US Environmental Protection Agency - Final Report, May 3, 2006. (6) WERF ID# CR-83155901-2, Attachment B (7) Global Water Research Coalition Members and Partners (http://www.globalwaterresearchcoalition.net/gwrcmembers.htm) (8) Water Reuse Foundation Committees (http://www.watereuse.org/Foundation/committees.htm)

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The WQRP has an unambiguous and focused design that was developed on the model devised by the OW/ORD Strategic Planning and Research Coordination (SPRC). (1,2) The Board of Scientific Counselors (BOSC) final draft report states that the "The overall impression of the Subcommittee is that ORD's Water Quality Research Program is addressing its goals in an efficient and effective manner and that this research contributes significantly to the Strategic Goals of the EPA." (3)

Evidence: (1) WQ Program Design Model (2) OW/ORD Strategic Planning and Research Coordination (SPRC) Action Plan for Aquatic Ecology and Water Quality, Oct., 31, 2002 (3) Board of Scientific Counselors (BOSC) Review of the Office of Research and Development's Water Quality Research Program at the US Environmental Protection Agency - Final Report, May 3, 2006

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The WQRP is designed so that its outputs (methods, models, data) directly address clients' sound science needs for effective decision making. (1,2) The Multi-Year planning process seeks input from key stakeholders to ensure both intramural and extramural resources are appropriately targeted. In addition, the MYP process allows the program to minimize duplication and unintended subsidies. (1,3,4) The Board of Scientific Counselors (BOSC) final draft report concluded that the program clients regularly participate in research planning, and that the program's research outputs were used by clients and stakeholders. (5)

Evidence: (1) WQRP 2003 MYP http://www.epa.gov/osp/myp.htm (2) WQ Program Design Model (3) STAR Research Program http://cfpub.epa.gov/ncer_abstracts/index.cfm/fuseaction/research.search/rpt/abs/type/3: (4) WERF ID# CR-83155901-2, Attachment B (5) Board of Scientific Counselors (BOSC) Review of the Office of Research and Development's Water Quality Research Program at the US Environmental Protection Agency - Final Report, May 3, 2006

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has developed two new output measures that provide a systematic way of quantifying research performance and impact. The first measure counts the number of times an article reporting research results or products from the program was cited in other publications. The second measure evaluates "high impact," the percentage of program publications that are accepted in relevant prestigious journals and their subsequent impact on the field. "High impact" journals are an indication of quality and influence. Each analysis will evaluate the program's publications from the last ten-year period.

Evidence: For the first measure, the criteria and the "highly cited" rankings are provided by "Thomson's Essential Science Indicator (ESI)". For the second measure, the criteria and the "impact factor" rankings are provided by "Thomson's Journal Citation Reports (JCR)". The program has made significant progress on development of an outcome measure and received provisional OMB approval subject to the establishment of an acceptable evaluation methodology and targets and baselines. The Agency has committed to including these measures in its 2007 GPRA documents.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has obtained quantitative, verifiable baselines for its long-term output measures. Targets have been developed based on data gathered on citations of its research products. However, targets are not considered ambitious because the measures are outputs that have been put in place while the program completes development of its outcome measures.

Evidence: Targets are set at 1% increase every two years for both measures.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: In their Multi-Year Plan, the Water Quality Research Program sets out an annual schedule for completion of a set of research outputs and technical support activities. The program has three new annual output performance measures that assess the percentage of these activities that are achieved each year. Each annual measure ties directly to one of the program's three research goals and they directly support the outcome measures that are currently under development.

Evidence: The program's three goals are based on the three sequential areas of research: (1) to support water quality criteria development; (2) to develop diagnostic tools that aid in establishing causal relationships between pollution and water quality impairments; and (3) to provide information that supports sustainable watershed management practices through the demonstration of technologies, the application of decision tools and for forecasting restoration and benefits of management practices. The Agency has committed to including these measures in its 2007 GPRA documents.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baselines and targets for the annual milestone measures are created through the multi-year planning process. A limited number of research and technical support activities are negotiated with regional and program office clients. The Multi-Year Plan (MYP) is updated every 3-4 years with new outputs put into place for out-years (3-4 years in advance).

Evidence: The measure is calculated in a given year as percent outputs completed relative to the number proposed in the MYP (previous commitments are not dropped from the MYP, even in cases of delays or non-completion). Targets below 100% would be ambitious because output negotiations assume full utilization of available resources, and these conditions may change with time (e.g. key personnel may leave for other positions or resource limitations may affect specific programs) and significant coordination across all organizational elements is required to meet all outputs on schedule. The program has set targets at 100%; therefore these are ambitious.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Water Quality Research Program partners include grantees and contractors as well as partners through cooperative agreements. The program has evidence that it works with grantees to produce results that contribute to program goals. (1,3,4) In addition, the program has evidence that cooperative agreement and contract partners are required to report on performance. (2,5) However, evidence for these partners' performance is lacking. Grantees receive funding through EPA's Science To Achieve Results (STAR) grant program. Solicitation topics for STAR grants are developed within the program and link to agency goals and program activities and milestones (3). The Request for Applications (RFA) is developed by a group that includes representatives from all relevant ORD laboratories/centers and EPA Program Offices to ensure relevance. Project Officers within Office of Research and Development are responsible for tracking how STAR grantees are supporting the program goals, part of which is done through the review of grantee annual reports (4).

Evidence: (1) EPA Order 5700.6 A1: Grants Administration Division Policy on Compliance, Review, and Monitoring (2) EPA Order 5700.7: Environmental Results under EPA Assistance Agreements (3) Example of a STAR Water Quality Research RFA http://es.epa.gov/ncer/rfa/archive/grants/02/02newwatclass.html (4) Examples of progress reports: http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5863/report/2004, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5864/report/2003, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5866/report/2003, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5865/report/2003, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5848/report/2005 (5) WERF Assistance Agreement #CR-83155901-2, Section 8

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: An independent, expert panel evaluated the Drinking Water Program in 2006 and will reevaluate the program every 4 to 5 years. (1) Program evaluations are conducted by a Board of Scientific Counselors (BOSC) subcommittee. The BOSC and BOSC subcommittees are comprised of a distinguished body of scientists and engineers drawn from academia, industry, non-EPA government or state agencies, and the environmental field. . The BOSC was established under the Federal Advisory Committee Act (FACA) to provide unbiased and independent counsel to EPA's Office of Research and Development. The BOSC designated federal officers use FACA procedures to develop the committee and to manage the review process. The BOSC reviews address not only the scientific relevance and quality of the program but also evaluate program performance and improvements made since the previous review.

Evidence: (1) Board of Scientific Counselors (BOSC) Review of the Office of Research and Development's Water Quality Research Program at the US Environmental Protection Agency - Final Report, May 3, 2006 (2) The BOSC Charter, review schedule and past reports are available online at; http://www.epa.gov/osp/bosc/charter.htm (3) In order to maintain independence and prevent conflicts of interest, panelists submit financial information using the following form: http://www.epa.gov/sab/sge_course/pdf_sge/epaform3110_48annual.pdf; ORD has used this information to ask panelists to recuse themselves.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The Water Quality Research Program's system for budget-performance integration does not enable full assessment of its performance-resource mix. However, total funding and performance information is presented as a stand-alone program in EPA's budget execution system and annual budget requests to Congress. The program does not generate any costs or requirements that must be absorbed by other programs and budget documents demonstrate the linkage between total program resources and performance goals. (1) Also, the Agency accounts for all direct and indirect costs, including work year costs, for this program in the annual operating plan and in the annual budget materials provided to OMB.

Evidence: (1) EPA's annual Congressional Justification includes a stand-alone description of the program which describes how program resources will be used to achieve program goals, under the Program Project "Research: Water Quality" in the Science and Technology Account.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The Water Quality Research Program has taken meaningful steps to address strategic planning deficiencies, specifically with respect to performance measurement, independent reviews and development of a stand-alone Research Coordination Team (RCT). (1,2,3) The WQRP has made significant progress on development of outcome-oriented long-term goals around which the Multi-Year Plan is structured. (3) The program has also established recurring program-level independent, expert reviews that incorporate the R&D investment criteria of relevance, quality and performance. (1) The program is working to address several recommendations from this evaluation. Finally, the WQRP is establishing a stand-alone Research Coordination Team (RCT) as part of efforts to improve its planning processes. Previously, the WQRP shared a RCT with the Drinking Water Research program. (2)

Evidence: (1) Board of Scientific Counselors (BOSC) Review of the Office of Research and Development's Water Quality Research Program at the US Environmental Protection Agency - Final Report, May 3, 2006 (2) Water Quality Research Planning White Paper discussing MYP revisions in response to BOSC findings, including development of a stand-alone Research Coordination Team consisting of members from ORD, OW and the Regions (3) The Office of Research and Development has also been working with OMB and the BOSC to develop outcome performance measures for Research and a sound measurement methodology.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: EPA's Water Quality Research program is generally focused on doing research to support regulatory decisions. Other programs, both in the federal government and outside, are doing water quality research, though not for regulatory support. Despite this difference, EPA could learn from comparing the potential benefits of its efforts tothose of other programs in the federal government and outside that do water quality research. The program has not made this comparison

Evidence: Some other organizations doing Water Quality research include the Water Environment Research Foundation (www.werf.org), The Global Water Research Coalition (http://www.globalwaterresearchcoalition.net/), and the National Water Program (http://www.usawaterquality.org/default.html) a partnership between the United State Department of Agriculture (USDA) Cooperative State Research, Education and Extension Service (CREES) and Land Grant Colleges and Universities.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The Water Quality Research Program (WQRP) follows an internal prioritization process to guide budget requests and funding decisions. (1,2) External advisory bodies such as the EPA Science Advisory Board and the Board of Scientific Counselors are consulted and their recommendations are considered in priority-setting. The Multi-Year Plan (MYP), which guides programmatic research and resource allocation, explicitly states the program's goals and identifies a focused set of priorities. (3) The program's priorities are strongly influenced by client needs. The EPA program offices rank their research needs, which are then compared to the current research program and used to adjust the program's funding decisions. (4,5)

Evidence: (1) OSP ranking criteria, draft 12/2/02 (2) OSP FY2007-2008 contingency plan development process, draft 10/4/04 (3) Water Quality 2003 MYP http://www.epa.gov/osp/myp.htm (4) Disinvestment in toxics criteria research - Congressional Justification FY 2007 Change from FY 2006 President's Budget (5) OWOW Top Ten Research Needs

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program collects performance information, including from grant partners. (1,2,3,4,5) However, the program did not adequately demonstrate how it uses performance information to make program adjustments and improve performance.

Evidence: (1) IRMS/GPRA data (2) Grants Terms & Conditions http://es.epa.gov/ncer/guidance/ (3) Examples of Grant Progress Reports: http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5863/report/2004, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5864/report/2003, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5866/report/2003, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5865/report/2003, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5848/report/2005 (4) Site Visit Trip Reports (5) STAR Progress Review Proceedings Documents http://es.epa.gov/ncer/publications/workshop/

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The WQRP incorporates program performance into personnel performance evaluation criteria. Senior managers are accountable for specific performance standards relating to program goals, including progress toward achieving the targets and timelines described in the multi-year plan. (1, 2) The program conducted nearly 30% of its research extramurally through contracts, grants and interagency agreements in 2006. (5). STAR Grantees are held accountable for cost, schedule and performance results through semi-annual progress reporting requirements (3) Cooperative agreement partners are required to provide progress reports and quality management plans that specify requirements for rectifying situations should deficiencies be uncovered. (4) The program did not demonstrate how contractors are held accountable for cost, schedule and performance results.

Evidence: (1) Example performance agreement of WQ program manger showing quantifiable performance standards which hold managers accountable for cost, schedule and performance results (2) ORD Performance Metric for Senior Managers (3) Examples of Grant Progress Reports: http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5863/report/2004, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5864/report/2003, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5866/report/2003, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5865/report/2003, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5848/report/2005 (4) WERF Assistance Agreement CR-83155901-2 (5) The program provided a breakout of its intramural vs. extramural funding since 1996. In 2006, the program had $0.9 million in STAR grants, $14.3 million in other extramural funding, and $35.5 million in intramural funding.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: The program does not establish prospective schedules for obligations that correspond to the resource needs of the program plan. The program has operating plans at the aggregate program level and does not have excessive unobligated balances. (1,2,3) Project officers track financial expenditure information (intramural and extramural) following the agency policy on award monitoring. (4,5)

Evidence: (1) Enacted Operating Plans from 2001 to 2005 (2) Budget and Execution Reports from 2001 to 2005 (3) Summary of unobligated balances 2002 to present; as of June 23, 2006 the program had obligated 92% of its operating plan resources (4) EPA Policy on Compliance, Review, and Monitoring (EPA 5700.6) (5) Terms and Conditions for all Science To Achieve Results (STAR) Assistant Agreements (http://es.epa.gov/ncer/publications/terms/tscs99.html)

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The Water Quality Research Program has an output efficiency measure (with baselines and targets) that tracks the total publications per program full-time equivalent (FTE). In addition, the program is working on a more meaningful, outcome efficiency measure. The program also has procedures to improve internal efficiency and effectiveness. For example, through the broader Office of Research and Development, the program is working to achieve administrative efficiency in the Total Cost of Ownership initiative. (1)

Evidence: (1) The goal of the Total Cost of Ownership initiative is to "develop more effective, efficient ways to manage computer infrastructure operations and maintenance" (http://www.epa.gov/ord/orma/part3/Q_3.4/2%20-%20EPA%20%20ORD%20%20CIO%20%20Total%20Cost%20of%20Ownership.mht)

Does the program collaborate and coordinate effectively with related programs?

Explanation: The Water Quality Research Program actively coordinates its research agenda with other Federal and non-Federal research organizations. The Program seeks formal input from related programs to support its multi-year research plan revisions. (1) The WQRP coordinates with key stakeholders such as the EPA Office of Water (OW), the National Regional Science Counsel (NRSC), EPA Regions, States, and non-governmental organizations. (1,2,3,4,5,6,7) In addition to seeking stakeholder input in planning WQRP research, the program also provides input into other research organizations such as the Global Water Research Coalition and the Water Reuse Foundation. (3,8,9)

Evidence: (1) OW/ORD Strategic Planning and Research Coordination (SPRC) Action Plan for Aquatic Ecology and Water Quality, Oct., 31, 2002) (2) Board of Scientific Counselors (BOSC) Review of the Office of Research and Development's Water Quality Research Program at the US Environmental Protection Agency - Final Report, May 3, 2006 (3) WERF Assistance Agreement # CR-83155901-2 (4) State/USEPA Office of Research and Development Water Quality Partnership Project Cooperative Agreement April 1, 2005 (5) OW/ORD Strategic Planning and Research Coordination (SPRC) Action Plan for Aquatic Ecology and Water Quality, Oct., 31, 2002) (6) Client research needs lists, BOSC Materials Tab D, slides 22-26 (7) Envisioning the Agenda for Water Resources Research in the Twenty-First Century http://books.nap.edu/catalog/10140.html (8) Global Water Research Coalition Members and Partners (http://www.globalwaterresearchcoalition.net/gwrcmembers.htm) (9) Water Reuse Foundation Committees (http://www.watereuse.org/Foundation/committees.htm)

Does the program use strong financial management practices?

Explanation: The Water Quality Research program follows EPA's financial management guidelines for committing, obligating, reprogramming, and reconciling appropriated funds. Agency officials have a system of controls and accountability (EPA's Resources Management Directives System), based on GAO, Treasury and OMB guidance as well as generally accepted accounting practices (GAAP), to minimize improper payments. The program is served by Funds Control Officers (FCOs) that have documented experience and/or training in EPA's budget execution and financial management systems. The program has no material weaknesses as reported by the Office of the Inspector General (OIG) and has procedures in place to minimize erroneous payments.

Evidence: EPA's Annual Reports and Financial Statements, including audit opinions, are available at: http://www.epa.gov/ocfo/finstatement/finstatement.htm. In their latest FMFIA report (FY 2005), the Office of Research and Development (the office that houses the Land Research program) certified that management controls were adequate and reported no material weaknesses. The program provided training records for their Funds Control Officers that demonstrate competence in financial management.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The Water Quality Research program, housed within the Office of Research and Development, has taken several steps to address management deficiencies. In September 2005 the WQRP hired a program director with sufficient authority and responsibility to manage the budget, establish a research agenda, develop measures of accountability, to conduct program evaluations and take necessary actions to improve quality management. Key management elements of the program have received independent, external review by a Board of Scientific Counselors (BOSC), an independent review panel convened regularly by EPA's Office of Research and Development. (1)

Evidence: (1) Board of Scientific Counselors (BOSC) Review of the Office of Research and Development's Water Quality Research Program at the US Environmental Protection Agency - Final Report, May 3, 2006

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The Water Quality Research program must follow EPA's policy for competition of grants and assistance agreements, which includes external scientific peer reviews to rate applications based on scientific merit. (1) The program has demonstrated that grants are awarded through competitive processes. (2) However, the program has not demonstrated that cooperative agreements and contracts are competed. The program advertises contract opportunities on the Fed Biz Ops (www.fbo.gov) website. However, while this provides a central, transparent repository for contract opportunities, it is not a requirement to be a fair and open competition. Information about research grant opportunities and application processes are articulated on EPA's web site. (2,3) To attract new investigators, research solicitations are posted on the EPA web site and on Grants.gov for at least 90 days, emailed to institutions and individuals that have indicated an interest in receiving them, distributed at scientific conferences, and disseminated to researchers by other federal agencies. EPA does not accept renewal applications for grants, cooperative agreements or contracts - researchers can only get further funding through new competition.

Evidence: (1) Policy for Competition of Assistance Agreements," Classification No. 5700.5A1, 01/11/2005. This internal Order establishes EPA policy and requirements for the competition of assistance agreements (including grants and cooperative agreements). (2) Example of a STAR Water Quality Research RFA http://es.epa.gov/ncer/rfa/archive/grants/02/02newwatclass.htm (3) All Office of Research and Development funding opportunities: http://www.epa.gov/ord/htm/grantopportunity.htm

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The Water Quality Research program project officers monitor grantee cooperative agreement, and contract partner performance through annual progress reports, expenditure reports and formal site visits. (1,2,3,4,6,7) EPA offices are required to conduct evaluative reviews on at least 10% of active recipient institutions each year.

Evidence: (1) EPA's Policy on Compliance, Review, and Monitoring [EPA 5700.6]. (2) NCER website for annual reports, publications, activities, and Workshop Proceedings (www.epa.gov/ncer) (3) Site visits for water quality research grantees (4) WERF Assistance Agreement #CR-83155901-2 (5) State/USEPA Office of Research and Development Water Quality Partnership Project Cooperative Agreement April 1, 2005 (6) US Infrastructure Inc. Contract #EP-C-04-064, monthly progress report for August 2006, work assignments 1-1 to 1-7. (7) US Infrastructure Inc. Contract #EP-C-04-064 account balance and account history query from EPA's Financial Data Warehouse.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: An annual progress report is submitted by each grantee and posted on the EPA NCER website. (1,2) Project Officers also monitor cooperative agreement performance through annual progress reports and contractor performance through monthly progress reports. Results of agreements and contracts are made available through publication in scientific journals. (3)

Evidence: (1) NCER information about grantee reporting requirements and STAR grant Progress-Review Workshops http://es.epa.gov/ncer/guidance/tscs99.html (2) Examples of Grant Progress Reports: http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5863/report/2004, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5864/report/2003, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5866/report/2003, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5865/report/2003, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/display.abstractDetail/abstract/5848/report/2005 (3) Site visits for water quality research grantees and cooperative agreements, discusses project officer site visits as well as review of progress reports

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The WQRP funds internal research and an external, competitive (STAR) grants program. (1) The internal program funds are not allocated via a competitive grants approach. Rather, funds are distributed to high priority areas, identified by the program's primary clients in the Office of Water, through the Research Coordination Team composed of representatives from OW, ORD and the Regions. (2) Internal non-competitive funding is justified because of the unique research facilities that are available in the labs utilized by the program. (3)

Evidence: (1) FY05 STAR $0.9M, Non-STAR $44.1M, Extramural portion $9.7M; FY06 STAR $0.9M, Non-STAR $49.8M, Extramural portion $10.3M (2) OSP FY2007-2008 contingency plan development process, draft 10/4/04 (3) Unique facilities include the wet weather flow facilities in Edison, the pilot treatment facilities in Cincinnati, and the microcosm simulated stream flow facilities also in Cincinnati

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Water Quality Research Program is currently using bibliometric analyses as a proxy measure for research performance. The analysis conducted for 2005, to establish a baseline, found that 14% of the water quality research publications are highly cited papers. Approximately the same proportion (13%) is classified as "high impact." The next analysis will be conducted in 2008.

Evidence: The program provided detailed backup for its analyses including tabulated listing of citations that met the "highly cited" and "high impact" criteria.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The Water Quality Research Program delivered 100% of its annual products contributing to the water quality criteria development and application long-term goal in 2003 through 2006. The program did not meet its 100% targets in 2004 or 2005 for the annual multi-year plan (MYP) measure contributing to the diagnostics long-term goal; however, it met its target in 2006. The program has not met its annual 100% targets since 2004 for the annual MYP measure contributing to the watershed management long-term goal; however, they have showed improvement since 2004.

Evidence:

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Performance data for the program's new efficiency measures shows an increase of approximately 2.6% from FY 2004, the baseline year, to FY 2005. The program submitted data indicating that approximately 24% of resources in 2005 were Extramural, with approximately 12% of publications originating from Extramural FTE. These figures will be important in future years in order to assess true efficiency increases vs. resource shifts that may appear as changes in efficiency.

Evidence:

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Limited evidence indicates the program compares favorably; however, more direct comparisons are required. Water quality research publications were more highly cited than the average paper when using the Essential Science Indicators criteria. (1) In addition, the BOSC review found that "the Program has played and continues to play a leadership role in a number of areas of water quality research, such as probabilistic monitoring, biological criteria, stressor identification, etc. and is taking a leading position in emerging issues such as pharmaceuticals and personal care products (PPCPs) in the environment." (2)

Evidence: (1) Bibliometric Analysis (2) Board of Scientific Counselors (BOSC) Review of the Office of Research and Development's Water Quality Research Program at the US Environmental Protection Agency - Final Report, May 3, 2006

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: EPA's independent Board of Scientific Counselors (BOSC), reviewed the Water Quality Research Program in 2006. The charge questions (1) to the reviewing sub-committee incorporated the federal R&D criteria. The BOSC report concluded that "ORD's WQRP overall is in very good shape. It is conducting high quality research focused on well-articulated goals. It has strong leadership and a well-trained, diverse, motivated, and productive staff. Based on responses from clients, the Program appears to be serving them well. The Program has accomplished much with the resources available to it. Although the Subcommittee clearly was impressed with this Program, this report provides a number of comments that the Subcommittee hopes are constructive and a number [of] recommendations which may help to ensure the Program's continued success and communicate its accomplishments more effectively. (2)

Evidence: (1) Charge to BOSC (2) Board of Scientific Counselors (BOSC) Review of the Office of Research and Development's Water Quality Research Program at the US Environmental Protection Agency - Final Report, May 3, 2006