Detailed Information on the
Global Change Research Assessment

Finalize ambitious long-term outcome measures that assess the utility of the program's research products and services with respect to the outcome goals of its clients.

Reassess meaningfulness of current efficiency measure in light of recent National Academy of Sciences (NAS) report on efficiency measurement.

Develop and implement a protocol for more frequent review and use of financial and performance tracking data to improve budget-performance integration.

Identify appropriate targets for bibliometric analysis measures by benchmarking with other agencies.

More clearly define the program's framework and mission to help focus assessment efforts and provide structure for setting priorities.

Measure: Percentage of Global Change research publications rated as highly cited publications

Explanation:The criteria and the "highly cited" rankings will be provided using "Thomson's Essential Science Indicator (ESI)".

Measure: Percentage of Global Change publications in "high impact" journals

Explanation:The criteria and the "impact factor" rankings will be provided using "Thomson's Journal Citation Reports (JCR)".

Measure: Percent progress toward completion of a framework linking global change to air quality

Explanation:The program has outlined a series of linkages that need to be defined before an overall framework for air quality can be established. This measure tracks progress on developing those linkages.

Measure: Percentage of planned outputs delivered

Explanation:Annual research outputs will be outlined in the program's revised Multi-Year Plan. This measure will track progress toward completing those milestones across the program.

Measure: Percent variance from planned cost and schedule (New measure, added August 2007)

Explanation:This measure captures the ability of the program to increase cost effectiveness based on the extent to which it delivers annual research outputs relative to the amount of funds spent. Using an approach similar to Earned Value Management, the data are calculated by: 1) determining the difference between planned and actual performance and cost for each Long-Term Goal, 2) adding these data together to generate program totals, and 3) dividing the Earned Value of all work completed by the Actual Cost of all program activities."

Is the program purpose clear?

Explanation: Although the program has made progress in redefining its mission and narrowing the focus of program activities, the purpose is too broadly defined. According to the program's research strategy, is "to provide scientific information to stakeholders and policy makers to support them as they decide whether and how to respond to the risks and opportunities presented by global change" [1, 2]. This is an important and necessary task however its broadness does not provide a robust framework for setting priorities and defining program boundaries.

Evidence: [1] 2001 Research Strategy of the EPA Global Change Research Program, [2] There is no statutory authority that specifically authorizes this program and therefore there is no statutorily-defined purpose. However the program does address needs outlined in the Global Change Research Act of 1990.

Does the program address a specific and existing problem, interest, or need?

Explanation: There is an existing need to provide decision-makers with reliable science-based information to make informed judgments regarding policies and actions that respond to the risks and opportunities presented by climate variability, climate change, and related systems [1]. EPA's Global Change Research program helps to address this need and has taken steps to focus its efforts on aspects of this need that align within EPA's mission [2].

Evidence: [1] Climate Change Science Program (CCSP) outlines this need in their description of U.S. Global Change Research Program program elements: http://www.usgcrp.gov/usgcrp/ProgramElements/human.htm. [2] Revisions to the program's research strategy and draft revisions to its Multi-Year Plan reflect focused efforts to address issues relevant to EPA's mission: Air Quality and Water Quality Focus Area Overviews for EPA's Global Change Research Program (submitted to the 2005 Board of Scientific Counselors (BOSC) for review).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: EPA's Program is part the CCSP, which was established to coordinate planning and implementation across federal agencies involved in climate change science. The planning and implementation of EPA's program is integrated by the CCSP with other participating Federal departments and agencies to "reduce overlaps, identify and fill programmatic gaps, and add integrative value to products and deliverables produced under the CCSP's auspices." [1, 2] EPA contributes uniquely to this program in the area of decision-support. EPA coordinates with other CCSP agencies to develop and provide timely, useful and scientifically sound information to decision makers [2]. EPA also coordinates with States and internationally to help avoid duplication and redundancy [3].

Evidence: [1] 2003 CCSP strategic plan summarizes the organization and management of the program as well as demonstrates limited duplication of efforts. (quote from pg 29 of highlights document: http://www.climatescience.gov/Library/stratplan2003/vision/ccsp-vision.pdf) [2] CCSP Research Elements show the integration of CCSP agency efforts (http://www.climatescience.gov/infosheets/factsheet2/default.htm). [3] The program works to limit redundancy through collaborative research and fostering dialogue within the climate science community. An example would be the USEPA-Canada co-sponsored symposium on North American climate change and weather extremes: http://www.climate.org/PDF/Research%20Agenda.pdf, http://www.climate.org/PDF/PROCEEDINGS.pdf

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program has no major design flaws and there is no evidence that a different design would improve the program's effectiveness or efficiency. The Global Program employs a good balance of intramural and extramural research which is based on competitive merit-review and independent expert advice [1].

Evidence: [1] The 2005 BOSC review "??found no evidence that the allocation of tasks between intramural and extramural researchers was anything other than appropriate". Review of the Office of Research and Development's Global Change Research Program at the U.S. Environmental Protection Agency, March 2006, pg. 21: http://www.epa.gov/OSP/bosc/pdf/glob0603rpt.pdf

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: EPA's Office of Air and Radiation (OAR) and Office of Water and the U.S. Climate Change Science Program (CCSP), which is run by the Department of Commerce, are the primary beneficiaries of the global change research program's efforts. To ensure that the program addresses problems that are timely, relevant, and useful to EPA offices, it engages those customers in a systematic process to target the program's research efforts [1]. Once research priorities are identified through this process, resources are allocated to specific laboratories and centers to complete the research. Based upon the needs identified by OAR for example, the program is working to reduce uncertainties in and improve the understanding of the linkages between global change and air quality in the United States. In general, stakeholders are provided opportunities to identify issues that are particularly relevant to them and remain engaged throughout the process [2]. To ensure the program's efforts are adequately integrated with other agencies and directly support the needs and goals of the CCSP, EPA is an active participant on the CCSP Interagency Committee and related work groups that prioritize research and assessment activities. [3, 4]

Evidence: [1] The program has established a "Research Coordination Team" (RCT) that consists of Global Change Research program staff and staff from other EPA offices (primarily the Offices of Air and Water), including representatives from EPA regions. The program uses this team for planning, priority-setting, and as a source of real-time information on program performance. [2] The program employs other mechanisms beyond the RCT, such as workshops, to ensure continued relevance of their work to stakeholders: "Proceedings and Recommendations from the Workshop on the Impact of Climate Change on Air Quality in the United States", April 2002. [3] A description of EPA's integrated CCSP efforts can be found through the CCSP website: http://www.usgcrp.gov/usgcrp/agencies/epa.htm; [4] Example of how the program engages the research community to help target its resources so that they will address CCSP customer needs: As part of the CCSP Interagency Working Group on Ecosystems the program co-sponsored a workshop that engaged the research community in the development of research priorities for the CCSP. The results of this effort were used in future planning efforts. Report: "Ecosystems and Climate Change Research Priorities for the U.S. Climate Change Science Program: Recommendations from the Scientific Community", Special Series No. SS-92-06,University of Maryland Center for Environmental Science, Chesapeake Biological Laboratory, Solomons, MD, USA; http://www.usgcrp.gov/usgcrp/Library/ecosystems/eco-workshop-report-jun06.pdf

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has developed two new output measures that track publications as a way of representing research performance. The first measure counts the number of times an article reporting research results or products from the program was cited in other publications [1]. The second tracks the percentage of program publications that are accepted in relevant, prestigious journals and their subsequent impact on the field [2]. These "High impact" journals are an indication of quality and influence. Each analysis will evaluate the program's publications from the last ten-year period. The program is making progress on development of new outcome measures, including the establishment of an acceptable evaluation methodology as well as targets and baselines. Once complete, those measures will be the primary performance measures for the program.

Evidence: [1] For the first measure, the criteria and the "highly cited" rankings are provided by "Thomson's Essential Science Indicator (ESI)". [2] For the second measure, the criteria and the "impact factor" rankings are provided by "Thomson's Journal Citation Reports (JCR)".

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has established quantitative, verifiable baselines for its long-term output measures. Targets have been developed based on data gathered on citations of its research products. However, targets are not considered ambitious because the measures are outputs that have been put in place while the program completes development of its outcome measures.

Evidence:

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: One of the main focus areas for the program is developing a better understanding the potential links between global change and air quality. To accomplish this, the program has outlined a modeling framework to shed light on potential influences and outcomes. The first annual measure for the program tracks the progress toward completion of this framework in percentage terms. The outline defines clear ordered steps (linkages) that need to be developed and further identifies research activities that support the completion of the linkages. The program also defines specific research activities and products for other focal areas such as water quality and ecosystems. The second annual measure for the program looks at progress in completing these products across the entire program.

Evidence: [1] Information about the programs air quality assessment activities can be found on EPA's website: http://www.epa.gov/appcdwww/apb/globalchange/quality.htm

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has established a baseline and ambitious targets for its air quality framework output measure. The targets are ambitious given the complexity of the modeling and research that is needed to make progress on the framework. The program does not have a baseline or targets for its annual milestones measure. The program is in the process of revising its multi-year plan and the baseline and targets for the annual milestone measures will be developed once the revision is complete. A limited number of research and technical support activities will be negotiated with regional and program office clients and with input from the Climate Change Science Program (CCSP). The Multi-Year Plan (MYP) is updated every 3-4 years with new outputs put into place for out-years (3-4 years in advance).

Evidence: Targets for the air quality framework measure require at least 15% incremental progress toward final completion every year. Completion is expected in 2010.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Program partners include grantees, contractors, and certain agencies and non-federal entities (through cooperative agreements). The program works with these partners to produce results that address program milestones and contribute to goals. The major partners are grantees that receive funding through EPA's Science To Achieve Results (STAR) grant program. Solicitation topics for STAR grants are developed within the program and link to agency goals and program activities and milestones [1]. To help ensure relevance, the Request for Applications (RFA) is developed by a group that includes representatives from all relevant ORD laboratories/centers and EPA Program Offices. Project Officers within Office of Research and Development are responsible for tracking how STAR grantees are supporting the program goals, part of which is done through the review of grantee annual reports [2]. RFAs for cooperative agreements are normally developed within a specific laboratory or center to satisfy a more specific need [3]. Project Officers in the Labs/Centers track how cooperative agreements are supporting the program goals [4]. Contracts are used to support individual scientist's specific projects. A Statement of Work is written by the scientist which outlines needs and responsibilities that tie directly to program needs [5, 6].

Evidence: [1] Example RFA's that demonstrate connection to program activities and agency/program goals: Consequences of Global Change for Air Quality: Spatial Patterns in Air Pollution Emissions: http://es.epa.gov/ncer/rfa/archive/grants/03/current/2003_global_change.html Consequences of Global Change For Air Quality: http://es.epa.gov/ncer/rfa/2006/2006_star_gcaq.html, and RFA associated with the UC Davis report: http://es.epa.gov/ncer/rfa/2004/2004_air_poll_em.html. [2] A detailed progress report for a University of California, Davis grantee was provided as evidence. Many grantee reports are available on EPA's website: http://cfpub.epa.gov/ncer_abstracts/index.cfm/fuseaction/search.welcome. [3] "A Request for Applications for a Cooperative Agreement to Provide Assistance to Conduct an Assessment of the Impacts of Climate Change and Variability in the Middle and Upper-Atlantic Region of the United States," NCEA/Global Change Research Program Solicitation, NCEA-02-01, Global Change Research Program, February 14, 2002 and associated example progress report from Pennsylvania State University for Cooperative Agreement No. R-830533-01, dated January 7, 2005. [4] Scope of Work documents and evaluation criteria for contractors provided that demonstrated link to program needs and milestones. Example "Identifying Research and Management Needs for Evaluating Species Invasions Associated with Global Change and Their Impacts on Ecosystem Services". [5] Program staff is responsible for ensuring that for contracts all required deadlines are met and deliverables received through regular reviews and close out procedures. Reports and a "Close Out" document for a contract were provided as evidence.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The Global Program recently underwent an independent evaluation by EPA's Board of Scientific Counselors (BOSC) which was of sufficient scope, quality, and independence [1]. The review qualitatively evaluated the relevance, quality, performance, scientific leadership, and resources of the Program. The Program plans to conduct such reviews every four years. The program has undergone review in the past however it is not clear that those reviews fully met quality or scope criteria for this question. A 1997 Expert Panel review identified significant deficiencies in EPA's 1997 Research Strategy for the Global Program [2]. A 2001 Peer Review Panel reviewed the revised Global Research Strategy [3].

Evidence: [1] 2005 BOSC Report: "Review of the Office of Research and Development's Global Change Research Program at the U.S. Environmental Protection Agency", March 2006: http://www.epa.gov/OSP/bosc/pdf/glob0603rpt.pdf [2] Eastern Research Group, Inc., "Expert Panel on Global Change Research Strategy: Review Workshop," Meeting held in Washington, DC, on July 23-24, 1997, Report submitted to U.S. Environmental Protection Agency, Office of Research and Development, Office of Science Policy, Lexington, MA, September 30, 1997. [3] Eastern Research Group, Inc., "Report of the Peer Review Workshop for EPA's Global Change Research Strategy," Meeting held in Washington, DC, on February 15-16, 2001, Report submitted to U.S. Environmental Protection Agency, Office of Research and Development, Office of Science Policy, Lexington, MA, April 6, 2001.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The program's system for budget-performance integration does not enable full assessment of its performance-resource mix. No integrated presentations clearly include partner contributions. Total, top line funding is presented as a stand-alone line item in EPA's budget execution system and annual budget requests to Congress, the latter of which also discusses overall program performance. The program also generates internal lab-specific operating plans that link activity funding to program goals [1]. The program does not generate any costs or requirements that must be absorbed by other programs and budget documents demonstrate the linkage between total program resources and performance goals [2]. Also, the Agency accounts for all direct and indirect costs, including work year costs, for this program in the annual operating plan and in the annual budget materials provided to OMB.

Evidence: [1] An example operating plan for the Global Change program's efforts at EPA's National Center for Environment Assessment laboratory was provided. The program's Integrated Resource Management System tracks also performance but does not include full resource accounting. [2] EPA's annual Congressional Justification includes a stand-alone description of the program which describes how program resources will be used to achieve program goals: http://www.epa.gov/ocfo/budget/2007/sciencetech.pdf

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The Global Change Research program has taken steps to address deficiencies in strategic planning and is working to develop improved outcome goals that better reflect program performance. The Global Change Research program's has established a Research Coordination Team (RCT) as part of efforts to improve its planning processes. With guidance from this team, program efforts have been streamlined and redefined. The program also works closely with the Climate Change Science Program (CCSP) to ensure the strategic direction of the program is aligned with the overarching priorities. Examples of specific actions taken to address deficiencies include the termination of monitoring network efforts that lacked usefulness to stakeholders and the shift in overall program direction to emphasize decision support and linkage to EPA's mission [1, 2, 3, 4].

Evidence: [1] The program terminated UV monitoring network activities that lacked relevance to long-term programmatic goals and were no longer necessary for program stakeholders. Evidence included a February 5, 2004 fact sheet sent to the CCSP from regarding termination of the network. [2] A 1997 Expert Panel review identified significant deficiencies in strategy for the Global Program. Consequently, EPA redirected the program towards assessment and emphasized linkage with the agency's overall mission. The Program discontinued exploratory, process-oriented research that was duplicative of other agencies' work: "Expert Panel on Global Change Research Strategy: Review Workshop - Summary Report," September 30, 1997. [3] A new Research Strategy was written and finalized in 2001 that focused the program more on key issues important to program beneficiaries: 2001 Research Strategy of the EPA Global Change Research Program; http://www.epa.gov/ncea/pdfs/glblstrtgy.pdf. [4] This new strategy received a positive peer review: Report of the Peer Review Workshop for EPA's Global Change Research Strategy, April 2001.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: This program does not pursue multiple options for achieving goals and therefore internal comparisons can not be completed. It is not an industry-related R&D program.

Evidence:

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The program has established a Research Coordination Team (RCT) to help develop and define priorities however the priority-setting process needs further improvement [1]. The principles and framework for the priority-setting process are not always clear and further work is needed to define and focus research priorities particularly water quality, ecosystems, and human health research areas. Through the RCT, program scientists and managers work closely with representatives from EPA's program offices to identify priorities and to track progress and adjust activities on an ongoing basis. The use of this process has clearly helped the program. The program also uses input from the Climate Change Science Program, the Board of Scientific Counselors, and expert workshops to guide planning and balance priority decision-making.

Evidence: [1] 2005 BOSC Report also noted priority-setting as an area needing improvement: "The Subcommittee also, however, identified a number of areas in which future performance can be improved to meet the Program's evolving mission and responsibilities, as detailed in the body of the report. These include: (1) a more rigorous approach to priority setting; ??": "Review of the Office of Research and Development's Global Change Research Program at the U.S. Environmental Protection Agency", March 2006: http://www.epa.gov/OSP/bosc/pdf/glob0603rpt.pdf.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program does collect performance information from key partners and from intramural efforts however the program has not demonstrated how this information informs day-to-day program management, such as in resource allocation decisions, to improve performance [1]. The program does compile milestone progress information in their Integrated Resource Management System (IRMS) and this information is used to inform the annual planning process and to update the Multi-Year Plan. The program has also shifted its focus over time based on feedback from external experts on research needs and program performance as well as to reflect the changing needs of program clients.

Evidence: [1] Key program partners include grantees and recipients of cooperative agreements (CA). Grantees and CA recipients do provide annual progress reports and final reports on results (including significant accomplishments), which are evaluated by program staff.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The Global Change Research Program incorporates program performance into personnel performance evaluation criteria [1]. Individuals within the program are identified as responsible for program results. These managers are held accountable for through mid-year and end-of-year performance reviews. Contractors are explicitly held accountable for deliverables, costs, and schedules in evaluation criteria and in the statements of work. Performance of grantees and cooperative agreement partners is linked to program activities and needs in the Requests for Applications [2]. The program follows a detailed post-award management plan for cooperative agreement partners and grantees, which helps ensure consistent management and oversight of performance [4]. The program's project officers are responsible for seeing that agreements are awarded and managed according to this plan and applicable government regulations.

Evidence: [1] The program's National Program Director is held accountable for the overall performance of the program through performance agreements. Example Performance Appraisal and Recognition System [PARS] agreement for a senior manager was provided. [2] Example 2006 Global Request For Applications that demonstrates link to program activities and goals: "Consequences of Global Change for Air Quality http://es.epa.gov/ncer/rfa/2006/2006_star_gcaq.html. [3] Calendar Year 2006 Post-Award Management Plan for all ORD Assistance Agreements outlines requirements for grants and cooperative agreements to help ensure proper, consistent accountability.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: The program does not establish prospective schedules for obligations that correspond to the resource needs of the program plan. The program does establish an overall high-level spending plan that identifies key agency participants (EPA labs) that will contribute to meeting the program's goals [1]. More detailed operating plans are established for some intramural laboratory funding that tie annual milestones and activities to long-term goals however they do not consistently include prospective obligation schedules below the annual level [2]. Obligations are tracked to these operating plans. Project officers track financial expenditure information (intramural and extramural) through the agency's Integrated Financial Management System, following the agency policy on award monitoring and using EPA's EASYLite system [3, 4]. Also, the program does not have excessive unobligated balances.

Evidence: [1] The program's internal overarching spending plan for 2008 was provided and demonstrates ties to long-term goals, establishes priorities. Documented information on the potential impacts of lower than budgeted funding was provided which also ties to Long-Term Goal and identifies impacts to specific Laboratories/Centers. [2] Internal FY 2006 Operating Plan for the National Center for Environmental Assessment was provided. It includes full anticipated expenditure levels for given activities but does not indicate how those expenditures are spread over time. [3] Example IFMS reports were made available for review including for Harvard University cooperative agreement and for University of California, Davis grant. [4] Partner funds are tracked according to EPA's Policy on Compliance, Review, and Monitoring, which requires submission of annual progress reports and compliance with federal requirements. Project Officers use the EASYLite Electronic Approval System (http://oasint.rtpnc.epa.gov/fmc2_prv/easylite.welcome) that allows them to approve invoices on line. This system gives instant validation of account totals, ensures account balances can not be exceeded, and indicates the amount actually paid by the Treasury. Project officers review and maintain copies of the invoices and confirmation of payment (via emails) from the finance center and documentation of payments are maintained on line in the EASYLite system.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has proposed an efficiency measure that would track cost and schedule variance for the program's air quality framework development however further work is needed to refine the measure and define a baseline and targets to determine if the measure is implementable. The program does continuously seek opportunities to improve the efficiency and cost effectiveness. For example, the program has identified scenario-building as an area where improved processes and consistency could lead to efficiency gains and is taking steps to further evaluate the opportunity. The program also regularly participates in Office of Research and Development-wide efficiency improvement efforts including competitive sourcing actions and a Total Cost of Ownership initiative to consolidate computer infrastructure and maintenance [1].

Evidence: [1] As an example of competitive sourcing efforts, two clerical positions in the National Center for Environmental Assessment went through a competitive sourcing procedure in 2005: http://www.epa.gov/oarm/source/2005/report05.pdf.

Does the program collaborate and coordinate effectively with related programs?

Explanation: EPA's Global Change Research program is part of the multi-agency Climate Change Science Program (CCSP), which was established to coordinate planning and implementation across federal agencies involved in climate change science [1]. The Global Change Research program collaborates and coordinates extensively with CCSP agencies, particularly through the five CCSP Interagency Working Groups that EPA is a member of. EPA's Global program is co-producing two Synthesis and Assessment documents for CCSP which requires the program to work with other agencies (DOE, NASA, and NOAA) and provides evidence of their successful collaboration [2]. In addition, the program has issued joint solicitations that leverage resources between agencies to allow activities of mutual interest to be undertaken more efficiently [3, 4]. The program also collaborates internationally [3, 5]. Within EPA, the Global Program collaborates and leverages resources in a similar way with other research programs and with Program Offices [6].

Evidence: [1] Strategic Plan of the U.S. Climate Change Science Program (CCSP): http://www.climatescience.gov/Library/stratplan2003/default.htm. Working group structure and agency focus areas are outlined in Chapter 16. [2] Prospectus for CCSP Synthesis & Assessment Product #4.6: http://www.climatescience.gov/Library/sap/sap4-6/sap4-6prospectus-draft.htm [3] EPA, the National Weather Service, and Canada co-sponsored "United States - Canada Symposium: North American Climate Change and Weather Extremes / A Research Agenda": http://www.climate.org/PDF/Research%20Agenda.pdf [4] EPA/DOE joint research solicitation - "Nonlinear Responses to Global Change in Linked Aquatic and Terrestrial Ecosystems and Effects of Multiple Factors on Terrestrial Ecosystems": http://es.epa.gov/ncer/rfa/2005/2005_nonlinear_responses.html [5] In support of the Administration's Bilateral Climate Dialogs led by the State Department, the Global Program has partnered with NOAA to support research on climate impacts: "Proceedings of the International Conference on the Adaptation to Climate Variability and Change", January 5-7, 2006, New Delhi: http://www.i-s-e-t.org/Proceedings-final.pdf [6] Program demonstrated collaboration with EPA's air program in the Community Multiscale Air Quality model development effort and with EPA's water program in the augmentation of the publicly-available BASINS model.

Does the program use strong financial management practices?

Explanation: The Global Change Research program follows EPA's financial management guidelines for committing, obligating, reprogramming, and reconciling appropriated funds. Agency officials have a system of controls and accountability (EPA's Resources Management Directives System), based on GAO, Treasury and OMB guidance as well as generally accepted accounting practices (GAAP), to minimize improper payments [1]. The program is served by Funds Control Officers (FCOs) that have documented experience and/or training in EPA's budget execution and financial management systems [2]. The program has no material weaknesses as reported by the Office of the Inspector General (OIG) and has procedures in place to minimize erroneous payments [3].

Evidence: [1] EPA's Annual Reports and Financial Statements, including audit opinions, are available at: http://www.epa.gov/ocfo/finstatement/finstatement.htm. [2] The program provided training records for their Funds Control Officers that demonstrate competence in financial management. [3] In their latest FMFIA report (FY 2005), the Office of Research and Development (the office that houses the Land Research program) certified that management controls were adequate and reported no material weaknesses.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Program managers regularly review progress and discuss issues related to program management. Managers also use external reviews to help identify issues [1]. The program also follows an office-wide plan established in 2005 to help ensure proper financial management and is implementing an office-wide policy designed to address deficiencies in Conflict of Interest reviews [1, 2]. In 2004, the Office of Research and Development (ORD) established a National Program Director (NPD) for the Global Program to address issues identified in the management structure of the program. The NPD was given clear responsibility for managing the Global Program with a cross-ORD perspective, focusing on what science and research is needed and was given budgetary authority to manage resources, which was a key deficiency identified in the previous structure [3].

Evidence: [1] For example, a new policy was developed to address issues with conflict of interest reviews that were identified by GAO: "Federal Research: NIH and EPA Need to Improve Conflict of Interest Reviews for Research Arrangements with Private Sector Entities," (#05-191), February 2005. No corrective actions were required of the Global Program. [2] In 2005, the Office of Research and Development (ORD) began implementing a Management Multi-Year Plan (MMYP) to ensure that it is in compliance with the reporting and implementing requirements of the Federal Managers' Financial Integrity Act (FMFIA). [3] Issues with the management of the program were identified in 1997: "Expert Panel on Global Change Research Strategy: Review Workshop - Summary Report," September 30, 1997. Subsequently changes were made to restructure and improve oversight. The latest step in this ongoing improvement process has been the granting of resource management authority to the program's most senior official (National Program Director): 2004 memorandum from ORD Assistant Administrator to EPA Deputy Administrator establishing Budget Authority for ORD's New National Program Directors.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All Global program grants and the large majority of cooperative agreements and contracts are awarded through competitive processes. The processes for grants and cooperative agreements include external scientific peer reviews, which to rate applications based on scientific merit [1]. Most contracts are competed through the GSA. The program develops the Statement of Work for the contract, which is then submitted to GSA for competition [2]. Information about research opportunities and application processes are articulated on EPA's web site [3, 4]. To attract new investigators, research solicitations are posted on the EPA web site and on Grants.gov for at least 90 days, emailed to institutions and individuals that have indicated an interest in receiving them, distributed at scientific conferences, and disseminated to researchers by other federal agencies. EPA does not accept renewal applications for grants, cooperative agreements or contracts - researchers can only get further funding through new competition.

Evidence: [1] Policy for Competition of Assistance Agreements," Classification No. 5700.5A1, 01/11/2005. This internal Order establishes EPA policy and requirements for the competition of assistance agreements (including grants and cooperative agreements). [2] EPA has very specific requirements for the use of the GSA contract schedules. The EPA requirements can be found in the Contracts Management Manual (dated Sept. 15, 2005); Chapter 8, Section 1. [3] All Office of Research and Development funding opportunities: http://www.epa.gov/ord/htm/grantopportunity.htm. [4] Global program grant topics/RFA's: http://es.epa.gov/ncer/science/globalclimate/solicitations.html.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Grantees and cooperative agreement partners are required to provide expenditure reports to the program. These data expenditure data are available through the agency's central financial system [1]. Project officers within the global program monitor performance, including submission of annual progress reports and documented compliance with federal requirements [2]. Prior to award, project officers review budgets and often request additional explanations or modifications. Formal site visits are also conducted to monitor the progress of active grants- at least 10% of active recipient institutions are visited each year. Grantees provide a list of publications, presentations and other activities on an annual basis and at the end of their grant period [3]. In addition, grantees are required to participate in annual progress review meetings to provide and discuss scientific results and progress [4]. For contracts, global Program staff review monthly progress reports that provide information on progress being made towards completion of the tasks, as well as detailed cost information. Actual contract expenditures are compared to the original Task Order to ensure that funds are spent as intended [5].

Evidence: [1] Integrated Financial Management System (IFMS) system reports for the grantees and cooperative agreements were provided (i.e. University of California, Davis coop example). [2] Examples include quarterly progress reports for cooperative agreements with Pennsylvania State University and Texas A&M University. [3] Results and presentations are available on the program website: http://cfpub.epa.gov/gcrp/db.cfm [4] Examples include "Fire, Climate, and Air Quality Kick Off Meeting" with STAR grantees: http://es.epa.gov/ncer/publications/workshop/04_03_06_agenda.html and "Proceedings of Joint Progress Review for U.S. EPA STAR Grants: Regional-Scale Stressor-Response Models and Consequences of Global Change for Aquatic Ecosystems": http://es.epa.gov/ncer/publications/workshop/pdf/11_03_05_proceedings.pdf [5] Contractor documents included task orders/statements of work, and progress reports (e.g. Tellus Institute)

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Grantee reports on results are collected annually and are made available on the EPA website [1]. These annual reports are distributed to EPA staff to disseminate to interested parties. Grantees also present results at national and international scientific conferences held annually. Project Officers monitor cooperative agreement performance through annual progress reports and results of these agreements are also made available through publication in scientific journals. The Global Program also uses its official website and the EPA Science Inventory to disseminate performance data from cooperative agreements and contracts in an accessible manner [2]

Evidence: [1] Global Change Research grantees' reports are available by topic: http://es.epa.gov/ncer/science/globalclimate/research.html. [2] The program is developing a website that provides more details about program activities, including results from grants and cooperative agreements. Research projects, results, publications, and presentations are available: http://cfpub.epa.gov/gcrp/db.cfm. This website is based on the EIMS data system, which enables scientists to update metadata records on their projects (or projects they are managing) on a real-time basis. This system is also linked directly to the EPA Science Inventory, so that updates to the Global Website are automatically reflected in the Inventory.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The Global Program intramural funding is not competitively awarded but the program has established a non-competitive process that maintains quality and has identified the unique capabilities of the EPA labs that receive funding through that process. To determine research needs the program utilizes its Research Coordination Team (RCT) and considers outside expert advice and input from the Climate Change Science Program (CCSP) [1, 2]. The program, with guidance from the RCT, applies established criteria to determine whether program requirements can be best addressed through intramural or extramural research [3]. For intramural work, the program allocates work responsibilities and budgets among EPA Labs and Centers by considering the specialized skills and expertise of each Lab and Center [4].

Evidence: [1] The Research Coordination Team is composed of client representatives and members from EPA's Laboratory and Centers. The team plays an important role in planning, priority-setting, and reviewing progress for the program. [2] In some cases, the workshop is also used to identify those research activities most appropriately undertaken by EPA, given its particular expertise within the research community. For example, an Air Quality Workshop was held December 2001, at the outset of the program's air quality framework efforts to help prioritize needs and identify the most appropriate entities to address them. [3] The criteria used for evaluating how best to address program research needs are articulated in the program's revised draft Research Strategy, December 2001, p. 11. [4] The program has evaluated how each unique EPA laboratory can satisfy special needs of the program. A description of how the labs satisfy program needs is included in the program's research strategy (revised draft, December 2001, pp. 11-14).

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Global Change Research program is currently using bibliometric analyses as proxy measures of research performance. The analysis conducted for 2005 to establish a baseline found that 21.0% of the global program's publications were highly cited papers and nearly 23% were classified as "high impact." The next analysis will be conducted in 2007.

Evidence: The program provided detailed backup for its analyses including tabulated listing of citations that met the "highly cited" and "high impact" criteria.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program has made progress on the development of the global change/air quality framework. Since the baseline year (2001), the program has missed its targets twice (2002 and 2003) but has surpassed targets set for the last three years.

Evidence: The program provided detailed backup and explanation for the Climate Change/Air Quality framework annual output measure.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program does not have an approved efficiency measure.

Evidence:

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: No analyses have been undertaken to specifically compare EPA's Global Change Research Program with other similar climate research efforts however the program has performed favorably within the Climate Change Science Program (CCSP). Historically the program has constituted approximately 1% of the CCSP budget [1] and has consistently taken on a share of the responsibility for addressing Global Change Research Act requirements [2]. The program has been effective at producing the necessary products. In addition, EPA's researchers and grantees are frequently cited in peer-reviewed scientific journal articles, indicating the relevance of EPA's work to the broader climate research community [3].

Evidence: [1] EPA's program is the fourth smallest (in terms of funding) of the 13 CCSP agencies: "FY 2007 Report to Congress on Federal Climate Change Expenditures": http://www.whitehouse.gov/omb/legislative/fy07_climate_change.pdf. For 1999 through 2001, EPA constituted approximately 1% of the U.S. Global Change Research Program (CCSP's predecessor) budget. [2] EPA produced 3 of the 21 (14%) Regional Assessments and 1 of the 5 (20%) Sector Assessments for the 2001 First U.S. National Assessment: Mid-Atlantic, Great Lakes, and Gulf Coast Regional Assessments and Health Sector Assessment. EPA is currently the Lead Agency for 3 of the 21 (14%) CCSP Synthesis and Assessment Products (4.1, 4.4, and 4.6), and is contributing to 7 others (33%). Note these percentages are not weighted by complexity. [3] The program provided a detailed explanation and bibliometric analysis of Global Change Research program papers in support of the two long-term goals.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: One external review of the program meets the criteria and therefore can be considered as evidence for this question. The results of the Board of Scientific Counselors (BOSC) review completed this year were overall favorable. Though certain areas of improvement were identified, the BOSC found that "The research program produced significant findings that have been appropriately disseminated both in the peer-reviewed literature and more generally, and have led to better understanding of, and potential adaptation to, global change" [1]. Some individual program activities have undergone independent peer reviews and though they do not meet the criteria for this question, they have provided important feedback that helped the program ensure high scientific quality and effectiveness of program components [2].

Evidence: [1] Review of the Office of Research and Development's Global Change Research Program at the U.S. Environmental Protection Agency, March 2006, pg. 21: http://www.epa.gov/OSP/bosc/pdf/glob0603rpt.pdf. [2] For example, peer reviews have been completed for the Community Multiscale Air Quality modeling system that is central to the entire Climate Change/Air Quality Assessment process: http://www.cmascenter.org/r_and_d/cmaq_review_process.cfm?temp_id=99999.