Detailed Information on the
National Oceanic & Atmospheric Administration: Ecosystem Research Assessment

Assess the portfolio of research within NOAA's Ecosystem Research Program in order to clarify the role of each of the Program's components and eliminate redundancies.

Modify planning and management processes so that research activities meet the highest priority science needs and provide a balanced response to local, regional, and national issues.

Measure: Cost per site characterization.

Explanation:One ERP goal is to improve ecosystem characterizations by making them less expensive, faster, and more accurate. As new technologies automate sample collection and processing, and economies of scale are achieved, the cost per site characterization should decrease. This annual efficiency measure will improve ERP management by providing information on whether new methods are meeting management's characterization needs in a more cost effective manner.

Measure: Return on investment from the discovery and application of new sustainable coastal, ocean, and Great Lakes products.

Explanation:Society benefits from the discovery, exploration, and development of new sustainable coastal and ocean resources (i.e. aquaculture, marine natural products, health, pharmaceuticals.) For example, in 2003 ERP support of eight pearl farms, four demonstration and training pearl hatcheries, 15 giant clam farms, and 20 sponge farms helped expand the value of Hawaii's aquaculture industry to $25.2 million and 630 jobs. ERP efforts to develop new drugs from marine organisms have resulted in discovery and description of more than 1,000 compounds that may be vitally important to the health industry. This measure tracks cumulative economic benefits generated by NOAA's investment in new products from coastal, ocean, and Great Lakes ecosystems.

Measure: Percent of U.S. Large Marine Ecosystems with science-based warning systems that decrease human health risks.

Explanation:Science-based warning systems (i.e. elevated fecal coliforms, contaminated shellfish beds, harmful algal bloom landfall) decrease human health risk by providing information to resource managers, public health officials, and the public on which to issue pubic health warnings or change behaviors. The goal is to increase the percentage of the ten U.S. Large Marine Ecosystems with operational science-based warnings systems from 10% in 2004 to 50% in 2012.

Measure: Percentage of ERP rated as "Effective" by comprehensive external review of research, outreach, and education performance.

Explanation:ERP component programs are subject to comprehensive external review. The unit of measure would be the percentage of total ERP funding accounted for by component programs that were rated "Effective" by a comprehensive review of research, outreach, and education performance every five years. The number reported is the dollar value of ERP programs that have been reviewed within the past five years expressed as a percentage of the total ERP funding.

Measure: Cumulative number of coastal, marine, and Great Lakes ecosystem sites adequately characterized for management.

Explanation:Sound management of coastal and ocean ecosystems requires scientifically-based information on their condition. Characterization includes identification of the physical location, spatial extent, and biological, chemical, and physical characteristics. NOAA ecosystem characterizations improve understanding of the history, current state, and future condition of ecosystems. They are cornerstones to ecosystem-based management and the basis for many coastal and ocean management tools including forecasts, assessments, and management plans. NOAA will use several factors to guide decisions about which characterizations to carry out: user community priorities; adequacy of indicators; significance of issue; and consequences of management action/inaction. Characterization of an ecosystem will be measured as uncharacterized, substantially characterized, or adequately characterized. NOAA has initiated a process to divide each of the Large Marine Ecosystems (LMEs) into subecosystems; ecosystem sites are being used as a proxy unit of measurement until that effort is completed. At that point, the ERP will measure the percentage of ecosystems adequately characterized for management. The long-term target is that NOAA routinely provides adequate characterizations for management in all LMEs.

Measure: Percentage of tools, technologies, and information services that are used by NOAA partners/customers to improve ecosystem-based management.

Explanation:This measure tracks ERP success in translating research findings into information, tools, and technology that improve the use and management of coastal, ocean, and Great Lakes ecosystems. Examples of tools include: land cover data, benthic habitat maps, environmental sensitivity index maps. Technologies refer to the transfer of new or underused approaches for addressing coastal management (e.g., remote sensing, biosensors, AUVs, genetic markers for fishery stocks) and resource development (e.g. culture systems for aquaculture, marine pharmaceuticals). This includes the application of technology to coastal resource management through synthesis, integration, training, and the development of new management tools. Information services would include technical assistance, education materials and curricula, extension and training. Tracking the accessibility and use of information by target audiences will allow ERP to identify and expand its most effective programs and products.

Measure: Percent of grants awarded on time.

Explanation:As an annual efficiency measure, ERP is committed to continuing improvement in processing of competitive grants through electronic submission and processing of awards. ERP's performance measure is the percentage of total NOAA grants processed in time to meet NOAA's standards. Grants include competitive, non-competitive, formula, congressionally mandated, soft earmarks, and institutional. The standards for determining on-time performance differ by the type of review. From the receipt of application, a Full Merit Review process has 250 calendar days until the award date (signature by the Grants Office), a Mail Review has 220 calendar days, and a Panel Review has 190 calendar days. The NOAA target is for 25% improvement over the next 6 years from 60% to 85% or a 4% improvement each year.

Measure: NOAA safety incident rate.

Explanation:NOAA's greatest asset is its highly valued personnel. Unnecessary losses either out on the field or in an office environment are detrimental to operational capability. An effective environmental and safety and health compliance program can reduce harm to human health and minimize environment risks. This also reduces financial risks associated with harm to people and pollution/contamination of property, and civil and criminal risks (e.g. legal liabilities) associated with noncompliance and violations. The recorded Incident Rate is calculated as the total number of Recordable Incidents times 200,000 divided by YTD Hours worked. According to the OSHA formula, the 200,000 number is the equivalent of 100 people working for one year. Thus, an Incident Rate of 1.95 indicates that there were 1.95 injuries per 100 employees in a given year. NOAA tracks this measure on a monthly basis. In concurrence with the President's Safety, Health and Return to Employment (SHARE) initiative, NOAA/ERP's goal is to reduce incident rates by 3% annually.

Measure: Cumulative number of coastal, marine, and Great Lakes issue-based forecast capabilities developed and used for management.

Explanation:NOAA is developing discrete forecast models that allow resource managers to make decisions based on predicted environmental and socioeconomic impacts related to a particular issue. Managers will use these issue-based forecasts to understand the impact of ecosystem stressors (climate change, extreme natural events, pollution, invasive species, and land and resource use) and to evaluate the potential of various options to manage those stressors. These forecasts will be based upon field and laboratory studies, existing data, and models predicting environmental conditions under different scenarios. Forecast capabilities will be counted for each ecosystem as they become operational. Similarly, multiple, distinct forecast capabilities could be counted within a single ecosystem (i.e. NOAA may forecast HABs, pink shrimp harvest, and hypoxia in the Gulf of Mexico). Progress toward this goal has been documented since 2001 and includes: Eastern Gulf of Mexico and Gulf of Maine harmful algal bloom alerts (2001), pink shrimp harvest and Gulf of Mexico hypoxia forecast model development (2002), transfer of an operational oyster mortality forecast capability to the US Army Core of Engineers (2003), transfer of an operational Eastern Gulf of Mexico harmful algal bloom alert capability to NOAA's Coastal Services Center (2004), transfer of the Great Lakes Forecasting System to NOS CO-OPS and NWS (2005), and preliminary forecasts for domoic acid in Pacific NW razor clams, coral bleaching, MSX oyster mortality, and real time jellyfish prediction in the Chesapeake Bay.

Measure: Percent of U.S. Large Marine Ecosystems with integrated environmental and socioeconomic predictive models that address the priority information needs identified by regional managers.

Explanation:Resource managers need a suite of models that can simulate the complexity of an ecosystem and provide a sound scientific basis for decision-making. NOAA's developing forecasting systems link research, modeling, and monitoring within an ecosystem framework to help coastal and marine resource managers 1) understand the impacts of multiple human uses on marine ecosystems, and 2) forecast the consequences of management decisions. To ensure a focus on high-priority issues and the successful transfer of resulting forecast products to the management community, NOAA is working with managers in each of the 10 U.S. Large Marine Ecosystems to identify priority ecosystem forecasting needs (based on the significance of the issue and consequences of management action/inaction). The goal is to increase the percentage of the ten U.S. Large Marine Ecosystems with integrated environmental and socioeconomic predictive models from 10% in 2004 to 50% in 2012.

Is the program purpose clear?

Explanation: The National Oceanic and Atmospheric Administration's (NOAA's) Ecosystem Research Program (ERP) provides scientific information and tools for ocean, coastal, and Great Lakes management. ERP provides research to meet NOAA's legislative and executive mandates to base policy decisions on sound science; to enable coastal resource managers to support society's needs and protect marine mammals and endangered marine life; and to provide credible, unbiased science to the public.

Evidence: 1. ERP consists of: the National Ocean Service's (NOS) National Centers for Coastal Ocean Science (NCCOS)and the Oceans and Human Health Initiative; six programs of NOAA's Oceanic and Atmospheric Research (OAR)??Sea Grant, Atlantic Oceanographic and Meteorological Laboratory [AOML, in part], Great Lakes Environmental Research Laboratory [GLERL], Pacific Marine Environmental Research Laboratory [PMEL, in part], Ocean Exploration, and National Undersea Research Program (NURP); and portions of National Marine Fisheries Service (NMFS) Office of Protected Resources (OPR) research programs. 2. ERP mission requirements are articulated in numerous statutes, executive orders, and international agreements concerning coastal and ocean ecosystems, including the National Environmental Policy Act, the Harmful Algal Bloom and Hypoxia Research Control Act, the Marine Mammal Protection Act, and the recently enacted Oceans and Human Health Act. 3. NOAA Strategic Plan Ecosystem Goal (pp. 4-5, www.spo.noaa.gov/pdfs/NOAA%20Strategic%20Plan.pdf) 4. NOAA 5-Year Research Plan. "Research in NOAA: Toward Understanding and Predicting Earth's Environment, A Five-Year Plan: Fiscal Years 2005-2009", January 2005 (pp. 13-20, nrc.noaa.gov/Docs/NOAA_5-Year_Research_Plan_010605.pdf). 5. Program Charter for Ecosystem Research Program [www.oarhq.noaa.gov/erp/documents/erp_2005_prog_charter.pdf]

Does the program address a specific and existing problem, interest, or need?

Explanation: ERP provides the research foundation for science-based ecosystem management. Two recent blue ribbon reports [Report of the U.S. Commission on Ocean Policy (USCOP) and the Pew Oceans Commission Report] detailed problems and threats to aquatic ecosystems including habitat destruction, invasive species, and land-based sources of pollution. ERP provides the data and science for managers and ocean-related education and outreach. The Administrations' U.S. Ocean Action Plan calls for use of the "best science and data to inform our decision-making" in order to "advance the next generation of ocean, coastal and Great Lakes policy."

Evidence: 1. An Ocean Blueprint for the 21st Century, Final Report of the U.S. Commission on Ocean Policy; www.oceancommission.gov/documents/full_color_rpt/welcome.html) 2. U.S. Ocean Action Plan (ocean.ceq.gov/actionplan.pdf) 3. America's Living Oceans: Charting a Course for Sea Change, Final Report of the Pew Oceans Commission (www.pewtrusts.org/pdf/env_pew_oceans_final_report.pdf) 4. National Research Council, Priorities for Coastal Ecosystem Science (1994), National Academy Press, Washington DC. 106 pp. [www.nap.edu/books/0309050960/html/] 5. National Academy of Public Administration. Courts, Congress, and Constituencies: Managing Fisheries by Default (2002) [www.napawash.org/pc_management_studies/recent.html]

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Some amount of overlap and duplication exists between ERP and other Federal programs, as well as within the different components of ERP itself. A report by the U.S. Commission on Ocean Policy noted the need for greater interagency coordination on ocean issues. Although NOAA is the lead agency in applied marine and coastal ecosystem research, other Federal, state, and private organizations have similar or shared responsibilities under Acts such as the Endangered Species Act, Marine Mammal Protection Act, and Coastal Zone Management Act. The ERP is working to ensure that these efforts are complementary and efficient by routinely assessing its research priorities and related programmatic activities, and refining its direction to minimize duplication and leverage limited resources. ERP is also working to coordinate its activities with partner organizations through a variety of interagency task forces, for example with the Joint Subcommittee on Ocean Science and Technology as called for by the U.S. Ocean Action Plan. Within NOAA, the ERP was established in part to try to address redundancies and duplication that exist in the different program components.

Evidence: 1. The Planning, Programming, Budgeting and Execution System is described in NOAA's 5-Year Research Plan (see Q. 1.1) 2. Where there is possible overlap of responsibilities, agencies work together guided by interagency MOU's and other agreements. Examples include: a. Aquatic Nuisance Species Task Force [www.anstaskforce.gov/TF%20Definition.htm] b. US Coral Reef Task Force (www.coralreef.gov/charter.cfm) c. Collaborative Agreement on Ecology and Oceanography of Harmful Algal Blooms (ECOHAB) [(es.epa.gov/ncer/rfa/current/2003ecohab.html] d. Steller's Sea Lion Recovery Team / Plan [www.fakr.noaa.gov/protectedresources/stellers/finalrecovery92.pdf] 3. Program Charter for Ecosystem Research Program (see Q. 1.1)

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The Ecosystem Research Program was created in June 2004 by NOAA to enhance the integration of NOAA's ocean and coastal research programs. NOAA's program planning process requires ERP to annually identify gaps in its capacity to meet mission requirements, and determine the appropriate balance between intramural and extramural research among the component programs so that ERP can best achieve the common set of long-term program goals. ERP's internal resources include several NOAA laboratories and research programs that can respond to immediate research needs (including those required by regulatory and judicial mandates); sustain long-term monitoring and modeling capabilities; and assure that research is forward-looking and responsive to programmatic needs. In addition, ERP is able to provide support for, and leverage funding from, state, university, and non-governmental partners that complement and augment the internal research capabilities; provide critical expertise in areas not fully represented inside the agency; and share new ideas and technologies. NOAA is currently undergoing an extensive external review of all activities related to ecosystem research and science (including those within the Ecosystem Research Program) and ERP will be responsive to the results of that review. That review will consider whether the balance between internal and external research conducted and/or sponsored by NOAA is appropriate for its mission needs.

Evidence: 1. The "Role of our Research Partners" is described in NOAA's 5-Year Research Plan (see Q. 1.1) 2. ERP programs dedicate resources both internally and externally as appropriate when designing long-term research programs. For example, the VENTS Program at the Pacific Marine Environmental Laboratory relies on internal NOAA scientists for physical, chemical, and geological sciences, but partners with the academic community to bring in the biological expertise, taking advantage of the relative strengths of the institutions involved in the program. The Harmful Algal Bloom Programs within ERP use NCCOS Laboratories to provide expertise in toxin identification and remote sensing, while relying on the academic community for the oceanography and ecology necessary to understand bloom development. 3. Framework For An External Review Of NOAA's Ecosystem Research And Science Enterprise [www.sab.noaa.gov/Doc/Ext_Rev_of_NOAAs_Ecosystem_Research_and_Science_Enterprise_Framework.pdf]

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: ERP has a mixed record with respect to targeting research funding. The program receives a no for this element because of questions related to NOAA's ability to influence how resources are targeted in formula funded, institutional programs such as Sea Grant. The program also receives a number of Congressional add-ons and direct earmarks. Though currently only 9% of ERP funding, earmarks can interfere with the program's ability to target resources so that they directly and efficiently support the program's purpose. To the extent that the program has control over the use of funds, ERP targets two categories of user groups: those who use the end products of research, and those who are recipients of financial awards. Program targeting begins with stakeholder input to NOAA's Strategic Plan. ERP validates the effectiveness of its products and services with end users through surveys, consultation, and external reviews. Products and services are changed based on these surveys, and subsequent consultations determine whether the changes improved the services.

Evidence: 1. ERP solicitations for financial awards include a clear statement of the program's objective and how the research will support that objective. The ERP and Line Office Annual Operating Plans guide ERP to achieve its long-term outcomes and those of the NOAA Strategic Plan. 2. For FY 2005, 9% of ERP's appropriation was earmarked for specific recipients, but the subject areas were consistent with ERP priorities. Congress earmarked 7% of ERP funds to be spent on Oceans and Human Health and those funds were distributed competitively. An additional 22% of the FY2005 budget represented increases over the President's budget to ERP programs that were expended in the same manner as base funds in direct support of ERP priorities. 4. NOAA Sea Grant funding procedures [www.oarhq.noaa.gov/ERP/documents/prog_core_funding_procedures.pdf]

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: ERP's outcomes and performance measures are based on the NOAA Strategic and 5-Year Research Plans. ERP's four long term Program Outcomes reflect the program mission: 1) Resource managers use the best available science to make ecosystem-based decisions; 2) A well informed public acts as an effective steward of coastal, marine, and Great Lakes resources; 3) Human health risks decrease as a result of improvements in the condition of coastal, marine, and Great Lakes resources; and 4) Resource managers and society benefit from the development, demonstration, and transfer of technology that ensures sustainable use of marine resources. ERP's four long-term performance measures reflect outputs but focus the Program's activities on those long-term outcomes: 1) Percent of U.S. Large Marine Ecosystems with integrated environmental and socioeconomic predictive models that address the priority information needs identified by regional managers; 2) Percent of U.S. Large Marine Ecosystems with science-based warning systems that decrease human health risks; 3) Return on investment from the discovery and application of new sustainable coastal, ocean, and Great Lakes products; and 4) Percentage of ERP rated as "Effective" by comprehensive external review of research, outreach, and education performance.

Evidence: 1. NOAA Strategic Plan (see Q. 1.1) 2. NOAA 5 year research plan (see Q. 1.1) 3. See Performance Measures Section

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: ERP has set ambitious targets. For example, one of ERP's long-term targets is to provide integrated environmental and socioeconomic forecast models to managers in all 10 U.S. large marine and freshwater ecosystems by 2012. These models will allow resource managers to make science-based decisions on management of large and complex ecosystems (e.g., the Great Lakes and the Gulf of Mexico). Developing functional models will depend on research progressing on a number of fronts. The full list of performance measures is included in the Performance Measure Section.

Evidence: The scale, complexity, and accuracy of the large marine and freshwater ecosystem models will depend on research progressing on a number of fronts. For example, model development for the Integrated Assessment of Hypoxia in the northern Gulf of Mexico was supported by over five years of research and the integration of the ecological and human dimensions took an additional two years to complete. The assessment examined: the distribution, dynamics, and causes; ecological and economic consequences; sources and loads of nutrients transported by the Mississippi River to the Gulf of Mexico; effects of reducing nutrient loads; methods for reducing nutrients loads; and the social and economic benefits of such methods. This model and others like it allow resource managers to make science-based decisions on management of large and complex ecosystems (e.g., the Great Lakes and the Gulf of Mexico).

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: ERP has developed annual measures to reflect progress in characterizing coastal and marine ecosystems, and developing forecast capabilities and tools, and technologies for use by coastal managers. In addition, ERP tracks program efficiency in developing characterizations, improving workplace safety, and in assessing the overall effectiveness of its components through external review.

Evidence: 1. See Performance Measures Section

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Performance targets are developed through NOAA's five-year planning process. Targets for FY03-FY05 were set by ERP with input from other NOAA programs and external stakeholders. Preliminary targets were also set for FY06-11. Out-year targets will be evaluated annually and modified to reflect the latest identified needs and scientific progress as new research results and management issues arise.

Evidence: 1. See Performance Measures Section

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: For individual competitive grant programs, the Federal Register Notice soliciting grant applications delineates the scope of the funded research, ensuring that research supports NOAA program goals. For institutional grant programs (e.g., Sea Grant and NURP) recipients develop strategic plans that are explicitly tied to the NOAA strategic plan and program goals. Starting in FY2006, Sea Grant institutional awards will indicate the ERP performance measures that are supported, and Sea Grant programs will report on those measures in their annual reports. Responsiveness to NOAA's needs is one of the criteria considered during the four-year performance review of each state Sea Grant program that determines the level of future funding. For other grant awards, ERP staff monitor new and competing grant applications for content relevant to ERP goals, and mandated annual progress reports are reviewed to track progress toward goals and the ERP performance measures. Where possible, earmarks are handled as cooperative agreements to ensure that the partners' activities are linked to ERP goals.

Evidence: 1. Federal Register notice example [ww004.pdf] 2. Sea Grant National Strategic Plan showing NOAA performance measures [www.miseagrant.umich.edu/sp/pdf/2005MSGSP-part1.pdf page 6] 3. MOU and/or cooperative agreement examples a. ECOHAB: The Ecology and Oceanography of Harmful Algal Blooms (www.redtide.whoi.edu/hab/nationplan/ECOHAB/ECOHABhtml.html) b. National Plan for Algal Toxins and Harmful Algal Blooms (www.esa.org/HABPlan/index.php)

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: ERP programs are subject to regular evaluation by external and independent specialists at numerous points during their life cycle. Evaluation activities range from external peer review of grant applications to external program reviews facilitated by Federal Advisory Committees such as the NOAA Science Advisory Board and the National Sea Grant Review Panel. NOAA's Science Advisory Board (SAB), a Federal Advisory Committee whose members are external to NOAA and without conflicts of interest, oversees the review process for most programs. SAB activities ensure that NOAA's science programs are of the highest quality and provide optimal support to resource management, and environmental assessment and prediction. ERP intramural science is generally evaluated every five years. Reviews evaluate effectiveness, quality, comprehensiveness of science, management, and relevance to NOAA's mission and stated beneficiaries for stated objectives. Where relevant, programs are also reviewed on the effectiveness of education and outreach. The state Sea Grant programs are reviewed by analogous Program Assessment Teams led by a member of the National Sea Grant Review Panel (a FACA Committee). The emphasis of those reviews is on program impact and connection with users. Other external reviews also occur: The National Research Council has reviewed several programs within the ERP for effectiveness and adequacy of the science; the NRC is presently reviewing the Sea Grant Program evaluation process. Marine Mammal science benefits from regular interaction with, and review by, Regional Scientific Review Groups. The Center for Coastal Environmental Health and Biomolecular Research was reviewed in 2001 by a panel of independent, distinguished scientists. Independent evaluations are periodically conducted on the state of ocean science and the needs of coastal managers in terms of adequacy of the science.

Evidence: 1. Science Advisory Board charter (www.sab.noaa.gov/Charter/charter.html) 2. Sea Grant Program Assessment Manual (2005): www.seagrant.noaa.gov/other/greenbook_doc/2005_patmanual.pdf 3. CCEHBR 2002 Review [www.oarhq.noaa.gov/erp/documents/ccehbr_review_2002.pdf] 4. National Academy reports: Coastal Ocean Program (1994; www.nap.edu/books/NI000001/html/), Sea Grant (1994; www.nap.edu/books/NX005518/html/), Ocean Exploration (2003; www.nap.edu/books/0309086310/html/) 5. Framework for an External Review of NOAA's Ecosystem Research and Science Enterprise (www.sab.noaa.gov/Doc/FRAMEWORK.pdf)

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: NOAA has made progress using a Planning, Programming, Budgeting, and Execution System (PPBES) to better define how the components of the Ecosystem Research Program contribute to annual, long-term, and strategic goals. This process helps to guide research and programmatic activities through an annual review of priorities and budget allocations to ensure the program is on track to meet its strategic objectives. The annual budget requests include the full cost of attaining performance goals and all programmatic increases include specific performance information. Budget and performance data are integrated and the impact of funding decisions on performance is apparent. The program should continue to increase the transparency of how its base programs align with performance goals.

Evidence: 1. NOAA section of the Department of Commerce Budget Submission. www.osec.doc.gov/bmi/budget/06BIB%5CNOAAallnoPM.pdf

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: ERP engages in several coordination and planning processes to address potential strategic planning deficiencies. ERP has developed a set of long-term performance measures that support the major objectives of NOAA's strategic plan. Integrated ERP annual measures have evolved from the measures used by the component programs in previous years. ERP partners will be reporting on these measures along with the internal ERP programs and laboratories. ERP is implementing a trackable performance measure database that will be used in budget development to more clearly tie the budget request to the accomplishment of those annual and long-term performance goals. As part of the strategic planning process, ERP evaluates the balance between intramural and extramural research to maximize those capabilities and capacities to reach NOAA's strategic goals. ERP is working with the Joint Subcommittee on Ocean Science and Technology (JSOST) to establish national priorities for coastal and ocean research, with the Coral Reef Task Force to set research priorities for coral reef ecosystems, with the Invasive Species Task Force to identify their research priorities, and with other interagency groups to maximize coordination of potentially overlapping programs. NOAA's Research Review highlighted the lack of a research strategy for NOAA's research programs. In response NOAA developed and published "Research in NOAA: Toward Understanding and Predicting Earth's Environment, A Five-Year Plan: Fiscal Years 2005-2009" in January 2005.

Evidence: 1. NOAA Business Operations Manual (see Q. 1.4) 2. NOAA Strategic Plan (see Q 1.1) 3. NOAA 5-Year Research Plan (see Q 1.1) 4. Understanding Global Ecosystems to Support Informed Decision-Making: A 20-Year Research Vision: http://nrc.noaa.gov/Docs/Final_20-Year_Research_Vision.pdf

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: NOAA's Program Planning, Budgeting, and Execution System was established in 2003 to: assess Program goals, activities, directions, and gaps; determine continued relevance of the Program to NOAA; and ensure the quality, integration, and productivity of Program activities. The evaluation team is led by the ERP Program Coordinator and comprised of representatives of several program areas within the ERP. In addition to this internal process, ERP competitively awarded research undergoes external independent review that includes comparisons of the potential benefits of each proposal submitted. Program managers then consider and select from among alternative proposal combinations to ensure that supported research contributes to Program goals, is cost effective, and avoids duplication. Potential benefits of ERP efforts are also periodically compared with similar programs run by the National Science Foundation. As requested in the 2002 National Sea Grant College Program, NOAA has submitted three annual reports to Congress that describe overlapping ocean and coastal research interests between the agencies and specifies how such research interests will be pursued by the programs in a complementary manner.

Evidence: 1. Sea Grant Program Assessment Manual guidelines on strategic planning [www.seagrant.noaa.gov/other/Password_files/strat_planning.pdf] 2. "Ocean and Coastal Research Activities of the National Oceanic and Atmospheric Administration (NOAA) and the National Science Foundation (NSF): Report on interagency activities in ocean and coastal research in FY 2005 and plans for coordination in FY 2006" (Report to Congress, February 2005)

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The NOAA planning process provides a detailed procedure for prioritizing budget requests based on the Annual Guidance Memorandum from the NOAA Administrator. The memorandum is developed based on Administration priorities, input from the Ecosystem Goal Team, public meetings, consultation with outside constituencies, and deliberation by internal NOAA Councils. ERP then refines its research priorities for budget requests further using a broad range of inputs which are then screened and ranked. ERP solicits external input from NOAA constituents and from the scientific community, through numerous workshops, meetings, and programmatic documents to refine specific research needs. These needs are then screened for relevance and ranked for importance to NOAA's mission and specific programmatic needs. These research needs become the basis for developing ERP budget requests which are refined through the NOAA planning process.

Evidence: As part of the ERP annual Program Baseline Assessment, ERP prioritizes all Program Capabilities based on an assessment of which capabilities are most critical to reaching ERP's long-term outcomes. These priorities are used to guide budget and funding allocation decisions. 1. Develop forecasts to predict ecological (and socioeconomic) impacts. 2. Develop technologies and tools. 3. Identify causes and consequences of changes in ecosystem condition. 4. Explore and characterize ecosystem health. 5. Strengthen stewardship through outreach and education. More information about ERP priorities can be found in the NOAA Annual Guidance Memorandum (www.noaa.gov/images/agm.pdf) and the NOAA 5-Year Research Plan (draft) (see Q 1.1).

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: ERP, like all of NOAA, collects and reports on performance measures quarterly. Quarterly reports from the Line Offices are also required under their annual operating plans. For program partners, ERP follows the regulations and guidelines specified in the Department of Commerce (DOC) Grants Manual regarding the collection of grantee performance data. Quarterly and annual grantee progress reports monitor key aspects of performance such as progress towards goals and milestones, presentations and publications. Contracts are handled in accordance with the Federal Acquisition System, which requires reporting on goals and milestones. Based on these regular reporting requirements, ERP manages program partner performance and takes corrective action, including redirection of funding or shifting personnel, as necessary. In instances where Congress specifies the contractor, final reports are required and the agency offers the results to Congressional staff inquiries. In the case of non-performance under an earmark, we have worked with Congressional staff to improve performance or change the recipient of the earmark.

Evidence: 1. DOC Interim Grants Manual [www.osec.doc.gov/oebam/pdf/CGA_manual.pdf] 2. Examples of grantee reports a. Florida Sea Grant 2003 Annual Report [nsgl.gso.uri.edu/flsgp/flsgpq03002.pdf] b. Ocean Exploration Galapagos Rift Exploration Cruise Report [oceanexplorer.noaa.gov/explorations/02galapagos/galapagos.html] 3. Federal Acquisition Regulation (www.arnet.gov/far/) 4. NOAA Acquisitions Management Division Manual (www.ofa.noaa.gov/%7Eamd/contracting.html) 5. Evidence of program management a. Summary of four years of Sea Grant Program Assessments [www.seagrant.noaa.gov/other/Password_files/tollreport.pdf]

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ERP managers are held responsible for achieving ERP objectives within budget. Program managers' performance plans include accomplishment of specific outcomes related to Line Office and ERP outcome measures. Unsatisfactory progress can, and does, result in a negative performance evaluation. In addition some ERP programs are part of OPM's pay banding demonstration program, which more closely links pay to performance. Program partners and contractors are held accountable for achieving program results through the terms of the recipient's grant award as specified in the DOC grants manual. Grant recipients are required to account for cost, schedule, and performance results as part of their reporting requirements associated with quarterly, annual, and final reports. When necessary, corrective actions are taken. If expectations are not being met, ERP is able to take the appropriate action, including not renewing funding for a particular research focus if it is not leading to desired results or accomplishments.

Evidence: 1. DOC Interim Grants Manual [www.osec.doc.gov/oebam/pdf/CGA_manual.pdf] 2. COTR Manual [www.ofa.noaa.gov/7Eamd/cotr.html]

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: ERP funds are spent for the intended purpose as directed by annual program plans and tracked by program databases. Funds are generally obligated in a timely manner, according to the schedule defined by the DOC Grants Manual and Congressional directives. The spending of grants is strictly regulated by the NOAA Grants Office, which monitors and approves each grant expenditure, planned expenditures versus obligations are compared, and discrepancies are analyzed. If discrepancies are found, the program manager works with the grant recipient to correct the discrepancy. Delays in the appropriation process have led to delays in expenditures, which may affect research and research partners. Such delays are a special problem for aquatic research. Vessel reservations must be made two to three years in advance due to a limited number of appropriate vessels, and the need to maximize their use. Field seasons are also dictated by research questions and physical conditions.

Evidence: 1. DOC Interim Grants Manual [www.osec.doc.gov/oebam/pdf/CGA_manual.pdf]

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: NOAA uses established Federal guidelines that require competitive bidding unless there is a compelling justification for a sole source contract. ERP is achieving greater efficiencies in administrative and technical support areas. Examples include streamlining grants processing through organizational changes, development of on-line grant application, and where possible increases in the size and length of grants to reduce administrative costs. Gains in IT efficiencies have been achieved through consolidation of functions and competitive sourcing. ERP also achieves efficiencies by pooling resources and collaborative outsourcing on projects involving multiple entities (within NOAA or externally) that are supporting projects in similar topical areas. Programs have also increased efficiency by developing databases to track individual projects and their performance.

Evidence: 1. Federal Acquisition Regulation (www.arnet.gov/far/) (see Q. 3.1) 2. NOAA Acquisitions Management Division Manual (www.ofa.noaa.gov/%7Eamd/contracting.html] (see Q. 3.1) 3. NOAA Grants Online: www.ofa.noaa.gov/~grantsonline/index.html

Does the program collaborate and coordinate effectively with related programs?

Explanation: Within NOAA, the ERP was established to ensure internal coordination and collaboration. Collaboration with external partners can range from straight financial support to long-term cooperative funding of important research topics, as well as in-kind research collaboration.

Evidence: For example, research on harmful algal blooms (HABs), involving NOAA, EPA, NSF, the Office of Naval Research, and NASA, is coordinated through an Interagency Agreement. Under the agreement, the agencies coordinate in a structured process to assign the funding of HAB research projects in accordance with the agency mission, capability, and resources. This has resulted in significantly improved HAB forecasts for states, localities, tribes, and the general public. Another example is a collaborative program between Sea Grant and the National Weather Service to provide warnings to the public on the dangers of rip tides and how to avoid them. As a result of this internal and external collaboration, warning signs have been placed on public beaches throughout the country. The Steller Sea Lion Recovery Team provides a third example of how NOAA collaborates with external partners. The Team, in place since 1990, currently has members from 12 different organizations (e.g., NOAA Fisheries, Alaska Department of Fish and Wildlife, U.S. Fish and Wildlife Service). Recovery Teams assist the development and implementation of Recovery Plans, which include a prioritized implementation schedule of research and management needs.

Does the program use strong financial management practices?

Explanation: NOAA follows prescribed Department of Commerce-wide financial management and accounting policies, procedures, and controls. In 2003, the Department received an unqualified opinion on the FY 2003 consolidated financial statements, the highest rating available. ERP planning and spending is done via Annual Operating Plans and the ERP has routine execution reports. In FY05 the ERP started collecting and tracking Program financial management data.

Evidence: 1. FY 2003 Department of Commerce Financial Report, Performance Accountability Report (www.osec.doc.gov/bmi/budget/FY03PAR444.htm) 2. Commerce/NOAA CAMS documentation [www.ofa.noaa.gov/~cams/documentation.htm 3. NOAA Grants Management: www.ofa.noass.gov/~grants/

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The ERP was created to address a NOAA-wide deficiency in the coordination of planning, execution, and evaluation of NOAA ecosystem research. By consolidating ecosystem research into a single program (ERP), NOAA improved the management and coordination of NOAA ecosystem research activities. ERP performance measures improve NOAA's ability to track, evaluate, and defend the impacts of ocean and coastal ecosystem research. NOAA is taking additional steps to increase coordination and collaboration with other Federal ocean science programs through participating in interagency groups such as the Joint Subcommittee on Ocean Science and Technology. Line offices have also taken a variety of actions, such as developing performance-based strategic and implementation plans, consolidating common management functions to save personnel resources (e.g., IT, grants) and consolidating coastal research to better support management needs.

Evidence: 1. NCCOS Strategic Plan (coastalscience.noaa.gov/documents/strategicplan.pdf) 2. Examples of consolidating management functions (e.g., the IT document) (http://www.rdc.noaa.gov/~foia/asdhome/noaacirc/Nc03_htm.htm) 3. Memo from the administrator establishing NCCOS (www.oarhq.noaa.gov/erp/documents/memo_admin_est_nccos.pdf)

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All ERP grant proposals, whether competitive or not, are reviewed by independent mail reviews and/or a separate panel of expert reviewers for scientific merit and relevance to the program and NOAA's mission and objectives. In FY2005 ERP will award 62% of its grant funding on a competitive basis, including those funds awarded to institutions but then competed following procedures established by NOAA. About 17% of ERP's grant funds are earmarked and awarded without competition. An additional 21% of ERP funds is awarded to institutional programs as specified by law (Sea Grant) or appropriations language (National Undersea Research Program). Cooperative agreements with the joint institutes are competed every five years with a maximum potential for a five-year renewal.

Evidence: 1. DOC Interim Grants Manual [www.osec.doc.gov/oebam/pdf/CGA_manual.pdf] (see Q. 1.5) 2. NOAA Sea Grant funding procedures [www.oarhq.noaa.gov/ERP/documents/prog_core_funding_procedures.pdf]

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Following standard NOAA procedures for grants, grantees make regular progress reports to ERP program officers, which are reviewed for consistency with the grant award. Program officers also make site visits. NOAA oversees whether grants are 1) awarded/obligated to the correct person and 2) whether the grantees are using their awards for the intended purpose. The DOC Grants Manual requires monitoring to ensure that the terms and conditions of awards are fulfilled. Quarterly and/or annual reports are required from all grantees. Programs also report on progress through NOAA's quarterly and monthly reporting.

Evidence: 1. Code of Federal Regulations (15CFR Part 14, section 14.51)- www.gpoaccess.gov/ecfr; 2. Joint Institutes (cooperative agreements) report on research progress annually (www.ogp.noaa.gov; www.oarhq.noaa.gov/OSS_JI.html) 3. DOC Grants interim manual (see for example chapter 10 page 4) [www.osec.doc.gov/oebam/pdf/CGA_manual.pdf]

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Grantees and Joint Institutes (cooperative agreements) regularly report research findings as a condition for their awards. These findings are made available on NOAA websites and/or through scientific literature, hard copy reports, and CDs. For ERP institutional programs (Sea Grant, NURP), program impacts are available in aggregated form on the two national web sites or on the individual web sites for each state Sea Grant program or NURP Center. ERP also provides research results for many different types of audiences: it issues press releases highlighting accomplishments and publications; publishes documents summarizing a body of research on a topic on the web and through hard copy; and sponsors special sessions at national conferences which have highlighted grantee research efforts. Where appropriate, data sets with explanatory material are provided to centralized and publicly accessible databases (e.g., the National Oceanographic Data Center) and libraries (e.g., the NOAA central library).

Evidence: 1. Website Examples a. Sea Grant (www.seagrant.noaa.gov) b. Ocean Exploration (OE) (www.oe.noaa.gov) c. National Centers for Coastal Ocean Science (www.cop.noaa.gov/current_activities.html) d. Office of Protected Resources (www.nmfs.noaa.gov/pr/) 2. Examples of hard copy and CD dissemination a. Decision Analysis Series (www.cop.noaa.gov/pubs/das.html) 3. Press releases a. coastalscience.noaa.gov/news/press.html b. www.seagrantnews.org/

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: As noted earlier, 62% of ERP FY 2005 grant funding was awarded on a competitive basis, 21% of ERP funds is awarded to institutional programs as specified by law (Sea Grant) or appropriations language (National Undersea Research Program), and 17% of ERP's grant funds are earmarked and awarded without competition. All ERP grant proposals, whether competitive or not, are reviewed by independent mail reviews and/or a separate panel of expert reviewers for scientific merit and relevance to the program and NOAA's mission and objectives. For contracts, ERP issues a Request for Proposals with specific requirements to be met. Proposals are evaluated both in comparison to competing proposals and in terms of their absolute merit. For Interagency Agreements, ERP requires formal and written justification for acceptance of new work. The work must address ERP objectives and goals. For earmarked funds, proposals are reviewed by technical experts and are revised based on expert comments. Outside reviews are conducted periodically for laboratories and programs to assess scientific and technical merit as well as administrative efficiencies. The 21% of ERP funds awarded to institutional programs (Sea Grant and the National Undersea Research Program) are then competed by those institutions to meet research and information transfer priorities set at the National and Program level. These Institutions and Centers have unique capabilities that contribute to ERP's mission and goals.

Evidence: 1. Commerce Business Daily (cbdnet.gpo.gov/) 2. Example of interagency agreement (see Q 1.3 ECOHAB) 3. Number of publications in peer-reviewed journals [www.oarhq.noaa.gov/erp/documents/pubtotals.pdf] 4. National Sea Grant Program Authorization Legislation[www.sga.seagrant.org/legis/auth/sec_1126.htm]

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The ERP program can demonstrate progress on only two of its four long-term goals. While ERP is a recently unified program, the long-term goals of the ERP represent an integration of the goals of the component programs. As such, the programs that make up ERP have already made some achievements toward ERP's long-term outcomes. However, performance measures were not used to track this progress and the evidence is anecdotal.

Evidence: See Performance Measures Section

Does the program (including program partners) achieve its annual performance goals?

Explanation: ERP receives a small extent because it can only demonstrate achievement of two of its annual goals. However, ERP is a recently unified program, the annual goals of the ERP represent an integration of the goals of the component programs. The programs that make up ERP can also demonstrate success is meeting previous component-specific annual goals.

Evidence: See Performance Measures Section

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: ERP has made some progress in improving its efficiency, but cannot yet demonstrate progress in achieving its most significant annual efficiency goals. ERP tracks safety (the reported incident rate) as an efficiency measure. Improved workplace safety results in improved productivity. It also reduces financial risks associated with harm to people and pollution/contamination of property, and civil and criminal risks (e.g. legal liabilities) associated with noncompliance and violations. Between June 2004 and June 2005 NOAA decreased its incident rate from 1.97 to 1.59 injuries per 100 employees in a given year. This exceeded the target of 1.89 which was set based on the President's Safety, Health and Return to Employment goal of reducing incident rates by 3% annually. In addition, ERP has put in place strong financial controls and significantly improved management and planning. Examples include improved planning through the establishment of ERP, the new "grants on line" process, and significant gains in IT efficiency.

Evidence: 1. FY 2003 Department of Commerce Financial Report, Performance Accountability Report (www.osec.doc.gov/bmi/budget/FY03PAR444.htm) 2. NOAA Grants Management: www.ofa.noaa.gov/~grants/

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Direct comparison to other programs on ERP's mission effectiveness is difficult. However, the U.S. Ocean Commission Policy report highlighted a number of areas where NOAA is the lead agency (see evidence) and specifically cited Sea Grant as being worthy of increased support. Another indicator of the high regard for the performance of ERP programs is that they have been copied internationally. For example, Korea and Indonesia have started Sea Grant programs modeled on the US example; other countries have established an Extension Program modeled after Sea Grant. In addition, Portugal and Germany are adopting ERP's scientific approach for forecasting nutrient pollution and its impacts. Scientific quality audits and reviews of ERP programs that have been conducted have been favorable (see question 2.6). A broad array of partners in the public and private sectors has sought out our expertise and research capabilities, including other Federal agencies who transfer funds to ERP programs. ERP internal scientists compete successfully for funding, despite the additional responsibility to provide technical support for agency missions that may not result in peer-reviewed publications.

Evidence: 1. Merit Review process in DOC Interim Grants Manual [www.osec.doc.gov/oebam/pdf/CGA_manual.pdf] (see Q. 1.5) 2. Ocean and Coastal Research Activities of the National Oceanic and Atmospheric Administration (NOAA) and the National Science Foundation (NSF): Report on interagency activities in ocean and coastal research in FY 2005 and plans for coordination in FY 2006 (See Q 2.RD1) 3. U.S. Ocean Commission Policy Report [www.oceancommission.gov/documents/full_color_rpt/welcome.html]: "NOAA is the clear lead civilian ocean agency" (p. 112); NOAA should become the lead agency for "marine debris management" (p. 267), "management of deep water coral communities" (p. 326), "marine aquaculture" (p. 334), "implementing and operating the IOOS [Integrated Ocean Observing System]" (p. 398), "the transition of proven ocean-related technologies into operational applications" (p. 415), and "the establishment of "national virtual marine technology centers" (p. 426). NOAA should work with the Navy to "rapidly advance U.S. coastal and ocean analyses and forecasting capabilities  using all available physical, biological, chemical, and socioeconomic data" (p. 434). "Congress should significantly expand the National Sea Grant College Program as part of doubling ocean and coastal research funding." (p. 385) 4. "The Sea Grant Approach to Coastal and Marine Research, Extension, and Education: A Review of International Experience and Opportunities" (NOAA International Activities Office Background Paper #1); www.ia.oar.noaa.gov/Programs/sea_grant/seagrant.htm 5. The AQUARIUS undersea laboratory, owned by NOAA's Undersea Research Program (NURP), and managed by the University of North Carolina at Wilmington was chosen for use by NASA. Because of its unique nature and superb safety record, the AQUARIUS was chosen over sites proposed by research institutions and the commercial diving industry.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: As discussed in the response to Question 2.6, ERP programs receive periodic independent evaluations, generally every five years. These independent reviews have consistently shown the current programs to be effective and achieving results. NOAA's Science Advisory Board (SAB) external reviews have found ERP's programs to be effective in meeting NOAA's mission. There is also evidence that current research is effective and achieving results for intended beneficiaries. Research results are regularly published in peer-reviewed journals, a test of scientific integrity and relevance.

Evidence: 1. Science Advisory Board charter (www.sab.noaa.gov/Charter/charter.html) 2. Sea Grant Program Evaluation Process: Report of the Sea Grant Review Panel's Program Evaluation Committee (www.seagrant.noaa.gov/other/greenbook_doc/tollreport.pdf) 3. "Program Assessment - A New Performance-Based Program Evaluation Paradigm for the National Sea Grant College Program" (presented at the Eastern Evaluation Research Society Annual Meeting, April 2003) (www.seagrant.noaa.gov/other/greenbook_doc/prog_assess.pdf)