Detailed Information on the
Forest Service: Invasive Species Program Assessment

Continue to implement the integrated invasive species strategy based on input from the Regions and other customers; improve outreach and delivery of research and management information; and work to ensure states and other cooperators link their proposed activities to the Forest Service's Strategic Plan or annual performance measures. The Forest Service will also undertake other such actions as necessary to implement the desired outcomes of this plan.

Develop an analysis of the costs and benefits of treatments undertaken to reduce the impacts of invasive species and implement its use in program management. The Forest Service will also undertake other such actions as necessary to implement the desired outcomes of this plan.

Issue and track specific invasive species program performance targets to increase accountability of National Forests and partners to monitor and report treatment efficacy and long-term program outcomes. The Forest Service will also undertake other such actions as necessary to implement the desired outcomes of this plan.

Measure: Percent of acres at risk of mortality due to insect pests and diseases, or of infestation by invasive plants on which risk is reduced.

Explanation:Target is to reduce risk on 15% of 25 million "at risk" acres over the next 5 years. "At risk" acres are those which face 20% mortality over the next 15 years due to invasive insects and diseases. Risk is reflected in an insect and disease mortality risk map using updated predictive models, soil and climate condition layers, and improved inventory composition data for forested stands (a single consistent framework with 1 plot per 6,000 acres nationwide). This is useful for broad scale assessment, allows the program to establish priorities nationally among pests and across regions, so funding is focused on pests that contribute the most to risk of mortality.

Measure: Number of priority acres successfully treated for invasive species annually.

Explanation:Acres to treat are selected by science-based risk maps, risk assessments, forest plans, and other scientific methods. Success is measured as completing treatments in accordance with applicable scientific and regulatory standards.

Measure: Cost per acre treated to reduce risk.

Explanation:This measure reflects the ratio of total program costs to the number of priority acres succesfully restored or protected.

Measure: Number of treated acres monitored annually

Explanation:Post-treatment monitoring determines treatment efficacy and provides an adjustment factor used in calculating outcome and efficiency measures. Sample variance is low, and suggests 20% monitoring is quite adequate. In addition, 100% of pesticide applications on NFS lands will be monitored.

Measure: Score of Research and Development (R&D) customers reporting satisfaction with accessibility, relevance, outcome and cost effectiveness of tools developed, delivered, and used.

Explanation:R&D in the Forest Service seeks to provide quality products and services as assessed through evaluations from their customers.

Measure: The 5-year running average of the number of Research and Development (R&D) tools developed, delivered, and used.

Explanation:One critical role of research is to translate scientific developments into useful tools to resolve management problems. A tool, in this context, is defined as any method, technique, technology application service, model system, science synthesis, database, evaluation or monitoring protocol, prototype, operational application, and decision support system that is developed, maintained or revised to address current, and emerging invasive species issues. In 2007, this will reflect a 5-year running average. Between 2003 and 2006, the average is not based on 5 years.

Measure: Cost per Research and Development (R&D) tool developed, delivered, and used

Explanation:Dollars expended to develop and deliver a R&D tool. (5-year running average cost). The baseline and target are $426k, the average cost to develop a management tool. In 2007, this will reflect a 5-year running average cost. Between 2003 and 2006, the average cost is not based on 5 years.

Is the program purpose clear?

Explanation: The purpose of the Forest Service invasive species program activities is to reduce, minimize or eliminate the potential for introduction, establishment, spread and impact of invasive species across all landscapes and ownerships. Forest Service programs that contribute to addressing invasive species include Research and Development, Forest Health Protection, National Forest System and International Forestry.

Evidence: USDA Forest Service Strategic Plan for Fiscal Years 2004-2008; Forest Service National Strategy and Implementation Plan for Invasive Species Management (http://www.fs.fed.us/rangelands/ecology/invasives.shtml); FY 2006 Forest Service Budget Justification; Executive Order 13112; Forest Service Research and Development Invasive Species Logic Model; Information flowchart; USDA Forest Service Legal Authorities for Invasive Species Management.

Does the program address a specific and existing problem, interest, or need?

Explanation: The threat from invasive species is scientifically acknowledged, impacting human health, the environment and the economy of the nation. Forest health and productivity are at great risk, as increased wildfire occurrence has been linked to increased invasive species infestation impacts on the landscape. Economic impacts in the United States from invasive species are estimated to be, at a minimum, $138 billion annually (Pimentel, et. al., 2000) and invasive species are negatively affecting hundreds of millions of acres of native ecosystems, and associated native plants and animals.

Evidence: FICMNEW Weed Fact Book, 1998; Pimentel, et. al., 2000; The Nature Conservancy, 1996; Nowak, et. al., 2001; USDA Forest Service Legal Authorities for Invasive Species Management; National Invasive Species Council (NISC) National Invasive Species Management Plan.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The Forest Service is the sole entity authorized by statute to directly respond, control, monitor and conduct research in invasive species on all Federal lands, including 193 million acres of National Forest System lands, or to assume responsibility for invasive species responses and control when those species become established in rural and urban communities and forest lands. To avoid duplication with state and other efforts, the Forest Service provides planning and other assistance to State Foresters or Departments of Agriculture and other cooperators to accomplish management of invasive species. This design minimizes redundancies of efforts on both national forest and non-Federal forest lands. Excessive overlap of research is avoided by Forest Service collaboration with partners including academic institutions, and the research of this program provides value beyond research conducted by other entities. The Forest Service also works with States to ensure compliance with relevant regulatory standards required by other Federal or state entities that establish these standards.

Evidence: Management of national forests is delegated to the Chief of the Forest Service in Title 7, Code of Federal Regulations, Section 2.60. See also: USDA Department Regulation 9500-10; Forest Service briefing papers; Forest Service congressional testimony

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: There is no evidence that an alternative form of program design such as regulation would be more effective in meeting program goals, nor does the evidence indicate that structural changes are needed to increase efficiency or efficacy. The Forest Service is addressing needed program improvements such as performance measures through the program's current design. In 2004, the Forest Service examined its invasive species program and developed a National Strategy and Implementation Plan for Invasive Species Management to provide guidance to maximize effectiveness and efficiency within administrative and policy constraints. Key focus areas for the invasive species program are: 1) Science-based prioritization of invasive species problems; 2) Improved collaboration on solutions for those problems; 3) Streamline procedures; and 4) Enhance communication and education.

Evidence: Forest Service National Strategy and Implementation Plan for Invasive Species Management (http://www.fs.fed.us/rangelands/ecology/invasives.shtml).

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Within the authorities and jurisdictions of the Forest Service, the invasive species program is designed to focus resources on the National Forest System and other forest land ownerships in the private, local, State, Tribal, Federal and international sectors. Funds are targeted to appopriate cooperators and beneficiaries, including landowners and visitors to the National Forests, to meet program purposes. No alternative Federal source provides funding for program purposes, nor does program funding provide a subsidy to cooperators or beneficiaries. The invasive species research and development program relies on partnership capabilities, such as with universities, to provide scientific information and technology to land managers which address priority invasive species problems.

Evidence: Forest Service National Strategy and Implementation Plan for Invasive Species Management (http://www.fs.fed.us/rangelands/ecology/invasives.shtml); FY 2006 Forest Service Budget Justification; National Invasive Species Council (NISC) National Invasive Species Management Plan; FY 2004 Forest Service Accomplishment Report; Forest Health Protection Futuring Panel Report.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The Forest Services long term goal is to reduce, minimize of eliminate the potential for introduction of invasive species across all landscapes and ownerships. To address this goal, the Forest Service has developed a performance measure that tracks program performance in reducing the risk of mortality due to insect pests and diseases, or infestation by invasive plants on which risk is reduced over the next five years. Prioritization of which acres to treat is based on science-based risk maps, accessibility, probability of success, cost-effectiveness of treatment, and local capacity. A purpose of the Research program is to produce tools that will increase the effectiveness of restoration and/ or protection efforts on priority acres on federal and cooperative lands. Forest Service Research has developed an outcome based performance measure that tracks program performance as measured by customer satisfaction with research tools developed, delivered and used. The 2006 baseline score for customer satisfaction with the relevance, quality and timeliness of these tools was 72, and is consistent with the scores for other natural resource agencies. The "yes" answer for this question is based on an acknowledgement that 1) significant improvements are still needed in the transparency of economic and environmental effects of treatments that Forest Service is committing to making, and 2) the program will continue to competitively allocate funds each year from its national headquarters to address highest priority risks.

Evidence: 2006 Mortality Risk Map: The Forest Service developed an insect and disease mortality risk map using updated predictive models, soil and climate condition layers, and improved inventory composition data for forested stands (a single consistent framework with 1 plot per 6,000 acres nationwide). This is useful for broad scale assessment, allows the program to establish priorities nationally among pests and across regions, so funding is focused on pests that contribute the most to risk of mortality. The risk map is the work of 5 Regional teams, with input from almost 100 scientists and pest specialists. They developed the 189 pest models that reflect the best peer-reviewed science available for each disturbance agent layer and characterizes the combined effects of multiple stressors. A General Technical Report, as well as a report for more general audiences, are now in development. 2006 Research and Development Customer Satisfaction Survey: Score of Research and Development (R&D) customers reporting satisfaction with accessibility, relevance, outcome and cost effectiveness of tools developed, delivered, and used.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has target of reducing the risk on 15% of the estimated 25 million acres at risk of mortality due to insect pest and diseases, or of infestation by invasive plants, over the next five years. Program targets are focused only on completed treatments that successfully reduce risk, rather than solely acres treated. Forest Service Research has adopted the American Customer Satisfaction Index (ACSI) to set targets for customer satisfaction. Research and Development's initial target score is 72. The average score received by government agencies that have used this index over the past 7 years is 70.3. This target will be adjusted based on the results of first survey, which is in progress and will be conducted once every 3 years. In 2009, the target will be 74. Baselines exist for both outcome and output measures. Risk is ongoing, so definition of an end target is not applicable. Given these factors and the scope of activities, targets within the timeframes are ambitious but realistic.

Evidence: USDA Forest Service Strategic Plan for Fiscal Years 2004-08; 2006 Mortality Risk map; ASCI Overall Federal Government Scores with Historical Scores of Agency Measured 1999-2005 (http://www.theacsi.org/government/govt-05.html); Deputy Chief letter to customers introducing FY 2006 customer satisfaction survey and survey instructions; FY2006 Forest Service Program Direction; 2006 Research and Development Customer Satisfaction Survey.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The agency has developed annual output measures that demonstrate performance toward achieving long term goals. These annual output measures provide discrete, measurable and quantifiable progress, and include the number of acres treated for invasive species and the number of acres monitored annually. Because of the dynamic nature of forest health, only longer-period timeframes provide meaningful information concerning risk trends, so annual updates of risk are not suitable and annual outcome measures are not appropriate. Rather, given the large but finite number of acres upon which program actions can occur, annual output measures focusing on acres treated provide insight on the results of annual and critical data needed to calculate the long-term outcome. The Forest Service has developed a measure of research tools developed and delivered and in turn used by customers. This measure, combined with the customer satisfaction index, allows the program to track tends in the adoption and use of the research tools it develops, in turn permitting the program to assess its progress in scientific understanding and its application. The agency has also established measures to evaluate program efficiency which include cost per priority acre restored and/or protected and cost per tool developed, delivered and used. This permits the program to continually focus on the efficient attainment of long-term goals.

Evidence: FY2006 Forest Service Budget Justification; FY 2005 Forest Service Program Direction; FY2006 Forest Service Program Direction.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The annual output measure for acres treated shows a decline of 36% below the baseline level although available resources have remained fairly constant. While this may accurately reflect a programmtic shift to devote more resources to early detection and rapid response activities which do not result in many acres, the reduction appears to consitute a rebasing and does not provide a challenging or amibitious target. Targets for the research component of the program are ambitious. The Forest Service established a baseline of 106 research tools developed, delivered and used in FY2003. This target was exceeded in 2004 and 2005 because of effective partnerships, the production of multiple tools from a line of research, and the development of a more comprehensive invasive species research program. Therefore, Research's target was increased to 133 in 2006, which is the most recent three-year average. This target will be refined to the five-year average in FY2008 so that the data reflect the larger program.

Evidence: USDA Forest Service Strategic Plan for Fiscal Years 2004-08; FY 2006 Forest Service Budget Justification; FY 2003 and FY 2004 Forest Service Accomplishment Reports; 2004 Insect and Disease Condition Reports, Forest Health Protection; R&D Forest Inventory and Analysis P3 Understory Vegetation Indicator Document; Research Work Unit Five-year Charter; Integrated Goal Strategy Team Document; FY 2006 and FY 2007 Forest Service Program Directions.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: On non-Federal watersheds, State Foresters and Departments of Agriculture are the Forest Service's primary partners, but most State Forester or Departments of Agriculture workplans do not link their proposed activities to the Forest Service's Strategic Plan or annual performance measures. Program allocations also do not link to the Forest Service's Strategic Plan. However, on national forests, the partners program support the program goals and report accomplishment in furtherance of those goals. Forest Service Research and Development ensures commitment from academic and state organizational partners when establishing cooperative agreements that support the program.

Evidence: FY 2004 Forest Service Accomplishment Report; Forest Health Protection Futuring Panel Report; Species Specific Strategic Plans, Forest Health Protection; Forest Health Protection reports and risk maps on Sudden Oak Death and Emerald Ash Borer; Research & Development Cooperative (Coop) Agreement, Grant Agreement, Accomplishment Report; Economic Action Grant to State of Pennsylvania.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The General Accounting Office (GAO) audited USDA's invasive species programs on several invasive species in 2000, 2001, 2002 and 2003. GAO recently completed an audit of the emerald ash borer, Asian long-horned beetle and sudden oak death programs, which has not yet been published. In May 2002, the National Association of State Foresters (NASF) conducted a review of the entire State and Private Forestry program including invasive species. Research's invasive species program will be evaluated by an External Peer Review process, This is a rigorous, formal, systematic paneled review modeled after other USDA research agency methods. The initial review process will be conducted during the 2006, starting a regular five year interval review cycle. Information from the review will help research identify strategic areas most in need of improvement, as well as areas of strength. Additional examples of independent evaluation of Research and Development includes the logic modeling efforts (2003) to be responsive to the Investment Criteria Process, as well as, our regular five-year cycle of of station level program reviews (Technical Assistance Visit process or TAVs).

Evidence: National Association of State Forester's Review of State & Private Forestry Washington Office Programs; General Accounting Office reports on invasive species 2000, 2001, 2002, 2003; Research & Development Stakeholder Review of Logic Model for Invasive Species; Rocky Mountain Research Station Listening Session Strategic Plan; Forestry Research Advisory Panel Report to the Secretary of Agriculture's Blue Ribbon Panel on America's Forest Research; Forest Service Research & Development Strategic Program Area (SPA) Review Team Handbook.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The Forest Service cannot ensure that the accomplishment information it provides to Congress and other interested parties is consistent, valid, and supported, rendering it largely ineffective in managing Forest Service operations. Also, the Forest Service's Performance and Accountability Report was based on inaccurate, inconsistent, and unverified data, and thus unreliable for making sound decisions. Consequently, resource allocations do not reflect desired performance levels nor are the effects of funding and other policy changes clear. The Forest Service has made some improvements in its planning processes but has not effectively implemented these improvements. For example, in FY 2006 managers must link project budget requests and accomplishment reporting to agency long-term strategic goals.

Evidence: USDA Office of Inspector General Audit Report No. 08601-1-Hy, Forest Service Implementation of the Government Performance and Results Act, found that standards defining performance vary between regions and national forests and even among the districts within a single national forest. USDA Forest Service Strategic Plan for Fiscal Years 2004-08; Forest Service FY 2004 - FY 2006 Budget Justifications; FY 2007 USDA Forest Service Budget Justification (http://www.fs.fed.us/aboutus/budget/); FY 2004 - FY 2006 annual program directions; FY 2006 USDA Forest Service Program Direction; FY2006 Invasive Species Workplan

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The Forest Service National Invasive Species Strategy and Implementation Plan outlines program objectives and provides recommendations for implementation milestones. To address needed improvement in program metrics, the Forest Service is developing quantitative goals and performance measures for the program. The agency is incorporating these new measures into its performance and accountability system, but additional work will be required. To address additional performance and accountability needs, each geographic region of the Forest Service is to work with research counterparts to develop a regional plans to identify priorities. To addresses outcomes and management needs, the research component has developed a logic model for invasive species research that, in conjuction with an external peer review process, will lead to the development of a Forest Service Research and Development Invasive Species Strategic Plan to identify research priorities and produce desired tools that address strategic goals and improve performance and accountability.

Evidence: USDA Forest Service Strategic Plan for Fiscal Years 2004-08; Forest Service National Strategy and Implementation Plan for Invasive Species Management (http://www.fs.fed.us/rangelands/ecology/invasives.shtml); Forest Service Research and Development Invasive Species Logic Model.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: The Forest Service Research and Development staff conducts species specific research on current known and potential invasive species as well as their habitats. The research program is accomplished through the agency-wide strategic planning and budget processes involving the Research Executive Team (FSRET). FSRET conducts annual assessments and compares potential benefits of Forest Service Research and Development (R&D) efforts against other similar research according to the nature and magnitude of current and anticipated issues and information needs required by land managers for effective natural resource management. The Forest Service compares research and development findings annually with participant agencies and organizations of the National Invasive Species Council and research scientists, who have validated the Forest Service program's credibility. Forest Service scientists also compare potential benefits against other programs when participating in professional organizations and related meetings. For example, invasive species research scientists meet annually at interagency research forums and workshops with different federal, state, and local agencies, universities, and other invasive species-related organizations to analyze and compare potential benefits and research products in a regional forum that fosters independent verification.

Evidence: National Invasive Species Council (NISC) National Invasive Species Management Plan; Forest Service Sudden Oak Death Table of Responsibilities; Annual Gypsy Moth and Other Invasive Species Research Forum Proceedings.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: Prioritization for the Invasive Species R&D program is accomplished through the R&D-wide strategic planning and budget processes involving the Research Executive Team (FSRET). FS research and development (R&D) prioritizes research annually according to the nature and magnitude of current and anticipated issues and information needs required by land managers for effective invasive species management. Development of research priorities is based upon OMB R&D investment criteria for relevance and applicability; support of technology development needed to produce forest sustainability. The Forest Service annual program direction documents R&D priorities each year and includes research on invasive species as one of the national priorities. Invasive Species R&D priorities are: prevention and detection of invasive species, detection, response and eradication of invasive species, management and mitigation, and restoration and rehabilitation of impacted ecosystems. Requests and allocations reflect this process and these priorities. The Forest Service Research Executive Team (FSRET) consisting of the Deputy Chief, Station Directors and Washington Office Program Directors develop national priorities for the Research Stations based on the Forest Service Strategic Plan. The Research Stations also prioritize projects within stations and within programs.

Evidence: Forest Service Research Executive Team (FSRET) Directions to Staff and Stations

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: While the Forest Service regularly collects program performance information and tracks participation and performance of agency program partners and grantees, a key component of its performance data system will not be deployed until FY 2007. This element, the Performance Accountability System (PAS), will integrate key budget, financial and performance information from existing data systems to provide customer-focused performance information to aid decision making, including priorities and resource allocations. The Forest Service has deployed the other portion of the system, WorkPlan, for local to national level project development, budget tracking and performance monitoring and reporting. Baseline data for each element of the program's performance is not complete. However, management efficiency will be measured as cost per acre restored or protected. A strategy that describes how Research and Development will contribute to the Agency's invasive species goals will be finalized within 6 months of the External Peer Review (Summer 2006). This 15 year strategy will improve program focus, identify high priority research and capacity needs, and clearly define our program goals and associated measures. The strategy will include input from internal and external partners, and will be adjusted to provide feedback from the External Peer Review. Information from the review will help in the identification of strategic areas most in need of improvement, prioritization of the budget, and the focus of IT and capital resources investments.

Evidence: USDA Office of Inspector General Audit Report No. 08601-1-Hy, Forest Service Implementation of the Government Performance and Results Act. FY 2006 Forest Service Program Direction; FY 2005 Performance and Accountability Report (March, 2006) pages B 8-10 (http://www.fs.fed.us/plan/par/2005/docs/par-annual-perform-2005.pdf); Performance Reporting Regional Allocations; Research Station Data on Coop Agreement and Contracts; Forest Health Protection State Health Highlights (various state examples); Forest Service Research and Development Strategic Program Area (SPA) Review Team Handbook.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: All agency line officer (Regional Foresters, Forest Supervisors and District Rangers) performance standards and evaluation criteria include requirements in support of national performance measures related to agency strategic goals and objectives and assigned annual performance plan targets. Agency managers are accountable for costs, schedules and performance results as documented in the project budget and performance accountability WORKPLAN data system and in performance evaluations that assess performance under GPRA strategic goals, including improving watershed conditions. Accomplishments are reported and tracked through WORKPLAN and beginning in FY 2007, reported through the performance and accountability system (PAS) database. Research line officer performance standards and evaluation criteria include requirements on GPRA goals, and the attainment of annual performance targets. Cost, schedule, and results of research work units are evaluated every 5 years. Grants are monitored for performance results. Research cooperative agreement instruments outline performance standards and require periodic status reporting in order to obtain funding.

Evidence: Information flowchart; FY 2004 Forest Service Accomplishment Report; FY 2007 Budget Justification Partnerships Special Exhibit, page 15-87.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: The report of an independent auditor on the Forest Service's FY 2004 and FY 2005 Financial Statements stated the "USDA Forest Service does not obligate all transactions required by appropriations law" as well as its belief that the USDA Forest Service may not be in compliance with the Anti-Deficiency Act, 31 U.S.C. 1517. Other systems appear to be functioning well, however. For example, Federal and partner funds are obligated in a timely manner and reviewed on a quarterly basis for accuracy, timeliness, and intent. These process will be reviewed as part of the Forest Service's internal controls as prescribed by Circular A-123. Forest Service policy requires a project to charge expenditures to the fund code appropriated by Congress for the purpose of achieving the goals of the intended project. Additionally, the WorkPlan system requires units to plan every project, including identifying the funds that will be used, and links the project to the established performance measure associated the funds chosen.

Evidence: In its Forest Service Financial Statements for Fiscal Years 2005 and 2004 (Report No. 08401-5-FM, November, 2005), the independent auditor Obligation testwork performed over approximately 132 transactions disclosed that about twenty percent of tested transactions were not obligated as required by appropriation law prior to payment, including temporary travel, GSA automobile leases, and probable contingent liability type transactions. It also reported that to maintain administrative control of funds, the USDA Forest Service made sub-allocations to its organizational components, and stated that at the end of FY 2005, they understood that the USDA Forest Service's Region 5 had obligated funds in excess of its sub-allocation by approximately $4.0 million. The Forest Service is working to resolve these issues. Circular A-123 internal control narratives and control matrix; FY 2006 Program Direction; Audit Report, Forest Service FY 2004 and FY 2005 Financial Statements.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has taken a number of actions to improve efficiency. The Forest Service developed an appropriate efficiency measures to track unit costs. The Forest Service recently completed a competitive sourcing study on IT that is leading to enhanced efficiencies and reduced costs to the program. The agency utilizes the Forest Service Activity Tracking System (FACTS) to track accomplishments across all levels of the Forest Service. The agency is identifying strategies for providing incentives to units with high levels of efficiency to optimize treatment benefits and has identified unit costs or timing targets. Programs such as IT and the Forest Health Technology Enterprise Team, which contribute to invasive species program, are reviewed for cost effectiveness. In 2004 and 2005, Forest Health Protection conducted reviews of four Regional offices and the Slow-the-Spread program. The program has expanded the use of the internet to reduce program delivery costs while providing information that can help the public participate in invasive species control, such as Exotic Forest Pest Information System (EXFOR), the Mortality Risk Map (now in the final stages of revision), and the new Sirex risk map.

Evidence: Forest Service National Strategy and Implementation Plan for Invasive Species Management (http://www.fs.fed.us/rangelands/ecology/invasives.shtml); FY 2006 Forest Service Budget Justification; FY2006 Forest Health Protection Economic Analysis; RFP for an economic analysis of gypsy moth Slow-the-Spread; Forest Service Research and Development Invasive Species Logic Model; Information flowchart; Forest Health Technology Enterprise Team (FHTET) Review; Regional Review; Slow-The-Spread (STS) Review; FACTS database; Sirex risk map http://www.fs.fed.us/foresthealth/technology/invasives_sirexnoctilio_riskmaps.shtml

Does the program collaborate and coordinate effectively with related programs?

Explanation: Close collaboration among programs within and external to the Forest Service is critical to program success and contributes to management and resource allocation actions. The Forest Service collaborates externally with the participating agencies and Departments in the National Invasive Species Council, international partners, States, Tribes, local governments as well as private non-governmental organizations. For example, the Forest Service collaborates with private nursery owners, universities, States and other Federal partners to address Sudden Oak Death. Many of these partners contributed information which was used to develop the Sudden Oak Death risk map. The map in turn is used by the Forest Service and its partners to prioritize survey and management activities.

Evidence: Executive Order 13112; USDA Forest Service Legal Authorities for Invasive Species Management; National Invasive Species Council (NISC) National Invasive Species Management Plan; Forest Health Protection State Health Highlights (various state examples); Federal Interagency Committee for the Management of Noxious and Exotic Weeds Brochure and Report; Inter-departmental MOU on Invasive Species Cooperation; Forest Health Protection FY 2004 Funding Allocations for Invasive Species Suppression, Prevention, and Restoration Projects; Project Prioritization Process; FY 2004 Forest Service Accomplishment Report; Sudden Oak Death Risk Map, Agency responsibility table from the Sudden Oak Death Strategic Plan.

Does the program use strong financial management practices?

Explanation: The program follows the Forest Service's financial management guidelines for committing, obligating, reprogramming, and reconciling appropriated funds. However, an independent auditor's report on FS' internal control structure over financial reporting identified five material internal control weaknesses, each of which has a direct relation to the program. The auditor's report on Forest Service laws and regulations contains instances of noncompliance with appropriations law and instances of noncompliance with the Federal Financial Management Improvement Act of 1996. The same report also indicated the "USDA Forest Service does not obligate all transactions required by appropriations law" as well as its belief that the USDA Forest Service may not be in compliance with the Anti-Deficiency Act, 31 U.S.C. 1517.

Evidence: FY 2002 (08401-1-FM), FY 2002 and FY 2003 (08401-3-FM), FY 2003 and FY 2004 (08401-4-FM) and FY 2004 and FY 2005 (08401-6-FM) Forest Service Audit Reports from KPMG LLP and reviewed by USDA Office of Inspector General (http://www.usda.gov/oig/rptsauditsfs.htm )

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The Forest Service Strategic Plan for 2004-2008 provides national direction for invasive species under Goal 2. The National Strategy and Implementation Plan for Invasive Species identified deficiencies in the Forest Service invasive species program, and proposed measures for improvement. A logic model was developed for invasive species research to strengthen the link between research and management issues. Integrated (National Forest System, State & Private Forestry, and Research and Development) Regional Invasive Species Teams have formed to prioritize regional needs, identify obstacles, and develop roadmaps for success in controlling invasive species. This process permits the program to correct identified deficiencies and to make necessary program adjustment to address them. The first Forest Service national conference on invasive species was held in June 2006 specifically to identify and address obstacles to success in managing invasive species, and to prioritize activities.

Evidence: USDA Forest Service Strategic Plan for Fiscal Years 2004-08; Forest Service National Strategy and Implementation Plan for Invasive Species Management http://www.fs.fed.us/rangelands/ecology/invasives.shtml; Forest Service Research and Development Invasive Species Logic Model; FY 2003 Forest Service Accomplishment Report; Economic Action Grant to State of Pennsylvania; National Invasive Species Conference invitation and agenda.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Annual invasive plant program grant proposals are submitted by Forest Service regional offices and the Northeastern area to the Washington Office. Those proposals are ranked according to specific evaluation criteria and funding awarded to the highest ranked proposals. Insect and disease project proposals are treated in a similar manner. More than 90 percent of awards are made through this process. Awards by law can be provided only to States, but projects consider participation of new parties including non-governmental organizations, as appropriate. Projects must demonstrate a federal role, be biologically effective and economically efficient. Nearly all funding is fully obligated annually. However, should the Forest Service implement an alternative allocation methodology as it now contemplating, competition may diminish, in turn reducing both the likelihood of having only the most meritorious projects selected and seriously compromising the competitive awarding of the majority of funds.

Evidence: An example of the competitive process is reflected under "Forest Health Protection" at: http://na.fs.fed.us/stateadvice/exhibitb/2006/exhibitb06.htm. Forest Service and California Department of Food and Agriculture Grant Agreement and Accomplishment Report; Forest Service and Pennsylvania Dept. of Conservation and Natural Resources Gypsy Moth Grant Agreement and Accomplishment Report; Southern Research Station Invasive Species Extramural Research and Grant Package; Sudden Oak Death Request for Proposals, 2006

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The Forest Service provides oversight of the activities of grantee and cooperative agreement participants and tracks their expenditures. Forest Service personnel work closely with cooperators/recipients and clarify regulations, policies, and procedures that may be unclear and/or new to them; determine the need for and, if applicable, conduct post-award site visits or meetings; review financial reports and ensure that payment requests (reimbursables and final requests) are reviewed and approved by the Program Official prior to certification for payment by the Certifying Officer; prepare modifications to the instrument as needed; and determine if corrective action is needed (for example, recommendation for debarment, suspension, financial controls, and so forth).

Evidence: Forest Service Manual Section 1580. Southern Research Station Invasive Species Extramural Research and Grant Package

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: States, territories, national forests, and other cooperators report annual accomplishments in the Performance Measurement Accountability System (PMAS) through the Forest Service Regions or the Northeastern Area, who then summarize regional or area-wide program and partnership accomplishments for publication in the Forest Service annual accomplishment report. Information is publicly available online.

Evidence: The 2004 accomplishment report is available online (http://www.fs.fed.us/spf/coop/library/2003%20CF%20Accom.pdf). The Forest Service also recently completed a year-long effort to revise cooperative forestry performance measures.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: Forest Service Research and Development (R&D) will increase the total R&D funding in extramural work based on merit from the current 13 percent to 20 percent over the next five years. For all other funding and to enhance accountability, R&D managers are required to demonstrate the extent to which their programs meet the following three tests: (1) managers must be able to articulate why this program investment is relevant and high priority; (2) managers must justify how funds will be allocated to ensure quality; and (3) mangers must be able to monitor and document how well this investment is performing. Forest Service R&D has begun an external peer review process to review the relevance, quality, and performance of specific R&D programs, including invasive species. This action addresses all R&D investment criteria, but especially focuses on quality.

Evidence: USDA Scientific Research Guidelines and USFS Northeast Research Station Quality Assurance/Quality Control Plan; USDA Forest Service Science Consistency Review Report and Letter to Stations

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Program long-term performance goals are new, so progress has yet to be measured. Some baseline data for 2005 are presented. Percent of acres at risk of mortality due to insect pests and diseases, or infestation by invasive plants on which risk is reduced is measured by accumulating annual accomplishments by the program. The target is to reduce risk on 15% of at risk acres over the next 5 years. An informal Research and Development customer survey demonstrated extremely high customer satisfaction with the publications produced, but this does not meet PART standards for a long-term performance goal. In FY 2006, an external organization is evaluating customer satisfaction with Research and Development tools through the American Customer Satisfaction Index.

Evidence: USDA Forest Service Strategic Plan for Fiscal Years 2004-08; FY2006 Forest Service Program Direction; FY2003 and FY 2004 Forest Service Performance and Accountability Report; ASCI Overall Federal Government Scores with Historical Scores of Agency Measured 1999-2005 (http://www.theacsi.org/government/govt-05.html); Research and Development Deputy Chief letter to research customers introducing FY 2006 customer satisfaction survey and survey instructions; Highlights of Invasive Species Research and Development Tools Adopted by Customers 2001-2005; 2006 Research and Development Customer Satisfaction Survey.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program has had mixed results meeting targets. The Forest Service meets some of its invasive species annual performance goals as reported in the FY 2007 Congressional budget justification and the FY 2004 Performance and Accountability Report. Output targets were exceeded in FY2005 for federal and cooperative land acres treated and acres monitored. Research and Development exceeded its annual target every year since 2003, because of an increased emphasis on invasive species and program consolidation. However, targets associated with the National Forest System portion of the program were not met.

Evidence: FY2007 Forest Service Budget Justification; FY 2004 Forest Service Performance and Accountability Report summary; FACTS, WORKPLAN

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Units costs have risen and continue to rise. Under the National Strategy and Implementation Plan for Invasive Species management, the program has focused more resources toward prevention, early detection, and rapid response activities. Although these activities do not lessen the cost per acre restored and/or protected, they are generally considered more cost effective than treating invasive species once they are established. Even assuming the benefits of cost-avoidance because the benefits of such activities are difficult to measure credibly, efficiencies are not realized. Annual costs are applied to acres successfully protected and/or restored, rather than to the total acres treated; thus they are tied to outcome performance. New technologies developed by research are improving the efficiencies of detecting, protecting and restoring acres. Research's cost per tool efficiency measure has decreased every year since 2003. These lower actual costs to USDA Forest Service, reflect Research's ability to leverage funding and partnerships, and to produce multiple relevant tools from one line of research. Further, the Forest Service has conducted a competitive sourcing competition on IT that is contributing to program efficiencies.

Evidence: Forest Service National Strategy and Implementation Plan for Invasive Species Management (http://www.fs.fed.us/rangelands/ecology/invasives.shtml); Forest Service Research and Development Invasive Species Logic Model; FY 2006 Forest Service Program Direction; FY2007 Forest Service Program Direction; Highlights of Invasive Species Research and Development Tools Adopted by Customers 2001-2005.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The Forest Service was the first member agency of the National Invasive Species Council, a group of Federal agencies that work on invasive species, to give top priority to the invasive species issue and develop a comprehensive strategic approach to address the invasive species problem. Independent evaluations from GAO and the USDA Inspector General have occurred in conjunction with congressional oversight: Results of Forest Service invasive species management compares favorably with other government and private programs due to the agency's 1) use of comprehensive analyses of the ecological, environmental, and economic consequences of alternatives for invasive species treatments; 2) characterization of how invasive species disturbances interact with other disturbance processes, such as windstorms, fires, and disease; and 3) development of guidelines for incorporating these tradeoffs into land and resource management plans.

Evidence: Forest Service National Strategy and Implementation Plan for Invasive Species Management (http://www.fs.fed.us/rangelands/ecology/invasives.shtml); R&D FIA P3 Understory Vegetation Indicator Document; General Accounting Office reports on invasive species 2000, 2001, 2002, 2003; Risk maps for invasive species.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Some individual components of the program are currently being evaluated. In FY 2006, the Research and Development Deputy Area plans to implement an external peer review of the effectiveness of the invasive species program at three to five year intervals. The review panel will score the program on its relevance, quality and performance. Research and Development used an investment criteria process with external stakeholders to review the invasive species program and make recommendations for future research direction, and uses scientific panel reviews that include stakeholders to focus research direction. Independent evaluations from GAO and the USDA Inspector General have occurred in conjunction with congressional oversight: GAO reported that agency actions are likely to contribute to preventing and controlling invasive species and recommended long-term goals toward which the program should strive. These have since been developed. In May 2002, the National Association of State Foresters conducted a review of the State and Private Forestry program, including invasive species, and found the program is effective.

Evidence: Forest Service National Strategy and Implementation Plan for Invasive Species Management (http://www.fs.fed.us/rangelands/ecology/invasives.shtml); General Accounting Office reports on invasive species 2000, 2001, 2002, 2003; National Association of State Foresters (NASF) Review of State & Private Forestry Washington Office Programs; Forest Service headquarters review of three western regional invasive species programs; Research & Development Stakeholder Review of Logic Model for Invasive Species; Rocky Mountain Research Station Listening Session and Strategic Plan; Forest Service Research and Development Strategic Program Area (SPA) Review Team Handbook; Forest Health Protection Regional Review.