Detailed Information on the
Research on Protection and Safety of Agricultural Food Supply Assessment

Re-evaluating the criteria that it uses to determine outyear targets for project quality, for the purpose of establishing more ambitious targets.

Review and revise the planning, implementation and external review processes of the National Research Program 5-year cycle.

Target resources to the highest priority threats to food safety and crop and livestock protection.

Ensure that research findings are used by USDA customers.

Enhance the tracking of measures, as well as the use of research and technologies, in the budget justification.

Conducting an independent external Retrospective Panel Review of the Veterinary, Medical, and Urban Entomology program during FY 2007.

Monitoring the long term measures to show the actual use of research outputs (new knowledge and technologies)

Continuing to submit Congressional earmarks and add-ons to the Office of Scientific Quality Review (OSQR) for independent external peer review to help ensure quality.

Measure: Additional research funds leveraged from external sources.

Explanation:There are several sources of additional funds to ARS including grants, trusts, reimbursables, specific agreements, and cooperative development research agreements (CRADA's). Grant funds are variable and inconsistent and are not a direct measure. Leverage (cost-sharing) can however, be directly measured by CRADA's, trusts and reimbursables, which indicate the level of partnerships between ARS and external parties interested in developing, transferring and commercializing ARS technology. The target is the percent of additional funding leveraged directly for research purposes. That is, additional funding over and above the total appropriated ARS budget, including the management overhead. The target is calculated from a running average of the previous 2 fiscal years. Targets for fiscal years 2006-2009 can only be estimated since the appropriated ARS budget is unknown.

Measure: Cumulative number of new technologies developed and used by ARS customers to detect, identify, control, and/or eradicate domestic and exotic diseases affecting economically important crops as determined by APHIS, ERS, EPA, NASA, DHS, Corn Growers Association, American Soybean Association, etc.

Explanation:The purpose of this measure is to develop new technologies that assist ARS customers to detect, identify, control, and/or eradicate diseases, insects, and weeds that cause the majority of crop losses. Recent examples of methods (brackets indicate customer who adopted technology) that have been transferred and implemented include: detecting Soybean rust (APHIS) and used to detect soybean rust in 9 U.S. States in 2004; detecting Bacterial Wilt [Ralstonia] a USDA Select Agent that infects a wide range of commercial plants: used by (APHIS) to quarantine shipments of host plants to limit the spread of this disease; the Sudden Oak Death host range list (APHIS) the basis for the quarantine program to control the spread of this disease outside California; and Asian Citrus Canker (APHIS) determined the critical distance needed to remove Citrus trees surrounding infected trees which is the basis of the eradication program.

Measure: Total program analysis through a retrospective independent external program review and assessment.

Explanation:This measure addresses Q. 2.6 in the PART analysis (Strategic Planning) framed by the R&D Investment Criteria. The score by independent expert panel reflects their judgment of the National Program in its totality at the end of the 5-year cycle (retrospectively). Beginning in FY 2004, about 20% of ARS' program will be evaluated annually for a series of factors that include: Relevance, Quality, and Performance. This summary evaluation combines data from 3 pilot assessments (conducted at the end of the 5-year program cycle for 3 National Programs) serves as the baseline score for future retrospective reviews. ARS is now implementing a policy to govern future retrospective reviews by external panels framed by R&D Investment Criteria. The average score for National Programs ending their 5-year cycle in FY 2004 was 78 out of a possible 100. This information is used in formulating the agency's budget and in making program management decisions.

Measure: Percent of projects receiving a passing score by independent, external, expert panel on the first submission through the OSQR review process.

Explanation:This measure tracks research project and program quality. To ensure quality, 20% of ARS' research projects are reviewed annually. Three classifications: "moderate," "minor," or "no revisions" are considered as passing grades and require no further panel review. All projects scoring "major revision" or "not feasible" are either terminated, or rewritten and resubmitted for re-review. In the last 5 years, all ARS projects have been reviewed, with 72.3% of the projects receiving passing grades in FY 2000. This compares to 76.7% in FY 2004. The target score per year between FY 2000 through FY 2004 could not be determined until the entire research portfolio had been evaluated, hence for these years a not determined score (ND) was given. Rounding of the median score for all ARS projects from FY 2000 through FY 2004 will be the target for FY 2005. The expert panel score (prospective) can be related to the 5-year independent evaluation score (retrospective) shown in Performance Measure 7.

Measure: Cumulative number of new diagnostic tests developed and used by ARS customers to identify, control, and/or eradicate economically important and/or easy targets for domestic and exotic diseases affecting animal and human health as determined by APHIS, ERS, FDA, DHS, DoD, American Association Veterinary Lab Diagnosticians, and the World Animal Health Organization.

Explanation:The purpose of this measure is to develop new diagnostic tests that assist ARS customers to detect, control, and/or eradicate diseases affecting animal and human health. Research focuses on Brucellosis, Foot and Mouth Disease, Avian Influenza, BSE, and Exotic Newcastle disease. Recent examples of successfully adopted technologies (brackets indicate customer who adopted technology) include methods for Avian Influenza (APHIS); Newcastle Disease (APHIS) implemented through the National Animal Health Laboratory Network; and BSE (APHIS). The BSE test was the 'gold standard' used to conclusively identify the BSE cow. The test is now used for 'definitive confirmation' on suspected positives detected by field tests conducted in the National APHIS-BSE surveillance program.

Measure: The percentage of annual research project milestones met.

Explanation:To measure the performance of the research program, in FY 2004, ARS adopted the OMB R&D Investment Criteria as its annual measure for all research. All research projects are now assessed annually to determine the number of milestones, as described in the 5-year project plans, which were met/not met. Information as to why a milestone was not met was collected and will be used for making program management decisions. By implementing this measure, ARS modified its Strategic Plan, incorporated the data requirements into its GPRA process, and modified its data collection process to report needed project and program information. Beginning in FY 2005, ARS has specifically included program milestones in all new National Program Action Plans.

Measure: Percentage of projects annually assessed to have direct relevancy in contributing to the achievement of ARS long-term goals.

Explanation:In FY 2004, ARS initiated a new assessment mechanism to measure relevancy of its research projects. All of ARS' research projects are now assessed annually against the Program Action Plans, the Agency's long-term goals, and the priority needs of U.S. agriculture. The results from the first review conducted in FY 2004, indicated that 97.1% of ARS' projects were conducting highly relevant research. Those projects considered of lower relevance (2.9%) were subsequently re-evaluated, and 41% were redirected to higher priority research. In FY 2005, a second evaluation was conducted and the percent of projects considered of lower relevance had been significantly reduced to 1.7%. Information from this evaluation process is used directly for making program management decisions and in formulating the agency's budget.

Measure: Cumulative number of new technologies developed by the Program and used by ARS customers to detect, identify and control the most critically and economically important foodborne contaminants causing illness, death, or chronic disease that impact public health and industry as determined by the FSIS, APHIS, ERS, CDC, FDA, DHS, DoD, Risk Assessment Consortium and Codex Alimentarius Commission.

Explanation:The purpose of the measure is to develop new technologies that assist ARS customers to detect, identify, and control food-borne diseases affecting human health. Research focuses on the pathogens Campylobactor, Salmonella, Giardia and Noroviruses that cause the overwhelming majority of foodborne illnesses: E. coli O157:H7; Listeria and Clostridium because of their high morbidity and mortality; mycotoxins; ricin toxin; botulinum neurotoxins; dioxins; pesticides; antibiotics; and hormones. Recent development examples include (brackets indicate customer who uses the technology: Salmonella (eLEXNET-FSIS); E. coli O157:H7 (FSIS, Maxwell Sensors-U.S. Army; Campylobacter (Silliker Labs-Worldwide, Iceland-EU); RNA viruses (CDC, FDA, State Health Dept's); Trichina (FSIS, APHIS, National Pork Board, EU); QuEChERS for multiple pesticides (FSIS, FDA, EPA and EU); Staphylococcal toxins (FSIS); fluoroquinolone-antibiotics (FSIS, FDA), beta-agonists and growth promoter hormones (FSIS).

Is the program purpose clear?

Explanation: The mission of the Agricultural Research Service (ARS) for Strategic Goal 3 is to provide research results that is aimed at protecting and securing the Nations's food supply and agricultural production systems and to reduce the instances of food borne illnesses. ARS research results on critical issues will be used by Federal & State governments & the private sector. Major activites include the protection of livestock production systems, crop production systems and food safety. The research is designed to ensure the health of our food animals; assure that the food supply is safe & secure for consumers & that food & feed meet foreign & domestic regulatory requirements; reduce losses caused by plant diseases while maintaining environmental quality; & provide technologies to manage pest populations below economic or environmental damage thresholds by the integration of compatible strategies. To ensure relevancy, ARS seeks input from its customers, partners, & users to prioritize its research agenda & focus it on the most critical issues & needs with a goal of providing cost-effective, solution-oriented technologies.

Evidence: The ARS Program to Enhance the Protection & Safety of the Nation's Agricultural & Food Supply (Program) was authorized under Pub. L 104-127, Title VIII, Section 801, "Purposes of Agricultural Research" to "improve risk management, improve the safe production & processing of U.S. food resources, & maintain an adequate & safe supply of food." ARS' research for Goal 3 is clearly defined by the current USDA/REE/ARS Strategic Plans. The USDA Strategic Plan 2002-2007 states: "USDA has unique & critical responsibilities to help ensure the safety of the U.S. food & fiber supply chain & the security of the U.S. agricultural production system," & "Research programs are essential to an effective U.S. agricultural biosecurity program to develop methods of early detection, rapid & accurate assessment, & immediate responses that prevent spread & control of harmful agents." The Bayh-Dole & Stevenson-Wydler Acts of 1980 requires all Federal agencies to transfer results of their research programs.

Does the program address a specific and existing problem, interest, or need?

Explanation: New science-based approaches for goal 3 programs are to protect & secure the food supply are necessary to meet the demands of new production systems & new threats to agriculture. Goal 3 program play a critical role, as identified by Congress & the Administration, in addressing the need to have a safe & secure food system protected at each step from production to consumption. The production, processing, & distribution system for food in the U.S. is diverse, extensive, & easily accessible. This open system is vulnerable to pathogens, toxins, & other contaminants through natural processes, global commerce, by unintentional or intentional means. Crop & livestock production systems must be protected from the ravages of diseases, whether domestic or exotic in origin.

Evidence: ARS' integrated, multi- & interdisciplinary research Program addresses issues affecting the safety of the U.S. food supply & the security of the agricultural production system. This Program is based on the needs expressed in USDA's National Health Objectives 2010 & incorporated in the ARS Strategic Plan 2003-2007. Research in this Program addresses priorities identified by a Homeland Security Council Biodefense Assessment & is included in the U.S. Food & Agricultural Defense Initiative. Research on biologically based integrated pest management is guided by the interdepartmental National Invasive Species Management Plan. This Program strengthens research on rapid response systems to foreign & domestic animal & plant disease agents, new & improved vaccines, & genes affecting disease resistance. New methods developed from this research are transferred to Federal & State agencies, private industries, & other stakeholders.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: ARS' intramural program is designed to conduct long-term, multi-disciplinary, high-risk strategic research that cannot be conducted by any other U.S. entity. With a $1B annual budget & >2,200 scientists working in >100 locations, ARS has unparalleled ability & capacity, authority, & expertise to respond quickly to immediate national emergencies &/or long-term critical needs of U.S. agriculture & food safety/security. This Program uses this capacity to integrate & coordinate with other Federal research efforts (including other USDA agencies, DHHS, DHS, DoI, EPA, DoD, NASA, NSF, Federal Interag. Cmte. on the Mgmt. of Noxious & Exotic Weeds (FICMNEW), & National Invasive Species Council), academia, State agencies, industry, & non-profit organizations. In this way, ARS fills critical gaps in research, complements the work of other research entities, & minimizes unnecessary duplication. ARS also manages critical U.S. germplasm repositories of plant, microbial & animal genetic material.

Evidence: To avoid unnecessary duplication, ARS starts its research planning cycle with a National Program Workshop involving customers & research partners to identify high priority issues. Program managers search literature & the Current Research Information System to identify any other research being done before a new project is initiated. ARS coordinates its research with other Federal & State research agencies, academia, & industry. ARS has core capabilities that do not exist elsewhere, such as the National Plant, Animal, & Microbial Germplasm collections; a network of overseas biological control laboratories; & unique diagnostic & systematic capabilities in support of action agencies. For example, ARS is the only research entity that has the special biocontainment facilities & the legal authority to conduct research on such economically devastating diseases as foot & mouth disease (in coordination with DHS at Plum Island, NY) & plant pathogens (Ft. Detrick, MD) on the Select Agent list.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The overarching goal of this Program is to protect & secure the Nation's food supply & agricultural production systems & to improve public health by reducing the incidence of foodborne illness. However, the Program's ability to efficiently & effectively manage its resources is complicated by Congressional priorities & pass-through funds specifically targeted for local needs. Similarly, ARS is unable to close less-productive laboratories because of Congressional resistance. However, this extra Congressional funding plays a positive role in ARS' overall research agenda. Congressional add-ons are fit into the priorities within the established Action Plans for that research program. While Congressional add-ons may not coincide with Adminitration policy, they do undergo the same agency review for quality and performance as do other projects. ARS realizes a cost-sharing benefit by leveraging these funds to extend our research program & capabilities. These research activities are required to undergo external independent peer review & are subject the same standards of relevance, quality, & performance as are all other ARS projects.

Evidence: Unrequested Congressional priorities have grown steadily over recent years. FY 2001-2005 Congressional priorities totaled $174.7 million, 15% of ARS' FY 2005 appropriations. Despite this, the Program is uniquely designed to respond effectively & efficiently to current & emerging critical needs. These priority & pass-through funds are managed through existing research units, focused on relevant issues, peer reviewed as appropriate, & held to the goals established in ARS' Strategic Plan.. For example, funding for the National Alliance for Food Safety & Security (NAFSS) is used to address issues related to Listeria & E. coli O157:H7 that supplements & is specifically integrated with the ARS food safety program. This research is subject to in-depth & rigorous external peer review annually. ARS uses funding to the NAFSS to address specific research objectives that are currently unavailable or beyond its capability to conduct in-house.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: This Program actively seeks input from & is responsive to the needs of its customers (e.g. APHIS, FSIS, FDA, producers, & food processors), partners (e.g., CSREES, DOE, NSF), & stakeholders (e.g., Congress, the Administration, & commodity associations). This input becomes the basis for priorities identified in the Program's Action Plan that guides it over its 5-year cycle. The Program is subject to periodic evaluations, including frequent customer/stakeholder interactions that assure the Program's continuing relevance & focus on high priority national needs. The Program's goals are tied to outcomes (new technologies) that provide real-world solutions to identified problems & needs. This is the basic tenet of the ARS mission to address the needs of U.S. agriculture & consumers (intended beneficiaries). In the Measures tab, ARS commits itself to producing 90 new technologies that will be adopted & used by appropriate customers by 2011.

Evidence: ARS' research is aggressively articulated by various means, including Web sites, such as Food Safety Research Information Office (FSRIO) (www.nal.usda.gov/fsrio/fsresearch.htm) & www.invasivespecies.gov; scientific & lay publications & presentations; reports; conferences, & workshops; & reports to Congress. Beneficiaries include APHIS, FSIS, EPA, & NRCS, who use the Program's research for developing & implementing regulations. For example, when BSE was discovered, the test APHIS used to confirm the infection was developed by ARS. Additionally, tissue samples were sent to ARS for confirmation. Also, APHIS used ARS-developed detection methods to identify & confirm the presence of Soybean rust in 9 Southern states last year. EPA is using ARS research in determining the safety of fungicides proposed for controlling Soybean rust. FSIS has incorporated ARS-developed protocols for the analysis & quantitation of chemical residues in animal tissues as part of their regulatory programs.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The Goal 3 program area, which comprises five individual National Programs, has a limited number of quantifiable long-term performance measures carefully selected to focus on research that contributes to protecting & ensuring a safe, secure, & adequate food supply; ensuring animal health by improving disease & insect pest detection, prevention, & control; & developing control strategies & technologies to manage &/or reduce losses caused by plant diseases, weeds, & insect pests. ARS' long-term research Programs are specifically targeted to mitigate the immediate & long-term threats to the Nation's agriculture & food supply, as identified by Federal agencies & ARS customers & end-users to protect public health & the security of U.S. agricultural production systems.

Evidence: This research program produces technologies (diagnostic tests) that are being used by APHIS, FS, NRCS, CDC, & the Bureau of Land Management. These new methods can also be used by these & other Federal & State agencies to protect against agroterrorism. Data from these agencies are published annually & are a good measure of the impact of ARS research. ARS' long-term goals over the next 5 years include developing over 90 new technologies that will be adopted & used to assist ARS customers to detect, identify, & control foodborne diseases affecting animal & human health & eradicating diseases, insects, & weeds that cause the majority of agricultural production losses.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The nature of the challenges confronting U.S. agriculture is complex & the research program to address these challenges, by necessity, requires multi-year objectives & actionable strategies. For goal 3 programs, specific objectives/strategies, as detailed in the ARS Strategic Plan 2003-2007 & the USDA's National Health Objectives (NHO) 2010, are established to focus on the most critical & economically damaging threats to human health & the safety & security of the production system. Research baselines are clearly defined & long-term targets are established to build upon ARS' capabilities to address these issues. Research outputs, & subsequent technology transfers, hold the promise for developing practices &/or new technologies that will be adopted & used in support of the targets. Targets show significant increases during the timeframes indicated.

Evidence: The Program has targeted the development of at least 8-10 new detection methods, diagnostic tests, & technologies for the most significant pathogens relative to health & economic consequence each year through 2011. These long-term targets include development of a rapid, cost-effective, user-friendly field test for BSE; development & release of new rust-resistant soybean varieties; &, through the use of genomics, the development of detection tests & intervention technologies to control Listeria, Campylobacter, & E. coli O157:H7. The work of this program directly contributes to protecting public health & making a safer & more secure agricultural production system.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: ARS has adopted the OMB R&D Investment Criteria as its specific annual performance measures that focus on relevancy, quality, performance, & efficiency. ARS uses these annual measures to ensure that the research stays on target to meet the Program's long-term goals. Each measure is assessed separately. Continued relevancy is assessed against the customer-identified priorities & goals as described in the 5-year Action Plan. Quality is determined through an external independent peer-panel project review. Performance is determined by the number of project milestones met or not met each year. Total Program Analysis is determined through a retrospective external peer panel project/program review. Efficiency is measured by leveraging funds from external sources interested in developing, transferring, & commercializing ARS technologies.

Evidence: Annual measures are the building blocks of the research that is needed to achieve the Program's long-term goals. Each measure is specifically designed, targeted, & tracked to ensure that the research is of the highest quality & performance, while making the most efficient use of available resources to move the science closer to solving the issues identified in the long-term goals. For example, one project within this Program focuses on the genome sequencing & annotation of various Campylobacter species that will allow for the development of international "gold standard" detection methods & development of universally accepted intervention strategies, ultimately leading to enhanced U.S. export trade in poultry products. Tracking the annual measures (relevancy, quality & performance) keeps the scientific work focused on the long-term objectives.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Quantifiable baselines and ambitious targets are established for all annual measures. Annual milestones are the target measures for individual research projects, and milestones increase incrementally in difficulty and complexity throughout the project's 5-year life. All measures are carefully crafted to meet the established goals of the individual project. ARS expects each project to show continued progress and to produce useable results (tech transfer). Annual tracking of project milestones achievement is valuable input to the agency's program management process, including decision-making regarding whether a project is continued, modified, redirected, or terminated. Thus, these baselines and annual measures are important tools used to gauge the relative success and progress of an individual project and its contribution to the overall progress of the Program. '

Evidence: Since FY 2004, baselines and annual performance targets have been identified in each research project plan, and progress towards meeting these measures is tracked through annual project reports. Beginning in FY 2005, baseline and annual targets are included in all new National Program Action Plans, the President's budget submission, and ARS' Strategic Plan, GPRA Annual Performance Plan, and Reports. Successfully meeting these targets is an annual performance standard for each individual ARS scientist. In an analysis of the FY 2004 project milestone information, ARS found that 15% of the annual milestones were not met. Of those milestones not met, approximately two-thirds, or 10% of the total, were due to unpredictable and unavoidable circumstances. The single greatest mitigating circumstance was the unanticipated loss of essential personnel due to illness, death, retirement, transfer, or unfilled vacancy. Thus, achieving 90% of total can be considered an ambitious real-term target.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: ARS enters into research partnerships with external organizations when a mutual benefit is clearly demonstrated that is consistent with the ARS mission & its annual & long-term goals. These partnership agreements (CRADAs, SCAs, etc.) clearly define the scope of the research, milestones, measures, outcomes, & the mechanism for monitoring performance to ensure support for the overall Program goals. Each partnership agreement is overseen by an Authorized Departmental Officer's Designated Representative (ADODR) to ensure progress toward meeting the annual & long-term goals of this Program.

Evidence: ARS provides funds to external research entities through contracts & agreements for cooperative work in support of this Program's research goals. Prior to ARS' approval, research agreements must include performance measures relevant & complementary to the annual & long-term goals of this Program. As appropriate, these research partners are involved in National Program Workshops & planning ongoing research activities. ARS requires its external research partners to provide Annual Project Reports (AD-421s) & identify accomplishments as they relate to the Program's goals. In addition, where appropriate, a contractor or agreement partner may be required to file a quarterly financial & progress report, as well as a 5-year summary report. These accomplishments are reported in the GPRA APR process. Information on annual meetings, workshops, & quarterly & annual reports are available on file, or at www.nps.ars.usda.gov/programs.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: At the onset of the Program cycle, relevancy of the research agenda is established in consultation with a broad cross section of customers & partners. From this input, ARS develops an Action Plan to establish the 5-year research agenda. Each project undergoes a prospective independent external peer-review for quality by the Office of Scientific Quality Review (OSQR) before implementation & each National Program undergoes a retrospective assessment by another external review panel near the end of the 5-year cycle to evaluate for relevancy, scope, quality, performance, impact, & effectiveness. The concluding findings critique the Program's methods, results, & findings, & are documented with a report. This assessment is also used to create a framework for the subsequent 5-year Action Plan. In addition, Annual Project Reports (AD-421s) are reviewed annually by the Area Office & NPS to assess continued performance, relevancy, & progress.

Evidence: The overall program process that ARS has devised covers the life of each project within the program from inception to completion with opportunities for modifications & adjustments to refine or redirect research focus & direction throughout the 5-year cycle. ARS has developed a comprehensive objective peer review process specifically designed for intramural research, which has been adopted by several countries, i.e., Canada, France, Finland, & Brazil. The Action Plans & Annual Reports for the 5 National Programs under Goal 3 are available at www.ars.usda.gov. Also, ARS maintains on its Web site a comprehensive & detailed description of the OSQR peer review process, including review schedules & detailed documentation describing the scope & mechanism of evaluation. This information is available at www.ars.usda.gov/osqr.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The ARS budget-planning process ensures that the direction of the Agency's research activities meet annual & long-term performance goals, as outlined in the ARS Strategic Plan FY 2003-2007. The detailed explanatory notes developed during the budgetary process highlight how ARS budget requests are tied to annual & long-term performance measures. In addition, performance measures from both the strategic & annual performance plans are constantly monitored to determine the maximum benefit from appropriated funding toward meeting pre-established goals. ARS budget requests depend on the review & approval of USDA's Office of Budget & Program Analysis & OMB. This provides a secondary executive review to ensure alignment of funding & performance. ARS also maintains the ability to rapidly respond to changes in policy & legislative decisions by altering its research directions &/or redirecting existing resources or requesting new funding to address emerging or critical priorities. However, the USDA budget justification does not show an integrated connection between funding and performance measures.

Evidence: At the Administration/USDA level, evidence of budgetary linkage to accomplishment of the long-term & annual performance goals are documented within the ARS FY 2006 Explanatory Notes, ARS Strategic Plan, & GPRA documents. National Program Staff (NPS), which is responsible for setting & monitoring research progress, priorities, & objectives, identifies budgetary needs & implements funds to ensure that sufficient resources are available to meet the annual & long-term needs of the project & the overall Program. Funding for projects failing to meet performance standards are redirected to areas of higher priority, e.g., in 2004, at one ARS location, 4 of 9 projects were eliminated to improve the overall performance of the Program. ARS' ability to redirect existing funds is exemplified by refocusing resources from a food safety project (pathogen detection) to a higher priority food security issue (bacterial toxins) in FY 2005.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: In FY 2003, ARS developed a new Strategic Plan that specifically incorporated both PART & the Budget Performance Integration (BPI) initiatives in the President's Management Agenda. The previous Strategic Plan was deficient in providing a good basis for developing a performance-based budget. The new Strategic Plan aligns ARS' priorities with the outcome-oriented USDA Strategic Plan. Beginning in FY 2005, the ARS budget request followed the goals & performance measures contained in its Strategic Plan. This new ARS Strategic Plan makes it easier to align the National Programs with agency goals & objectives. The preceding Strategic Plan did not. This alignment under the new Plan focuses the agency towards providing tangible results useful to American agriculture, to ensure a safe food supply, & a secure crop & livestock production system.

Evidence: ARS has taken steps to correct its strategic planning deficiencies by revising the ARS Strategic Plan FY 2003-2007 to address the needs of the BPI initiative, PART, GPRA, REE & USDA Strategic Plans' goals & objectives (available on the ARS website at www.ars.usda.gov). The ARS Strategic Plan now links budget requests with specific performance measures at the Program level. Also, each project within the Program now has annual milestones that must be met to fulfill the overall Program goals & objectives. The project's annual reports (AD-421) now include information on performance indicators (milestones met/not met, see Performance Measure 6).

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: This Program is coordinated through NPS with complementary programs across the Federal Government, academia, industry, producers, processors, & consumer groups (end-users) to ensure scientific relevance. Relevance to the USDA mission is regularly evaluated & coordinated with other Federal, industry, & university programs both nationally & internationally. A wide range of formal & informal mechanisms are used to ensure an ongoing dialogue with customers & partners (including researchers from academia & the private sector). The Program avoids unnecessary duplication through continual interactions with its research customers & partners. Where possible & appropriate, customer collaboration on specific research projects is undertaken, thus increasing quality & performance, & ensuring efficiency & cost-effectiveness.

Evidence: ARS was specifically requested by FSIS to assist that agency to obtain data for development of the Listeria Rule for ready-to-eat foods. Because of the potential conflict-of-interest between sister agencies, ARS developed an alternative plan, validated by both FSIS & FDA, to be conducted by the NAFSS (www.nafss.com). To address the issue of antimicrobial resistance in animals, ARS & other Federal agencies (APHIS, FSIS, FDA) have met with industry working groups to develop tracking systems & research approaches to reduce its impact on human health (www.FDA.gov). The Sclerotinia Initiative Working Group discussed various research options & alternatives to eliminate this disease in major crops, including sunflowers, soybeans & peanuts. A comprehensive list of performance measures & targets for this research were evaluated & documented against the strategic plan &, where appropriate, were redirected (www.whitemoldresearch.com).

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The Program has a series of priorities clearly defined in the 5-year Action Plans which guide future budget requests & funding decisions. Partners & customers, however, continually provide advisory input. Where necessary, that input helps the Program to re-prioritize its research within its base budget & identifies future funding directions. The BPI is linked directly to both ARS' & USDA's Strategic Plans, & makes clear the impact of funding on performance. ARS has incorporated the OMB R&D Investment Criteria into its Strategic Plan which provides mechanisms that ensure budget requests are relevant to the needs of U.S. agriculture. The ARS Annual Budget Team receives input from various sources to determine the priorities that go into the Agency's budget request. These sources include annual & long-term research needs documents from advisory bodies both in government (FSIS, APHIS, FDA), & in industry, e.g., American Meat Institute (AMI) & National Cattleman's Beef Association (NCBA).

Evidence: ARS receives initial input (on guidance, relevancy, & research priorities) from various sources - e.g., REE, FSIS, APHIS, commodity groups, & academic boards (NAS, American Academy of Microbiology). Budget estimates undergo review by the USDA BPI panel which oversees the development of a performance-based budget. Internally, ARS follows the R&D Criteria in conducting annual project reviews for relevance, quality, & performance. Program managers make funding decisions & redirect resources within the base budget to areas of higher priority. For example, the new ARS BSE budget request resulted from input from the White House, Secretary of USDA, APHIS, ERS, FAS, FSIS, & industry, as the potential for an outbreak within the U.S elevated the disease to a critical concern. Also, ARS redirected part of the food safety base funding from pathogen detection to higher priority food security at the request of FSIS, FDA, DoD, & DHS & redirected funds to its animal health program to combat TSEs.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: As part of the process to develop specific research Project Plans, ARS establishes milestones, which become the baseline for measuring performance for each year of the 5-year project cycle. Project research leaders submit Annual Progress Reports (AD-421) that provide performance information regarding completion of milestones. This timely & credible performance data is used by NPLs & Area Directors to monitor project performance. NPLs analyze these data & use it to redirect or realign projects, if evidence suggests such a step would enhance the quality & performance of the research. NPS provides summary reports on progress towards meeting milestones that are widely disseminated in the GPRA annual performance report & the PART analysis.

Evidence: Information from Project Plans, Annual Reports (AD-421), & internal review documents are used in managing the Program & deciding whether to realign, redirect, reduce, or terminate [Program Adjustment Decision Item (PADI)] a project to maintain quality & performance. For example, in the past 2 years, ARS has substantially realigned its research focus & operations at many of its major Research Centers (including Gainesville & Tallahassee, FL; Wyndmoor, PA; Plum Island & Ithaca, NY; Fargo, ND; & Athens, GA), resulting in more efficient program management & use of resources. This realignment of personnel, fiscal, & facility resources has been accomplished to better fit the Program's objectives in the Action Plans. Additional realignments are being considered at other locations to improve quality, performance, &/or efficiency. ARS projects are terminated when the mission is completed, the underlying national need for the research has changed, or performance fails to meet ARS standards.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: NPLs (program managers) & Area Directors (line managers) are held strictly accountable through an annual evaluation process. In addition, the impact of the Program is evaluated on a regular basis to ensure that the goals of the Program are being met & that those results are meeting program objectives in a timely manner. All collaborative research agreements are continuously monitored by the program managers to ensure research activities are meeting the project goals & milestones on time & in a fiscally responsible manner. NPLs responsible for the Program have specific program management performance standards that are included in their annual evaluations.

Evidence: Managers & scientists are subject to annual performance reviews. Also, ARS scientists are subject to an in-depth peer review on a 3- to 5-year cycle under the Research Position Evaluation System. The annual reports from each research project, which contain performance results, are reviewed by both the Area Office & NPS. The Performance Plan for NPLs holds them accountable for evaluating their Programs for "?? relevance, direction, & rate of progress toward providing solutions to specific high-priority national problems & research objectives." The Performance Plan also requires them to develop "?? recommendations to line mangers for strengthening the Agency's research program." NPLs are evaluated on these & other criteria annually.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Research project plans have specific goals & objectives that must be met within a period not to exceed 5 years. ARS allocates, obligates, & manages agency funds at the project level to help ensure appropriate use. As part of the annual performance appraisal, scientists & NPLs are held accountable for research progress & the timely & appropriate use of resources. NPLs & scientists have the incentive of performance awards if their performance exceeds stated expectations, as outlined in their Annual Employee Performance Plan. ARS uses the Foundation Financial Information System (FFIS) accounting system to enhance accountability. ARS undergoes annual financial & programmatic audits by USDA/OIG & GAO that review timeliness of obligating & reporting, accuracy in accounting, & intended use of funds.

Evidence: The FFIS accounting system is used to process & manage financial data to meet stringent budget & fund-control needs, as well as complex multifund & reporting requirements. This Program does not have carryover authority; therefore, all funds are expended or obligated by the middle of the 4th quarter of each fiscal year. ARS, as an agency, uses its own local obligation tracking system in conjunction with FFIS to ensure funds are obligated up to the assigned funding level without exceeding it. Historically, ARS obligates more than 99% of its assigned funds. In FY 2004, the ARS obligation rate was 99.7%. Scientists & managers are subject to annual performance reviews in which they are held responsible for financial management & compliance of their assigned fiscal resources. ARS has received clean audit opinions from OIG & GAO for several years.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: Recent scrutiny by ARS exposed IT structure deficiencies specifically in cyber security, network connectivity, & e-mail configurations. In FY 2002-3, ARS analyzed the costs & benefits of alternative solutions for improving e-mail & web hosting services; recommended e-mail & web hosting solutions that best met the agency's needs; & developed an efficient & cost-effective IT Consolidation Project Plan to implement the infrastructure improvements. The agency has implemented competitive sourcing strategies (A-76) to reduce administrative costs & improve overall efficiency. To operate the agency as efficiently as possible & maximize the resources at the scientific bench, ARS has consistently managed with an overhead of 10% or less.

Evidence: The IT Consolidation Project will be completed in FY 2005. Major milestones include unifying 52 different e-mail systems; consolidating web hosting; updating infrastructure (i.e., telecommunications network, security, organizational configuration) to support ARS' new e-mail & web hosting design; & providing a plan for building the enterprise architecture & estimating related costs & staffing. The benefits of the IT Consolidation Project will provide ARS with a secure, scalable, robust network, enterprise e-mail, & web hosting capabilities for customers. These measures will greatly improve the efficiency of the computer technology that supports ARS scientists & research facilities & decrease the redundancy, overlapping, & unproductive use of staff time. Another example of the Program's efforts to increase efficiency is the outsourcing of specific research tasks to partners through cooperative agreements to address agricultural issues needing immediate attention.

Does the program collaborate and coordinate effectively with related programs?

Explanation: ARS research is intramural, not grants-based, & is integrated & coordinated with related programs across the Federal Government, including, FDA, NIH, CDC, EPA, Nat. Acad. of Sci. (NAS), DHS, other USDA agencies, academia, & industry groups. Many of these groups fund extramural research that is complementary to the work of ARS. Whenever possible, ARS collaborates with these organizations to meet the needs of regulatory agencies & the agricultural community. These efforts assure close coordination of ARS research to eliminate program overlap & duplication. To maintain this close coordination, ARS continually reevaluates the Nation's research needs based on changing priorities.

Evidence: Program managers regularly meet with ARS partners to ensure coordination of effort. The Program receives continual input from FSIS, APHIS, CSREES, & FDA, & conducts joint planning workshops to assess industry & commodity group needs & review the progress of pertinent ARS research. These collaborations have led to Program & resource redirections/realignments to avoid overlap & to help sharpen the focus of existing research needs. In the FY 2006 budget request, ARS proposed new & expanded research initiatives (e.g., BSE, food security, emerging & exotic diseases, & invasive species) which were developed in consultations with various Federal & State agencies, industry, & academia. For example, CAHFSE (Collaboration in Animal Health & Food Safety Epidemiology) is a program with APHIS, FSIS, & the swine industry to examine the effect of prophylactic use of antibiotics in animals to help understand the development of resistance to antibiotics in humans.

Does the program use strong financial management practices?

Explanation: The Program expenditures are continuously tracked using the Agricultural Research Information System (ARIS) & the Foundation Financial Information System (FFIS) to monitor scientific & fiscal progress. ARS maintains extensive financial management policy & procedures on funds management & accountability, including resource management, funds control, & semiannual reviews of unpaid obligations.

Evidence: ARIS tracks scientific effort, funding, objectives, milestones, progress, & accomplishments across the Program. FFIS is used to process & manage financial data to meet stringent budget & funds controls, as well as complex reporting requirements. Implementation of FFIS & its associated data warehouses offers the integration & capacity to improve the delivery of timely & meaningful financial management information, putting ARS in compliance with legislation, including the CFO Act of 1990. FFIS helped ARS achieve a clean OIG audit of its financial statements for FY 2004. In addition, ARS is part of the USDA Consolidated Audit. For the last 3 years, USDA has received an "unqualified (clean) audit opinion." In FY 2005, ARS implemented a new Program Decision Resource Allocation Memoranda (PDRAM) process that provides specific guidance to field managers on the direction of research programs & the allocation of resources (fiscal, personnel, & equipment) to achieve program goals.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Five years ago, ARS completely revised its research portfolio management structure to strengthen its ability to ensure coherent & relevant research. The ARS 5-year program planning cycle now includes an external retrospective assessment of the relevance, quality, performance, & management of the Program in its totality. This assessment is used to direct the development of the next 5-year action plan. The NPLs, with input from customers, stakeholders, & partners, establish the National Program research agendas to ensure that program goals remain relevant. Area Directors are responsible for assuring performance & scientific quality. In addition, each new project undergoes a rigorous, independent external peer review that provides scientific credibility to the Program's research agenda & identifies management deficiencies at the project level. Area Offices & NPS conduct in-depth external location reviews during the National Program cycle to identify management & program deficiencies.

Evidence: To ensure relevance, quality, & performance, program realignments & redirections PADIs are implemented to address & correct program management deficiencies. For example, in recent years, ARS substantially improved Program management by realigning the food safety & animal health programs at Athens, GA; the crop protection program at Ithaca, NY; & the livestock protection research at Plum Island, NY. These actions improved management of personnel, fiscal, & facility resources to better address the Programs' objectives as outlined in the Action Plans. Additional reorganizations are under consideration at other locations (e.g., Wyndmoor, PA) to improve performance, quality, & efficiency.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The intramural program, which is 100% non-competitive, allocates funds to projects based on research priorities and an assessment of quality using established standardized criteria. Congressional priorities and pass-through funding are incorporated into existing projects & peer reviewed as part of that project. Each project is reviewed & approved by an independent external peer panel (OSQR) to ensure quality. Research plans are revised to reflect the review panel's input. Projects that are required to make major revisions are re-reviewed. The panel may indicate that a project is not feasible. Under those circumstances, NPS & the Area Office jointly consider whether the project should be revised or discontinued & whether the scientist(s) & funds should be redirected. Periodic location reviews & retrospective program assessments monitor quality throughout the 5-year program cycle.

Evidence: To ensure the quality of research in this Program, each of the projects undergoes periodic reviews by independent external peer panels. The OSQR process invites leading scientists from industry, academia, & other Federal agencies to evaluate the Program for the quality of the science proposed in each Project Plan & the credentials of scientists proposed for the project (www.ars.usda.gov/osqr). To date, OSQR has reviewed 409 projects in this Program & ranked them in 5 categories: No revision (45); Minor revision (131); Moderate revision (143); Major revision (80); & Not feasible (10). Overall, 78% of projects reviewed scored a passing grade. All projects scoring either Major revisions or Not feasible (22%) were reconsidered &, where appropriate, were rewritten, resubmitted, & re-reviewed through OSQR. Projects scoring "Minor revisions" were revised, reviewed, & approved by NPS & Area Offices.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Yes. The long-term performance of the Program is measured by providing research and technologies, and by tracking their adoption and use by industry & Federal & State agencies to protect and ensure an adequate, safe, and secure supply of food. In the FY 2004 GPRA Annual Performance Report for Goal 3, ARS had 17 indicators of progress aligned under 8 performance goals. 64 significant accomplishments are contained in this report against the 17 indicators. These accomplishments address the most critical concerns as expressed to ARS by our customers & partners, e.g., APHIS, FS, NRCS, CDC, the Bureau of Land Management and State agencies. ARS receives direct feedback on the implementation and use of its technologies. This information is also included in the annual reports of the various agencies.

Evidence: The ARS FY 2004 Annual Performance Report is available on the ARS website. Significant research accomplishments are outlined in the ARS Explanatory Notes. USDA's ability to rapidly respond to recent threats to the safety & security of the U.S. food & agricultural production system were made possible by ARS-developed detection technologies. In the last 2 years, 18 new technologies were developed & incorporated into inspection procedures by Federal & State regulatory & action agencies, & private industry. Specific examples include: Confirmation of BSE in Washington State; Eradication of recent avian influenza & exotic Newcastle disease outbreaks; Detection of soybean rust in 2004; Development of microarray technology to rapidly detect Listeria, Campylobacter, & E. coli O157:H7; Detection methods for chemical contaminants; & Adoption by producers of an areawide pest management program for leafy spurge, dramatically reducing the amount of herbicide used to manage this invasive weed.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Yes. ARS has adopted the OMB R&D Investment Criteria as its annual measures, & modified its Strategic Plan, GPRA data requirements, & data collection process to report project & program information. Whenever possible, ARS uses independent external panels for review purposes. However, it is not physically & fiscally possible to do so in every instance. In FY 2004, ARS began annually evaluating all research projects for relevance & performance against customer-identified priorities & goals in the 5-year Action Plan. This was in addition to OMB's suggestion that ARS modify its AD-421 (annual project report) to collect data on how the research units met their established milestones. Also, beginning in FY 2005, ARS has specifically included program milestones in all new National Program Action Plans. The quality review process (OSQR) reports tangible results. These quantitative data & baselines (which are available on file) are summarized in the evidence column. However, several measures did not compare actual accomplishments with a specific target.

Evidence: In FY 2004. 97.1% of ARS' projects were determined to be directly relevant to the agency's goals. Since that review, 41% of the projects found to be of lesser relevance were redirected; the remaining projects are currently under review & may face redirection or termination. However, this was less than its target of 100%. Through the AD-421 annual reporting process, 85.3% of milestones were met. Based on the first round of the OSQR peer panel review for quality, 76.7% of ARS projects initially attained a passing grade (as described in 3.RD1 Evidence) The FY 2004 data in the various categories, described above, will be the baseline for future program evaluations. Beyond the R&D criteria, ARS met or exceeded additional annual measures for external independent total program assessment, agency overhead, & leveraging funds from external sources. Additional information & future targets are described in the Measures tab.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: As part of ARS' ongoing collaboration with its partners & customers, the agency has also identified areas of research & resources that can be shared to meet common goals & needs of U.S. agriculture. Trusts, reimbursables, & CRADAs that ARS has entered into indicate the level of partnership with external parties interested in developing, transferring, & commercializing our technologies. Also, ARS is using the A-76 process to outsource employees at Beltsville. Specifically, this Program demonstrates efficiency & cost effectiveness by co-locating with university partners, through shared laboratory equipment, the formation of core facilities, & sharing support staff. The Program periodically reviews its operations for opportunities to further improve efficiency & cost effectiveness.

Evidence: Over the last 4 FYs, ARS increased its leveraged dollars by 20% ($79.8M to $95.9M), from 8.2% to 8.7% of appropriated funding. Also, ARS consistently maintains its overhead at 10% or less of appropriations. An A-76 study at Beltsville initiated in FY 2004, affecting 180 employees, will achieve significant cost savings. To improve efficiency & cost effectiveness, this Program operates facilities co-located at academic institutions. In 2004, ARS moved one of its Food Safety Labs into a new joint facility at UMD Eastern Shore (1890) Center of Excellence. Several Program research centers or locations have restructured to maximize efficiencies & reduce costs. In FY 2004, the Program restructured the SE Poultry Lab & the Russell Research Center in Athens, GA, creating new units & realigning research activities to address new priorities in food safety & animal health. The ARS research activities at Plum Island, N.Y. have undergone a major realignment in response to the creation of DHS.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: ARS is the only intramural research agency in USDA that addresses all areas under Goal 3. Apart from CSREES, there is no comparable research entity in government or in the private sector that addresses the broad research scope of protecting & ensuring the safety of the Nation's food supply, plant & animal health, & public health. CSREES, however, funds research through an extramural, grant-based program with shorter-term objectives. Private sector research is typically profit-motivated & geared to specific commercial products or goals. ARS research has broad applicability to a wide range of issues & problems that affect the agricultural economy &/or public health. Only ARS has the capacity to conduct research that impacts all issues within the farm to table continuum. ISI Essential Science Indicators lists USDA as the most cited research institution in agricultural sciences in the world.

Evidence: ARS is distinctive & unique in its ability to address the full scope of national agricultural priorities with both short- & long-term, high-risk research impacting animal & crop protection & safety & security of the food supply. This range of research is not addressed by any entity in the United States. ARS research covers the full continuum from basic to applied research & the active transfer of developed technologies to potential users. Relevant examples of short- & long-term, high-risk research that distinguish ARS from other research organizations include: the Areawide Pest Management programs; the National Plant, Animal, & Microbial Germplasm System; the research capacity to work with threat agents to agriculturally important livestock, plants, & microbes under high containment; & several overseas biological control laboratories.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Initially, each ARS project is evaluated by an independent external panel conducted by OSQR to ensure project quality & validate the Program's priorities. At the end of the 5-year cycle, an independent external panel retrospectively evaluates the Program's progress for scope, focus, productivity, & impact as described in the Action Plan. Scientists are regularly held accountable for their performance by peer panels to determine productivity & impact of their work. In-depth site reviews are primarily conducted by external reviewers to evaluate program quality, effectiveness, & impact. Program Workshops provide another independent, external review at the end of the program cycle & input for the next 5-year Action Plan. Wherever possible, ARS relies upon independent, external reviews for assessments; however, with over 1,000 projects, this is not feasible to do in every instance.

Evidence: In FY 2004, 76.7% of projects reviewed by OSQR passed the first review. The remaining 23.3% were revised & re-reviewed. Projects are only evaluated twice & if they do not receive a passing score, the scientists are reassigned to other research. In FY 2004, 20% of the total ARS research program was evaluated retrospectively & the average score for all ARS National Programs ending their 5-year cycle was 78/100. The food safety component received an 81/100 score. The livestock & crop protection components are currently undergoing review. Both the OSQR & retrospective review scores were graded superior (Federal scale)/good (academic scale) since scores of 75-79 are a B- & scores of 80-84 are a B (grading systems data available). These evaluations are baseline scores for future reviews & used to formulate ARS' budget & to make program management decisions. An independent review by JIFSR indicated that ARS had the most efficient & effective Federal food safety research program.