Detailed Information on the
USDA Research: Economic Opportunities for Producers Assessment

Re-evaluating the criteria that it uses to determine outyear targets for project quality, for the purpose of establishing more ambitious targets.

Review and revise the planning, implementation and external review processes of the National Research Program 5-year cycle.

Worked to ensure that funding is targeted to highest priority initiatives and projects.

Completed the development of the annual measures.

Modify long term measures to show the results of the actual use of research outputs (discoveries and new technologies) rather than just the number developed.

Revise the long term performance measures (percentage of biofuels used and improvements in the agricultural productivity index) to a measure that would track the number of technologies developed and used.

Conducting an independent external Retrospective Panel Review of the ARS Crop Production program during FY 2007.

Continuing to submit Congressional earmarks and add-ons to the Office of Scientific Quality Review (OSQR) for independent external peer review to help ensure quality.

Measure: Cumulative number of new technologies developed and used by ARS customers to reduce the cost, improve the efficiency, increase the yield, and increase the sustainability of production and conversion of biobased feedstocks to biofuels.

Explanation:The purpose of this measure is to develop new technologies that assist ARS customers to produce and convert biobased feedstocks to biofuels that improve energy security, enhance the environment, and improve the rural economy. Recent examples of technologies that have been transferred and used include: Dry-grind ethanol computer model used by fuel ethanol producers to lower cost of production; industrial microorganisms, capable of fermenting biomass sugars in high yields to fuel ethanol, used enzyme and ethanol production companies that are working to develop commercial cellulosic biomass to ethanol production; a high-throughput screen to evaluate superior corn varieties for fuel ethanol production is being used by a non-profit producers' organization as a standard analytical method for testing corn samples; and a process to produce biodiesel fuel from 'high-acid acid oil' is being used by a start-up biodiesel production company.

Measure: Cumulative number of new technologies developed and used by ARS customers to increase productivity and efficiency of animal agriculture, and enhance the economic value and well-being of agricultural livestock and poultry by improving the nutritional quality and yield of meat, milk, egg, and other animal by-products while decreasing the environmental footprint of production systems.

Explanation:The purpose of this measure is to develop new technologies that provide ARS customers with genetic resources, genomic tools, and high quality animal products that ensure an ample food supply, enable production of more healthful food, and provide beneficial animal by-products. Recent examples of technologies (brackets indicate customer who adopted technology) that have been transferred and implemented include: DNA test for the calpain gene that allows improvement of beef tenderness [Genetic Solutions, MMI Genomics, Merial, Ltd.]; sets of molecular markers that allow biological traceability of cattle [Genaissance]; and an on-line instrument grading system for beef and pork carcasses [Excel, Tyson]

Measure: Relevancy: Measure of the relevancy of programs in contributing to achievement of long-term goals

Explanation:Relevancy: 20% of the ARS Research Program will be reviewed each year by one or more independent panels of scientists, stakeholders, partners, & customers to assess program relevancy in contributing to the Agency's long-term goals & addressing the priority needs of American agriculture. A measure is being developed for the panels to use in their assessment. The base or target will be the average score for the programs reviewed in a given year. The indicator is the number of workshops held during the year at which stakeholders, partners, and customers addressed the relevancy of the program. The number of workshops is a function of the 5-year program cycle. The workshops are held in the first year of that cycle. The programs are not evenly distributed in terms of the state they are in the cycle.

Measure: Project quality: Result of review by independent peer panels. This measure tracks the percentage of reviewed project proposals that require moderate, minor or no revisions.

Explanation:Quality: To ensure quality, 20% of the ARS Research Program will be reviewed by independent external peer panels each year under the agency's Scientific Review process.

Measure: Performance: Percent of projects achieving research milestones as described in project proposal.

Explanation:Performance: To ensure that the ARS research program is performing well, 100% of the research projects will be assessed each year against milestones in projects generally planned on a 5-year cycle. Baseline will be established in FY 2004; target will subsequently be set.

Measure: Performance: Program achievements as evaluated by stakeholders & customers using the National Program Action Plan as a criterion.

Explanation:Performance: To ensure that the ARS research program is performing well, each of the National Programs will be assessed at the end of their 5-year cycle against objectives set in their Action Plans. The Baseline was established for each National Program at the beginning of its 5-year cycle. Scores for accomplishments will be determined by an external panel of customers & stakeholders. About 1/5 of the National Programs will be scored each year. The indicator is the number of independent, external program reviews that were conducted in the year.

Measure: Percent of construction acquisition & upgrade projects with negative cost variance of less than 10% of the approved project plan.

Explanation:The purpose of the program is to construct, upgrade, maintain, & operate research facilities in a wide range of scientific fields, ARS' core function. A key indicator that facilities construction is going as planned is the variance of cost from the approved project plan. Performance measurement requires regular monitoring & reporting of project cost. The Facilities program has a 90% target associated with this measure, which allows for some variance across ARS programs due to unanticipated or external factors. This measure will no longer be used. The agency has developed new efficiency measures. The new measures will be utilized.

Measure: Cumulative number of new technologies developed and used by ARS customers to increase productivity and efficiency of producing important crops, and to enhance the economic value of agricultural crops by improving the nutritional quality of food and feed ingredients as determined by APHIS, ERS, EPA, FDA, NIH, FAS, DHS, National Oilseed Processors' Association, North American Millers' Federation, National Bakers' Association, etc.

Explanation:The purpose of this measure is to develop new technologies that provide ARS customers with genetic resources, genomic tools, and plant natural products that ensure an ample food supply, enable production of more healthful food or feed ingredients, and provide beneficial plant products. Recent examples of technologies (brackets indicate customer who adopted technology) that have been transferred and implemented include: release of agronomic NuSun sunflower varieties with oil quality properties that give food manufacturers a more healthful alternative to vegetable oils with high 'trans-isomer' content [McDonalds, Frito Lay, etc]; sets of molecular markers that identify important genes that govern soybean productivity and quality [Monsanto Co.]; and development of guayule, a domestic source of natural rubber that provides latex for hypoallergenic medical care products [Yulex Corp.].

Is the program purpose clear?

Explanation: The mission of ARS under Goal 1 is to conduct research that will enhance economic opportunities for agricultural producers & benefit consumers, both domestic & foreign. The research is designed to increase productivity & efficiency, develop new technologies/products & new uses from agricultural commodities & byproducts, enhance profitability, & expand market opportunities for U.S. agricultural products. To ensure relevancy input from customers, stakeholders, & partners is solicited & used in setting the program's research agenda, which is documented in the Action Plans. Prospective & retrospective independent reviews are conducted at appropriate times in the program cycle to ensure quality & performance.

Evidence: The ARS Research Program to Enhance Economic Opportunities for Agricultural Producers (hereafter referred to as the 'program') was authorized under Pub. L 104-127, Title VIII, Section 801, 'Purposes of Agricultural Research, Extension, & Education:' (1) 'enhance the competitiveness of the U.S. agriculture & food industry in an increasingly competitive world environment', (2) 'increase the long-term productivity of the U.S. agriculture & food industry...' & (3) 'develop new uses & new products for agricultural commodities, such as alternative fuels & develop new crops.' The USDA Strategic Plan 2002-2007 states, 'Expanding markets for agricultural products is critical to the long-term economic health & prosperity of our food & agricultural sector.' That is the mission of this research program.

Does the program address a specific and existing problem, interest or need?

Explanation: The rate of extinction of lines & strains of food animals & crops is rapidly accelerating. Maintenance of genetically diverse animals & crops is necessary to quickly adapt to changing conditions. The prosperity of U.S. agriculture depends on the preservation of plant & animal germplasm collections in a healthy, secure, & easily accessible form. These genetic resources provide the means to enhance crops by deploying molecular tools & genomic strategies for selection of germplasm with high quality & valuable traits for commercial use. U.S. agricultural competitiveness depends on cutting edge research in the areas of bioinformatics, genetics, & genomics; & the continued collection, preservation, safe storage, & use of critical genetic resources. The development of new uses & new biobased products & fuels will also contribute to the economic viability of U.S. producers.

Evidence: ARS conducts an integrated research program that addresses issues affecting economic value & the domestic & foreign market share of U.S. agricultural products. This research addresses a broader Departmental and Administration goals as identified in the USDA Strategic Plan for FY 2002-2007 ' 'Expanding markets for agricultural products is critical to the long-term economic health & prosperity of our food & agricultural sector. In order to maintain agricultural competitiveness, there is a need for enhanced efficiency of crop & livestock production, improved animal well being, & reduced impacts on the environment. The program also addresses problems of agricultural surpluses & byproducts through improved utilization & enhancing value of low value products, including development of functional foods, biobased products, & biofuels.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The scope of ARS' intramural research program enables it to conduct long-term, innovative, high risk, multi-disciplinary strategic research that cannot be conducted by any other agency or industry in the U.S. ARS, as an intramural agency, has the unique ability, authority, & expertise to respond quickly & effectively to critical national &/or international issues & emerging problems. The program is effectively coordinated with other research efforts conducted across the Federal Government (including APHIS, CSREES, DHS, NIH, DOE, EPA, GIPSA, NSF, & FDA), academia, & industry. ARS is often able to fill critical gaps in research & thus maximize complementarity with the work of other research entities & minimize unnecessary duplication of effort. Examples of very successful partnerships with other agencies include the sequencing of the bovine, honeybee, & chicken genomes (with NIH & CSREES) & the National Plant Genome Initiative with NSF, & the biobased products/biofuels work with DOE.

Evidence: To avoid unnecessary duplication or redundancy ARS starts its research planning cycle with a national program workshop that brings together customers, stakeholders, & partners (universities & private sector research organizations) to identify the high priority researchable issues. Then a thorough literature search & a search of the Current Research Information System are conducted to identify any other research being done in a given area before a new project is initiated. Because of proprietary interests, it is generally not possible to identify all ongoing private sector research. ARS works to coordinate its research with other Federal or State research agencies. ARS has core capabilities that do no exist elsewhere such as the Dairy Herd Improvement database & the National Plant, Animal & Microbial Germplasm System.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Even though ARS has a strong interal merit review process, as shown in the following write up, the answer received a "no" becuase a portion of the projects are appropriated through Congressional earmarks that allocate the funding by purpose and to specific locations. The overarching goal of ARS research is to promote the economic viability of American agriculture & enhance its competitiveness in the global market. The ARS program to enhance economic opportunities for agricultural producers is uniquely designed to respond to current & emerging high priority national & regional needs. Although earmarked projects may not represent the most effective use of available funding, such funding is integrated into existing projects within the National Programs &, as appropriate, subjected to rigorous external peer review conducted by the Office of Scientific Quality Review (OSQR) to ensure high quality. These projects are managed in the same way & to the same high standards as the rest of ARS' research program. As an intramural research agency, ARS is uniquely designed to conduct long-term, high risk, innovative, strategic research to improve agricultural competitiveness that cannot be conducted elsewhere, while retaining the flexibility to address new or emerging issues.

Evidence: This program is designed so as not to limit its effectiveness & efficiency to responding to current & emerging needs. Its focus has been developed & validated by various mechanisms: including stakeholder meetings; in-depth, external Program reviews, & rigorous external peer review of each research project conducted by the Office of Scientific Quality Review (OSQR). The value of this program's approach is a free, rapid, & unbiased dissemination of research information & technology to potential users. The program is effectively managed, & has a track record of working collaboratively with other Federal agencies, industry & universities. ARS employs a form of matrix management to provide both programmatic guidance & line management for its widely dispersed locations. The objective of the ARS management structure is to have the headquarters based NPS provide the programmatic vision & research direction to the ARS scientists. Responsibility for implementation, day-to-day management, & ensuring that quality research is conducted, falls to the Area Offices.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The research program is developed after extensive discussions with customers (producers & processors of various agricultural products, Federal & State action & regulatory agencies), stakeholders (commodity & trade organizations), & partners (universities, private sector, & international research organizations) to ensure the relevancy of the work to be done. The priorities identified at the program workshops become the basis of the Action Plans which guide the projects that constitute the research program over a 5-year cycle. The individual project plans then undergo a rigorous external peer review conducted by OSQR to ensure the quality of the research to be conducted. Ongoing customer/stakeholder interactions assure the continuing relevance of ARS' research. The program is subject to periodic evaluations and assessments. These evaluations & assessments are the basis for assuring the appropriate alignment of resources to achieve programmatic objectives that effectively address high priority national needs.

Evidence: Several mechanisms have been created to coordinate & target the work of a number of Federal research agencies to promote economic opportunities in rural areas. The Under-Secretary for Research, Education, & Economics (REE) is the Chairperson of the government-wide Interagency Working Group on Domestic Animal Genomics & a member of the Interagency Working Group on Plant Genomics, which has conducted stakeholder listening sessions to gain input on ARS genomics research. ARS chairs the USDA Biobased Products & Bioenergy Coordination Council & receives input from the Biomass R&D Technical Advisory Committee. These government-wide Interagency Working Groups identify areas of research that are uniquely appropriate for the Federal Government to conduct. The program's Action Plans are available at www.ars.usda.gov.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: ARS employs a balanced strategy to tackle problems affecting expanding economic opportunities for U.S. producers. This program has a limited number of quantifiable long-term performance measures that have been carefully selected to focus on the following outcomes: (1) to ensure an adequate & affordable supply of high-quality agricultural products to meet domestic & foreign needs & (2) to ensure the profitability of the U.S. agricultural enterprise. ARS meets its long-term measures through linkages to existing measures of the performance of the agricultural economy. U.S. agricultural output increased at an average annual rate of 1.88% over the period 1948-1999 although evidence in recent years has shown a slowing of the rate to about 1.5% per year. The Index of Agricultural Productivity (1996 = 100) demonstrates continuous growth, forecast at 113 for 2004 & projected to reach 123 in 2010. Research contributions to the gains in total factor productivity & account for all of the growth in output.

Evidence: The specific long-term performance measures for this program are: (1) % of consumption of biobased transportation fuels & (2) improvement in the index of total factor productivity (TFP) as published annually by the USDA Economic Research Service (ERS). The TFP is measured as the ratio of the index of total farm outputs to the index of total farm inputs (1996 index = 100) & is the best measure of the impact of agricultural research on improved efficiency of livestock & crop production. The use of these specific performance measures requires ARS reliance upon data provided by ERS, NASS, FAS, & the Office of Energy Policy & New Uses (OEPNU).

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The recent ERS briefing paper referenced in evidence, stated 'Advances in agricultural productivity have led to abundant & affordable food & fiber throughout most of the developed world. Public & private agricultural research has been the foundation & basis for much of this growth & development. Technology transfer holds promise for improving incomes & welfare throughout the world.' The ARS long-term goals are to build upon past successes in improving agricultural productivity & are ambitious in that they exceed historic averages. For biofuels ARS' long-term goal is to substantially increase the market share for these biobased fuels.

Evidence: Data compiled by ERS covering the years 1948-1999 showed an annual average gain in TFP of 1.88% (Ahearn et al., 1998, Agricultural Productivity in the U.S.), however, this increase slowed to 1.5% annually during the last decade of this period. The target for this program is to increase the Total Factor Productivity Index from 113 in 2004 to 123 by 2010. The current share of biobased products, according to OEPNU, is 0.5%. The ARS target is to increase this 8-fold by 2010.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: ARS has adopted the OMB R&D Investment Criteria as its specific annual performance measures for each long-term goal. Annual measures will focus on relevancy, quality, & performance. Relevancy- will be stakeholder workshops that formulate complete plans with clear research goals & priorities. Quality-will increase the % of projects receiving passing scores from 78% to 85% from external OSQR peer panels. Performance-retrospective assessments of program performance will be documented at the end of each 5-year program cycle by external review panels & stakeholder workshops. The results of retrospective performance assessments will be used to ensure relevancy of research projects in the next 5-year program cycle. One Facilities-related efficiency measure is identified in this document.

Evidence: We have modified the ARS Strategic Plan FY 2003-2007 to incorporate the OMB R&D Investment Criteria into our GPRA & PART processes. We are modifying our data collection process for the research units (AD-421) to collect the information which is not currently available. To ensure that the research program is: relevant to our customers, stakeholders, & partners, 20% of the overall ARS research program will be evaluated for relevancy each year; of high quality, 20% of the ARS research program will be evaluated for quality each year by OSQR; & is performing well, 100% of the ARS research program will be assessed each year against indicators to measure performance, 20% of all National Programs will be evaluated each year against their Action Plans, which guide the work of the research projects for 5 years. In some instances the targets, measures, &/or data collection procedures are still being developed. We anticipate being able to report substantive data in each area for the 2007 PART analysis.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: To ensure the quality, relevance, & performance of this program, ARS is collecting data to establish meaningful baselines to implement the OMB R&D Investment Criteria as its annual measures for this program. The target for: Relevance ' is to continuously assess the relevancy of this research program, at least one national program workshop will be held for each major subcomponent of the program at the beginning of the 5-year program cycle, Quality ' is to reach by 2010 the goal of having 85% of the research projects reviewed by OSQR receive a rating of moderate, minor, or no revisions required (current baseline for this program is 78%) (All projects scoring either not feasible or major revision are rewritten, resubmitted, & re-reviewed through OSQR, Performance ' we are developing mechanisms, such as the ability to track whether research projects meet their milestones each year, technology transferred to users, whether National Programs meet the objectives in their 5-year Action Plans, etc. to measure general performance. Performance goals & targets are under development. The answer received a "no" because targets were not indicated in most instances.

Evidence: ARS has adopted the R&D Criteria as its annual measures, modified its Strategic Plan, & is modifying its data collection to obtain information not currently available. To ensure that the research program is: relevant to our customers, stakeholders, & partners, 20% of the overall ARS research program will be evaluated for relevancy each year; of high quality, 20% of the ARS research program will be evaluated for quality each year by OSQR; & is performing well, 100% of the ARS research program will be assessed each year for performance & 20% of the National Programs are evaluated each year against the objectives in their Action Plans. Some targets, measures, &/or data collection procedures are still being developed. We anticipate being able to report substantive data for each annual measure in the 2007 PART analysis. ARS will work with ERS & OEPNU to develop & implement assessment instruments that apply economic value to the program's annual outcomes. These new measures will be piloted to the livestock production subcomponent.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: ARS enters into research partnership with external organizations only when a mutual benefit is clearly demonstrated consistent with the ARS mission & its long-term & annual goals. These partnership agreements (CRADAs and SCAs) clearly define the scope of the research, milestones, measures, outcomes, & also define the mechanism for monitoring performance to ensure support for the overall goals of the program.

Evidence: ARS provides funds to external research entities through over 1,400 Specific Cooperative Agreements, in order to work cooperatively in support of this program's research goals. Prior to ARS approval, research agreements must include performance measures relevant to the long-term, & annual goals of this program. As appropriate, these research partners are involved in national program workshops & planning ongoing research activities. ARS requires its external research partners to provide an annual progress report (AD-421) & identify accomplishments related to the program goals. These accomplishments are reported in the GPRA APR process. Information on annual meetings, workshops, quarterly & annual reports with stakeholders & collaborators are available on file, or at the National Program Staff (NPS) web site (www.nps.ars.usda.gov/programs).

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The research agenda for this program is established by NPS after consultation with a broad cross section of customers, stakeholders, & partners. This process ensures the relevancy of the research program. All research projects undergo review every 5 years through the Office of Scientific Quality Review (OSQR) before implementation. The Peer Review Process conducted by OSQR involves independent & expert scientific peer review of project plans. Members of the review panel are external to ARS. The OSQR review process is a critical component for evaluating research project quality & effectiveness. In this way, OSQR contributes to the quality of the national program. Additionally, a retrospective program review is conducted towards the end of the 5-year program cycle to assess continued relevancy, progress, & performance. Project Annual Reports are carefully reviewed by Area Offices and NPS. Projects also undergo periodic external reviews (location reviews) organized by Area Offices. Goals & targets for program assessments are under development.

Evidence: The peer review schedule for this program is available on the OSQR web site (www.ars.usda.gov/osqr). Documentation describing the type of evaluation & criteria for selecting peer reviewers is also available on the OSQR web site. Reports on other reviews by NPS & Area Offices are available on file.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The ARS budget is developed, in part, using information derived from the Annual Resources Management Plan (ARMP) which is a structured approach to establish a framework for integrated resource planning & to improve program & resource accountability. NPS is responsible for setting research priorities & allocating resources in conformity with research objectives. ARMPs feeds into the budget development & the budget execution processes. The ARS Strategic Plan FY 2003-2007 was crafted to address the needs of the Budget Performance Integration (BPI) initiative, PART, GPRA, USDA Strategic Plan goals & objectives, & to begin to link budget requests with specific performance measures. The ARS budget is presented according to the BPI initiative.

Evidence: The Annual Resource Management Planning (ARMP) process is a comprehensive review of all available resources (human, financial, facilities, etc.) for the upcoming fiscal year. These resources are assigned to specific research projects in this program & reviewed to ensure that sufficient resources are available to meet the needs of the project & the overall program. ARS also perform projections of costs for outyears to ensure that the projects within this program remains viable. Each research project has goals & objectives, the achievement of which links directly to achieving the broader annual & long-term program goals. The ARMP process enables ARS to identify all costs associated with each project and better align them with the budget's goals & objectives.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: In FY 2003, ARS promulgated a new Strategic Plan that specifically incorporates both PART & the Budget Performance Integration initiatives in the President's Management Agenda. The previous agency Strategic Plan was deficient in that it did not provide a good bases for developing a performance-based budget. The new Strategic Plan aligns ARS' priorities with USDA's Strategic Plan. Beginning in FY 2005, the ARS budget request followed the goals & performance measures contained in its Strategic Plan. This new ARS Plan reflects the national program structure which was also a major deficiency in the preceding Strategic Plan. The new Plan is designed to better align the research program to producing tangible results that are useful to American agriculture.

Evidence: The ARS Strategic Plan FY 2003-2007 was crafted to address the needs of the Budget Performance Integration initiative, PART, GPRA, REE & USDA Strategic Plan goals & objectives, & to begin to link budget requests with specific performance measures. The new Strategic Plan is available on the ARS website at www.ars.usda.gov.

If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals?

Explanation: This program is continually coordinated through NPS with complementary programs across the Federal Government, with academia, industry, producers, processors, & consumer groups to ensure scientific relevance. Relevance to the USDA mission is regularly evaluated & coordinated with other Federal, industry & university programs. A wide range of formal & informal mechanisms are used to ensure an ongoing dialogue with customers, stakeholders, & partners (including researchers from academia & the private sector).

Evidence: Customers, stakeholders, & partners (including researchers from academia & the private sector) are regularly involved in program reviews & planning. Stakeholder input on relevance & priorities is regularly sought & documented. Annual meetings with industry stakeholders (including AMI, NCBA, Pork Producers, U.S. Poultry & Egg Assn., National Assn. of Animal Breeders, Renewable Fuels Assn., National Biodiesel Board, National Corn Growers Assn., American Soybean Assn., & Corn Refiners Assn.) to assure the relevancy of research & complementarity to other programs. Examples of coordination of Federal efforts include the Interagency Working Group on Domestic Animal Genomics & the Interagency Working Group on Plant Genomics, the USDA Biobased Products & Bioenergy Coordination Council & the Federal Interagency Biomass R&D Board & the Biomass R&D Technical Advisory Committee.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: Input from customers & stakeholders is obtained in developing the research agenda for this program. External reviewers conduct prospective & retrospective reviews at different points in the 5-year program cycle. The goal of these processes is to evaluate, prioritize, & implement a mission oriented research program that is tied to guiding budget requests & funding decisions. The new BPI is linked directly to both the Agency's & Department's Strategic Plans. The BPI makes clear the impact of funding on performance. In addition, ARS has incorporated the OMB R&D Investment Criteria into its Strategic Plan & provides several mechanisms to ensure that the funds requested in its budget request are relevant to the needs of U.S. agriculture. The ARS Annual Budget Team takes input from all of these sources to determine the priorities that go into the Agency's budget request.

Evidence: The USDA BPI panel (chaired by the Deputy Secretary, & including the Department's Chief Financial Officer, Chief Information Officer, Budget Director, & others) oversaw the development of a performance-based budget for FY 2005 will do so again for FY 2006. The BPI panel provides the Agency with guidance on priorities, Administration initiatives, & relevancy in developing its budget. Each agency followed the prescribed BPI format in developing & linking its budget request to the USDA Strategic Plan's goals & performance measures. In addition, the REE Advisory Board meets quarterly to provide oversight to ARS in prioritizing its budget to benefit the larger science & technology enterprise.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: This program regularly solicits input from customers, partners, & stakeholders in developing Action Plans (research agendas) at the beginning of the 5-year program cycle to ensure program relevance. All research project plans are then evaluated by the OSQR peer review panels of internationally/nationally recognized, independent, external scientists to ensure quality. Regularly scheduled program & location reviews are undertaken to ensure program performance. Each of the 395 research projects in this program submits an annual progress report (AD-421) which is reviewed by their Area Office & NPS to determine if they are meeting the annual milestones in their Project Plan. NPS analyzes the data received from the projects & provides summary reports which are widely disseminated. Projects & programs are subject to redirection or realignment by NPS if the evidence suggests such a step would enhance the relevance, quality, & performance of the research.

Evidence: The ARS programs' Action Plans, Annual Reports, & Internal Review documents are available on file. Project & program realignments & redirections (Program Adjustment Decision Item (PADI)) are conducted to maintain focus & efficiency. For example in recent years, ARS has substantially realigned its value-added research at all 4 of its Regional Research Centers (Wyndmoor, PA; Albany, CA; Peoria, IL; & New Orleans, LA), which resulted in more efficient program management & use of resources. This realignment impacted personnel, fiscal, & facility resources to fit the programs' objectives outlined within the Action Plans. Additional realignments are pending at other locations to improve relevancy, performance, quality, & efficiency. ARS programs are terminated when their mission is completed, or, when the underlying public need for the program has changed. Recent examples: termination of dairy forage research programs at University Park, PA; Wooster, OH; & Columbia, MO; & the redirecting of programs in methyl bromide alternatives to plant disease research.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: National Program Leaders (NPLs) (the program managers) & the Area Directors (line managers) are held strictly accountable through an annual evaluation process. In addition, the impact of the program is regularly evaluated to ensure that the goals of the program are being met, & that the results are meeting program objectives & customer expectations in a timely manner. All collaborative research agreements are continuously monitored by the program managers to ensure that the research activities meet the needs of the program in a timely & fiscally responsible manner.

Evidence: Both scientists & managers are subject to annual performance reviews. ARS scientists are subject to an in-depth peer review on a 3- to 5-year cycle under the Research Position Evaluation System (RPES). The annual reports from each research project are reviewed by both the Area Office & NPS. The NPL responsible for the program has specific program management performance standards which are evaluated on an annual basis.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Research project plans have specific goals & objectives that must be met within a period not to exceed 5 years. ARS allocates, obligates, & manages Agency funds at the project level to help ensure appropriate use. As part of the annual performance appraisal, scientists and NPLs are held accountable for both research progress & the timely & appropriate use of resources. NPLs & scientists have the incentive of performance awards if their performance exceeds the stated expectations, as outlined in their Annual Employee Performance Plan. ARS uses the Foundation Financial Information System (FFIS) accounting system to enhance accountability. ARS udergoes annual financial & programmatic audits by USDA/OIG & GAO that review timeliness of obligating & reporting, accuracy in accounting & intended use of funds. ARS has received clean audit opinions from OIG & GAO for several years.

Evidence: The FFIS accounting system is used to process & manage financial data to meet stringent budget & fund control needs, as well as complex multifund and reporting requirements. This program does not have carry-over authority, therefore, all funds are expended or obligated by the middle of the 4th quarter of each fiscal year. ARS, as an agency, uses its own local obligation tracking system in conjunction with FFIS to ensure that funds are obligated up to the assigned funding level without exceeding it. Traditionally, ARS obligates more than 99.6% of its assigned funds. In FY 2003, the ARS obligation rate was 99.64%. Scientists & managers are subject to annual performance reviews in which they are held responsible for financial management & compliance of their assigned fiscal resources.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: Recent scrutiny by ARS exposed IT structure deficiencies specifically in cyber security, network connectivity, & e-mail configurations. In FY 2002-3, ARS analyzed the costs & benefits of alternative solutions for improving e-mail & web hosting services; recommended the e-mail & web hosting solutions that best met the Agency's needs; & developed an IT Consolidation Project Plan for efficiently & cost-effectively implementing the infrastructure improvements. The Agency has implemented competitive sourcing strategies (A-76) to reduce administrative costs & improve overall efficiency. In order to operate the agency as efficiently as possible & maximize the resources at the scientific bench, ARS has consistently managed with an overhead at 10% or less.

Evidence: The IT Consolidation Project was initiated in FY 2003 & will be completed in FY 2005. Major milestones include: unifying the 52 different e-mail systems; consolidating web hosting; updating infrastructure (i.e., telecommunications network, security, organizational configuration) to support ARS' new e-mail & web hosting design; & providing a plan for building the enterprise architecture & estimating the related costs & staffing. The benefits of the IT Consolidation Project will provide ARS with a secure, scalable, robust network, enterprise e-mail, & web hosting capabilities for ARS customers at the enterprise level. These measures will greatly improve the efficiency of the computer technology that supports ARS scientists & research facilities; decrease redundancy, overlapping, & unproductive use of non-scientific staff time. Another example of the program's efforts to increase efficiency includes outsourcing specific research tasks to partners through cooperative agreements to more rapidly address pressing agricultural issues.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program is well coordinated with related programs across the Federal Government: for example, USDA-CSREES, USDA-GIPSA, USDA-APHIS, DOE, National Academy of Sciences (NAS), National Science Foundation (NSF), Department of Homeland Security (DHS), with university & industry groups funding &/or conducting research to promote the competitiveness of U.S. producers in global agricultural markets. Customers, stakeholders, & partners also include State & local governments, consumer organizations, & professional societies (American Society of Agronomy, American Oil Chemists Society, & Federation of Animal Science Societies). Examples of very successful collaboration & coordination with other agencies include work on biobased products & biofuels with DOE; the sequencing of the bovine, honeybee, & chicken genomes (with NIH & CSREES); & the National Plant Initiative with NSF.

Evidence: The program managers regularly hold workshops & meetings with ARS' primary customers & stakeholders to ensure close coordination of effort. For example, ARS receives input from GIPSA & provides an annual report to GIPSA under an MOU describing how ARS is addressing GIPSA's research needs regarding grain quality measurement. ARS & the American Bakers' Assn. conduct joint annual workshops to assess industry needs & review progress of pertinent ARS research. These other collaborative efforts assure a close coordination of ARS research with other Federal, State, & private partners, customers, & stakeholders to eliminate program overlap & duplication. Program reviews & interagency joint program planning documents resulting from these meetings are available on file.

Does the program use strong financial management practices?

Explanation: The program & project expenditures are currently & continuously tracked using the Agricultural Research Information System (ARIS) & the Foundation Financial Information System (FFIS) to follow scientific & fiscal progress. ARS maintains extensive financial management policy & prodecures on funds management & accountability, including resource management, funds control, & semiannual reviews of unpaid obligations.

Evidence: ARIS allows centralized tracking & multi-point entry of scientific effort, funding, objectives, milestones, progress, & accomplishments across the program. The system also initiates approvals for all research program implementations & adjustments, & all research documentation & publications. FFIS is used to process & manage financial data to meet stringent budget & funds control needs, as well as complex multifund & reporting requirements. Implementation of FFIS & its associated data warehouses provided the integration & capabilities needed to improve the delivery of timely & meaningful financial management information & compliance with legislation, including the CFO Act of 1990. FFIS enabled ARS to achieve a clean USDA-OIG audit opinion of its financial statements for FY 2003.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program leaders, in consultation with customers, stakeholders, & partners, establish the national program research agendas, ensuring that program goals remain relevant & achievable. Area Directors are responsible for ensuring that performance & scientific excellence are achieved. Several years ago, ARS completely revised the way its research portfolio is managed to strengthen its ability to manage coherent & relevant research. The ARS 5-year program planning cycle includes an external retrospective assessment of the relevance & performance of the program accomplishments. This assessment also feeds into the development of the next 5-year action plan. In addition, each new research project in the program undergoes a rigorous, independent external peer review (conducted by OSQR), which provides significant scientific credibility to the program's research agenda. The Agency is realigning its National Program Staff to strengthen the management of its research program. New performance goals, measures, & targets are under development.

Evidence: To ensure quality, relevance, & performance; program realignments & redirections (Program Adjustment Decision Item (PADI)) are conducted as needed to address & correct program management deficiencies. For example in recent years, (1) ARS substantially realigned its value-added research at the Regional Research Centers, which resulted in more efficient program management & use of resources; (2) several livestock production research locations have been realigned to address deficiencies & enhance efficiency in the animal genomics part of this program; & (3) the resources & organizational structure of the National Peanut Laboratory at Dawson, GA, were redirected & realigned to address deficiencies identified by customers & stakeholders. These actions impacted personnel, fiscal, & facility resources, in order to better address the programs' objectives as outlined in the Action Plans. Additional reorganizations are pending at other locations to improve relevancy, performance, quality, & efficiency.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The program allocates funds to specific research projects based on identifiable &/or Congressionally mandated research priorities. "Earmarks" are incorporated into existing projects &/or managed as a specific Cooperative Agreement through an existing project & peer reviewed as part of that project. Each research project must be reviewed & approved by an external peer panel of leading experts (OSQR) to ensure quality. Research plans are revised to reflect the review panel's input. Projects that are required to make major revisions are re-reviewed by OSQR. The panel may indicate that a project is untenable & should not be continued. Under such circumstances both NPS & the Area Office jointly consider whether the project should be completely revised or discontinued, & the scientist & funds redirected into a more appropriate research area. Retrospective Program Assessments are being piloted & performance measures are being developed to ensure that high quality is maintained throughout the program cycle.

Evidence: To ensure the quality of the research conducted in this program, each of the research projects undergoes periodic reviews by independent, external peer panels. To date, 29 OSQR panels have reviewed 351 projects in this program & ranked them into 1 of 5 categories: no revision (NR 43), minor revision (MNR 129), moderate revision (MDR 103), major revision (MJR 67), not feasible (NF 9). Overall, 78% of projects reviewed scored moderate revision or higher & 22% of the projects required major revisions or were NF. All projects scoring either not feasible or major revision were rewritten, resubmitted, & re-reviewed through OSQR, & all but 4 were subsequently approved. Even projects scored "minor revisions" are revised, reviewed, & approved by NPS & Area Offices. Detailed information for the OSQR Review Process & guidelines are available on the ARS website (www.ars.usda.gov).

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: This program, focused on enhancing the agricultural sector, made substantial progress towards achieving its long-term goals. The program is working to achieve an additional 8-fold increase in the market share of biofuels by 2010 & increase animal & crop production efficiency. The Index of Agricultural Productivity captures the growth in outputs not accounted for in growth of inputs. The Index of Agricultural Productivity (1996 = 100) was 104.4 in 2002, the most recent year for which the Index was available. This is up from 103.6 in 2001. Biobased transporation fuel (ethanol and biodiesel) use increased from 0.5% of total fuel (gasoline and diesel) use in 2002 to 1.9% in 2004. This is consistent with acheiving the target of 4% by 2010. ARS uses its Strategic Plan & the GPRA process to track & report on progress of this program. In the FY 2002 GPRA Annual Performance Report for Goal 1, ARS had 29 indicators of progress aligned under 12 performance goals. Sixty-seven significant accomplishments are contained in this report against the 29 indicators. These accomplishments demonstrate that progress was made toward meeting the program goals. The ARS FY 2002 Annual Performance Report is available on the ARS website www.ars.usda.gov. Therefore, even though there is no data to demonstrate future acoomplishments for these measures that are under development, the question was given a "small extent" rather than a "no" since the agency provided data to show past progress.

Evidence: There has been a dramatic increase in the use of biofuels in recent years & ARS research has made a significant contribution to this effort. Fuel ethanol & biodiesel are major components in biobased transportation fuels. Historical information shows that the use of fuel ethanol increased from 175 M gallons in 1980 to 2.81B gallons in 2002, to 3.4B gallons in 2003. Biodiesel use has increased from 500,000 gallons in 1999 to 25 million gallons in 2002, to 30 million gallons in 2003. Likewise, agricultural productivity has continued to grow in recent decades, although the rate has slowed.

Does the program (including program partners) achieve its annual performance goals?

Explanation: ARS has adopted the OMB R&D Investment Criteria as its annual measures for this program. ARS has modified its Strategic Plan & incorporated the data requirements in its GPRA process. It has also modified its data collection process to secure project and program information curently not available. For relevance, ARS is holding National Program Workshops involving researchers, customers, stakeholders and partners covering 20% of the ARS research program each year (10 workshops were held in FY 2003). ARS is collecting data and establishing baselines for its annual performance goal, quality - 234 projects were reviewed between April 2003 and March 2004, or slightly over the 20% target. For performance, 100% of the ARS research program will be assessed each year, against performance indicators and targets that are under development. ARS anticipates being able to report substantial data for relevance and performance in the 2007 PART analysis and in GPRA. A "small extent" was given because the progress was not consistent.

Evidence: The Quality review process (OSQR) is well established & able to report tangible results as outlined in the Explanation column. ARS has, at OMB's suggestion, modified its AD-421 (annual project report) process to collect data on how the research units are meeting their established milestones & thus better measure Performance. We are planning to include specific program milestones in future National Program Action Plans with will give us another tool for measuring Performance. We are also collecting additional data to help us report more effectively on program Relevancy. Specific Baselines, Targets, & Performance Measures for Relevance & Performance are being developed. This data will be available in time to be reported in the 2007 PART analysis & in GPRA.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: A major objective of ARS is to keep overhead costs to a minimum, thereby maximizing the resources devoted to its research activities. The Agency's overhead represents approximately 10% of appropriations. In recent years, the agency has kept its Headquarters & Area Office overhead cost below 10% at a time when growth in the research program, the scientific workforce, & in mandated additions to Headquarters responsibilities, such as OSQR & OCIO, etc. have required across the board restraint in administrative units. In addition, this program demonstrates efficiency & cost effectiveness by collaborating &/or co-locating with university partners, through purchasing shared laboratory equipment, the formation of core facilities, & the sharing of support staff. The program periodically reviews its research operations for opportunities to further improve efficiency & cost effectiveness. ARS continually looks for opportunities to enter into interagency collaborations & partnerships to increase the effectiveness & efficiency of its research programs.

Evidence: Each year the Agency reports its overhead which consistently has been at 10% or less. This research program has facilities co-located with Iowa State University, Mississippi State University, Colorado State University, University of Illinois, Cornell University, University of Minnesota, University of Missouri, Washington State University & others. Several research centers & locations, including the the National Peanut Laboratory at Dawson, GA, the value-added programs at the Regional Research Centers, & the livestock program at BARC restructured their research & management functions to maximize efficiencies, reduce costs, & strengthen the focus of their research program.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: ARS is the only intramural research agency in USDA & there is no comparable research entity in Federal, State, local government, or in the private sector that address agriculture & food issues at the national level. In recent years there have been a number of studies by the National Research Council, REE Advisory Board, & various task forces. None of these studies specifically or directly addressed this question.

Evidence: What makes ARS unique is its ability to address the full scope of national agricultural priorities with long-term, high risk research impacting animal & crop production & biobased product utilization that cannot be addressed by any single entity in the United States. Relevant examples of long-term, high risk research that distinguish ARS from other research organizations include: the Dairy Herd Improvement Program, the National Plant, Animal & Microbial Germplasm System, and the Super Slurper technology.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Each research project is reviewed by an independent external peer panel at the beginning of the 5-year program cycle to ensure the quality of the project plans (OSQR Review). Towards the end of the 5-year program cycle, the program will undergo a Program Assessment (Portfolio Review) that will include an external retrospective panel of scientists and stakeholders, followed by one or more National Program Workshops comprosed of a broad cross section of customers, stakeholders, partners, scientists and ARS managers who review the accomplishments of the preceding five years. The Workshops will then identify current & emerging problems that will form the basis of new Action Plans that will lay out the research agenda for the next 5 years. During reviews, one National Program panel found that research accomplishments had a highly favorable impact on end users. Another determined that its research was of high scientific quality and high impact. New project plans are developed consistent with the new Action Plan. In addition, each scientist in the program is reviewed by a peer panel every 3 to 5 years to ensure the quality of the ARS scientific workforce.

Evidence: An important part of ARS 5-year program cycle is retrospective reviews by external panels made up of scientists & customers/stakeholders, as appropriate, & a 2nd round of national program workshops which will evaluate the program's past performance & identify research priorities for the next 5 years. ARS created & implemented its current program structure about 5 years ago so the 'oldest' programs are just now entering the retrospective assessment phase of the program cycle. In FY 2004, two National Programs, Manure and By-Product Utilization (NP 206) and Plant Biological and Molecular Processes (NP 302) were reviewed and National Program Workshops held. The NP 206 panel found that the research accomplishments had a highly favorable impact on end users. The NP 302 panel determined that the program's research was of high quality and high impact. The findings of both panels are used to plan program direction in the next five year cycle. Action plans are now being developed for boh programs to reflect this input. Beginning in FY 2005, ARS will review approximately 20% of the research programs each year (about three to five National Programs) and will report the results in the FY 2007 PART analysis and GPRA.