Detailed Information on the
Protection and Safety of Agricultural Food Supply (Grants) Assessment

Improving the focus of grant recipient reporting to place more emphasis on outcomes.

Increasing planning and coordination with the Agricultural Research Service regarding the collection of stakeholder input.

Develop measures that show the actual use of discoveries and technologies that are developed by the program. In addition, the program needs to develop targets related to extension activities.

Find more innovative and cost effective ways to review grant proposals on an agency wide-basis.

Re-evaluating the efficiency measures, proposing new ones if appropriate.

Continuing to improve efficiency in the grant review process through the utilization of Grants.gov, the web-based peer review system, and virtual panels where appropriate.

Measure: Portfolio Review Score

Explanation:Goal performance is externally assessed on a 5 year basis to determine progress toward solving targeted national problems related to each Strategic Objective. The score reflects expert assessment on 3 PMA dimensions of relevance, quality and performance, along a continuum from exceeds expectations, to meets expectations, to needs improvement. The reported score is a mean of quantitative scores for the Goal''s objectives.

Measure: Methods that Reduce Food Contamination and Growth of Foodborne Organisms

Explanation:The number of contamination reducing methods (interventions, mitigations) for priority, high public health risk, and economically important microbial pathogens and contaminates that have been developed and used.

Measure: Internal Portfolio Review Score

Explanation:Goal performance is internally assessed annually by NPLs to determine progress toward solving targeted national problems related to the Objective. The score reflects expert assessment on 3 PMA dimensions of relevance, quality and performance, along a continuum from exceeds expectations, to meets expectations, to needs improvement. The reported score is a mean of quantitative scores for the Goal''s objectives.

Measure: Proposal Review Time in Days

Explanation:Average number of calendar days between receipt of proposal and funding date funing awarded for competitively reviewed proposals. Average is calculated by summing the total number of days for all major competitively reviewed proposals and dividing by their total number.

Measure: High Consequence Plant and Animal Pests and Diseases

Explanation:The number of high-consequence pests, bacterial, parasitic, and vital pathogens, and disease threats that affect animal, plant and human health and impact the national economy that the integrated National Plant Diagnostic Network and the National Animal Health Laboratory Network diagnostic labs are prepared to detect and diagnose possible outbreaks.

Measure: Cumulative Dollars Saved for Grant Review

Explanation:Dollars saved reflects the average salary saved by calculating the number of calendar days saved annually between receipt of proposal and date funding awarded for competitively reviewed proposals, then multiplied by the average daily salary for CSREES employees. The annual savings can be derived be comparing the change in cumulative savings between two consecutive years.

Is the program purpose clear?

Explanation: The mission of the Department of Agriculture's programs for Goal 3 is to enhance the protection & safety of the Nation's agriculture & food supply. The Cooperative State Reseach Education and Extension Service (CSREES) funds grants for research, edcucation and extension programs via coordinated work under three portfolios deemed critical by expert review panels: Reduction of Food-borne Disease; Plant Protection, and Animal Protection. Multiple funding authorities support this work which advances food protection through science-based knowledge, education and delivery of information and technologies addressing the most critical threats, and used widely by farmers, ranchers, and producers to reduce the number and severity of agricultural pest and disease outbreaks in plants, animals, and food products, ensuring that the US will maintain its status as the most productive country.

Evidence: CSREES Goal 3 education, research & extension work is clearly & explicitly defined by the USDA Mission & Strategic Plan (2002-07), REE Mission & Strategic Plan (2002-08), & CSREES Mission & Strategic Plan (2004-09). Responsibilities are authorized by: Food Security & Rural Investment Act of 2002; Research Facilities Act; Conference Report for Agricultural Risk Protection Act of 2000; Competitive, Special, & Facilities Research Grant Act; Agricultural Research, Extension, & Education Reform Act of 1998; Federal Agriculture Improvement & Reform Act of 1996; Food, Agriculture, Conservation & Trade Act of 1990, Title XVI; Food, Agriculture, Conservation, & Trade Act of 1990, Title XXV; National Nutrition Monitoring & Related Research Act of 1990; National Agricultural Research, Extension & Teaching Policy Act Amendments of 1985; National Aquaculture Act of 1980; National Agricultural Research, Extension, & Teaching Policy Act of 1977; Rural Development Act of 1972.

Does the program address a specific and existing problem, interest, or need?

Explanation: A safe, secure food supply is a priority of Congress & the Administration. CSREES grants for its Goal 3 programs serve to increase food safety, plant protection, & animal health & welfare via research, education, extension, & integrated projects, such as Integrated Pest Management & Plant & Animal Diagnostic Networks. Topic clusters deemed critical by expert reviewers that address a secure food supply include: production of an adequate & wholesome supply of animal & food products; reduction in pests, internal & external parasites; & loss in productivity due to toxic chemicals, pesticides, & poisonous plants. Development of analyses of plant & animal health risks & their management responses developed to reduce such risks guide users' decisions on production; enhance US global competitiveness; ensure individual producer survival & sustainability in a dynamic, competitive world; & enhance food security, public health & control of zoonotic diseases.

Evidence: CDC estimated 76 mil. people get sick,>300,000 are hospitalized & 5,000 die from food borne illnesses. These remain a major public health challenge. Minor Use Animal Drug Program; National Animal Health Laboratory Network; National Plant Diagnostic Network. The Porcine Reproductive Respiratory Syndrome (PRRS) research program links over 60 researchers & extension specialists in an integrated approach to reduce a $600 mil. annual disease problem. The 1st vaccine developed for PRRS originated from CSREES support. The portfolio contributed to important new leads in degrading prion proteins using keratinase. The National Animal Health Laboratory Network provides early detection of high consequence animal diseases through establishment of 12 laboratories in collaboration with APHIS & the American Association of Veterinary Laboratory Diagnosticians. National Plant Diagnostic Network links plant & animal disease diagnostic facilities across the country.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Goal 3 enhances protection & safety of ag. & food supply through coordination, consultation & input from other agencies & stakeholders, in cooperation with state partners. NPLs coordinate with partners, other providers & constituents to ensure work is not redundant or duplicative, but complementary & supplementary- ensuring comprehensive, validated knowledge for constituents. However, external panels state some redundancy is needed- trying multiple approaches can solve problems quicker. In meetings with USDA counterparts (ARS/ERS/FSIS scientists), NPLs seek input for new grants announcements that will complement work of other agencies/include colleagues on CSREES proposal peer reviews. e.g., CSREES/NSF jointly implemented animal genome projects, eliminating duplication; CSREES/ARS/ Amer. Assoc. of Veterinary Immunologists discuss needs for specific immunological reagents; CSREES/FSIS/FDA evaluate foodborne illness data to dev. more effective regulatory strategies.

Evidence: Goal 3 objectives enhance the protection & safety of ag. & food supply through coordination, consultation & input from other agencies & stakeholders. National Program Leaders coordinate with partners, & constituents to avoid redundancy or duplication ensuring comprehensive, validated knowledge for relevant constituents. Through meetings with USDA research counterparts (ARS/FSIS scientists/ERS/economists), NPLS seek input for Requests for Applications to complement the value of work of other providers, & including them on peer reviews panels. CSREES/NSF jointly implemented the animal genome projects; CSREES/ARS/Ame. Assoc. of Veterinary Immunologists address lack of veterinary specific immunological reagents; CSREES/FSIS/FDA eval. foodborne illness & dev. measures on effective regulatory strategies. http://www.csrees.usda.gov:8765/query.html?fs=http%3A//www.csrees.usda.gov/funding/rfas/04_nri.html&ws=0&rq=0&qm=0&charset=iso-8859-1&col=about+business+csrees+funding+news.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Optimally designed portfolios address needs of producers, growers, students, teachers, and professionals in food safety, animal and plant protection. Portfolios integrate research and extension in a comprehensive, non-duplicative package. High standards of relevance, quality & performance specified in the PMA are achieved through independent peer/merit review by NPLs/external expert panels. Funding is never awarded to inferior proposals and is deferred pending correction of deficiencies. Proposed 2006 budget includes phased elimination of formula funded/earmarked research & increased use of competitive grants, including NRI. Projects directly funded by Congress align with and contribute to Goal 3 achievement because NPLs work closely with earmark recipients to enhance planning & execution of projects to ensure relevance & quality.

Evidence: The 2006 budget request proposes no earmarks & replaces formula funds with more for the NRI (to $250 mil.) & $75 mil. competitive grants for regional, state, local priorities. While earmarks are routinely eliminated in budget requests, all subsequent earmarked projects are subjected to merit-based evaluations & measured against standards established for all CSREES grants. NPL scientific expertise, independent review & stakeholder input address national needs. This reflects changes in the state of science & priorities for comprehensive, effective, efficient investment in protection of ag & food supply. Peer review ensures relevance, quality & productivity of activities. Integrated education, research & extension define science in monitoring, detection & mitigating food, animal, plant threats. Plant & animal protection are distinct from topics in plant & animal production PARTed in 2004.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Many CSREES beneficiaries are defined in authorization legislation, through formula grants to land grant institutions, as well as through appropriated earmarks. CSREES products reach intended beneficiaries and/or otherwise address Goal 3 directly. Collaboration with other agencies (including ARS, ERS, FSIS, FDA) complement and supplement work not covered in other agency mission areas. No other USDA research agency has the extended mission of education and extension. Triple targets are reached, as CSREES ensures a supply of researchers and educators and ensures research results used by farmers, producers, and the general public. e.g., to reduce the annual multi-billion dollar trade deficit in shrimp, CSREES supported development of the shrimp-farming industry in the US via a line-item appropriation for the United States Marine Shrimp Farming Program (USMSFP). Research is directed at developing a sustainable commercial shrimp industry. Recently the USDA/USMSFP provided direct access to reliable supplies of high-health and genetically improved shrimp stocks.

Evidence: CSREES designed & packaged Strategic Goal 3 portfolios in collaboration with other USDA agencies, including ARS & ERS, to complement & supplement work not covered in their agency mission areas avoiding redundancy & duplication of efforts/funding, to university served community (researchers, students, industry, producers, & the general public). The soybean entomological entomological regional projects (S-74, S-157, S219, S-255 & S-28) were a clear example of how collaborative research efforts generated results for the end-users. Pest control recommendations from those efforts were packaged & transferred to producers through the extension service. The newly adopted strategies moved from the reliance on "hard" pesticide usage to a newer & more environmentally friendly & target oriented pest management methods.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: CSREES programs, including those for Goal 3, provide assistance entirely through grants related to research, education and extension. Grants provided through Goal 3 relate to enhancing the protection and safety of the nation's agriculture and food supply. Strategic Goal 3 has a limited number of quantifiable long term outcome based performance measures that focus specifically on ensuring public health and well being and the protection of plants and animals, and animal welfare and well-being through improved production, management, handling and processing of plants, animals, and food products, and providing an adequate, affordable supply of safe products to meet domestic and global needs for safe, affordable, nutritious food. Measures show the development and use of contamination reducing methods for serious pathogens, the development and availability of pest and pathogen tests in case of an outbreak, and external assessments to determine progress toward solving national problems.

Evidence: The overall long term performance measure established for Strategic Goal 3 is an aggregate 5-year Portfolio Review Score with a goal of 80 for 2005. The actual 2005 score received was 86 as documented in CSREES, FY 2006 Explanatory Notes. The other specific long term measures are: (A) the number of contamination reducing methods (interventions, mitigations, etc.) developed for critical high human health & economic consequence threats, & available for adoption & to be used, & (B) the number of high consequence animal & plant pest & disease threats that National Plant & Animal Health Laboratory Networks diagnostic laboratories are prepared to detect & diagnose. Both measures are baselined, revised in 2004, to reflect recent research accomplishments. See PART Measures tab; see CSREES Statement of Agency Goals & Objectives, Exhibit 4-1, 2005 Budget Submission.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The panel saw as ambitious the agency targets over the next 5 years. Specifically, these targets are: a 5% increase in external review score, substantial increases in the number of contamination reducing methods & high consequence pest & disease threats that diagnostic labs are prepared to detect & diagnose (electrostatic sprays for produce, interventions in lairage & transportation in cattle & swine, electron beam tech., vaccines for cattle for salmonella & E coli); an increase in active programs for the detection of high consequence animal pathogens & plant pests through state plant diagnostic labs that are securely linked with the National Animal Health Lab. Network & the National Agricultural Pest Info. Sys (in addition to Foot & Mouth Disease, Classical Swine Fever, Exotic Newcastle Disease, High Pathogen Avian Influenza, Bovine Spongiform Encephalopathy, Chronic Wasting Disease; Sudden Oak Death, Soybean Rust, Ralstonia solanacearum race 3 biovar 2).

Evidence: Recently completed panel review judged long-term targets to be ambitious. Long term targets & 5 yr. time frames are: 1) 5% inc. in the external review score (86 to 88); 2) an inc. (6 to 20 by 2010) in the # of contamination reducing methods & the existing processing tech., electrostatic sprays, interventions in lairage & transportation in cattle & swine, electron beam tech., vaccines for cattle for salmonella & E coli; 3) an inc. (9 to 14 by 2010) in active programs for detection of high conseq. animal pathogens & plant pests: Foot & Mouth Disease, Classical Swine Fever, Exotic Newcastle Disease, High Pathogen Avian Influenza, Bovine Spongiform Encephalopathy, Chronic Wasting Disease; Sudden Oak Death, Soybean Rust, Ralstonia solanacearum race 3 biovar 2 through state plant diagnostic labs securely linked with National Animal Health Lab. Network & National Ag. Pest Info. Sys. See PART Measures tab; CSREES Exhibit 2-1, Statement of Agency Goals & Obj., FY 2005 Budget. Submission.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: CSREES has specific annual performance & efficiency targets that measure progress toward achieving long term goals. Annual measures include: (A) output (quality, relevance & productivity of agency funded education, research & extension work related to Strategic Goal 3, & (B) efficiency (cost & time to review proposals & award funding to grantees). These measures & targets are consistent throughout the agency for all Strategic Goals & Objectives. Annually, the programs are assessed internally based on the performance measures. These internal scores will be presented to external panels convened every five years to demonstrate progress toward the program's long-term goals.

Evidence: Annual performance of Strategic Goal 3 objectives is assessed to determine progress toward solving targeted national problems & achieving long term goals. Annual internal performance review scores for each objective reflect expert assessment of the PMA dimensions of relevance, quality & performance, on a continuum from exceeds expectations to meets expectations, to needs improvement on each of 14 categories. Efficiency measures monitor progress in reducing cost per proposal reviewed, & time per proposal processed, reviewed & awarded funding. Efficiency baselines for cost per proposal reviewed ($530) & mean time to review proposals & release funds (220) days were established in 2002 & recomputed in 2004 to reflect increased transportation & lodging costs & other variables beyond the control of the Agency, a 25% increase in honoraria, & the increased use of more effective but more costly on-site, rather than ad hoc, reviewers.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The recently completed PREP (expert review) judged long-term targets to be ambitious. Following the completion of the USDA & CSREES Strategic Plans, portfolios of work were established for each of five Strategic Goals, with annual measures for performance & efficiency. Baselines are established on recent performance (revised in 2004), & ambitious targets have been established to routinely & continuously improve performance, & reduce costs & time. Targets are ambitious because they have been set higher than previous years with incremental increases for the next 10 years (budget dependent) to reduce contamination & food-borne organisms as well as reduce high consequence plant & animal pest & diseases.

Evidence: Annual performance scores for each objective are baselined at meets ambitious expectations. Annual performance targets call for a cumulative percentage point improvement in exceeding expectations. Efficiency baselines for cost per proposal reviewed & mean time to review proposals & release funds were established for 2002 & baselined in 2004. The cost per proposal reviewed measure reflects transportation & lodging costs & other variables beyond the control of the Agency, honoraria, & the increased use of more effective but more costly on-site, rather than ad hoc, reviewers; annual targets call for holding review & processing costs to a real increase of 2%, & an average 2% reduction in review time. See PART Measures tab. Honoraria costs are rising due to the need to be more competitive with other higher-paying research agencies (such as NSF) in attracting highly qualified proposal reviewers.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: CSREES enters into education, research, and/or extension partnerships only when mutual benefit is clearly demonstrated & it is consistent with CSREES mission, Strategic Plan & long term & annual goals. All submitted proposals & plans of work specify the scope of work, milestones, measures, outputs & outcomes, along with performance monitoring procedures. Submission of proposals & acceptance of awarded funds requires contractual commitment to working to accomplish annual & long term goals.

Evidence: CSREES reviews over 5000 proposals & funds over 2000 proposals, cooperative agreements & other transactions yearly. All proposals & plans of work (including formula, special, other mandated funding) must include measures relevant to CSREES long term & annual performance goals & targets. NPL review & monitoring of performance assures continued commitment of partners. Funds are withheld or deferred in the event that the contracted commitment is not maintained. NPLs work together & with peers in other agencies to develop RFA guidance that explicitly specifies long term & annual goals. As an example of government partners working toward common goals, over the past 3 years Cooperative Extension Service partnering with Farm Service Agency built a mutually beneficial working relationship between government agencies in need of plant & animal production training for recently hired Farm Loan Officer Trainees www.joe.org/joe/2004april/ent.shtml#rb5

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: All Goal 3 portfolios are independently assessed & evaluated by panels of outside experts every 5 years, with a 2005 baseline score. Outcomes will be assessed internally on an annual basis. The assessments provide useful comments & recommendations to help NPLs & implementing partners generate meaningful outcomes over the long term. Other external reviews, evaluation studies & analyses of CSREES programs are encompassed in evidence presented to the review panels along with self-review documents. Review panel members underwent scrutiny for lack of bias, conflicting interests, & other such issues related to independence. The panels addressed the full scope of CSREES research, education, & extension work in Goal 3 described down to the level of each topic area & contributing programs. Having external panels meet on a 5-year basis is common practice in research, & allows sufficient time for advances in research results.

Evidence: The results of the three independent expert portfolio review evaluations for CSREES Strategic Goal 3, conducted in February & March, 2005, are documented in self-study reports & can be found in the Long Term Measures tab. All Goal 3 R&D Portfolios received independent review scores of exceed expectations: Reduce the Incidence of Foodborne Illnesses & Contaminants, 83; Animal Protection, 95; Plant Protection, 80.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The Budget & Performance Integration (BPI) process ensures a direct and continuous link between long range and annual goals, defined in the 3 Strategic Plans, and the annual Departmental Performance Plan, and REE Quarterly R&D performance reports submitted to the Under Secretary. Annual measures are established but long term measures are in the process of being assessed. Designation of explicit topic areas that define the portfolios are exclusive and non-duplicative, and make it possible to directly link budget lines with goals, objectives, portfolios, and problem areas in a transparent manner. However, the Budget justfication does not clearly link the impact of funding decisions on key performance measures.

Evidence: Annual & long-term performance goals & accomplishments were an explicit component of the FY 2006 President's Budget Request. See Agency budget requests, Explanatory Notes, tables & exhibits, & accompanying information. New classification & reporting systems currently under development or in pilot testing will continue to strengthen the agency budget & performance linkages & increase transparency in the future.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The portfolio review process identifies the opportunities for needed changes on an annual basis through the use of core teams. Reviews take place annually. Timetables are often based on deadlines related to the issuance of RFA's (Requests For Applications) and are reviewed annually. The alignment of CSREES programs in program areas that are consistent with the USDA research agencies and USDA's overall strategic plan was an example of this process. Significant efforts were undertaken to fully align the new CSREES Strategic Plan with the REE & the USDA Strategic Plans. Comprehensive Problem Area codes have been developed for all agency work, & are being used to align areas of CSREES higher education, research, & extension work under the portfolios that support the strategic objectives. Budget Performance Integration (BPI) enhances agency budget request preparation, justification, & reporting based on need, strategic planning, & overall performance.

Evidence: CSREES demonstrates comprehensive linkages of all Strategic Goal 3 portfolios between congressional authorizations & appropriations, the Strategic Plans, the departmental & agency mission, the departmental Performance Plan, & specific areas of science & education. All three of the strategic plans are posted together on the agency web site at www.csrees.usda.gov/about/strat_plan.html. The CSREES OneSolution data management & reporting system comprehensively links strategic planning, Budget & Performance Integration (BPI) & output, outcome & impact reporting. The agency recently submitted legacy & major accomplishment expectations to the Secretary through the office of the Under Secretary. Also see CSREES Customer Service Satisfaction Standards.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: Constituents, stakeholders and partners are involved in planning, development and review. NPLs consult with academic and industry scientists, USDA, advisory groups (National Ag. Research, Extension, Education and Economics Advisory Board; Council for Ag. Research, Extension and Teaching; Council for Ag. Science & Technology) and industry to assure complementarity with programs of others with similar goals. Coordination with USDA, interagency work groups, other departments adds value and extends benefits of programs with similar goals. Portfolio review uses independent outside experts to review CSREES investments in research, extension and education progress toward national problems and offer opinions and recommendations on new efforts to increase benefits. To receive funding from competitive sources, independent reviewers from outside rank high priority projects that are of high quality and expected to yield the most value. This allows for comparison of benefits among alternative proposals.

Evidence: CSREES designed & packaged Goal 3 programs in collaboration with other agencies (FSIS, APHIS, ARS, ERS) to complement work of others & supplement the work not covered in their agency mission areas to reach intended audiences & beneficiaries. R&D leaders confer quarterly with REE agencies. See REE Quarterly R&D Performance Management Plan & Quarterly Reports. CSREES' mission to support research, education, & extension is an asset in specifically utilizing & adding value to research efforts of other federal agencies & departments, as well as other public & private sector providers. Goal 3 portfolio reviews found there is a suitable process for determining the proper set of intended beneficiaries, & the program is sufficiently flexible to adjust quickly as the pool of intended beneficiaries, needs & conditions change. See Review Panel Report, Objective 3.1 2005; Objective 3.2A Portfolio Panel Review Report, 2005; & Objective 3.2B Portfolio Panel Review Report, 2005.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The Agency develops annual budget & programmatic priorities based on existing & emerging needs, recent successes in research, the state of science, educational capacity, & direct input from partners, constituents & other providers, as well as other USDA agencies, the Secretary and the Office of Managment and Budget CSREES secures inputs from stakeholders through both formal & informal processes. CSREES held a series of workshops including the USDA-CSREES Stakeholder Priorities Workshop for Animal Agriculture; Fair 2002 Implementation Partnership - USDA-ARS & the USDA-CSREES; the 2002 Joint ARS/CSREES National Aquaculture Program Planning Workshop. An overall priority is to provide additional funding for competitive reseafch grants through thre National Reseach Initiative, and to propose notto continue funding for earmarks to specific locations and purposes. The comprehensive BPI process makes the linkages & potential gaps & overlaps apparent as the budget request is being prepared, allowing for refinement of the budget request, strengthened justifications, & appropriate program changes, before the budget request is submitted. See Dodge Kelsey & Pense, "A Model for Gathering Stakeholder Input for Setting Research Priorities at the Land Grant University" In addition, CSREES is publishing documentation of its review process in the "web" by September 2005.

Evidence: NPLs & peers collaborate on dev. of RFAs, incl. the prioritization, science & edu., & integration of portfolio work in the agency. Comprehensive National Research Initiative RFA Planning Worksheets are prepared for each priority issue, clearly defining mission-relevant problems to address in short, intermediate & long term goals with stakeholders. Ag. Research, Edu. & Extension Reform Act (AREERA) requires recipients of formula funds to collect stakeholder inputs yearly. CSREES held a series of workshops with customers & constituents in 2003. This information, with Portfolio Review recommendations is integrated in the BPI process in collaboration with OBPA & OMB to guide Agency budget requests & funding decisions. In line with the President & Congress' desire for more accountability of public funds, CSREES redirected formula based funds for the Animal Health & Disease, Section 1433 Research Program & $75M to universities to competitively awarded grants. See FY 2006 Explanatory Notes.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Recipients of competitive & line item grants are required to submit annual progress & final reports through the Current Research Info. Sys. (CRIS). This information is used to revamp the RFA's for future years, to make recommendations to specific grantees, or to provide a different approach across a program area. Since research and extension programs have different purposes, baseline and performance data would not be the same. Research NPLs use this info. to monitor progress, make recommendations, measure impact & make changes in priorities & focus, & program admin. CSREES collects info. on progress of research formula awards through CRIS & collects info. on portfolio of ag. research & extension activities through the plan-of-work reporting process & other proprietary databases, with NPL oversight. Extension work reports through Plans of Work & annual reports. CSREES is developing a reporting sys. or add-on feature to CRIS for extension work to better integrate performance data across edu., research & extension. The OneSolution data & reporting system will inc. timeliness & quality of data/info., reduce reporting burden on key program partners, & inc. program performance mgmt. OneSolution is estimated to save the Agency $20.2 mil. over 7 years; benefit cost ratio of 1.89.

Evidence: Terms & Conditions for competitive & line item awards specify annual & final report requirements. Reports determine progress of each grant & if additional funds should be provided. NPLs use information to compile reports, measure program impacts & respond to inquiries from management, public, & Congress. See cris.csrees.usda.gov/. Research projects supported with formula funds are reported in CRIS as per Administrative Manuals for Hatch, Evans-Allen, McIntire-Stennis & Animal Health Research Programs. Some programs have proprietary database reporting systems. All 1862 & 1890 Institutions are required to submit 5-Year Plans of Work & Annual Reports of Accomplishments & Results (See Guidelines for State Plans of Work & Annual Report Guidelines). Reports measure institutional impacts in addressing critical issues identified through stakeholder input. See OneSolution Initiative, Business Case, Final Draft March 3, 2005; & CSREES Customer Service Satisfaction Standards.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: CSREES managers are accountable at all levels of the agency. Office of Extramural Programs is fiscally accountable; program managers are subject to Financial Assistance Regulations. Partners submit Cash Transactions, Financial Status Reports & undergo audit, per OMB Circular A-133. Formula grantees submit a Cash Transactions Report, annual Financial Status & documentation to sustain the report. NPLs are accountable for schedule & performance results & monitor progress by evaluating & measuring program impacts. CSREES performance standards relate to strategic goals & objectives. Project periods (3-5 years) ensure time to carry out work on schedule & as approved from peer- or merit- reviewed proposals & plans of work. NPLs consider budget adequacy; partners obtain prior approval for budget changes. NPLs review & approve plans of work from the land grant institutions, arrange for & coordinate review of programs not formally covered by plan of work reporting.

Evidence: Budget Office develops Control Tables with NPL assignments for the fiscal year. Office of Extramural Programs develops guidelines & training to assist NPLs in solicitation, review, & award of grants, an effort of a workgroup of NPLs & administrative staff to streamline & simplify grants administration & solve problems. Grant recipients are provided an annual allocation letter that provides amounts of grant awards, & if applicable, matching requirements; funded proposals generally have a 3 to 5 year lifespan & specific performance parameters that are contractual. Institutions with a match requirement submit a Certification of Offset before formula funds are released. Guidelines for Plans of Work, Annual & Termination Report Guidance, & Administrative Guidance for Integrated Research & Extension Activities provide grantees with compliance & reporting instructions for administration of funds, completing work on schedule, & accomplishing the research & extension goals.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Funds are, in all cases (unless Congressionally mandated otherwise as No Year or Two Year), required to be obligated in the year appropriated. Audits, reviews & mandatory reporting, along with active post-award management, ensure that funds are spent for the intended & agreed upon purposes.

Evidence: "CSREES prepares a schedule of obligations, monitors unobligated balances, compares obligations with scheduled estimates, & makes appropriate adjustments. NPLs can defer the dispersal of obligated funds if deficiencies or deviations are detected. Awardees are required to take proactive corrective action in order to secure the release of funds. There is one instance in which funds were not obligated in FY 2004 & it was not related to Goal 3. Since this occurrence, CSREES has developed a report in CREEMS to prevent this from happening again. This is the only instance in quite a while for CSREES. In FY 2004, CSREES did not obligate about $85,000 in Integrated Water Quality Funds. The amount can be verified by checking the 06 Explanatory Notes. Since this occurrence, CSREES has developed a report in CREEMS to prevent this from happening again. This is the only instance in quite a while. "

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: CSREES has an internal grants management system, CREEMS, that assists management & staff by providing budget, expenditures & statistical data related to the administration of grants (e.g., time to process a proposal from receipt to award; time to process a post-award action). The system serves as a management tool for tracking action items & administrative compliance requirements. The system is CSREES' innovation in integrating all grant processes (programmatic, administrative, & fiscal) in one database. Annual efficiency measures & targets compare proposal review time & costs for continual improvement. The OneSolution data & reporting system procedures will increase the timeliness & quality of performance related data & information, reduce the reporting burden on key program partners, & increase program performance management. It is estimated to save CSREES $20.2 mil. over 7 years, with a benefit cost ratio of 1.89. Note that the costs do not include the use of the internal grant management system. At this point, USDA has not determined the extent to which electonic management will reduce review costs.

Evidence: See the Annual Efficiency Measures tab. For CREEMS documentation, see www.csrees.usda.gov/business/other_links/egov/csrees/creems.html. The Agency has a comprehensive IT Strategic Plan to increase the use of information technology to increase cost & time efficiency, see www.csrees.usda.gov/about/offices/pdfs/strategic_plan_03-08_Final.pdf. See also CSREES Customer Service Satisfaction Standards. The Agency also has a specific Information Technology Strategic Plan, revised July 2004, www.csrees.usda.gov/about/offices/pdfs/strategic_plan.pdf; for OneSolution documentation & analysis of projected benefits, see OneSolution Initiative, Business Case, Final Draft March 3, 2005.

Does the program collaborate and coordinate effectively with related programs?

Explanation: CSREES programs & portfolios are highly integrated & coordinated both within the agency & externally. There is a high degree of coordination between related areas of science, education & extension. Coordination through CSREES NPLs with complementary programs in USDA, other federal departments, universities, state agencies, industry, & with producers & other stakeholders lead to adjustments in program management & resource allocations. NPLs & their counterparts from other federal agencies & institutions collaborate on the development of RFAs & guidance for Plan of Work, including priorities. NPLs within CSREES units & their counterparts in APHIS, ARS & the American Association of Veterinary Immunologists (AVVI) launched an initiative in the NRI Animal Protection Program to address the gap in availability of immunological reagents specific for veterinary species (see Animal Protection, Section IV-13).

Evidence: CSREES constituents, stakeholders & partners, along with others are routinely involved in program planning, dev. & review. NPLs collaborate with scientists & educators within/beyond the Department, & with advisory groups, industry, producers & citizens to maintain the relevancy of edu., research & extension to avoid redundancy with the programs of others. CSREES funded programs with NIH, NSF, DOE, other non-profit organizations, & numerous international sources to make genome sequences of food born pathogens available. Students/faculty exchange program with international institutions of higher edu. to facilitate the use of new tech. & collaboration in research. CSREES/FDA co-funded a microbial contamination of produce research project. Coordination occurs within USDA, with interagency work groups, & with other departments to add value & extend the benefits of CSREES efforts & of other programs.

Does the program use strong financial management practices?

Explanation: All grantees must comply with Federal financial assistance regulations, specific USDA & CSREES implementing regulations. Audits, reviews & mandatory reporting, with active post-award management, ensure that funds are spent for agreed upon purposes. Grant recipients are required to report expenditures each quarter 45 days after close of the quarter via PSC-272 Federal Cash Transactions Report, via DHHS-PMS. Draw down requests are denied for failure to submit this report. If an institution fails to submit technical & financial reports by June 15th, the 4th quarter formula funds & subsequent formula allocations are withheld until all requirements have been satisfied. The program has had a clean audit opinion for the last three years.

Evidence: USDA received an unqualified audit opinion for its FY 2004 Financial Statements & certified their financial statements by November 15, 2004, the deadline imposed by OMB. CSREES has no material weakness. Financial management is conducted as per OMB Circulars for Federal financial assistance, USDA regulations & the Administrative Manual, CRIS Instructions, CREEMS instructions, & Agency records indicating the receipt & approval of financial & technical reports. CSREES implemented the Federal Financial Information System after FY 2002. Prompt action is taken to correct any material weaknesses, if identified by auditors. Under the CSREES eGovernment Executive Board, the CREEMS Funds Management Module is one of the projects supported in terms of resources by the Board. In the last 2 years, ISTM has supported the funds management component of the grants process, helping greatly with reporting & business processes & therefore avoiding material weakness with the CREEMS audit.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Deficiences that are identified are corrected through recommendations made by managers, such as adding electronic reviews and reducing grant review time. As opportunities are identified, committees are establihsed to make recommendations. In addition to past efforts that resulted in a clean audit opinion, CSREES has: 1) a number of improvement processes underway to upgrade its management efforts; 2) taken action to improve general administrative, managerial, & fiscal functions; post-award management of its entire grants sys., including formula, line-item, & competitive grant mechanisms; & planning & accountability efforts; 3) set up an oversight staff within the Office of Extramural Programs to provide followup to OIG audits, OIG hotline complaints, GAO evaluations, IPIA & FMFIA compliance requirements, liaison with the Planning & Accountability staff on the POW, site visits, & analysis of trends in areas for which better internal controls & business processes need to be established. OEP works closely with other units in the agency to determine management deficiencies & dev. processes & reports to determine these deficiencies. New annual efficiency measures have been developed, with historical baselines & 2005 targets.

Evidence: Transition to e-Gov will result in paperless proposal & plan submission, review, approval & funding authorization to reduce processing time. Guidelines for review & approval of formula funding have been prepared. CSREES programs transition to electronic submission this year. A Program Administration Coordination Effort (PACE) addresses management, administrative, & fiscal issues to increase grant award effectiveness. PACE has completed Service Standards that set temporal guidelines for planning, awards management & time-critical administrative & management activities. A group is establishing guidelines for post-award management. Planning & Accountability provides guidance, analysis, policy recommendations for strategic & short-term planning, budget justification, performance measurement & evaluation, contributing to review & improvement of administrative functions & assisting NPLs assess accomplishments & determine changes to improve performance & efficiency.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The vast majority of grants are awarded using an extensive, rigorous competitive review process that includes a panel manager, qualified reviewers, strict guidelines & safeguards to avoid conflict of interest. Research formula funds (proposed for a two year phase out in the FY 2006 budget), research, special or administrative grants (including earmarks) are subject to qualified expert assessment of merit prior to award of funding. In all cases, release of funds can be and is deferred pending submission and approval of an acceptable and merit review proposal of work against which performance is assessed. CSREES works with new and first-time recipients to ensure they understand Federal Financial Assistance regulations and polices and to ensure the proposed project is sound, relevant and meritorious. NPLs monitor awards to ensure progress on the approved proposals. To ensure success and fulfill requirements, CSREES can place additional controls on new grant recipients.

Evidence: In accordance with to statutes (c.f., Agricultural Research, Extension, & Education Reform Act of 1998) or other legislation & official policies, proposals & plans of work are subject to either peer review or rigorous merit assessment. The agency maintains an extensive data base of highly qualified grant proposal reviewers from government, academia, & industry, & provides strict guidelines, training, & NPL & administrative oversight to the proposal review & award recommendation processes. All reviews both peer & merit, are held to the same strict standards of relevance, quality & performance as per the PMA.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The Agency's CRIS reporting system sufficiently monitors grantee activities. All grantees are required to submit proposals or plans of work fully describing education, research & extension work for which all types of funding (including mandated formula & special grants) are requested. Grantees are required to submit Annual Reports of Accomplishments describing their progress in achieving their stated goals for each funded activity. Reports, entered in the CRIS cris.csrees.usda.gov/, are available to the public via the Web. NPLs & Awards Management Branch provide review & oversight of all funded activities, including mandated projects. NPLs provide program leadership & oversight, including site visits; agency administrative & support staff assist in program management & support the review process for each program. Expert peer or merit review panels are used to provide assessment of the efforts of grantees.

Evidence: Oversight practices & techniques provide CSREES with explicit knowledge of grantee activities down to the funded project level. The Administrative Manual contains explicit monitoring & reporting requirements. Grantee institutions submit annual Progress Reports; a report is required for each funded project included in the institution's approved work plan for the Federal Fiscal Year. A Termination Report must be submitted for each completed or terminated project. Reports must be submitted in conjunction with progress reports on active projects & include a summary of outputs & accomplishments over the life of the project. CSREES is implementing an enhanced reporting system to supplement CRIS to include extension work (www.csrees.usda.gov/business/other_links). The agency conducts site visits to a substantial number of grantees on a regular basis, & conducts annual audits of grantee performance. NPLs maintain close working relationships with grantees & their institutions.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: CSREES annually collects, compiles & disseminates grantee performance data down to the funded project level, & makes data publicly available in a transparent manner via the CRIS Web Site at www.cris.csrees.usda.gov. Grantees are contractually required to submit an Annual Report of Accomplishments, & Termination Reports, to describe their performance in achieving their stated goals, & to describe their progress on funded projects are entered into the CRIS database.

Evidence: The widely used CRIS is a comprehensive repository of performance data for funded projects from CSREES, & for work of related agencies with similar missions & activities. These are identified by problem area, subject of investigation, & field of science, & include proposal summaries, annual performance reports, outcomes & impacts, & termination reports. This information is searchable, available to the public, & has the capacity to produce various types of reports according to customer needs. See www.csrees.usda.gov/business/other_links/egov/csrees/cris.html. Another CSREES innovation, the Competitive Research, Education, & Extension Management System (C-REEMS), makes comprehensive information on proposals, their review & award management available. See www.csrees.usda.gov/business/other_links/egov/csrees/creems.html

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: Expert review panels have not discerned qualitative differences between formula funded & competitive funded research, & have indicated that all projects contribute to the goals of CSREES, regardless of funding mechanism. To ensure quality for non-competitive grants programs partners are required to submit annual plans of work & proposals subject to merit/peer review. At least 1 NPL & 2 outside reviewers review & approve the proposal prior to award. The same rigor that is used to review proposals for competitive programs is also used for non-competitive programs to ensure relevance, quality & performance towards solving national problems. CSREES ensures that non-competitive grant proposals submitted include all required administrative & programmatic elements. Each year CSREES publishes guidelines for NPLs on how to request proposals & review & approve these projects for funding. [Note: Of the total funding provided through goal 3 programs, 24% is provided through formula grants, 28% through competitive grants, 42% through earmarks and six percent through other grants.]

Evidence: Financial management is explicitly conducted as per OMB Circulars for Federal financial assistance, USDA regulations & the Administrative Manual, CRIS Instructions, C-REEMS instructions, & Agency records indicating receipt and approval of financial & technical reports. Portfolio review including formula & special grants assessment outcomes from panels provide quantitative & qualitative assessments that focus & guide future efforts. Research projects supported with formula funds are reported in CRIS as per Administrative Manuals for Hatch, Evans-Allen, McIntire-Stennis. For some extension formula programs there are proprietary database reporting systems. Institutions are required to have a peer review conducted for the Special Research Grants & a Merit review for Federal administration grants & to certify that they were done for both the research & extension grants. CSREES Customer Service Satisfaction Standards.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Two of the three long term measures have met their targets. For the other, all Goal 3 component portfolios have made clear and consistent progress in achieving its LT performance goals over the past 5 years per the new PREP review. Even though the portfolio has only been scored once under the new system and therefore there is no basis for comparing the PREP scores over time, the panel noted significant progress was made in; developing contamination reducing methods and increasing the ability to detect and diagnose high consequence pest and disease threats. New technologies, such as serotyping or emerging microarrays, have been used to connect several food-borne outbreaks to a common risk factor. New technologies have provided evidence that fluoroquinolone resistant Campylobacter in poultry can remain in the environment despite stoppage of the use of antibiotics. Research has been used in HACCP implementation to identify risk factors for increased contamination, and the potential for new detection methods or intervention strategies. [WC:974]

Evidence: The overall long term performance measure established for Strategic Goal 3 is an aggregate 5-yr. Portfolio Review Score. The goal was 80 for 2005. Each of the Goal 3 portfolios was rated exceeds expectations by independent external review panels & the actual aggregate score was 86.This is the highest received by CSREES to date. See CSREES, FY 2006 Explanatory Notes. Centers for Disease Control & Prevention show a 20% decline in bacterial foodborne illnesses. www.cdc.gov/washington/testimony/fs092000.htm. 4 major bacterial foodborne illnesses: Campylobacter, Salmonella, Listeria, E. coli O157, posted a 21% decline in the past 6 yrs. Campylobacter infections dropped 27%, infections from Listeria fell 35%, & Salmonella infections decreased 15%. E. coli O157 infections dropped 21%. Declines show sig. progress toward meeting Healthy People 2010 objectives for reducing the incidence of disease caused by these bacterial infections. (www.cce.cornell.edu/food/fsarchives/050602/incidence.html)

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program received a small extent becuase it only met its two efficiency measure targets. The third, which is the portfolio review, and over time will affect its long term porttfolio review measure will not be measured against its target until 2006. The external portfolio review is the 1st annual review and the results of the external reviews are being used as the baseline for internal reviews in years 2-5 of a 5 year cycle. Because this is the 1st year of the external review, no internal review has been conducted. The first internal review for portfolios under Goal 3 will be conducted in FY06. Nevertheless the Agency management has initiated portfolio modifications in response to the recommendations of the external review panel, resulting in program improvements

Evidence: Were an internal review to be conducted this fiscal year it would indicate that the program managers had made concrete portfolio adjustments in response to the recommendations of the FY05 review and result in an increase in the portfolio score. For example, the current draft of the NRI request for applications to be released in Oct. 05 will request additional research in food safety. In particular the Agency will seek to support a food safety comprehensive agricultural project aimed at targeted development of new technologies to address emerging disease threats.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Agency estimates that at the end of FY05 that it will meet or exceed the FY05 targets for its efficiency performance measures. The efficiency and cost-effectiveness of the Agency's programs are increasing through the reduction in time-to-completion of proposal solicitation, review and funding, and by minimizing peer vetting costs per proposal reviewed. Larger and longer term awards are reducing the relative cost of processing, days, and cost to grantees per award. Transition to full electronic solicitation, submission and processing of proposals has been initiated and will continue into 2006. This major change in the Agency's business process is anticipated to yield considerable efficiency and cost-effectiveness.

Evidence: Estimates at the end of the third quarter of FY05 indicate that the Agency has made improvements in both the reduction in time from submission to receipt of grant award and the associated cost. Efficiency benchmarks were reconfigured in 2004 to accurately reflect significant changes: increased airline transportation costs, a 25% increase in honoraria in order to remain competitive with other research agencies in attracting the best qualified peer reviewers, and increased use of more effective but more costly on-site, rather than ad hoc, reviewers. Even with these adjustments, CSREES cost per proposal reviewed is extremely efficient vs. comparable NSF costs of $650 per proposal. The continuing trend toward more focused, larger grant awards increases administrative efficiency. CSREES honoraria are low compared to other federal agencies with whom CSREES must compete for highly qualified reviewers.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The answer received a "small extent" because no study was provided to show a direct comparison between this programand other similar ones. However, Nevertheless the Agency has solicited panel members from Universities, Agencies, and the private sector entities with similar purposes and goals to conduct independent external assessments of the Agency portfolios. The high score from their independent and expert assessment as measured by the R&D criteria of relevance, quality and performance, is testimony to the strength of the CSREES programs relative to programs with similar goals.

Evidence: The Portfolio Review Expert Panels, comprised of entities with similar goals, reiterated their satisfaction with the relevance, quality, and performance of the programs. Individual portfolio scores were: Foodborne Illness, 83; Animal Protection, 95; and Plant Protection, 80. They emphasized the high quality of portfolio leadership. While they recommended a few refinements, the panels judged the portfolio to be of very high quality and assigned them high scores. One can infer that because the reviewers come from organizations with similar goals and purposes, their high scores imply that the Agency is performing well in comparison to the organizations from which they come.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Three Goal 3 portfolios were independently assessed by outside experts & recommendations help NPLs & partners. Research affects productivity & public health with a time lag; benefit streams typically extend beyond 30 yrs. Studies found rates of return of 40% to 60% for public R&D regardless of level of aggregation or geographical area. Extension reduces time lags; its major effect on productivity is in the short to intermediate period, although major technology adoption throughout the sector is a LT accomplishment due to the dynamic & constantly changing technology & management methods available. Number of citations reflects high quality research, as scientists most often cite important, seminal work in a field. CSREES funded research is used by others to conduct further work to develop vaccines, manufacture dairy equipment & identify disease syndromes. Farm Foundation evaluation noted land-grant universities receive high marks from Congressional aides for reliability & reputation.

Evidence: The ag-food sector operates at high efficiency supported by knowledge, techniques & tech. dev. at land grant universities. SeeNAS: NRI: Vital Competitive Grants. Ohio Ag. Research & Dev. Center, partly funded by CSREES, managed/controlled ag. pests; prevented, diagnosed & treated animal infectious diseases; controlled crop losses by pathogens. Research evaluated users/non-users of extension dealing with diseases; CES rapidly alerts farmers on pest, disease, pathogen identification & prevention for animals & plants www.joe.org/joe/2004april/ent.shtml#a8. Economists estimate rates of return to extension tech. transfer & mgmt./behavior change from 20% to >100% (40% to 60% for public R&D; 20% to >100% for Extension tech. transfer & mgmt./behavior change. The International Food Policy Research Institute's meta analysis of rate of return to ag. R&D reviewed 292 studies & showed the median Rate of Return at 48% for research, 63% for extension & 44% for all studies.