Detailed Information on the
Grants for Economic Opportunities and Quality of Life for Rural America Assessment

Improving the focus of grant recipient reporting on outcomes.

Increasing planning and coordination with the Agricultural Research Service regarding the collection of stakeholder input.

Continuing to emphasize the use of competitive, peer reviewed grants, and proposing that no new funding be provided for unrequested add-ons (earmarks).

Improving the efficiency of the grants review process by using "Grants.gov" (a web based peer review system), as well as virtual panels when appropriate.

Ensuring that all interested parties have the necessary access to grant information, as well as to continue to emphasize grant capacity building as appropriate.

Measure: Portfolio Review Score - Goal 2

Explanation:Improvements in the portfolio review score does not in itself ensure improvements in the economic opportunities and the quality of life in rural America. There are intervening factors that are beyond the control of the research. Therefore, the use of an output measure in this instance is appropriate. Goal performance is externally assessed on a 5 year basis to determine progress toward solving targeted national problems related to each Strategic Objective. The score reflects expert assessment on 3 PMA dimensions of relevance, quality and performance, along a continuum from exceeds expectations, to meets expectations, to needs improvement. The reported score is a mean of quantitative scores for the Goal's objectives.

Measure: Percentage of Cooperative Extension Educators trained and using evidence- Based programming in rural communities to facilitate informed decisions that increase economic opportunities and improve quality of life.

Explanation:Improvements in the delivery of extension does not in itself ensure improvements in the economic opportunities and the quality of life in rural America. There are intervening factors that are beyond the scope of extension. Therefore, the use of an output measure in this instance is appropriate. CSREES builds the capacity of intermediaries, such as Extension Educators, to bring evidence-based programs to communities, families, and individuals. The appropriate measure for this work is educator/intermediary behavior change. This measure indicates an increase in the percentage of Cooperative Extension Educators trained and using national or regional multi-state evidence-based programs and activities (e.g. 4-H Youth Development; Family Strengthening; Community Development; Health Education; Housing & Indoor Environments; Resource Management; SARE-Professional Development) to enable rural people and communities to improve economic opportunity and quality of life.

Measure: Cumulative Dollars Saved for Grant Review.

Explanation:Dollars saved reflects the average salary saved by calculating the number of calendar days saved annually between receipt of proposal and date funding awarded for competitively reviewed proposals, then multiplied by the average daily salary for CSREES employees.

Measure: Proposal Review Time in Days

Explanation:Average number of calendar days between receipt of proposal and funding date funing awarded for competitively reviewed proposals. Average is calculated by summing the total number of days for all major competitively reviewed proposals and dividing by their total number.

Measure: Internal Portfolio Review Score

Explanation:Goal performance is internally assessed annually by NPLs to determine progress toward solving targeted national problems related to the Objective. The score reflects expert assessment on 3 PMA dimensions of relevance, quality and performance, along a continuum from exceeds expectations, to meets expectations, to needs improvement. The reported score is a mean of quantitative scores for the Goal's objectives. The baseleine of 82 is the Portfolio Review Score as determined by the external panel.

Measure: Benefits to farmers changing their risk management behavior per the net dollar cost of the Risk Management Education program.

Explanation:The measure indicates the Risk Management Education Programs effectiveness in convincing farmers to adopt insurance and marketing practices designed to increase their profitability and reduce the variability of their income. Notes: (a) The actual values given below were calculated for extension efforts in Minnesota, Wisconsin, Iowa and North Dakota. (b) Dividing the financial benefits by the program dollar is designed to control for changes in funding. (c) The indicator may still be affected to some extent by weather and financial markets.

Measure: The number of farmers and ranchers that gained an economic, environmental Or quality-of-life benefit from a change in practice learned by participating in a SARE project.

Explanation:This measure assesses the SARE program's progress toward helping farmers and ranchers improve their knowledge of sustainable agriculture production and marketing practices that ultimately leads to improved profitability, environmental stewardship and quality of life. Notes: (a) As part of the program's requirements, this measure is calculated from periodic program impact studies. Similar statistics, however, can be estimated every year.

Is the program purpose clear?

Explanation: The purpose of the program is to advance knowledge that will result in increased economic opportunities & improved quality of life in rural America by supporting extramural research, education, & extension programs in the Land-Grant University System & other grantees. CSREES helps fund activities at the state and local level & provides program leadership through two coordinated research, education & extension portfolios which ultimately result in improved decision-making by governments, farms, businesses, households, & individuals. Both portfolios provide scientific insights, technology, products, & information that facilitate informed decision-making. The first portfolio focuses on decision-making at the institutional level: in firms, farms, & businesses; community organizations; local governments; other governing bodies; planning commissions; civic leaders. The second portfolio focuses at the individual & household level: on consumers, individuals, families & youth.

Evidence: The CSREES Strategic Plan identifies key policy & mgmt objectives integrated within budget priorities & accountable through annual performance plans http://www.csrees.usda.gov/about/offices/pdfs/strat_plan_04_09.pdf. Work of the agency is authorized by a number of laws which specify the purpose including Hatch Act, McIntire-Stennis Cooperative Forestry, Smith-Lever, Evans-Allen, Animal Health & Disease Section 1433; Smith-Lever Extension 3(d) Extension & other special grants; the National Research Initiative (NRI) Competitive Grants, Section 2(b), Act of 1965, 7 U.S.C. 450i (b); Special Research Grants, Section 2(c), Act of 1965, 7 U.S.C. 450i (c); Sustainable Ag. Research & Edu. Program (SARE); & other legislation including those supporting minority serving institutions & programs http://www.csrees.usda.gov/about/pdfs/funding_mechanisms.pdf.

Does the program address a specific and existing problem, interest, or need?

Explanation: Fifty-five million Americans live in rural areas. Most rural communities are declining economically & in quality of life. Population loss creates a downward spiral - loss of revenue, tax base, services, & vocational, social, recreational & other opportunities. This program increases seven "Community Capitals" (built, cultural, financial, human, natural, political, & social). Economically enhancing these "Community Capitals" is a priority of the Administration & Congress. The program improves the capacity of rural communities to understand & address the economic, demographic, & environmental forces affecting their communities, using research, education, extension & integrated projects such as the 4-H Youth Development Program. Program efforts directed to improving rural communities address human development & societal change, health & the environment, individual, family & community resources, technology & information delivery, & sustainable local economies.

Evidence: Causes of decline vary by region. In many rural areas, mechanization & technological changes over decades eliminated jobs on farms & ranches, in mines & timbering - without offsetting rural development. See http://www.wkkf.org/Pubs/Federal_Spending_for_Rural_00376_03977.pdf. Over 300 non-metro counties went from a population gain in 1990-95 to population loss in 1995-2000. New definitions & population data released in June 2003 data show growth in rural & small town Ame was less than 1/5 of the annual ave metro rate. Non-metro poverty rate remains higher than metro poverty rate. http://www.ers.usda.gov/briefing/Population. This portfolio improves Community Capitals by addressing industrial & ag restructuring, globalization, application of risk mgmt tools, immigration, resource mgmt, population shifts & amenity development. Leadership & human development in 4-H, family life, & other programs build human capital. See also http://csrees.usda.gov/newsroom/white_papers/cultivating.doc.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: CSREES's active coordination avoids redundancy & permits the development of multiple complementary solutions to such national problems as limited economic opportunities in rural America & the lack of science-based technology, products, & information which can negatively affect informed decision-making & quality of life. CSREES's work is integrated with other USDA agencies (ARS, ERS, RMA, FSA) & Federal agencies (NSF, HHS, CDC); States & the private sector to provide inclusive, non-duplicative, coverage through cross-agency committees & collaborations. Additional coordination devices include the Regional Rural Development Centers, Risk Management Education Centers & the Sustainable Agricultural Research & Education (SARE) Councils. Through these interactions CSREES ensures that its support for extramural research/extension/education complements in-house research and the programs of "action" agencies (e.g. USDA-RD), and leverages our unique partnership with land-grant universities.

Evidence: NPLs participate in review panels for competitive programs, federal interagency task forces & working groups, & stakeholder listening sessions to ensure programs are not redundant or duplicative. NPLs attend professional & scientific meetings to track trends/better coordinate priority setting; work with federal agencies, partners, other providers & constituents to ensure programming is complementary & supplementary; & meet with counterparts to get input for Requests for Applications & on peer review panels. e.g., CSREES works with Coalition of Organization on Disaster Edu. (CODE) www.disastereducation.org, composed of over 20 federal agencies & national not-for-profit organizations on disaster preparedness and in the Federal Interagency Working Group on Child Abuse & Neglect nccanch.acf.hhs.gov/topics/prevention/organizations/federal.cfm, Team-Up: Cancer Prevention Saves Lives www.cervicalcancercampaign.org, & other efforts. In each our unique role is to leverage our LGU partnership.[

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: This question received a "no" because approximately 30% of the funding is provided through Congressional earmarks. Earmarks do not allow full competition, and is therefore is a program deficiency. Despite that, regardless of the source of the funds, portfolios integrate research and extension in a comprehensive, non-duplicative package. The high standards of relevance, quality & performance specified in the PMA are achieved through independent peer/merit review by external expert panels and by NPLs, who coordinate peer reviews and perform oversight and post-award management. All proposals regardless of funding line are reviewed and scrutinized against criteria of excellence. Formula funds and special grants present challenges, but NPLs work closely with recipients to ensure that they meet the Administration's national priorities and the same relevance & quality standards as other funding lines. Funding may be deferred pending correction of deficiencies. CSREES guides formula funds to support administration priorities by reviewing plans of work and Hatch project proposals and by interacting with university leaders through groups such as Multistate Projects (which compete for regional funds) and the Partnership Working Group.

Evidence: CSREES has guidelines for reviewing all proposals. e.g., 'CSREES National Program Leader Guidelines for Reviewing Hatch, McIntire-Stennis, Evans-Allen, & Animal Health & Disease Proposals' (See also http://www.csrees.usda.gov/business/reviews.html). CSREES also has operational guidelines such as Ag. Risk Mgmt. Edu. Competitive Grants Operational Guide for the Regional RME Centers & Operational Guide for Trade Adjustment Assistance for Farmers & Plans of Work (POW) such as National Family and Consumer Sciences POW http://www.oznet.ksu.edu/fcs/fcs_csrees/index.htm. These resources help prevent flaws that would limit effectiveness/efficiency. See also http://www.montana.edu/wwwcxair/notebook.html ; http://cyfernet.ces.ncsu.edu/cyfres/browse_2.php?search=evaluation http://ase.tufts.edu/4hstudy_pyd/news.htm. See http://nimss.umd.edu/ for Multistate Projects and http://www.colostate.edu/Orgs/WAAESD/Partnerships/Partnerships.html for the Partnership Working Group.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Agency resources are targeted based on administrative priorities, the external REE Advisory Committee, stakeholder input & needs identified in numerous reports. They are the recipients of the expertise and knowledge transferred to them by intermediary community leaders, university researchers, and extension educators. This non-linear researcher-educator-extension agent-citizen transfer process involves a multiplier effect in which one intermediary can have a profound effect on many citizens. Objectives are met by having CSREES ensure a supply of researchers and educators as well as oversee the communication of research results to rural communities. Program logic models for most programs and for state plans of work specify intended beneficiaries and desired outcomes. Proposals for extension and integrated research/extension projects must include extension/outreach plans which review panels evaluate, and which grantees then are held accountable to achieve.[

Evidence: Portfolio NPLs coordinated with other Federal and USDA agencies to jointly target rural and underserved populations. See for example, Whitepaper, Cultivating Resilient Communities and Rural Prosperity in a Dynamic and Uncertain and Uncertain Environment, http://www.csrees.usda.gov/newsroom/white_papers/cultivating.doc For examples of how projects reach beneficiaries, see Rural Development research awards, http://www.csrees.usda.gov/newsroom/releases/2005releases/rural_dev_05.html, the "Results and Impacts" sections of the program pages on CSREES's web site (www.csrees.usda.gov/nea/emphasis_area.html) and the Science and Education Impacts on the CSREES web site (www.csrees.usda.gov/newsroom/impacts/impacts.html).

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Four specific quantifiable long-term outcome measures assess the full portfolio. 1) Expert review scores track performance of 100% of the portfolio. 2) The share of educators trained & using evidence-based programming in youth development, strengthening families, community development, health, & consumer behavior assesses intermediary outcomes as CSREES brings research-based programs to communities, families, & individuals to improve rural quality of life. 3) The number of farmers & ranchers gaining benefits from SARE shows behavioral changes to improve profitability & environmental stewardship. 4) Benefits to farmers in the Risk Management Education program show effectiveness in changing practices, leading to increased profitability. All outcome measures have baselines and targets that increase through 2010.

Evidence: The five long-term performance measures included directly support the program's purpose and reflect a variety of appropriate evaluation strategies. Every five years, an expert panel conducts an extensive review of all program activities, interim evaluations, and reported outcomes. These experts also make recommendations for portfolio improvement. In addition, CSREES tracks the training of educators and the programs they are offering, as an interim outcome measure addressing the extent to which CSREES educators meet their own goals and effectively train others. The other long-term measures specifically capture outcomes of programs targeted to farmers, ranchers, and agriculture students. Taken together, these measures collectively capture the full range of portfolio activities and programs and assist CSREES in being accountable for improving programming to improve economic opportunities and quality of life in rural America.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has ambitious long-term targets and timeframes for its existing long-term measures. Five outcome measures were set in 2005. These measures align with the portfolios and are sufficiently focused toward increasing economic opportunities and improving quality of life in rural America. The portfolio review score target is ambitious (set at a level to ensure continued improvement). Incremental change is increasingly difficult as the score rises, and the entire portfolio must improve to achieve a score increase. The other 4 outcome targets are ambitious because they all set an expectation of continuously increasing behavior change (by Extension educators, students, farmers), whereas human behavior is notoriously difficult to change.

Evidence: The agency targets for the existing long-term measures are: (1) a Portfolio Review Score of 87 in 2011 from a baseline of 82 in 2006; (2) 85% of Cooperative Extension Educators to be trained and using evidence-based programming in rural communities to facilitate informed decisions that increase economic opportunities and improve quality of life in 2010, from a baseline of 75% in 2005; (3) 12,500 farmers and ranchers that gained an economic, environmental or quality of life benefit from a change in practice learned by participating in a SARE project in 2010, from a baseline of 8,100 in 2004; (4) $322 in financial impact to farmers per net program dollar in 2010; 5) the gap between total available jobs and qualified agriculture graduates in 2010 narrowed to 4.5% from a baseline of 5.2% in 2005. See PART Measures tab.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: CSREES has three annual performance measures to track progress toward long-term goals. Annual internal assessment of the Portfolio is used to track progress toward achieving the target external Portfolio Review Score. Each year, NPLs review & self-score the activities & results of the Portfolio regarding relevance, quality & performance, which includes collecting data to assess progress toward the long-term target. The NPLs report on steps taken to address recommendations made by the external review panel. Thus, if the annual targets are met, the long-term target will be met as the final annual increment in the progression. The other two long-term outcome measures have annual targets that reflect steady progress toward the long-term target. Improvements in the portfolio scores should also improve the effectivesss and useability of research and extension products. The two efficiency measures have been developed to measure the avg cost/proposal reviewed & the avg days to review a proposal & award funding. Achieving these measures contributes to long-term outcomes by helping to keep CSREES' award processes efficient.

Evidence: Baselines and targets through 2010 have been established for all measures and data have been collected for 2005 and are beginning to be collected for 2006. The external panel that assigned the Portfolio Review Score provided recommendations that are used by NPLs in their portfolio management to make ongoing improvements to the portfolio, which will be reviewed annually in the self-assessment. Thus, the three annual measures (the internal portfolio review score; the average cost for proposal reviewed; and the average number of days to review a proposal and award funding) collectively ensure that portfolio management makes changes as needed for accountability, to improve programming, and to meet the long-term targets.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has baselines and ambitious targets for its annual measures. Following the completion of the USDA & CSREES Strategic Plans, portfolios of work were established for each of five Strategic Goals, with annual measures for performance & efficiency. In 2004, baselines were established for the year 2004. Ambitious targets have been established to continuously improve program performance in terms of reducing the costs & time spent reviewing proposals. The targets are ambitious because incremental improvement is increasingly difficult to maintain or increase as a measure approaches 100% (for the Portfolio self-assessment and efficiency scores). In addition, the increase in the annual cost of the review of a proposal is kept at or below the rate of inflation, while the length of time to review a proposal falls by15 percent duirng a six year period (FY 2004 to FY 2010).

Evidence: Annual performance scores for each objective have ambitious baseline targets. Annual performance targets call for a cumulative percentage point improvement. Internal reviews will be conducted in 2007. The target score is 83. In 2004 baselines for efficiency measures of cost-per-proposal-reviewed & mean time to review proposals/release funds were $530 for cost per proposal & 220 days average review time per proposal. This reflects transportation, lodging & other variables beyond the control of the Agency. For example, honoraria costs that must be paid to remain competitive in attracting highly qualified proposal reviewers are rising. Thus increased costs as a result of using more effective but costlier on-site reviews are anticipated. In spite of this, annual targets call for holding review & processing costs to a real increase of 2% in dollars & an average 2% reduction in review time. In 2005 review cost per proposal was $535 & time to review 216 days. See PART Measures tab.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: CSREES enters into education, research, and/or extension partnership only when a mutual benefit is clearly demonstrated and it is consistent with and relevant to CSREES' mission, Strategic Plan, long-term & annual goals. All submitted proposals and plans of work specify the scope of work, milestones, measures, outputs & outcomes, along with the performance monitoring procedures. Submission of proposals and acceptance of awarded funds requires a commitment by grantees and partners to work to accomplish annual & long-term goals, progress on which must be reported to CSREES on an annual basis.

Evidence: CSREES reviews over 5,000 proposals & funds over 2,000 proposals, cooperative agreements & other transactions yearly. NPLs work with peers to develop RFA guidance to explicitly define long term & annual goals. All proposals & Plans of Work (POW) regardless of funding authority must include measures relevant to CSREES long term & annual performance goals & targets. NPLs receive annual progress reports which they use to review & monitor performance of transactions to assure continued commitment of partners. Funds are withheld or deferred if the commitment is not maintained. For grants, cooperative agreements, & some formula funds, these progress reports are collected in the CRIS reporting system (cris.csrees.usda.gov). See Award Terms & Conditions http://www.csrees.usda.gov/business/awards/awardterms.html. Hatch, Smith-Lever, Evans-Allen, and 1890 Extension funds must report on progress against their POW annually and these reports must be approved before funds are released.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: All portfolios are independently assessed & evaluated by external panels of experts every 5 yrs. & compared to a 2005 baseline score. Outcomes are also assessed internally on an annual basis. Other external reviews, evaluation studies, analyses of CSREES programs, & agency's self-review documents are presented as evidence to the review panels. Portfolios are rated on OMB's criteria of relevance, quality and performance. Beyond a score, both of these assessments result in useful comments & recommendations to help NPLs & implementing partners generate meaningful outcomes over the long term. To ensure quality panel members are scrutinized for any bias, conflicting interests, or other issues related to independence. The panels address the full scope of CSREES research, education, & extension work down to the level of Knowledge Areas and contributing programs. Having external panels meet on a 5-year basis is common practice in research and allows sufficient time for advances to take place.

Evidence: The results of expert portfolio review evaluations are documented on the web (see http://www.csrees.usda.gov/business/reporting/portfolios.html for reports & process description). All of the R&D Portfolios received indep review scores. These portfolios received a score of 82 which exceeds expectations . To ensure quality, the "Guidebook for Preparation for Portfolio Review Expert Panel (PREP)" was developed. CSREES conducts other indep evaluations. The Economics & Social Sciences Research Center, Washington State University sponsored by the Western Center for Risk Mgmt. Edu. is conducting an evaluation on behalf of the entire Trade Adjustment Assistance Program for Farmers to establish baseline data through which annual & long-term targets will be refined. See wsare.usu.edu/pro/pdf/FRGrantSurvey05.pdf, www.sare.org/reporting/report_viewer.asp?pn=LNC01-185&ry=2005&rf=1, and wsare.usu.edu/pub/pdf/PDSurveyRep04.pdf for independent evaluations of SARE.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget & Performance Integration (BPI) directly links long range and annual goals as defined in the annual Departmental Performance Plan, the REE Quarterly R&D performance reports and three other Strategic Plans submitted to the Under Secretary. Explicit knowledge areas defining the program are exclusive and non-duplicative, thus directly linking budget lines with goals, objectives, the program and related projects in a transparent manner. CSREES works with USDA's Office of Budget and Program Administration (OBPA), which provides direction and administration of USDA's budgetary functions including development, presentation, and administration of the budget and analyzes program and resource issues and alternatives. The CSREES 2004-2009 Strategic Plan identifies key policy and management objectives integrated within CSREES budget priorities and, at the same time, provides accountability through a series of annual performance plans. http://www.csrees.usda.gov/about/strat_plan.html In addition, the Budget Explanatory Notes (Congressional justification) includes both diret and indirect funding for agency program goals.

Evidence: Annual & long-term performance goals & accomplishments were an explicit component of the FY2006 President's Budget Request (see Agency budget requests, Explanatory Notes, tables, exhibits, & accompanying information). The program aligns specific projects by knowledge area and funding line with the objectives of the CSREES & USDA strategic plans. Using CRIS now and the State POW System beginning sometime this year, NPLs can track the impacts of projects by knowledge area & by funding source. http://cris.csrees.usda.gov/star/system.html. The new electronic State POW System released this year as part of CSREES OneSolution data & reporting system will continue to strengthen transparent links between agency budget & performance measures. Planning & Accountability is working to strengthen links between budget justification & budget & performance integration submissions. http://www.csrees.usda.gov/about/offices/planning.html. See also http://www.csrees.usda.gov/business/reporting.html.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Ongoing strategic planning efforts were undertaken to fully align the CSREES Strategic Plan with the REE and the USDA Strategic Plans. Comprehensive Knowledge Area codes developed for all agency work align CSREES higher education, research, and extension portfolios with the USDA strategic objectives. Budget Performance Integration (BPI) ties the Agency s budget request preparation, justification, and reporting to needs, strategic plans, and performance measures. State POWs require 5-year strategic planning efforts including extensive stakeholder engagement, as directed by AREERA. POWs link funding to projected outcomes, and indicators used to measure progress. These plans are revised annually, creating rolling POWs always extending five years into the future. NPLs reviews of LGUs are timed to coincide with their strategic planning efforts. http://www.csrees.usda.gov/business/other_links/prog_reviews.html

Evidence: CSREES is linking all portfolios between Congressional authorizations and appropriations, the Strategic Plans, the Departmental and Agency mission, the Departmental Performance Plan, and specific areas of science & education. All three of the strategic plans are posted together on the agency web site at www.csrees.usda.gov/about/strat_plan.html. CSREES State POWs now being implemented as part of the OneSolution data management & reporting system to comprehensively link strategic planning, Budget and Performance Integration (BPI) as well as output, outcome & impact reporting. Training on strategic planning and evaluation was offered to partners and to NPLs in 2005 to help them in their strategic planning, logic model development, and choice of indicators. Training information is also posted on the web. http://www.csrees.usda.gov/business/reporting/planrept/training_fy0711.html

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: Stakeholders are involved in planning, development & review. NPLs consult with academic & industry scientists, USDA advisory groups (Nat'l Ag. REE Advisory Board; Council for Ag. Research, Extension & Teaching; Council for Ag. Sci. & Tech.) & industry to assure CSREES projects complement programs in other institutions. Coordination with USDA, interagency work groups, other departments add value and extend benefits of programs with similar goals. Portfolio reviews enlist outside experts to examine CSREES programs and benefits and monitor progress toward solving national problems. Their recommendations are then used to increase benefits of new efforts. Outside independent reviewers review projects applying for competitive funding, and rank projects high priority when they are of high quality & are expected to yield the most value. In the process, they compare the benefits of alternative proposals. External & internal analyses usually agree; if not, NPLs address external concerns before funding.

Evidence: CSREES designs programs in collaboration with other agencies (e.g., ARS, ERS, CDC, DOE, DOD, HHS, NIH, NSF) to complement & supplement work not covered in their agency mission areas. CSREES leaders confer quarterly with REE agencies (see REE Quarterly R&D Perf. Mgmt. Plan & Quarterly Reports). CSREES' mission to support research, education, & extension is an asset in adding value to research efforts of other federal organizations as well as providers in other sections. Overall portfolio reviewers found the programs compared well in terms of benefits with similar programs. The institutional decision-making portfolio was found to have very high significance of findings; satisfactory alignment with the current state of science; & high productivity by the panel. The households & families portfolio was found to have satisfactory significance of findings, high alignment with the current state of science; & high productivity by the panel. (See 2006 Self-Review Panel Reports).

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The Agency develops annual budget and programmatic priorities based on existing & emerging needs, recent successes in research, the state of science, educational capacity, and direct input from partners, constituents & other providers. CSREES secures inputs from stakeholders through both formal and informal processes. Examples include the Committee on Agricultural Safety and Health (NCR-197), Family Economics Research Coordinating Committee (NCR-52), the Extension Committee on Organization and Policy (ECOP), and the 4-H "National Conversation". See Dodge, Kelsey & Pense (2001), "A Model for Gathering Stakeholder Input for Setting Research Priorities at the Land Grant University." The comprehensive BPI process makes the linkages and potential gaps & overlaps apparent as the budget request is being prepared. This allows for the refinement of the budget request, strengthened justifications, and appropriate program changes before the budget request is submitted.

Evidence: NPLs and peers collaborate on the development of Request For Applications (RFA), including the prioritization, the science & education, & the integration of portfolio work in the agency. Comprehensive NRI RFA Planning Worksheets are prepared for each priority issue, clearly defining mission-relevant problems to address in short, intermediate & long-term goals with stakeholders. Ag. Research, Education & Extension Reform Act (AREERA) requires recipients of formula funds to collect stakeholder inputs yearly. Additionally, CSREES holds workshops with constituents on a regular basis that are designed to help them prioritize their needs (e.g., the stakeholder workshop "Future Directions & Research Priorities for the USDA Biotechnology Risk Assessment Grants Program" June 9-10, 2003). This information, & Portfolio Review recommendations is integrated in the BPI process in collaboration with OBPA & OMB to guide Agency budget requests & funding decisions. Progran priorities are listed in RFAs, portfolio self-study reports, & other documents.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Recipients of competitive & line item grants must submit annual progress & final reports through the Current Research Information System (CRIS). NPLs use CRIS data to monitor progress, make recommendations, and make changes in priorities & focus & program admin. CSREES collects info. on progress of research formula awards through CRIS & collects info. on portfolio of ag. research & extension activities through the State POW reporting process, with NPL oversight. Extension work reports through POW & annual reports. The on-line State POW System for extension work will help toward better integrating performance data across education, research & extension. The OneSolution data & reporting system will further increase timeliness & quality of data/info., reduce reporting burden on key program partners, & increase program performance mgmt. OneSolution is estimated to save the Agency $20.2 mil. over 7 years with an estimated benefit cost ratio of 1.89.

Evidence: Terms & Conditions for competitive & line item awards specify annual & final report requirements. Reports determine progress of each grant & if additional funds should be provided. NPLs use information to compile reports, measure program impacts & respond to inquiries from management, public, & Congress. See www.cris.csrees.usda.gov. Research projects supported with formula funds are reported in CRIS as per Administrative Manuals for Hatch, Evans-Allen, McIntire-Stennis & Animal Health Research Programs. All 1862 & 1890 Institutions are required to submit 5-Year Plans of Work & Annual Reports of Accomplishments & Results (See Guidelines for State Plans of Work & Annual Report Guidelines). Reports measure institutional impacts in addressing critical issues identified through stakeholder input. See OneSolution Initiative, Business Case, Final Draft March 3, 2005; & CSREES Customer Service Satisfaction Standards.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: CSREES has 3 inter-related but indep. internal units (Program, Awards Mgmt. Branch (AMB) & Funds Mgmt. (FM)) that oversee the approval, admin. oversight, & funds accountability. NPLs oversee projects & interact with grantees to ensure project performance and program results. NPL's have specific program management performance standards that are included in their annual evaluations. AMB examines each project to ensure eligibility, admin. completeness, & authorizes release of funds when standards are met. CSREES is subject to Financial Assistance regulations & grantees are required to manage & expend funds according to the award Terms & Conditions (based on OMB circulars, statutory requirements, USDA & CSREES regulations). Award recipients are required to report grant expenditures via a PSC-272, Federal Cash Transactions Reports, on a quarterly basis, & SF-269, Financial Status Reports, to submit progress & final technical reports, & to undergo annual audits per OMB Circular A-133.

Evidence: Budget Office develops Control Tables with NPL assignments for the fiscal year. The Office of Extramural Program (OEP) develops guidelines & training to assist NPLs in solicitation, review, & award of grants (e.g., RFAs, Award Terms & Conditions, Guidelines for State POWs). Funds Mgmt. (FM) releases grant funds as authorized by the Awards Mgmt Branch (AMB) & POW staff after appropriate & satisfactory programmatic review is complete & admin. requirements met. NPLs, AMB, & FM follow up with awardees to ensure required financial & technical reports are received & satisfactory progress is being made according to the award proposal or POW. 2005 performance standards for NPL, AMB, & FM staffs had cascading standards referring to CSREES strategic goals or the President s Mgmt. Agenda. Each yr the NPL & OEP staffs develop a Schedule of Awards with target dates to manage workload/evaluate Agency performance. See http://www.csrees.usda.gov/business/business.html for Grant guidelines & regs.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Funds are (unless No-Year or Two-Year) required to be obligated in the year appropriated which is monitored through C-REEMS & FFIS reports. Annual reports, some of which are included in the Explanatory Notes, are prepared by OEP staff & reviewed for accuracy. Each year the Agency drafts a Schedule of Awards outlining target dates, peer panel dates, & award dates. OEP staff & NPLs are held accountable for these dates. Audits (e.g., A-133 & OIG), reviews (e.g., outreach & administrative) & mandatory reporting by partners (i.e., technical & financial), along with active post-award management, ensure that funds are spent for the intended purposes. OEP has developed an internal Formula Distribution Handbook for Staff to follow. CSREES has developed a new POW Reporting System for formula funds for CSREES administrative & program managers with set standards & requirements Award recipients must expend funds to achieve only funded objectives and according to Award Terms and Conditions

Evidence: CSREES prepares a schedule of obligations, monitors unobligated balances, compares obligations with scheduled estimates, & takes appropriate action. The dispersal of obligated funds can be deferred if deficiencies/deviations are detected &/or administrative requirements are not met. Awardees are required to take proactive corrective action in order to secure the release of funds. Agency personnel periodically review the Status of Funds Report & other management reports to monitor the availability of grant funds. Post-award oversight conducted by CSREES includes follow-up for any outstanding A-133 audits & Financial Status Reports, (required upon termination of award & annually for formula funds). Staff perform outreach & administrative reviews as well as Improper Payment Information Act (IPIA) transaction tests. Awardees are required to spend funds only on funded objectives, follow budgetary guidelines & reporting requirements, & fully expend all funds within a finite period.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: CSREES has an internal grants management system, CREEMS, that assists management & staff by providing budget, expenditures & statistical data related to the administration of grants (e.g., time to process a proposal from receipt to award; time to process a post-award action). The system serves as a management tool for tracking action items & administrative compliance requirements. The system is CSREES' innovation in integrating all grant processes (programmatic, administrative, & fiscal) in one database. Annual efficiency measures & targets compare proposal review time & costs for continual improvement. The OneSolution data & reporting system procedures will increase the timeliness & quality of performance related data & information, reduce the reporting burden on key program partners, & increase program performance management. It is estimated to save CSREES $20.2 mil. over 7 years, with a benefit cost ratio of 1.89. The program has two annual efficiency measures, with baseliens and targets. They are the cost and the time associated with the review of a grant proposal.

Evidence: See the Annual Efficiency Measures tab. For CREEMS documentation, see www.csrees.usda.gov/business/other_links/egov/csrees/creems.html. The Agency has a comprehensive IT Strategic Plan to increase the use of information technology to increase cost effectiveness & time efficiency, see www.csrees.usda.gov/about/offices/pdfs/strategic_plan_03-08_Final.pdf. See also CSREES Customer Service Satisfaction Standards. The Agency also has a specific Information Technology Strategic Plan, revised July 2004, www.csrees.usda.gov/about/offices/pdfs/strategic_plan.pdf; for OneSolution documentation & analysis of projected benefits, see OneSolution Initiative, Business Case, Final Draft March 3, 2005. The PACE and eCRU Committees provide forums for program and administrative staffs to discuss Agency performance standards, processes to attain standards, and standard evaluation.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The portfolios are highly integrated & coordinated within the agency & externally. CSREES constituents, stakeholders, and partners are routinely involved in scientific research, education & extension program planning, development, and review. Ongoing coordination occurs within USDA, with federal interagency working groups, and with other partners. NPLs collaborate with scientists and educators within/beyond the Department and with industry, producers and citizens to maintain the relevancy of programs, to avoid redundancy with other programs, and to add value and extend benefits of CSREES and other agency program efforts. NPLs and their counterparts also collaborate on the development of RFAs & guidance for Plan of Work, including priority areas.

Evidence: CSREES coordinates work with DOD, DOE, EPA, NSF, NIH, HHS, FSA, other federal agencies & non-profit organizations. e.g., Risk Mgmt. Edu. Centers work with the Farm Service Agency & other agencies to help farmers improve profitability with analytical risk management tools. CSREES NPLs & their counterpart staff in the defense sector (DOD, Army, Air Force) develop programs to help meet the needs of youth & military families. State 4-H Military Liaisons work with the Army & National 4-H Headquarters to develop a coordinated response to the special needs of newly military children & of Army National Guard & Army Reserve Soldiers. After Hurricane Katrina, the CSREES Extension Disaster Edu. Network coordinated with DOES Building America Program to extend technical & financial support to housing programs as the states begin to rebuild much needed housing. CSREES coordinated with America Saves on the High School Financial Planning Program??, including processes for gauging productivity.

Does the program use strong financial management practices?

Explanation: All grantees must comply with Federal assistance & implementing regulations. Audits, reviews & mandatory reporting, with active post-award mgmt., ensure that funds are spent for agreed upon purposes. Grant recipients are required to report expenditures each quarter 45 days after close of the quarter via PSC-272 Federal Cash Transactions Report, through DHHS-PMS. Draw down requests are denied for failure to submit this report. If an institution fails to submit technical & financial reports according to predetermined due dates, the formula funds are withheld until all requirements have been satisfied. Funds also are placed on hold for competitive awards until budget & admin. requirements are met. Each year selected transactions are reviewed by the USDA OIG (i.e., financial statement audit). CSREES is working with the USDA Office of the Chief Financial Officer on the A-123 requirements for which CSREES will have to certify that its internal controls are adequate & functioning.

Evidence: CSREES has received unqualified financial statement audit opinions & has not reported any material weaknesses since FY 2002. For the FY 2006 IPIA cycle, CSREES performed 6 site visits with no major findings. CSREES via the eGrants Exec Board continues to support improvements in the Funds Management Module of CREEMS (i.e., enhanced reporting & more automation). Program, administrative & financial management is conducted per OMB Circulars for Federal assistance, USDA regulations & the Administrative Manuals, CRIS & CREEMS instructions, & Agency records indicating the receipt & approval of financial & technical reports (i.e., receipt & review of financial & technical reports) for all 3 funding mechanisms: competitive, line-item, & formula fundsSee Award Terms & Conditions http://www.csrees.usda.gov/business/awards/awardterms.html Terms & Conditions and the Formula Funds web pages at http://www.csrees.usda.gov/business/awards/formula.html.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: CSREES has implemented steps to address accountability issues & initiatives through OEP oversight, which provides oversight to OIG audits & hotline complaints, GAO evaluations, IPIA & FMFIA compliance, A-123 requirements, & liaises with Planning & Accountability for POW. Concerns about the POW process are being addressed through a committee. The A-123 process is expected to result in improvements. As a result of the Customer Satisfaction Survey (CSS) (Spring 2005), 3 committees have been formed with state partners (i.e., administrators, applicants, & business officers) to follow up on the Survey. Draft recommendations were made in May 2006. A Final Report will be presented in July 2006. In relation to mgmt. of portfolios, a new deputy administrator was hired to fill a key vacancy in the Economic & Community Systems unit. CSREES staff participate in various intra-agency & interagency committees that address streamlining & improving business processes (e.g., PACE and eCRU Committees).

Evidence: Transition to e-Gov will result in paperless proposal & plan submission, review, approval & funding authorization to reduce processing time & improve accountability. CSREES plans to require electronic submission for all NRI and SBIR grants as well as provide the option for all programs in FY 2007. Guidelines for review & approval of formula funding have been implemented. The 3 CSS committees have been meeting & have drafted recommendations for the July 2006 report. PACE and eCRU committees continue to meet, discuss & implement initiatives to improve CSREES business processes & enhance program & fiscal accountabilitySee Ag. Risk Mgmt. Edu. Competitive Grants Program Operational Guide for the Regional RME Centers and Operational Guide for Trade Adjustment Assistance for Farmers

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: CSREES receiving the Plan of Work, & a second for each project seeking McIntire-Stennis or Hatch Act funding. The CRIS reporting system monitors recipient research & some extension activities, & the newly initiated electronic POW system will be used to oversee extension & other block/formula funded work. Recipients of funds are required annually report on progress in achieving their stated goals for each funded activity. NPLs provide program leadership & oversight of approved projects associated with these funds. Oversight of grantees is also made through OIG audits, OIG hotline complaints, GAO evaluations, IPIA & FMFIA compliance, A-123 requirements, review of financial status reports, A-133 audits, & agency site visits to verify program, grant, & financial compliance. Expert peer review panels are used periodically to assess grantees' efforts and link them to ongoing one & five-year portfolio reviews.

Evidence: In accordance with section 103(e) of AREERA a merit review process is required by every 1862, 1890, and 1994 land-grant institution to obtain formula funds under section 3(b)(1) and (2) of the Smith Lever Act, and formula funds for research programs under sections 1444 and 1445 of NARETPA & section 3(c) and (2) of the Hatch Act of 1987 (Regular Formula Funds). Furthermore, a scientific peer review process is required for programs funded under section 3(c)(3)of the Hatch Act (Multistate Research Funds). Site visits, audits & dept. program reviews are conducted by NPLs on an annual basis. More recently the new role of "State Liaison" has been created in which NPLs are assigned to coordinate the activities of cooperating institutions within one or two states. In addition, CSREES organizes an Admin. Officers' Conference & other Agency forums for admin. & programmatic mgmt. of CSREES grant funds http://www.csrees.usda.gov/business/reporting/planrept/additionalguidance105204.html.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: CSREES annually collects, compiles and disseminates for all funding sources performance data down to the funded project level, and currently makes data publicly available in a transparent manner via the CRIS and POW web sites (for CRIS see www.cris.csrees.usda.gov). Future One Solution web sites will also be available. Grantees are obligated to submit Annual Reports of Accomplishments and Performance, and Termination Reports to describe their performance in achieving their stated goals as well as to describe their progress on funded projects.

Evidence: Three systems contain performance information on formula funding projects: CRIS, the Research, Education, and Economics Information System (REEIS) and the Competitive Research, Education, & Extension Management System (C-REEMS). CRIS and REEIS are searchable and available to the public. CRIS is a repository of performance data for funded projects in research and education that identifies programs, projects and activities by knowledge area, subject of investigation, field of science, and includes proposal summaries, annual performance reports, termination and outcome & impacts reports (ww.csrees.usda.gov/business/other_links/egov/csrees/cris.html). REEIS contains performance information about some Extension programs (www.reeis.usda.gov). C-REEMS is used for comprehensive information on proposals, their review & award management. Multi-state project results may be reported on separate public web-pages. In the near future, these systems will become a part of the One Solution system.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: An extensive, rigorous competitive review process including a panel manager, qualified reviewers, strict guidelines & safeguards to avoid conflict of interest. In all cases, release of funds can be and is deferred pending submission and approval of an acceptable and merit review proposal of work against which performance is assessed. CSREES works with new and first-time recipients to ensure they understand Federal Financial Assistance regulations and policies and to ensure the proposed project is sound, relevant and meritorious. CSREES conducts outreach to encourage new grantees through presentations at professional societies and other stakeholder meetings, and through annual grantsmanship workshops (http://www.csrees.usda.gov/business/training/cpworkshops.html). Specific grant programs do additional outreach, including targeted outreach to minority-serving institutions (e.g. Southern SARE's Office of Minority Outreach). However, a significant portion of funding for these programs is provided through appropriations in a non-competitive means, such as through earmarks to specific projects and purposes and through formula grants.

Evidence: In accordance with to statutes (c.f., Agricultural Research, Extension, & Education Reform Act of 1998) or other legislation & official policies, proposals & plans of work are subject to either peer review or rigorous merit assessment. The agency maintains an extensive data base of highly qualified grant proposal reviewers from government, academia, & industry, & provides strict guidelines, training, & NPL & administrative oversight to the proposal review & award recommendation processes. All reviews both peer & merit, are held to the same strict standards of relevance, quality & performance as per the PMA. CSREES has a new Conflict of Interest and confidentiality form that has been approved by OMB (see Form 2007 - http://www.csrees.usda.gov/funding/forms_standard.html).This form and associated procedures should serve to avoid potential breaches of confidentiality and eliminate conflicts of interest.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The Agency comprehensively oversees grantee activities throughout the grant life-cycle. All competitive grant applicants are required to submit proposals fully describing the intended edu, research & extension work. This application is peer-reviewed for scientific merit and budgetary feasibility. If the application is approved a grant is awarded. If necessary, CSREES can place additional controls on new grant recipients. The activity's information is then loaded into the Current Research Information System (CRIS) (www.cris@csrees.usda.gov). Grantees must submit annual Progress/Accomplishments Reports for each funded activity, and a Termination Report upon its completion. NPLs provide program leadership & oversight of approved proposals through site visits, meetings with P.I.'s, P.I. seminars at CSREES, and participation in board/Council meetings of regional programs. Expert peer review panels are used periodically to assess grantees' efforts and link them to ongoing portfolio reviews.

Evidence: Oversight practices provide CSREES with knowledge of grantee activities at the funded project level. The Admin. Manual & agency regs contains explicit monitoring & reporting requirements. Grantee institutions must submit annual Progress Reports for funded grants & projects according to specific award terms & conditions. Termination Reports must be submitted for each completed project. Progress reports on active projects incl. a summary of accomplishments. CSREES is implementing an enhanced reporting system to supplement CRIS & include extension work (www.csrees.usda.gov/business/other_links). Site visits, audits & dept. program reviews are conducted by NPLs on an annual basis. CSREES organizes an Admin. Officers' Conference & other Agency forums for admin. & programmatic mgmt. of CSREES grant funds. CSREES makes site visits to grantee institutions to check on administrative procedures, and to the host institutions for regional programs (e.g. SARE) to check on their subaward management.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: CSREES annually collects, compiles and disseminates grantee performance data down to the funded project level and makes this data publicly available in a transparent manner via the CRIS Web Site at www.cris.csrees.usda.gov. Selected findings, results and impacts are also featured on the CSREES web site's Program Pages (see http://www.csrees.usda.gov/nea/emphasis_area.html) and Newsroom (http://www.csrees.usda.gov/newsroom/newsroom.html), including "Science and Education Impacts" developed by CSREES and its Land-Grant University partners. Grantees are obligated to report their performance in annual or even semi-annual Reports of Accomplishments or Progress and in a Termination Report. The Accomplishment/Progress reports provide periodic updates of performance. The Termination Reports are required upon completion of a project and describe the entire life of the project.

Evidence: The widely used CRIS is a comprehensive repository of performance data for funded projects from CSREES. These are identified by knowledge area, subject of investigation, & field of science. They include proposal summaries; annual performance reports; outcomes & impacts; & termination reports. This information is searchable, available to the public, & has the capacity to produce various types of reports according to customer need (www.csrees.usda.gov/business/other_links/egov/csrees/cris.html). Another CSREES innovation, the Competitive Research, Education, & Extension Management System (C-REEMS), makes comprehensive information on proposals, their review & award management available. This system is being combined with CRIS and others systems in the "One Solution" initiative designed to reduce reporting requirements (www.csrees.usda.gov/business/other_links/egov/csrees/creems.html)

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: To ensure quality for other than competitive grants programs, partners are required to submit annual plans of work & proposals subject to merit/peer review. At least one NPL & two outside reviewers review & approve the proposal prior to award. The same rigor that is used to review proposals for competitive programs is used to ensure relevance, quality & performance for all CSREES programs. CSREES ensures that all grant proposals submitted include all required administrative & programmatic elements. Each year CSREES publishes guidelines for NPLs on how to request proposals & review & approve projects for funding. Expert review panels have discerned no differences between competitive and other funded research, & have indicated that all projects contribute to the goals of CSREES, regardless of funding mechanism. While CSREES funds grantees through competitive grants programs, the agency also receives appropriations for formula funds & Congressional earmarks. These are held to the same high standards as competitive grants & are required to submit annual plans of work, proposals or accomplishment reports subject to merit/peer review to ensure relevance, quality & performance. CSREES also ensures that proposals include all required admin. & programmatic elements. Funds are withheld until all NPLs concerns are met. Each year CSREES publishes guidelines for NPLs on how to request proposals, review & approve projects for funding. Expert review panels have found no differences between competitive & other funded research, & have indicated all projects contribute to goals of CSREES, regardless of funding mechanism. CSREES funding allocation, review processes, & results are reviewed by the National Agricultural Research, Extension, Education, and Economics Advisory Board.

Evidence: Management of these programs is explicitly conducted per OMB Circulars for Federal assistance. USDA regs. & Administrative Manuals, CRIS Instructions, C-REEMS instructions. Agency records indicate receipt & approval of financial & technical reports. Quantitative & qualitative assessments of outcomes of formula & earmarks by panels during portfolio reviews provide focus & guide future efforts. Research projects supported with formula funds are reported in CRIS (See Admin. Manuals for Hatch, Evans-Allen, McIntire-Stennis). Institutions are required to conduct peer reviews for earmarks. Grantees are required to conduct merit reviews for both the research & extension grants. See CSREES' 'National Program Leader Guidelines for Reviewing Hatch, McIntire-Stennis, Evans-Allen, & Animal Health & Disease Proposals', CSREES Customer Service Satisfaction Standards, 'Operational Guide for Trade Adjustment Assistance for Farmers'.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Portfolios have demonstrated adequate progress in achieving their long term performance goals over the past 5 years. As of mid-FY 2006, all outcome measures are on track to achieve their targets, although it may be too early to determine that for the measure on the portfolio review score: They have either achieved a previously-set target or -- where quantitative measures are new -- the program goals are not new, and they are grounded in logical program models, where the activities and outputs of recent years' investments can be expected to lead to the program outcomes over the next five years reflected in the new quantitative measures. Furthermore, the programs have solicited feedback (ranging from stakeholder input to formal program evaluations) which provides a firm basis to improve program performance sufficient to achieve the performance targets. The program received a "large extent" because it could was too earlly to determine the extent to which the portfolio review score was on track to meet its FY 2011 target.

Evidence: The 5-yr Portfolio Review Score of 82 in 2006 assigned by the external panels is evidence of a highly successful program in 2000-2004. . Panel'recommendations provide a strong basis for CSREES to raise the score to 85 by 2011. In 2005, 75% of extension educators were trained & using evidence-based programming to facilitate informed decisions that increase economic opportunities & improve quality of life. As program investments of the past 5 years pay off, new quantitative measures are expected to capture incremental change sufficient to meet the performance targets. The measure of 10,000 farmers having benefitted from SARE programs by 2005 is based on impact studies that provide feedback to improve program delivery (e.g. http://wsare.usu.edu/pro/pdf/FRGrantSurvey05.pdf). The Risk Management Education target for 2005 was $200 in financial impact to farmers per net program dollar.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program achieved all of the 2005 annual performance targets. So far, for 2006, CSREES met its annual 2005 efficiency measure targets and the annual outcome measure target, the Portfolio Review Score (see 4.1 above). It is also on track to meet the annual outcome target for the Internal Review Score, which will be measured in 2007. Grantees are held accountable to achieve the results of their funded projects, and to provide evidence of those achievements in annual and final reports; NPLs draw upon these reports and achievements in documenting results and performance goals at the program and portfolio levels. In programs with regional structure (e.g. SARE, Rural Development Centers, Risk Management Education), regional and national program staff work together to develop performance goals and to conduct program evaluations to collect performance data. The program received a "large extent" since there there is no target aganst which to measure progress for the internal portfolio review score in FY 2006.

Evidence: It is clear from the independent external expert panel review's comments and scores that the portfolios, its programs, and partners are likely to achieve their annual performance goals. Annual progress toward long-term outcomes have achieved or exceeded annual targets through 2005, and are on track to achieve 2006 targets. The efficiency target for time per proposal processed, 216 days, was met in 2005 and looks like it will meet its target of 212 days in 2006. The other efficiency target met its 2006 target of $535 per proposal reviewed, in 2005, a year early. The first Internal Review Score will take place in 2007, and is projected to meet its goal of 86 at the end of 2007.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Agency has met or exceeded many of its targets. In particular, both efficiency targets were met in FY2005. The efficiency of the Agency s programs and processes are expected to continue to improve by reducing time-to-completion of proposal solicitation, review and funding, and minimizing peer vetting costs per proposal reviewed. Factors that are playing a role in this are larger and longer term awards which reduce the relative cost of processing, days, and cost to grantees per award. Transition to the full electronic solicitation, submission and processing of proposals, continuing into 2006, is also anticipated to contribute to increased efficiency and cost-effectiveness. The 2006 Expert Panel Review reports and scores for the Portfolios also indicate considerable progress in outcomes. The program met both of its efficiency targets in 2005.

Evidence: Estimates at the end of the third quarter of FY05 indicate that the Agency has reduced the time from submission to receipt of grant award and reduced associated costs. Efficiency benchmarks were reconfigured in 2004 to accurately reflect significant changes: increased airline transportation costs, an increase in honoraria to remain competitive with other research agencies in attracting the best qualified peer reviewers, and increased use of more effective but more costly on-site, rather than ad hoc, reviewers. Even with these adjustments, CSREES cost per proposal reviewed is extremely efficient vs. comparable NSF costs of $650 per proposal. CSREES honoraria are low compared to other federal agencies whom CSREES competes for highly qualified reviewers.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Although, independent peer review panel experts rated the performance of this program well, suggesting it compares favorably to other programs with similar purposes and goals in both the government and private sector, many of these program are unique because of CSREES' relationship with the universty system. Panelists noted that CSREES occupies a unique niche derived from the broad grass-roots network built over more than a century of partnerships with federal, state, and local agencies & that CSREES uses this network to deliver information, services, and programs to diverse rural and at-risk populations. An independent assessment of SARE cited "Interviewees regarded SARE as a unique model within USDA that reflected a flexible, innovative & risk taking image." The National Academy of Sciences highlighted SARE for its exemplary stakeholder participation in 2 reports & recognized the regional rural development centers as a "model for effective interuniversity collaborative research."

Evidence: The Portfolio Review Panels, which were comprised of experts familiar with other programs having similar goals, reiterated their satisfaction with the relevance, quality, and performance of the programs. That these CSREES programs compare favorably to those other programs can be inferred from the panel members' comments and scores. The portfolio review score was 82, as noted above. The independent assessment of SARE was by Leadership Development Services, Chicago, IL, final report date May 24, 2002; the National Academy of Sciences studies citing SARE and the regional rural development Centers are "Frontiers in Agricultural Research: Food, Health, Environment and Communities" (2002, p. 90-91, 100) and "Publicly Funded Agricultural Research and the Changing Structure of U.S. Agriculture (2002, p. 67), see www.nap.edu

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The 2 portfolios were independently assessed by outside experts in Jan 2006. The program was found to effectively use research and educational funding to improve the productivity & public health of communities with a time lag. Studies of similar programs found rates of return of 40% to 60% for public R&D regardless of level of aggregation or geographical area with benefit streams that can extend beyond 30 yrs. Extension benefits tend to have shorter time lags with benefits to productivity accruing in short & intermediate periods. Quality is judged by the # of citations from scientists in the field. Funded research is also used by others to replicate programs & disseminate information in other U.S. communities. Findings are used by industry professionals for decision-making, program administration and governance. The high score from their independent and expert assessment as measured by the R&D criteria of relevance, quality and performance, is testimony to the strength of the programs.

Evidence: In addition to the independent panel reviews referred to throughout, cooperative extension agents and others have often performed independent evaluations on specific programs. For example, current 4-H programming emphasizing positive youth development is informed by recent studies showing 4-H alumni learned and applied life skills (see Portfolio 2.2 - self-review document); a longitudinal study of youth in 4-H conducted by researchers at Tufts began data collection in 2002 (http://ase.tufts.edu/4hstudy_pyd/news.htm). More evaluation results and more effort at collecting and employing independent evaluations, information and data are expected in the future. See also http://www.montana.edu/wwwcxair/notebook.html and http://cyfernet.ces.ncsu.edu/cyfres/browse_2.php?search=evaluation