Detailed Information on the
Agriculture Marketing Service - Research and Promotion Programs Assessment

Clarifying long-term and annual measures to better demonstrate progress toward performance goals.

Demonstrating how program budgets clearly link to performance goals and resources.

Developing a cost efficiency measure.

Measure: Number of new products and production technologies developed for industry.

Explanation:This measure quantifies the number of successful research efforts relating to the industry. Success here will often increase production efficiency or reduce production costs.

Measure: Number of successful research projects for producers and consumers.

Explanation:This measure quantifies the number of successful research efforts relating to both the consumer market and the commodity industry. Success here will eventually be realized with strengthened consumer demand. Note: Dip in data between 2003 and 2004 is due to beef data being unavailable for 2004 and 2005. The beef checkoff no longer purchases the research study that includes this data on an annual basis. Therefore, we do not have figures for 2004. The annual number is usually not available until March or April of the next year, thus data for 2005 is unavailable.

Measure: Number of peer reviewed commodity board evaluations that show quantitative financial benefits

Explanation:This is the number of programs that have had a positive evaluation within last 5 yrs (starting with 2003 data). Positive evaluation is based upon the independent reviews the boards are required to authorize and fund at least every five years.

Measure: Number of new products or uses for existing products developed for the consumer market.

Explanation:This measure quantifies the number of successful research efforts relating to the consumer market. Success here will eventually be realized with strengthened consumer demand. Note: Dip in data between 2003 and 2004 is due to beef data being unavailable for 2004 and 2005. The beef checkoff no longer purchases the research study that includes this data on an annual basis. Therefore, we do not have figures for 2004. The annual number is usually not available until March or April of the next year, thus data for 2005 is unavailable.

Measure: Number of new products and production technologies developed for industry.

Explanation:This measure quantifies the number of successful research efforts relating to the industry. Success here will often increase production efficiency or reduce production costs.

Measure: Average number of days to approve program/budget.

Explanation:This measure shows AMS responsiveness to the boards and efficiency in their oversight role.

Measure: Dollars of board administrative expenditures to total program budget.

Explanation:This measure shows program administrative efficiency. Legal mandates dictate that this measure does not exceed 5% (or $0.05), and the programs are significantly below that rate. Maintaining this level given program cost-pressures is challenging.

Is the program purpose clear?

Explanation: The purpose of Research and Promotion (R&P) programs is to: 1). strengthen the position of agricultural commodity industries in the marketplace; 2). maintain and expand existing domestic and foreign markets; 3). develop new markets and uses for agricultural commodities; and 4). assist producers in meeting their conservation objectives through funding research projects and developing best management practices. R&P programs do this by providing an orderly means for developing, financing, and carrying out generic promotion, research, and education regarding agricultural commodities. AMS has oversight authority over the R&P programs and regularly reviews budgets, plans, and projects to ensure that they act in accordance with governing regulations.

Evidence: 7 U.S.C. 7401. Per the statute, Congress established the overall R&P program purpose by stating, "It is in the national public interest and vital to the welfare of the agricultural economy of the United States to maintain and expand existing markets and develop new markets and uses for agricultural commodities through industry-funded, Government-supervised, generic commodity promotion programs established under commodity promotion laws." R&P programs are authorized in two ways: thirteen are authorized through specific stand-alone legislation (cotton, dairy, eggs, beef, pork, soybeans, avocado, honey, mushroom, popcorn, potato, mango, and watermelon) and four are authorized under the Commodity Promotion, Research, and Information Act of 1996 (lamb, blueberries, peanuts, and avocado).

Does the program address a specific and existing problem, interest, or need?

Explanation: R&P programs are designed to facilitate collective action among producers; to maintain and expand markets; to increase domestic and international demand for various agricultural commodities; and to conduct research to improve nutrition, food safety, and production. The impact from collective funding and decision making on spending priorities is much greater than would be possible from the efforts of individual agricultural producers. Agricultural producers propose establishment of an R&P program, pay the assessments that fund these programs, and have the opportunity to express approval or disapproval through periodic referenda. By law, all R&P programs are required to hold referenda to determine if those effected by the programs wish to initiate, continue or terminate a program. The percentages of participants that are required to conduct a referendum vary from program to program with a range from 10 to 30 percent.

Evidence: The Commodity Promotion, Research and Information Act of 1996 states that "The production of agricultural commodities plays a significant role in the economy of the [U.S.]. Thousands of producers in the [U.S.] are involved in the production of agricultural commodities, and such commodities are consumed by millions of people throughout the [U.S.] and foreign countries. Agricultural commodities must be of high quality, readily available, handled properly, and marketed efficiently to ensure that consumers have an adequate supply. The maintenance and expansion of existing markets and the development of new markets for agricultural commodities through generic commodity promotion, research, and information programs are vital to the welfare of persons engaged in the production, marketing, and consumption of such commodities, as well as to the general economy of the [U.S.]."

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: States, industry organizations and other entities conduct local and regional activities in addition to the Federally-supervised R&P programs. These organizations also collect assessments and use those funds to address issues related to production, handling, new product development, nutrition and food safety. There is continuing coordination between the Federally-operated R&P programs and industry and State leadership to assure that the various programs are complementary.

Evidence: The purpose of the question is to determine whether the program is designed to fill a unique role or whether it instead duplicates or even competes with other Federal or non-federal programs. According to PART guidance, a "No" answer should be given when there is more than one program that addresses the same problem, interest, or need, regardless of the size or history of the respective programs. For programs that partially overlap with others, a "No" should be given when major aspects of the program, such as its purpose, targeted beneficiaries, or mechanisms, are duplicative.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Congress has declared as policy, that an orderly procedure for financing and carrying out coordinated national programs of promotion should be established to strengthen various agriculture commodity positions in the marketplace and to maintain and expand domestic and foreign market uses for respective commodities. There is no evidence of major design flaws that prevent the R&P programs from meeting their defined objectives and performance goals. There is also no strong evidence that another approach or mechanism would be more efficient or effective in achieving the intended purpose. The results of periodic independent reviews and program industry referenda have been positive. The U.S. Supreme Court ruled in May 2005, in Johanns, Secretary of Agriculture, et al v. Livestock Marketing Association et al (125 S. Ct. 2055) that, because the mandatory funding under the R&P programs pays for the Government's own speech, it does not violate industry First Amendment rights.

Evidence: R&P program performance is monitored through periodic reviews and annual, long-term, and efficiency metrics. Beginning in 2004, USDA began coordinating a review and evaluation process of each R&P program as stipulated in Section 610 of the Regulatory Flexibility Act (610 Reviews), 5 U.S.C. 601 et seq.). The purpose of the 610 Reviews is to determine: (1) the continued need for the authorizing statutes; (2) the nature of complaints or comments from the public concerning the statutes; (3) the complexity of the statutes; (4) the extent to which the rules of the statutes overlap, duplicate, or conflict with other Federal rules, or with State and local regulations; (5) the length of time since the statutes have been evaluated; and (6) the degree to which technology, economic conditions, or other factors have changed. Several R&P programs are evaluated each year and 610 Reviews have been completed for the Cotton, Fluid Milk, and Watermelon R&P Programs. Based on the 2004 reports, USDA determined that all three R&P programs reviewed should continue without change. Programs for which 610 Reviews will be conducted in the coming years include: Beef, Blueberries, Dairy, Eggs, Avocados, Honey, Mushrooms, Peanuts, Popcorn, Pork, Potatoes, Lamb, Mango, and Soybeans.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Federal R&P programs are directed by industry-governed boards appointed by the Secretary of Agriculture. These boards are required to exercise prudent management of assessment funds and to remain accountable to the producers they represent. The boards plan and execute programs directed toward various types of consumers??retail, foodservice, and food manufacturing, both domestically and internationally. Promotion programs employ various media to reach and influence customers.

Evidence: AMS provides Federal oversight of promotion board activities to: 1). review independent financial audits to assure adherence with USDA policies relating to the investment of public funds; 2). ensure internal fund controls meet accounting standards and that funds are used only for projects and other expenses authorized in budgets that were approved by AMS; and 3). ensure that funds are not used for lobbying purposes.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has recently developed two new long-term measures that will become operational beginning in 2006. The program's long-term measures are: 1) the number of peer reviewed commodity board evaluations that show quantitative financial benefit (reflecting an expanded demand for products); and 2) the number of successful research projects for producer and consumer markets. These measures reflect to the purposes of the R&P programs that are primarily designed to: 1). strengthen agricultural commodities position in the marketplace; 2). expand consumption in domestic and foreign markets; and 3). develop new markets and uses for agricultural commodities.

Evidence: These measures, described above, will become operational in 2006. The measures build on, and improve, the existing system of each R&P board to set its own long-term goals, with independent evaluations (at least every five years) to determine whether the goals are being achieved. Current individual boards' measures include increased product use and improved benefit-costs for producers. These measures will apply to all of the R&P program boards.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: R&P programs have developed baselines and targets for the measures described in response to Question 2.1, though currently only the targets in support of one of the long-term measures is acceptable. The targets associated with the program's measure - number of boards demonstrating quantitative financial benefit - increases each year toward the goal of demonstrating positive results for all boards. However, the targets associated with the program's measure - number of successful research projects - require further refinement and definition to be acceptable according to PART guidance. As currently submitted, it appears that the measure displays the number of copies of research reports that the program has distributed.

Evidence: The targets for the new long-term measures are based on past data.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program's annual measures include: 1) the number of new products or uses for existing products developed for the consumer market; and 2) the number of new products and production technologies developed for industry. These measures directly relate to the long-term measure of successful research projects for producers and consumers and contribute to the long term measure: number of peer reviewed commodity board evaluations that show quantitative financial benefit to the industry. R&P programs experience positive results when new products are developed for the consumer market with subsequent promotion of those new products resulting in increased consumer awareness. Getting the food service industry to use more of the commodity, or to serve new products, increases demand in two ways: (a) by increasing sales through food service outlets; and (b) by increasing consumer demand for commodities and products at home.

Evidence: These measures will become operational in 2006. The measures build on, and improve, the existing system of each R&P board to set its own long-term goals, with independent evaluations (at least every five years) to determine whether the goals are being achieved. Current individual boards' measures include increased product use and improved benefit-costs for producers. These measures will apply across all of the R&P program boards.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baselines and targets have been established for the program's new annual measures identified in the response to Question 2.3; however, these baselines and targets require further clarification in order to obtain credit as being ambitious and demonstrating continued progress towards a long-term goal as opposed to annual outputs.

Evidence: The targets for the program's annual measures demonstrate a commitment to strengthen the market through the introduction of new consumer and producer innovations. The targets for the new annual measures are based on past data: (1) increases in new products and product uses for the consumer market each year (this measure and the associated targets need further clarification); and (2) increasing new products or production technologies for the industry. The targets demonstrate a commitment to strengthen the market through the introduction of new consumer and producer innovations. As an example, the Soybean R&P program's long-range plan identified the need to aggressively market soy-diesel products (increasing soybean usage by 3.5 billion bushels by 2010). Partnering with the U.S. Navy, the Soybean R&P program expanded utilization of soybean products through worldwide establishment of alternative fuel manufacturing facilities. As a result, after 10 years and a $30 million investment, biodiesel is the fastest growing alternative fuel. Reference: www.biodiesel.org/resources/faqs/

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: AMS R&P program managers are in continuous dialogue with their industry partners who originate and administer the R&P programs. AMS R&P program managers accomplish this by approving industry board budgets and contracts, attending meetings, and otherwise maintaining contact. R&P programs work with scientific advisory committees, academia, and expert advisors to make recommendations for research, program direction, and long-term strategies.

Evidence: AMS has notified the boards that new performance measures are being developed and will work with them toward accomplishing the annual and long-term goals of the program. AMS Guidelines for AMS Oversight of Commodity Research and Promotion Programs (updated May 2004), Sections II, III, and IV (pp. 2-4).

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: As specified by authorizing legislation, independent economic analyses are conducted annually for the Dairy and Fluid Milk R&P Programs and reported to Congress. The Federal Agricultural Improvement and Reform Act of 1996 (FAIR Act) requires an independent evaluation of promotion program effectiveness for other programs every five years. Promotion boards also frequently use consumer panels, focus groups, and marketing consultants to evaluate individual advertisements, plans, and campaigns to determine target markets, assess specific promotions, and evaluate the success of the R&P activities.

Evidence: Periodic evaluations that provide details on the benefit of R&P programs are required and have been completed for 11 programs. (1) An Economic Analysis of the Cotton Research and Promotion Program, Final Report, September 2001. (2) An Economic Evaluation of Generic Egg Advertising by the American Egg Board (3) Report to Congress on the National Dairy Promotion and Research Program and the National Fluid Milk Processor Promotion Program, July 1, 2004; Chapter 3: Impact of Generic Fluid Milk and Dairy. (4) Beef Demand and Rate-of-Return to the U.S. Beef Check-off: Two Independent Evaluation Approaches, September 2004. (5) An Economic Evaluation of the Pork Check-off Program, January 2001. (6) United Soybean Board Check-off Program, Return on Investment 1995-2001, June 2003. (7) National Potato Promotion Board: Program Evaluation 1995-1999, March 30, 2001. (8) Generic Promotions of Watermelons: Measuring the Impact on Watermelon Demand. December 9, 2002. (9) The Hass Avocado Promotion and Research Order: Offsetting Price Impacts with Advertising and Promotion: Working Paper no. 04-006; October 2004. (10) Evaluation of Popcorn Board Activities Outcomes, June 9, 2003. (11) National Peanut Board: An Update on Evaluation and Results, 2004.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Currently, budget requests are not tied to the accomplishment of annual and long-term performance goals. In addition, the program does not report all direct and indirect cost in a manner that could inform budgetary decisions or program planning. AMS is in the process of defining the relationships between the new annual and long-term performance targets and partner budgets that must be approved by AMS. AMS intends to monitor the measures and targets as part of the approval process in 2007 in the context of the Guidelines for AMS Oversight of Commodity Research and Promotion Programs. Through the budget approval process, AMS will ensure a direct relation between funding and expected performance and the process will provide evidence that the requested performance/resource mix will enable each R&P program to attain its performance goals.

Evidence: PART guidance requires the following for a "Yes" answer: the program must have effective budgeting in place that defines the relationship between 1) annual and long-term performance targets and 2) budget resources.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Board members discuss the steps at meetings and on conference calls. All R&P budgets, plans, and projects as well as program changes must be reviewed and approved by USDA. USDA participates in the Boards' strategic planning sessions and board meetings and reviews budgets, plans, and projects.

Evidence: USDA, AMS, "Guidelines for AMS Oversight of Commodity Research and Promotion Programs," Sections I, II, and VIII to be revised in 2007. The guidelines currently state that USDA will approve all budgets. Budgets must include detailed information regarding administrative expenses and written justifications for contract costs as well as a statement of objectives and strategy in each major program area (research, advertising, etc.), including reasons for significant changes from the preceding budget period. Per AMS guidelines, the agency will not approve amendments or additions to approved budgets, including shifting of program funds from one major area to another unless those shifts are consistent with governing bylaws. AMS is responsible for determining if advertising comparisons or nutritional claims are in compliance with USDA and FDA guidelines.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: AMS reviews R&P program budgets and receives regular reports on expenditures and assessments collected. AMS management monitors the internal budget review process each year to ensure that staff is meeting their goal of providing timely review of program plans and budgets. AMS provides guidance to program boards on the allocation of funds among programs, and supports market-oriented research, nutrition research and education, foodservice promotion, and other programs that enhance the marketability of products. AMS also conducts periodic administrative reviews of national research and promotion board operations to ensure that adequate records are maintained (examples include travel expenses of board members and staff, contract compliance, and investment of funds).

Evidence: USDA, AMS, "Guidelines for AMS Oversight of Commodity Research and Promotion Programs," Sections I, IX (C), (E), (I), as revised May 2004. Under agency guidelines, AMS is responsible for verifying that the program boards submit an independent evaluation of the effectiveness of their promotion programs as required by the Federal Agriculture Improvement and Reform Act of 1996.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: AMS managers are held accountable for performance standards such as addressing communication, customer service, fiscal management, and meeting required timeframes which support the AMS Strategic Plan. Federal expenditures are regularly reviewed and reported within the agency during and at the end of each fiscal year. Federal managers are also accountable to the R&P programs that reimburse AMS for agency oversight costs. AMS provides an estimated budget for Federal oversight costs to the R&P programs each year, and the Federal managers agree to operate within these budgeted amounts unless an unforeseen expense occurs. R&P program members are accountable to their industry constituents to assure that the funds are spent in accordance with the mission and goals of these Congressionally-mandated programs.

Evidence: AMS Directive 2100.1, "Administrative Funds Control System for Appropriations and Fee Supported Funds" dated February 12, 1999. Quarterly financial status reviews are conducted by the AMS Budget Office to monitor expenses incurred by AMS throughout the year. AMS employees maintain accurate time and attendance sheets to ensure each program is properly billed. AMS annually reports its administrative expenses for R&P programs to Congress. External to the program, the Beef and Avocado R&P programs reviewed AMS' oversight charges and internal controls to determine the accuracy and appropriateness of expenditures. No significant instances were found of inappropriate expenditures or inaccurate accounting. In addition, OPM reviewed and approved the linkage between AMS Strategic Plan and AMS Performance Agreements for Senior Executives and other program managers for the October 1, 2005, to September 30, 2006, appraisal period. For example, the performance plan for AMS Deputy Administrators are linked to the Agency's Strategic Plan and will be for the newly developed performance measures.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: AMS obligates Federal oversight costs in the USDA accounting system in a manner consistent with the overall program plan; establishes schedules that properly correspond to the resource needs of the program plan; and assures adequate procedures for reporting actual expenditures and comparing them with intended use. Details in billing documents are sent to the individual boards for reimbursement. In addition, all projects and budgets are approved by USDA to assure that they effectuate the purpose and intent of the authorizing statutes, and R&P Programs monitor the expenditures of industry board funds in accordance with program approved budgets.

Evidence: (A) The Guidelines for AMS Oversight of Commodity Research and Promotion Programs (updated as of May 2004), Sections IV(A) and IX(C) require: (1) AMS to review financial statements for each accounting period for proper accountability of funds collected and expended; and (2) AMS to review annual financial audits performed by certified public accountants to assure that they include statements regarding whether or not (a) funds were used for lobbying, (b) the AMS investment policy was adhered to, (c) internal controls over funds met accounting standards, and (d) funds were used only for projects and other expenses authorized in a budget approved by USDA. (B) USDA, AMS Agreement No. 12-25-MU-371, "Memorandum of Understanding between the Agricultural Marketing Service and the Foreign Agricultural Service Regarding Oversight of Federal Commodity Research and Promotion Programs," dated January 24, 2003, identifies the shared and respective oversight responsibilities for each agency for those boards that receive funding from FAS to conduct international commodity promotion activities.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has two measures of efficiency: (1) average number of days to approve program plans/budgets, and 2) dollars of board administrative expenditures to total program budget. R&P programs use competitive bidding or best-value contracting whenever possible. In addition to the competitiveness of bids, factors such as reputation, ability to respond to change, and knowledge of the industry influence the selection of contractors. R&P programs regularly review contractor performance. Contractors that do not perform adequately can be terminated for substandard performance.

Evidence: The Guidelines for AMS Oversight of Commodity Research and Promotion Programs (updated as of May 2004), Section II (Budget Approval) details the requirements for approval of program plans/budgets.

Does the program collaborate and coordinate effectively with related programs?

Explanation: R&P programs regularly collaborate with industry organizations, academic institutions, research firms, and other Federal agencies to discuss strategic intent and projected needs. Their goal in combining funding and coordinating projects is to leverage promotion dollars through larger, more broad-based projects. R&P programs often conduct joint promotions with other commodity programs, trade associations, and private businesses. In addition, many R&P programs provide funding to State industry organizations to supplement national promotion and research efforts. AMS works closely with program managers and other agencies such as the Foreign Agricultural Service (FAS), which supports international promotional programs; the Farm Service Agency (FSA), which conducts referenda and polls; the USDA Office of General Counsel, which provides legal services; and Customs and Border Protection in the Department of Homeland Security, which collects assessments on imported commodities. Other agencies are reimbursed for their expenses on behalf of the R&P programs.

Evidence: USDA, AMS Agreement No. 12-25-MU-371, "Memorandum of Understanding between the Agricultural Marketing Service and the Foreign Agricultural Service Regarding Oversight of Federal Commodity Research and Promotion Programs," dated January 24, 2003. AMS collaborates with FSA to conduct nationwide R&P program polls and referenda. R&P programs have formed a coalition, known as the Commodity Roundtable, to coordinate their promotion efforts and exchange ideas about how to more effectively carry out research and promotion. The Pork and Soybean R&P programs have contracts for joint promotions. Examples of Federal-State program cooperation include the sharing of national program funds with State and regional promotion entities (Beef, Pork, Egg, Hass Avocado, Soybean, and Dairy). The Egg R&P program has developed national radio spots and billboards into which individual State promotion organizations may insert their own messages and taglines.

Does the program use strong financial management practices?

Explanation: AMS oversight of R&P programs has not been determined to be material for purposes of the USDA-wide audit. However, each R&P program is required to obtain an annual financial audit by certified public accountants to ensure: 1). that internal financial controls meet accounting standards, 2). that program funds are used only for projects and other expenses authorized in budgets approved by USDA, and 3). that no funds were used for lobbying. The most recent audit of each research and promotion program found that all programs received a clean audit opinion. USDA also requires R&P programs to conduct periodic contract and compliance reviews.

Evidence: USDA reviews and approves engagement letters that provide the parameters of financial audits to ensure that they are performed in accordance with USDA guidelines and the Government Auditing Standards issued by the Comptroller General of the United States. This can be seen in the Rhea & Ivy acceptance to the Cotton Board's letter of engagement for auditing services (March 2004). The Guidelines for AMS Oversight of Commodity Research and Promotion Programs (updated as of May 2004) govern audit reviews. Sections IV(A) and IX(C) require: (1) AMS to review financial statements for each accounting period for proper accountability of funds collected and expended; and (2) AMS to review annual financial audits performed by certified public accountants to assure that they include statements regarding whether or not (a) funds were used for lobbying, (b) the AMS investment policy was adhered to, (c) internal controls over funds met accounting standards, and (d) funds were used only for projects and other expenses authorized in a budget approved by USDA. Certain programs have additional auditing requirements. For example, the Soybean and Pork R&P Programs hire independent contractors to conduct audits of State programs which receive national funds. All R&P programs adhere to the AMS investment policy to safeguard assessment funds. Investment policy requires these funds to be invested in interest-bearing accounts, certificates of deposits of a bank that is a member of the Federal Reserve System, or in obligations full guaranteed as to principal and interest by the United States for maturity periods of 1-year or less. These investments must be 100 percent collateralized and cannot be invested in unsecured instruments. R&P programs generally have a qualified custodian of funds (e.g., Wells Fargo) to manage investments and ensure compliance with the agency's investment policy.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: When USDA, R&P programs, or an outside agency (e.g., OIG or GAO) identifies management deficiencies, USDA works with R&P programs boards to rectify the deficiencies. R&P programs have committees that monitor, evaluate, and identify management or contractor deficiencies. In addition, AMS periodically conducts management reviews of R&P programs to identify any management deficiencies or organizational weaknesses.

Evidence: Guidelines for AMS Oversight of Commodity Research and Promotion Programs (updated May 2004): Section VIII(C). AMS reviews of R&P program boards can identify management weaknesses that lead to corrective action. For example, during a 2002 management review, AMS staff noted that one of the boards did not always take advantage of its available tax exempt status. To correct the problem, AMS recommended that accounts at major office supply companies be established with such status. AMS also instructed the board on the importance of purchasing from vendors that would honor this status.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program has recently established several valid measures to monitor achievement of long term outcomes: 1) the number of successful research projects for producers and consumers (demonstrates expansion and creation of new markets); and 2) the number of peer-reviewed commodity board evaluations that show quantitative financial benefit. Current results for all measures indicate that the program is achieving its long-term outcomes. Specifically, the number of successful research projects for producers and consumers has been increasing, which shows a commitment to creating new and better products for the consumer and opening new markets for the producers. Additionally, the evaluations showing of a quantitative financial benefit has increased from 5 to 9 of the 17 boards.

Evidence: A "Small Extent" is provided to recognize the program's efforts to develop acceptable long-term measures. However, the Agency must clarify the targets that correspond to the goal of increasing the number of successful research projects for producer and consumers. It is not clear what the data represents, how it is collected, and how "success" is determined. Furthermore, the data does not differentiate between "success" for consumer and industry.

Does the program (including program partners) achieve its annual performance goals?

Explanation: R&P programs have developed new annual measures that are designed to link annual performance goals to long-term outcomes such as research results, market impacts, and promotion results. However, results of these measures are uncertain in part because the baselines and targets that correspond to the program's annual measures require further clarification. It is not entirely clear how the program is benchmarking its progress in reaching long-term goals. For example, it is unclear what is being measured by the measures: "Number of new products or uses for existing products developed for the consumer market" and "Number of new products and production technologies developed for industry". The Agency needs to provide additional information on how the data represents a new product or new technology.

Evidence: The new measures will be considered for inclusion in the new AMS Strategic Plan (scheduled for revision in 2006).

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: USDA monitors the average number of days to approve program plans/budgets and is currently achieving an average of 10 days or fewer. The dollars of board administrative expenditures to total program budgets has remained stable at $0.04 (or 4%) which is beneath the mandated 5% level.

Evidence: The program has two efficiency measures; however, only one appears to be acceptable according to PART guidance. The efficiency measure: "number of days to approve program plans and budgets" demonstrates the timeliness with which AMS staff review and approve budget submissions. However, the second measure submitted: "dollars of board administrative expenditures to total program budgets" is more of a cost containment measure than an efficiency measure. By law, AMS is required to keep administrative expenses under 5% of the overall program budget. As currently submitted, the AMS goal is 4% for the measurement period. It appears that this measure maintains the status quo because each year program budgets grow based on new entrants being assessed fees and administrative costs increase with adjustments for inflation and increased salary cost. To obtain a "Small Extent" the program should develop a dollar value efficiency measure.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Though there are research and promotion programs at regional, State, and local levels, R&P programs are the only nationwide generic commodity research and promotion activities. Agricultural industry supports the establishment and continuation of R&P programs to address ongoing industry needs. The Mango R&P Program became effective in 2004, after USDA received the industry petition, worked with the industry to develop the program, and conducted a referendum in which the proportion in favor was 77 percent. The Lamb R&P Program became effective in 2002 and USDA conducted a delayed referendum in 2005 that showed an overwhelming response, in which more than 80 percent supported continuation. R&P programs have received recognition for excellence in competition with private industry. In 2004, the Beef R&P Program funded an advertising campaign: "Beef. It's What's For Dinner." For two years in a row, the beef campaign was awarded an EFFIE, one of the advertising industry's most coveted and prestigious national awards. The award is presented annually by the New York American Marketing Association in recognition of the year's most effective advertising campaigns. Additionally, according to data from a research test conducted in April 2005 by the MSW Research Group, of Lake Success, New York, entitled "2005 3-A-Day Weight Loss Advertising Test - "Best Measure." (commissioned by the National Dairy Promotion and Research Board) dairy industry television advertising consistently scores higher than the television industry norm for food and beverages with regards to influence and purchase persuasion.

Evidence: R&P programs have been established by Congress through stand alone legislation as well as through the general authority to create R&P programs provided to USDA through the Commodity Research and Promotion Act of 1996. R&P programs are initially requested by industry representatives, implemented through Federal rulemaking, and voted on by the assessed industries for establishment and continuance.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Independent - and in some cases peer reviewed - evaluations of program effectiveness are legislatively mandated for all R&P programs. These evaluations have shown that each dollar of promotion expenditure returns several dollars of increased grower return, ranging from $2.50 to $6.75 in studies completed to date. In most cases, analyses are planned for 5-year intervals, with the next set of evaluations to be completed in 2006. An analysis of the Dairy R&P program is done yearly. The results of these studies, their thoroughness, and their underlying methodology vary from promotion board to promotion board because the program lacks a uniform or basic set of criteria with which to evaluate the evaluations.

Evidence: The independent studies cited are in the response to Question 2.6. Ten of the studies used econometric analysis and eight of those computed benefit cost ratios showing positive returns to the promotions funded from assessments paid by producers. Two other studies cited results in terms of increased agricultural product prices. R&P boards make use of impact analyses to improve their programs. Studies completed for two other Boards (peanuts and popcorn) had no overall numeric result, but instead focused on surveys of consumer attitudes toward particular products (e.g. peanut butter) and specific promotion outputs summarized by target audience and type of media. These showed a number of examples of effective promotion results. The program receives a "Large Extent" in response to this question because of the lack of uniformity among the evaluations and also because there is not evidence of independent evaluations of program effectiveness for the following R&P programs: honey, mushroom, lamb, blueberry, mango (initiated in 2004), and avocado.