Detailed Information on the
Pesticide Data Program Assessment

Evaluate the methodology used to establish program performance targets for long-term and annual measures.

Develop additional annual and long-term performance measures that demonstrate progress toward a long-term programmatic outcome.

Modify Web-based Remote Entry system to improve the effectiveness of data transmission and the program's ability to capture additional information by strengthening security and implementing user suggestions.

Develop procedures to improve program operations by analyzing and addressing issures identified through annual customer surveys conducted by the International Standards Organization (ISO) on each of the cooperating State labs for ISO 17025 accreditation.

Measure: Percentage of children's food commodities on which comprehensive pesticide residue data is available and used for dietary risk assessment.

Explanation:PDP analyzes and reports pesticide residue data on food commodities with a primary focus on those foods most frequently consumed by children. Prior to the implementation of PDP, a database of residues on children's foods was non-existent. This measure reports progress, expressed in percentage, in the development of a comprehensive database derived from actual sampling, on pesticide residue levels in the major foods consumed by infants and children. The percentage is calculated by dividing the cumulative number of children's commodities tested by PDP by the number of commodities that are considered high consumption items for children, based on USDA food consumption surveys. Accomplishment nearing 100 percent could only be achieved with additional resources.

Measure: Increase in comprehensive pesticide residue data available and used for dietary risk assessments.

Explanation:This measure reflects the program's accomplishment in generating the data identified as priority by EPA. EPA grouped chemicals according to similarities (classes) in their action, chemical composition and agricultural uses to assess the cumulative effect of these chemicals when present in foods. PDP tests food commodities for residues of these pesticides so that the data is available when needed for EPA dietary risk assessments based on a multi-year schedule. Dietary risk assessments are used by EPA to approve pesticide (re-)registrations and to set pesticide application tolerances. PDP measures the percentage of priority pesticide/commodity pairs the program has reported against the number of priority pesticide/commodity pairs on EPA's multi-year schedule for dietary risk assessment. A pesticide/commodity pair is a specific commodity analyzed for a specific pesticide compound.

Measure: Number of children's food commodities included in the Pesticide Data Program.

Explanation:This measure reports the program's progress each year in meeting the Food Quality Protection Act (FQPA) requirement to increase sampling of foods most likely consumed by infants and children. Each year the program tests various forms of the food commodities most frequently consumed by children, retests commodities to keep the data current, and adds new commodities as resources allow and as testing methodologies are developed. The total number of high consumption children's food commodities is 58, based on USDA food consumption survey data reported by ARS.

Measure: Number of compounds reported by PDP laboratories.

Explanation:PDP analyzes commodities for residues of agricultural compounds to meet FQPA requirements for dietary risk assessments. EPA identifies the priority compounds needed each year. This measure reflects the total number of chemical compounds reported by the program for commodities--including fresh and processed fruit and vegetables, grains, meat, dairy, and water samples. The actual number of chemical compounds analyzed each year is dependent on how complex the test for a compound is, and the technological capacity of the laboratory.

Measure: Program cost per test (in dollars).

Explanation:To arrive at the cost per test, PDP divides the total program operating costs by the number of tests (pesticide/commodity data points) conducted. The target is set at the maximum acceptable cost. The number of tests (data points) generated by the program varies with the type of chemical compound and complexity of the test, within the technical capacity of the program. Each data point is the result for a specific compound in a specific sample. Each data point is a unique input for tolerance reassessment calculations.

Is the program purpose clear?

Explanation: The purpose of the Pesticide Data Program (PDP) is to collect pesticide residue data on raw and processed agricultural commodities in support of the pesticide reregistration process required by the Food Quality Protection Act (FQPA).

Evidence: The authority for the PDP program is in title III, section 301 of the FQPA.

Does the program address a specific and existing problem, interest, or need?

Explanation: The PDP program is the main supplier of data to EPA that accurately reflects pesticide residues on raw and processed agricultural commodities. This data is used by EPA to implement the FQPA and make regulatory decisions that affect agricultural production. The data supplied through the PDP program is tied to FQPA implementation time tables through cooperative agreements with State Departments of Agriculture.

Evidence: PDP provided 87 percent of the data used for the cumulative risk assessment of organophosphates which led to regulatory decisions affecting 22 crop protectants. PDP is currently generating data needed for cumulative risk assessment of pyrethroids, conazoles/triazoles, and other pesticide classes.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: PDP is the only program in the United States that is specifically designed to produce dietary risk assessment data on a national level. The data is representative of over 50 percent of the U.S. population and all geographic regions of the country. The data reflect population exposure to pesticides through dietary intake.

Evidence: PDP is designed to allow national estimates from the data obtained. PDP covers major agricultural producers as well as underserved rural areas. For example, community water systems in the PDP water survey include small water treatment plants that are not being surveyed by any other federal program.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program is not free of design flaws. The program could strengthen its effectiveness and efficiency through charging a fee to industry beneficiaries for partial or full cost of operations. Guidance in OMB Circular A-25 advocates charging a fee to certain groups when a service is provided by the government to an identifiable group. Currently, the Environmental Protection Agency (EPA) partially offsets the cost of the pesticide registration and re-registration program through user fees.

Evidence: OMB Circular A-25, states that, "It is the objective of the United States Government to .... promote efficient allocation of the Nation's resources by establishing charges for special benefits provided to the recipient that are at least as great as costs to the Government of providing the special benefits ....".

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: PDP sampling plans, developed by National Agricultural Statistics Service (NASS) statisticians, are designed to gather pesticide residue samples from food that is available for consumption by the U.S. consumer. The States initially chosen for participation in the program are major producers of fruits and vegetables in the United States, and as such, they already had the sampling and testing infrastructure needed by PDP. In addition, PDP studies are focused on the chemicals and fruits and vegetables identified by the Environmental Protection Agency (EPA) as part of its needs to set pesticide tolerances.

Evidence: State vouchers are carefully reviewed to ensure that charges are for allowable items and are in agreement with forecasted expenditures. Any discrepancy is quickly addressed with state personnel and corrected, if necessary.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program's current long-term output measures are: 1) Percentage of commodities consumed by children for which pesticide residue data is collect for risk assessment; and 2) Increase in pesticide residue data available for dietary risk assessments. These measures reflect the program's goal of providing residue data needed to support implementation of FQPA, with an emphasis on foods most frequently consumed by children.

Evidence: The ultimate outcome of the PDP program is supplying data on pesticide residue levels found on raw and processed agricultural commodities to the Environmental Protection Agency (EPA) to assist in the pesticide re-registration process. The program needs to develop long-term outcome measures that demonstrate the use of PDP data by EPA in the pesticide re-registration process, not simply output measures that evaluate the volume and timeliness of data collected. The program receives credit for output measures as a proxy for outcome measures because effective and efficient data collection is an important component of the program. In order to obtain full credit elsewhere in the evaluation, the program is urged to develop measures that focus on long-term outcomes (such as the use of the data).

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: PDP plans data collection activities in accordance with EPA's established timetables and timeframes for pesticide tolerance reassessment to meet FQPA requirements. PDP works with EPA to establish schedules for data generation, but PDP data requirements are often modified due to findings uncovered during ongoing re-registration activities. The targets and timeframes for the current long-term output measures are partially affected by program's data requirements. For example, the data for fresh/raw commodities must be collected for a minimum of two consecutive years (processed commodities only require one year of data). In addition, PDP must retest high consumption foods every five years because residue profiles change significantly over time, making the data obsolete after five years.

Evidence: The targets associated with the program's long-term measure, "Percentage of children's food commodities on which comprehensive pesticide residue data is available and used for dietary risk assessment" do not appear to be sufficiently ambitious. The targets decrease sharply in later years as opposed to the benchmarks established earlier for the program. Since the FQPA was passed, PDP has delivered data for 76 percent of the commodities listed as high consumption foods by children. The program has generated 86 percent of the data on pesticide/commodity pairs for dietary risk assessment that were identified by EPA as a priority. To accomplish this, the program expanded laboratory operations to accommodate additional sampling requirements.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Annual performance goals are measured by the number of food commodities consumed by children that are included in the program and the number of compounds reported by PDP laboratories in a fiscal year. These annual performance measures are used as benchmarks towards meeting the program's long-term measures of delivering residue data for foods most often consumed by infants and children and providing residue data for dietary risk assessments as it is needed for each phase of the pesticide tolerance assessment schedule. While these annual and long-term measures are outputs, the program is being given credit for these measures as a proxy while outcome-based long-term and annual measures are being developed. Additional measures will consider the use of the data as opposed to the collection and reporting of the data.

Evidence: As required by the Food Quality Protection Act (FQPA), the program gives priority to commodities identified by USDA consumption surveys as food items frequently consumed by children (http://www.barc.usda.gov/bhnrc/foodsurvey/fcid.html). Sampling of these commodities is repeated every five years or sooner if EPA's regulatory activities have resulted in significant changes in pesticide applications.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: PART guidance states that in order to obtain a "Yes" all of the targets associated with the program's measures should be ambitious - that is they must be set at levels that ensure continued improvement and realization of efficiencies. While the program's targets are included in the program performance plan, the targets associated with the annual measure, "Number of compounds reported by PDP laboratories" does not appear to be sufficiently ambitious because the targets appear to be arbitrary.

Evidence: PDP strives to deliver 100 percent of the data analysis goals stipulated each year in the program's planned performance targets. When PDP receives urgent requests from elsewhere within USDA and EPA (rapid response data needs), the program performs a cost-benefit analysis to determine impact to the agricultural industry if urgent data needs are not met and adjusts the program's goals accordingly.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Program operations are carried out by state partners through cooperative agreements with State Departments of Agriculture. Terms of the agreements include execution of program planning, adherence to operating procedures and a requirement to achieve ISO 17025 accreditation. Planning meetings are held with partners (states, EPA) to discuss long-range planning and develop semiannual plans.

Evidence: Program plans and standard operating procedures (SOPs) are sent to state partners well in advance of planning cycle. Plans and SOPs are also posted on the program's web site. State partners are required to show evidence of progress towards achieving ISO 17025 accreditation for their operations, which will ensure acceptance of data by international organizations.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: A recent evaluation of the program was conducted by the USDA Inspector General's Office (OIG) in 2004 (Report 01099-28-At, May 2, 2005). The audit referenced was a one-time review of the program by USDA's OIG. Based upon the program's small size, and its role in providing EPA with pesticide residue data in support of the pesticide re-registration process, periodic audits are considered acceptable. The audit referenced above was requested by the agency and completed in response to the program's initial PART assessment.

Evidence: The 2004 OIG review said: "PDP has continued to improve program operations by (1) implementing additional SOPs for program operations to provide uniformity in performing administrative, sampling, and laboratory activities; (2) developing an electronic database to serve as a central comprehensive data repository and allow for more efficient administration and reporting of data; (3) performing internal reviews to ensure program compliance; (4) developing a quality assurance program to ensure the reliability of PDP data; and (5) hiring additional technical personnel to expand program operations and monitor program activities. States participating in the program have also helped to achieve PDP's objectives by adhering to terms of agreements, executing program plans, and following SOPs. Based on these improvements and the results of our review, we believe that the program is operating as intended."

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: PART guidance states that in order to obtain a "Yes", the program must provide a presentation that makes clear the impact of funding, policy, or legislative decisions on expected performance and explains why the requested performance/resource mix is appropriate. A program with budget planning that is not tied to performance or strategic planning would receive a "No".

Evidence: The program's budget justification materials do not tie the budget request to the program's annual performance measures and do not identify all direct and indirect program cost.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The purpose of the question is to determine whether the program is on track to correct any strategic planning deficiencies that have been identified. In particular this question evaluates whether the program is working to adopt a limited number of specific, ambitious long-term performance goals and a limited number of annual performance goals that relate to achieving the long-term goals. A "No" is provided in response to this question because the program needs to develop additional annual and long-term measures that evaluate the outcomes as opposed to the current output meansres.

Evidence: The program has two long-term output measures: 1) Percentage of commodities consumed by children for which pesticide residue data is collect for risk assessment; and 2) Increase in pesticide residue data available for dietary risk assessments.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Program performance is measured by collective ability of PDP participants to deliver data within deadlines. The program has a strong quality assurance program that covers all aspects of data gathering, from sample collection to data reporting. PDP participants guarantee data by adherence to these strict QA protocols.

Evidence: Sampling performance is measured by participant's ability to collect a specified number of samples that meets PDP criteria. Laboratory performance is measured by ability to meet quality assurance/quality control protocols. Laboratories' performance is also measured through "proficiency check samples" and validation studies. Workload assignments are given according to laboratory area of expertise and capacity. Laboratory backlog is closely monitored and if warranted samples are rerouted to another laboratory.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The Agricultural Marketing Service (AMS) has assigned management of the PDP program to the Science and Technology-Monitoring Programs Office (MPO). The MPO Director is held accountable for financial oversight, schedules and performance of PDP. Performance is monitored through GPRA. Performance of program partners such as State science laboratories and contractor staff are monitored by MPO staff through audits and the Quality Assurance program.

Evidence: The MPO Director is held accountable for overall financial oversight, schedules and performance of PDP. Program participants are required to select individuals to oversee administrative, technical and quality assurance operations at each of the PDP facilities. These requirements are described in standard operating procedures (SOP-ADMIN2 Personnel and Organization). Performance of contractors is monitored through ongoing reports from PDP facilities. In addition, to help ensure the proper use of program funds, MPO's Program Analyst and Program Director monitor various financial documents for adherence to the terms of cooperative agreements. Equipment purchases over $5,000 must have prior approval.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Funds are obligated to fulfill the statistical sampling plan and are managed through cooperative agreements when appropriations are received. Amounts allocated to the states are proportional to planned workload, staffing levels and operational costs. Program partners are required to submit a budget with estimates of costs. Deviation from estimated expenditures must be cleared by MPO.

Evidence: Program partners submit periodic billing of expenditures. MPO reviews these bills for conformance with the terms of cooperative agreements, forecasted expenditures (standard form 424) and program plans. All purchases of single items over $5,000 by State and Federal partners must be approved by MPO. State expenditures that exceed $25,000 or 10 percent (whichever is lower) of forecasted expenditures must be approved by MPO.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The agency has procedures in place to achieve efficiencies and reduce overall program cost. For example, the program utilizes State partners rather than private sector laboratories to maintain the impartiality of the data. However, when an activity cannot be accomplished by the PDP State partners due to either cost or availability private sector contractors are utilized. The program ensures that statistically representative, unbiased samples gathered by private sector entities are consistently collected, handled, and shipped in a manner to ensure the integrity of the data produced. All program objectives and programmatic quality assurance criteria must be maintained in contract awards.

Evidence: In 2002 PDP enlisted the services of water utilities staff and awarded this contract to a private vendor to manage water sample collection and provide supplies, resulting in savings of $450,000. Additional examples of activities outsourced include administering proficiency test samples and sample collection kits. Similarly, collection of pear concentrate samples at processing plants that could not be done by State partners in a cost effective manner was awarded to a private contractor. Additionally, the program has one efficiency measure which monitors the dollar cost per pesticide data sampling test performed.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program is a collaborative State-Federal effort. The EPA and all ten PDP states provide feedback in program planning. The program also works with the USDA Office of Pest Management Policy (OPMP) and the USDA National Agriculture Statistics Service to address agricultural industry's and government's data needs.

Evidence: EPA and OPMP review and concur on PDP program plans. EPA participates in PDP annual planning sessions with State partners. Long-range planning is developed with EPA, OPMP and other stakeholders (i.e., industry and consumer groups). EPA also provides technical support for laboratory operations (e.g., methods, pesticide reference materials, etc.). Records of planning sessions, conference calls, and interagency meetings are filed by MPO. State sample collectors and scientists are represented in Program Advisory Committees.

Does the program use strong financial management practices?

Explanation: PDP uses the Federal Financial Information System and internal budget spreadsheets to track budget transactions.

Evidence: Invoices from clients are reviewed before they are processed for payment. Billing discrepancies and corrections are promptly resolved. Contracts that fail to meet criteria for deliverables are terminated. State partners are required to notify MPO of changes in staffing levels if these affect budget expenditures. Also, all purchases over $5,000 per item require the approval of the program Director.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Reviewing data output provides the first indication of management deficiencies for the program because poor management quickly translates into errors and or delays reporting data. State managers are promptly notified if in the course of data review (financial or scientific), recurring errors are spotted. If the problem is not corrected, financial, sampling or laboratory operations are scheduled for an audit by qualified headquarters staff. In addition, each program participant undergoes periodic reviews/audits of program operations. Any management or technical deficiencies observed are reported to AMS Compliance Office. These deficiencies must then be corrected and the corrective action documented and communicated to headquarters.

Evidence: As reported in the 1994 OIG report, poor management caused errors in data reporting and backlogs when the problem was not quickly corrected. PDP staff review data submissions sent daily via remote data entry. Any discrepancies, errors and/or omissions are promptly reported to program partners. Reports of outcome of program reviews including recommendations for improvement or corrective actions are sent to the state laboratory and AMS Compliance Office and are filed in program QA archives. Follow-up onsite reviews track implementation of corrective actions.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program has two long-term output performance measures: 1) Percentage of commodities consumed by children for which pesticide residue data is collect for risk assessment; and 2) Increase in pesticide residue data available for dietary risk assessments. Each of these measures is designed to report the program's performance in implementing the USDA data requirements under FQPA. PDP must continually evaluate and reevaluate commodities that are highly consumed by the general population as well as those that are most frequently consumed by children. As tolerances are reassessed, the number of allowable uses for a chemical changes, and so do the residue profiles.

Evidence: The program's current long-term output measures track the amount of targeted residue data that the program provides to EPA for use with the pesticide re-registration process. This data is required under provisions of the FQPA. Per PART guidance, a "Small Extent" is provided because the program needs to develop long-term outcome measures that demonstrate how this data is used and what outcome results from the use of this data.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program's annual goals, which relate to the long-term measures, are output related. The annual performance goals track the number of food commodities consumed by children that are included in the program and the number of compounds reported by PDP laboratories in a fiscal year. These annual performance measures are used as benchmarks towards meeting the program's long-term measures of delivering residue data for foods most often consumed by infants and children and providing residue data for use in assessing pesticide tolerance levels. The program designs its annual goals with the active participation of all partners during the formulation of commodity testing plans. These testing plans are designed to set achievable timetables for the testing, collection and analysis of pesticide residue data. The program should develop outcome measures that quantify the use of the data gathered and analyzed by the program rather than just the task of gathering data.

Evidence: Summaries of PDP data are published annually and posted on the Web site (www.ams.usda.gov/science/pdp/). A "Small Extent" is provided in response to a lack of annual outcome measures. Future annual measures (and additional credit) will relate to long-term measurements of programmatic outcomes, that is, the use of the data collected.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: PDP has developed a program-level efficiency measure: cost per commodity test. The efficiency measure is designed to demonstrate the program's ability to keep costs low while producing a high volume of testing results year to year. The program sets targets within a range that reflects differences in data requirements for each of the commodities being evaluated and the testing methods used. Results are monitored and measured against the targeted level. Variance in this measure can result from unplanned, urgent data requests.

Evidence: The program has increased its data production without major increases in program funding in most years which indicates improved cost effectiveness over prior years. The number of data points (tests) generated by PDP has increased from less than 1 million in 2003 to 1.3 million in 2006. At the same time, PDP has demonstrated the ability to accommodate urgent data requests (e.g., the triazole project and soybean rust survey) while continuing to generate data in a timely fashion. In FY 2003, the program implemented a resource-intensive triazoles survey, conducted as a rapid response to provide the data needed to register newer, safer replacement pesticides for organophosphates. That effort generated urgently needed data, but reduced the overall number of tests generated by the program. A "Small Extent" is provided because it is not clear how - or by what means - the program ensures efficiency given changes in funding level and urgent data requests.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: PDP is the main provider of dietary risk assessment data to the EPA for use in the registration and re-registration of pesticides in accordance with the FQPA of 1996. Because PDP is the main source of pesticide residue data to EPA, there are no similar programs - either in the public or private sector - with which to base a comparison. PDP procedures have been given to other data providers at the request of EPA. Program staff was invited to participate in an international risk assessment workshop for the European Union to discuss program design and procedures. However, based on the unique purpose of the program, it receives a "N/A".

Evidence: See answer to question number 1.1 for PDP.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: USDA's OIG conducted a review of program operations in 2004, which found that the program was operating as intended. Based upon the program's small size, and its role in providing EPA with pesticide residue data in support of the pesticide re-registration process, periodic audits are considered acceptable. In addition, Federal PDP staff conducts periodic reviews of program operations and laboratories to verify that sampling and laboratory operations are in accordance with established procedures and meet PDP requirements. Laboratories are also required to pursue and maintain ISO 17025 accreditation, which requires ongoing, onsite assessments by a second party, the American Association for Laboratory Accreditation (A2LA).

Evidence: (USDA OIG Audit: Report 01099-28-At, May 2, 2005) The audit found that the program continues to improve operations by (1) implementing additional SOPs to provide uniformity in performing administrative, sampling, and laboratory activities; (2) developing an electronic database to serve as a central comprehensive data repository and allow for more efficient administration and reporting of data; (3) performing internal reviews to ensure program compliance; (4) developing a quality assurance program to ensure the reliability of PDP data; and (5) hiring additional technical personnel to expand program operations and monitor program activities. Laboratories analyzing fresh and processed fruit and vegetables, meat, poultry, and grains are required to seek and maintain accreditation pursuant to ISO 17025. ISO 17025 accreditation requires full and independent reviews of laboratory and laboratory management by independent assessors. To date, several of the PDP laboratories have achieved full accreditation, three of the laboratories will apply for accreditation during 2007, and the remaining laboratories will apply for accreditation during 2008.