Detailed Information on the
Child Survival and Health for Latin America and the Caribbean Assessment

Performance: As part of foreign assistance reform and the resultant new Foreign Assistance Framework, USAID is working with the Department of State to establish new performance measures and indicators that will allow for results to be measured across countries, programs and partners. These new measures will be incorporated into operating units' operational plans for FY 2007.

Measure: Proportion of children under one year of age who received the complete series (three doses) of DPT (diphtheria, pertussis, and tetanus) vaccine before age 12 months.

Explanation:Proportion of children under one year of age who received the complete series (three doses) of DPT (diphtheria, pertussis, and tetanus) vaccine before age 12 months. Numerator: Number of children under 12 months who received three doses of DPT vaccine before age 12 months. Denominator: Total number of expected live births in a given year. This measure is linked to L-T Measure #8 below.

Measure: Numbers of countries which have USAID Family planning programs reaching at least 55% contraceptive prevalence using modern methods.

Explanation:This measure replaces an earlier indicator "Contraceptive Prevalence Rate all methods" because it reflects USAID's new graduation policy for family planning programs. USAID will be closing out of countries that have successful and sustainable family planning programs.

Measure: Percent of deliveries attended by skilled birth attendants

Explanation:

Measure: Number of individuals receiving Voluntary Counseling and Testing (VCT) services.

Explanation:Receiving Voluntary Counseling and Testing and knowing your HIV status has an impact on behavior that can directly impact HIV prevalence rates. This measure is linked to L-T Measure #10 below. In 2004 and 2005 due to policy issues, Brazil has not fully implemented the voluntary counseling and testing activity. This is the primary reason for the low performance results.

Measure: Dollars spent on donated Family Planning commodities in the LAC region in USAID presence countries per total dollars spent on Family Planning programs in the LAC region.

Explanation:Less dollars expended on donated family planning commodities allows more funds to be expended on technical assistance to help promote the improvement of contraceptive prevalence rates through sustainable host country government family planning programs and structures.

Measure: Under five mortality rate, on average, as measured by UNICEF in USAID presence countries

Explanation:This measure is linked to the DPT3 rate (measure #2 above), as vaccination rates are used as a proxy for overall child survival activities. The 1990 and 2002 baselines should be viewed in context with the DPT3 coverage rates (expressed in measure #2 as targets for FY 2004-2006).

Measure: Total fertility rates, on average, per Population Reference Bureau (PRB) data, in USAID Presence Countries

Explanation:This measure is linked to the Measure #3 above, Contraceptive Prevalence Rate (CPR - using all methods), which is a measure of the proportion of the population that is using contraception. CPR is a proxy for overall reproductive health programs. The 1990-2000 and 2003 Baselines should be viewed in context with the Contraceptive Prevalence Rates (expressed in Measure #3 as Targets for FY 2004-2006).

Measure: HIV Prevalence Rate - Average, per UNAIDS data, in USAID Presence Countries

Explanation:This measure is linked to the Measure #6 above, Number of Individuals Receiving Voluntary Counseling and Testing (VCT), assuming that receiving counseling and knowing your HIV status makes you less likely to transmit the disease. The number of individual

Is the program purpose clear?

Explanation: The program has a focused and well-defined mission, though the level of analysis used is based on the Latin America and Caribbean (LAC) Region, not the country level. At times, information is given for progress at the country level when information is not available at the regional level. The Child Survival and Health (CSH) account funds activities in support of immunization, oral re-hydration, health, nutrition, water and sanitation, displaced and orphaned children, prevention, treatment and control of, HIV/AIDS, TB, malaria, polio, as well as family planning/reproductive health. LAC has 16 field operating units (OUs) in three sub-regions, three field-managed sub-regional programs and two Washington-managed programs; 16 of these 21 OUs support CSH-funded health programs. The CSH programs in LAC are linked to U.S. foreign policy priorities to advance sustainable development by improving human health and reducing the spread of infectious diseases in the Hemisphere; helping people realize their reproductive intentions; and addressing the HIV/AIDS epidemic. LAC/CSH programs contribute to and fully support USAID goals. The CBJ and the Assistant Administrator cite improving health as an "excellent development investment" and have identified LAC priority health areas as maternal health, child survival and reproductive health, HIV/AIDS, and infectious diseases.

Evidence: 1) Secretary of State Statement in Bureau Performance Plan (BPP). 2) State-USAID Strategic Plan for FY 2004-2009. 3) USAID Administrator Natsios testimony to HACFO, April 9, 2003. 4) Assistant Administrator (AA) Franco testimony to SACFO, April 2, 2003. 5) AA Franco testimony to the House Committee on International Relations, February 27, 2003. 6) Congressional Budget Justification (CBJ) Overview.

Does the program address a specific and existing problem, interest or need?

Explanation: The program addresses a specific need and specific problems within the region. For example, the development challenges in the LAC region which are addressed under the CSH account include increased prevalence of HIV/AIDS, especially in the Caribbean, high infant/child mortality rates among disadvantaged population, pockets of low vaccination coverage, drug-resistant tuberculosis and limited access to quality family planning and reproductive health services. These challenges are validated through statistics from other development agencies including the World Bank and the UN Economic Commission for Latin America and the Caribbean. The LAC Bureau also performs health trends and analyses for most of the Western Hemisphere countries. The trends include data by country and sub-region on the status in nutrition, HIV/AIDS, maternal health, child survival, population growth, spread of infectious diseases, and health policy reform. At the field OU level, in-depth analyses are done to define and quantify specific needs and problems based on country circumstances. For instance, in Brazil, CSH emphasis areas address incidence of HIV/AIDS, tuberculosis and malaria. In Peru, the need is towards improving health institutions and services in a sustainable fashion. Thus, USAID Peru has a three-pronged approach to improve the quality of services and capacity of institutions, to effect behavioral change, and formulate and implement policy reforms.

Evidence: 1) AA Franco testimony to SACFO, April 2, 2003. 2) AA Franco testimony to the House Committee on International Relations on February 27, 2003; 3) Financial Times article by Mark Mulligan, "Early Signs of Recovery in Latin America." 4) CBJ Overview; Health Trends Analysis by LAC Bureau. 5) Dominican Republic and Brazil Strategic Plans. 6) Peru Strategy. 7) Management Letter between LAC Bureau and Missions - Peru Management Letter provided as example.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The program is designed so that is not redundant or unneccessarily duplicative of any other Federal, state, local or private efforts. As one of several development entities in a developing host country, and the lead United States Government (USG) foreign assistance entity, USAID coordinates with all of the other in-country USG entities under the aegis of the U.S. Ambassador, as well as with the host government and other prominent donors, to avoid duplication of assistance efforts and eliminate program redundancies, and promote country program synergies. For example, in El Salvador, the Centers for Disease Control and Prevention and USAID jointly designed and implemented a program to strengthen an integrated public health surveillance system. Preparation of the annual Mission Performance Plan (MPP) is coordinated by the U.S. Embassy and documents all activities of USG agencies that are carried out in that country. Country Team meetings, involving the principals of all USG entities and chaired by the Ambassador, are also held on a weekly basis. Further, USAID and other bilateral and multilateral donors, the relevant host Government Office, and implementing partners including the U.N. health agencies and the Pan American Health Organization meet regularly to share development experiences and successes, and maximize assistance outcomes. Finally, through the Global Development Alliance (GDA), LAC leverages private resources for jointly funded programs, e.g., those with the Gates Foundation on immunization and nutrition issues.

Evidence: 1) Mission Performance Plan (MPP) - Bolivia, El Salvador, and Honduras as examples. 2) CJB--Donor Collaboration section. 3) OU Example: International Assistance section of Brazil strategy (pg. 13-15). 4) Donor Collaboration section of Health Strategic Objective in Peru Strategy (pg. 72-73). 5) Global Development Alliance (GDA) Guidelines. 6) List of Public-Private Alliances with the Gates Foundation in LAC countries. 7) USAID grant with PAHO. 8) The Automated Directives System (ADS) - Sections 201.3.6 and 201.3.7 (pg. 26-38) provides Agency guidance for the preparation of Strategic Plans. Section 201.3.9.2 provides specific guidance on donor coordination analysis in the strategic planning phase.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The LAC/CSH program design is free of major flaws that would limit it's effectiveness or efficiency. As required by the Automated Directives System (ADS), LAC operating units (OUs) must develop a Results Framework (RF) for each strategic objective (SO), linking the development challenge being addressed to program implementation activities and milestones to be achieved. The SO is the most ambitious result in a particular program area that an operating unit (with its partners) can materially affect and for which it is willing to be held accountable. The RF is informed by various technical and resource analyses and include sector assessments, lessons learned, and gender and country conflict vulnerability analysis. The LAC Bureau's CSH program strategy design process is initiated at the OU level in consultation with the Global Health Bureau to ensure that LAC strategies with CSH activities are relevant to the specific country needs and circumstances. All OU strategies are subject to intensive agency-wide reviews. Final agency approval is contingent on the resolution of all technical and policy-related issues and concerns that were raised during the reviews, as well as consistency with the LAC Bureau's stated priorities and U.S. Foreign Policy interests.

Evidence: 1) ADS 201, pg. 30-33 provides Agency guidance regarding the content of Strategic Plans; ADS 201 pg. 42-48 provide guidance on mandatory and other analyses for developing Strategic Plans. 2) Examples of Strategy: USAID/Brazil HIV/AIDS Strategic Objective including Results Framework and Peru Health Strategic Objective Strategy 3) Example of LAC review of OU Strategic Plan: Peru Management Letter and Issues Paper as examples. 4) June 2003 Ecuador Scope of Work for a Health Assessment (see pg. 58 -64 and Annexes I-III of Peru strategy for a sample summary of the analyses. See pg. 77-86 of Brazil HIV/AIDS strategy for a sample listing of primary and secondary research and interviews conducted to complete these analyses. See June 2003 Ecuador Health Assessment Scope of Work, which includes review of health financing options.)

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The program is effectively targeted to ensure that resources reach the intended beneficiaries. Management in the LAC Bureau is decentralized in that individual strategies are developed and implemented by OUs. Widespread consultation with prominent partners, customers (including beneficiaries), and stakeholders is critical to the development and implementation of all OUs strategic plans. This consultation is done through meetings, workshops, retreats, and surveys. The results of these consultations are integrated into strategic plans and implementation, (i.e. results and activities) are influenced by feedback from stakeholders. In addition, Country and Assistance Checklists are completed to ensure that USAID assistance is targeted at eligible countries and eligible organizations.

Evidence: 1) Example of OU Strategy - Peru Strategy (Health Strategic Objective pg. 73) and Brazil HIV/AIDS Strategic Objective (see discussion of beneficiaries, target groups and target areas pg.23-25). 2) Report to Congress, " USAID Child Survival and Health Program Funds Progress Report (FY 2002). 3) Country Checklist; Assistance Checklist. 4) ADS 201.3.6 and 201.3.7 (pg. 26-38) provides Agency Guidance on the content of strategic plans including involvement of customers, stakeholders and partners (see pg. 73 of Peru Strategy for a summary description of Customer/Beneficiary consultations).

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: To facilitate performance monitoring and reporting under the Joint State/USAID strategies, the LAC Bureau in FY 2004 undertook an extensive effort to align and harmonize a set of contextual and regional performance indicators that would provide valuable performance information to managers both in the field and at Washington Headquarters. USAID's long-term performance goals are supported by outcome and/or output-related performance measures and regional indicators which are used to assess progress towards completion of the long-term goals, and allow Agency management to re-direct program where necessary. For example, in El Salvador, the Mission has four CSH indicators over a five-year period at a minimum: infant mortality rates, diphteria, pertussis, and tetanus (DPT3) coverage for children under 12 months, contraceptive prevalence rate, and number of latrines and water systems constructed or rehabilitated.

Evidence: 1) Joint State/USAID Strategic Plan for Fiscal Year 2004 to 2009. 2) USAID FY 2005 Annual Performance Budget, March 25, 2004. 3) LAC Bureau Approved Regional Indicators, May 14, 2004. 4) Agency FY 2003 Performance and Accountability Report. 5) Agency Guidance on Performance Management - specifically ADS 203.2 through 203.3.5. 6) Example of a PMP : El Salvador Performance Management Plan (PMP). 7) USAID's Expanded Response to HIV/AIDS( pg. 17-19). 8) Draft 2005 Annual Reporting Guidance, Section I, Sub-section A, Item 5 (Page 6) and Guidance Annex VIII, Page 29.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The LAC/CSH program does have ambitious targets and yearly timeframes for it's long-term measures. For example, in FY 2004 the LAC Bureau undertook an intensive, Bureau-wide effort to develop and build consensus around a set of contextual and regional performance indicators for their sectors related to Democracy, Education, Economic Growth, Health, and the Environment (for this account, only the Health sector relates to CSH). Contextual indicators were adopted to indicate areas where USAID's activities within the region are impacting larger social/economic trends (such as GDP), though movement or progress cannot be attributed to USAID alone. In addition, the LAC Bureau has selected a set of widely accepted and theoretically sound regional performance indicators that will be used to assess progress against annual targets set by the Washington Bureaus and the Missions in the field. Targets are expressed as percentages, where appropriate, or as actual figures in cases where an actual quantified number can be used. Operating Units (OUs) are delegated the authority to define quantitative and verifiable targets and timeframes for their respective Strategic Objectives (SOs, the Bureau's key, ambitious and challenging measures), however these targets are reviewed and vetted by the LAC Bureau and PPC, and are reported each year in the Annual Report (AR). The OU's PMP details targets by SO over the life of the program. OU's are responsible for data quality assessments to determine the measures' validity, reliability, timeliness, precision, and accuracy. Annual Targets are identified in the LAC Bureau's table of regional indicators for FY 2004, 2005, and 2006, as well as results from FY 2003. The LAC Bureau reviews targets at the SO level during the strategy approval process (approximately every five years). OUs are responsible for data quality assessments to determine the measures' validity, reliability, timeliness, precision, and integrity. Changes could also be introduced when circumstances change. For example, in El Salvador, the mission is amending the agreement with its implementing partner due to a decrease in child survival/maternal health funds; this will require a revision of targets and activities.

Evidence: 1) Joint State/USAID Strategic Plan for Fiscal Year 2004 to 2009. 2) USAID FY 2005 Annual Performance Budget, March 25, 2004. 3) LAC Bureau Approved Regional Indicators, May 14, 2004. 4) OU-level indicators - El Salvador Mission PMP as example; 5) ADS 203.3.5 for Agency Guidance on Data Quality Assessments.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: USAID's performance management framework is structured around a limited number of challenging but achievable strategic objectives and goals (developed jointly with the Department of State), as well as performance goals, targets and indicators that are logically linked to the long-term joint goals and objectives which enables USAID to demonstrate progress toward achievement. The LAC Bureau measures their performance against this structure through a set of discrete, quantifiable, and measurable annual targets and contextual and performance indicators, which are monitored by the LAC Bureau and are reported each year in the annual reporting process. At the OU level, the PMP includes annual performance measures which are used to assess progress towards the program's long-term goals and objectives. The PMP provides baseline data and annual targets, among other information. Performance of SOs is assessed and documented annually. For example, USAID/Brazil has two SO-level indicators for its CSH-funded HIV/AIDS program that are tracked annually: (1) HIV seroprevalence rate in pregnant woman aged 15-24, and (2) HIV seroprevalence rate among high-prevalence populations.

Evidence: 1) Joint State/USAID Strategic Plan for Fiscal Year 2004 to 2009. 2) USAID FY 2005 Annual Performance Budget, March 25, 2004. 3) LAC Bureau Approved Regional Indicators, May 14, 2004. 4) USAID/Brazil's HIV/AIDS PMP (p. 31-32) 5) OU level indicators - Brazil HIV/AIDS Strategy as example of indicators to be included in USAID/Brazil's HIV/AIDS PMP (see p. 31-32). 6) Draft 2005 Annual Reporting Guidance, Section I, Sub-section A, Item 5 (Page 6) and Guidance Annex VIII, Page 29.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has documented baselines and ambitious targets for its performance measures. For example, each OU's Performance Management Plan (PMP) includes baseline figures and annual targets. Each SO also defines Intermediate Results (IRs) that are incremental over time and contribute directly to the attainment of the objective. During this phase, illustrative targets are identified and these are finalized within a year of the SOs approval. As an example, Peru's Strategy defined the SO as "Improved Health for Peruvians at High Risk" and one of three IRs contributing to that SO is "Quality Services Accessible and Utilized." Annual indicators used by USAID/Peru to assess performance of the SO and the above IR include: (1) fertility gap, (2) adolescent pregnancy rate, (3) births attended by a trained health professional, and (4) contraceptive market share of private producers and suppliers. Baseline data and targets for these indicators were preliminarily defined. During the annual performance review, the OU will consider, among other things, whether targets should be adjusted in light of recent performance so as to remain ambitious.

Evidence: 1) Peru PMP and Strategy Illustrative Performance Measures (pg. 70 of Peru Strategy) as examples. 2) ADS 203.2, 203.3, 203.4 and 203.5 provide Agency Guidance on Performance Management and Data Quality. 3) LAC Bureau Approved Regional Indicators, May 14, 2004. 4) Draft 2005 Annual Reporting Guidance, Section I, Sub-section A, Item 5 (Page 6) and Guidance Annex VIII, Page 29.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: At the OU-level, Strategic Objective Agreements (SOAGs) with host Governments outline the terms of agreement, including commitment to shared goals of USAID/LAC OUs and counterpart Ministries. Solicitation documents for contractors and grantees such as Request for Proposals (RFP), Request for Applications (RFA), or Annual Program Statements (APS) incorporate SO and/or program objectives. Respondents to the RFP or RFA need to articulate how they plan to contribute to the achievement of LAC Mission Strategic Objectives. The extent to which they would be able to help achieve these shared objectives and goals is incorporated in the technical merit criterion of the selection criteria. Grants and Contract Agreements incorporate or reflect these shared commitments to common goals, typically in the scope of work section. These commitments are likewise reflected in Contractors and Grantees annual work plans. By the terms and conditions of their agreements, contractors and grantees provide quarterly progress reports to OUs that document their results to date. USAID managers verify information in these reports through monitoring, site visits, and meetings. The information in the OU's PMP includes results reported and verified from contractors and grantees. All of these activities are designed and carried out to ensure that USAID partners commit to and work toward the annual and/or long-term goals of the program.

Evidence: 1) Sample SOAG. 2) Brazil HIV/AIDS Request for Applications. 3) ADS 350 guidance on SOAGs. 4) ADS 303.5.5 b and c provide Agency guidance on Evaluation Criteria for Grants. 5) Program Approach, Design, and Implementation section of Brazil HIV/AIDS RFA, pg. 7. 6) Brazil HIV/AIDS RFA, pg. 17. 7) Program Description of Brazil RFA. 8) Monitoring, Evaluating and Reporting section of Brazil HIV/AIDS RFA, pg. 9.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Independent evalautions of sufficient scope and quality are conducted on a regular basis. Evaluations are intended to look at results achieved vs. targets/goals, program impact and effectiveness, and lessons learned among others. For example, in the past five years approximately 200 evaluations have been conducted for the LAC Bureau's programs, including CSH-funded activities. A break-out of evaluations of CHS-only activities is not available because evaluations are often cross-sectoral and activities can involve multiple funding sources. Situations that require evaluations include: input to inform a key management decision; performance data that indicate an unexpected result or finding which cannot be readily explained and therefore, needs further analysis; customer or partner feedback that suggest there are implementation problems or unmet needs; the need to re-examine program sustainability; cost-effectiveness or relevance; and to identify lessons learned. Evaluations are internal, independent or involve a combination of evaluators. The type of information sought determines how the evaluation should be approached. An independent evaluation is conducted by a unit or expert not directly associated with the activity or program, such as a consulting firm which is competitively bid. An example is the September 2002 final evaluation of Nicaragua's Child Survival project, which shows that program goals have been achieved. In addition, the USAID Office of the Inspector General (IG) conducts audits both on program performance and management effectiveness. For example, the May 2002 IG audit of USAID Bolivia's Financial Operations showed that all advances and disbursements were made in accordance with established policy and regulations, but check issuing data needed to be entered into the electronic payment tracking system. In addition to evaluations, the LAC Bureau conducts intensive reviews of programs approximately once every three years during which time participants from outside the OU (including staff from LAC/Washington, PPC, the Pillar Bureaus, State Department, etc.) have an opportunity to assess program performance and, as necessary, provide guidance for improving the effectiveness or impact of the program.

Evidence: 1) LAC list of evaluations, as compiled by USAID's Center for Development Information and Evaluation (CDIE). 2) Inspector General Audit Reports (see Bolivia Financial Operations audit as example). 3) Independent Evaluations of Nicaragua's Child Survival Project, pg. 8. 4) ADS 203.3.6 provides Agency guidance on evaluations.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: At the regional level, USAID/LAC does not yet develop budget requests that are explicitly tied to accomplishment of annual and long-term performance goals. The framework of regional and common agency performance goals and targets is too new for the agency to be able to integrate this performance data into their strategic budgeting. OMB commends USAID for its efforts in developing a strategic budgeting model (which factors in performance data) to support the LAC (and all regional bureaus) in their budget formulation. At the level of the Operating Units (OUs) budget requests are linked somewhat to the Annual Report of the OU, which summarizes program progress toward each Strategic Objective of the OU. Furthermore, the Congressional Budget Justification summarizes program progress by Strategic Objective, for each individual OU, bu this is done in a qualitative, anecdotal manner, and the agency has not yet reached the stage of incorporating their more quantitative performance data into budget presentations. That said, the CBJ is otherwise a transparent and complete presentation of resource needs.

Evidence: 1) OU Annual Report (AR) - Guatemala as an example. 2) LAC Review of AR - Guatemala as an example. 3) FY 2005/2006 Bureau Program and Budget Submission (BPBS) Guidance issued by the Agency's Policy and Program Coordination Office. 4) Congressional Budget Justification (CBJ).

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: USAID has taken extensive action to improve it's strategic planning and budgeting porcesses and tools. USAID's performance management framework provides the vital linkage between joint State/USAID strategic objectives, strategic goals, performance goals, contextual and common performance indicators at the regional level, and annualized performance targets. LAC conducts intensive reviews of OU strategies at least once every three years in accordance with USAID policy (ADS 203.3.10).

Evidence: 1) Joint State/USAID Strategic Plan for Fiscal Year 2004 to 2009. 2) USAID FY 2005 Annual Performance Budget, March 25, 2004. 3) LAC Bureau Approved Regional Indicators, May 14, 2004. 4) Agency FY 2003 Performance and Accountability Report. 5) Agency Guidance on Performance Management - specifically ADS 203.2 through 203.3.5. 6) LAC Timeline for Mission Program/Strategy Reviews. 7) Intensive Review memos - Jamaica and Haiti as samples. 8) AA Franco-Cronin Memo, re: Central America and Mexico Regional Strategy. 9) ADS 203.3.10 - Planning, pg. 39. 10) Draft 2005 Annual Reporting Guidance, Section I, Sub-section A, Item 5 (Page 6) and Guidance Annex VIII, Page 29.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: USAID regularly collects timely and credible performance information from key program partners. Funding recipients are required to manage, monitor, and report performance information on a regular basis. This information is submitted to OUs on a quarterly basis and feed into the Annual Report (AR) and PMP. OUs indicate in the AR the status of targets for CSH. The LAC Bureau, in turn, reviews OU performance via the AR review process. When targets are not met, various options may be considered depending on the specific circumstances and issues involved: the contract or grant agreement may be modified, additional funds may be delayed or not provided, management changes may be introduced, discussions may take place with other partners (other donors or host-country) regarding their commitment, etc. If goals are exceeded the OU will decide whether to set new targets or to focus on other interventions. The AR review process in turn feeds into Bureau decisions about programming, budgeting, and staffing, among others.

Evidence: 1) Operating Unit Annual Report (AR), with Guatemala AR as sample. 2) LAC Bureau Review of Guatemala AR. 3) Operating Unit Performance Management Plan (PMP), with Ecuador as a sample. 4) USAID's ADS 202.3.6; USAID adherence to 22 CRR 226, "Administration of Assistance Awards to US Non-governmental Organizations," Section 226.51, "Monitoring and Reporting Program Performance"; ADS 303.7 Mandatory Reference for Grants. 5) Draft 2005 Annual Reporting Guidance, Section I, Sub-section A, Item 5 (Page 6) and Guidance Annex VIII, Page 29.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: USAID federal managers and program partners are held accountable for cost, schedule, and performance results. A cognizant technical officer (CTO) is designated for each LAC activity to manage contractors and grantees. CTO responsibilities include, among others, the technical review and approval of vouchers, monitoring contractor/grantee performance and financial status, and overseeing sub-agreements. The Agency recently re-established a training program to certify CTOs responsible for managing contracts and grants. The courses are designed to provide CTOs with the basic skills and knowledge to effectively carry out the role of a CTO. Under USAID's personnel evaluation process, CTOs are evaluated for effective management of agreements, including the quality of technical guidance provided to contractors and grantees. The Agency also offers a course for CTO supervisors to enable them to more effectively provide on-the-job training and to better monitor and evaluate the performance of a CTO. For contractors and grantees, past performance is a criteria in the award. An example is Brazil HIV/AIDS RFA which requires applicants to provide a description of all contracts, grants, and cooperative agreements which the applicant has implemented involving similar or related programs over the past three years with USAID and with other organizations (both commercial and governmental). Institutional capability and past performance is 25% of the total selection criteria in this case. Performance-based type of contracts and agreements are used to focus contractors on achievement of program results. For example, Guatemala's Health SO has the contractor responsible for achieving five specific results: increased use of the Maternal and Child Health services provided by the Ministry of Health (MOH) and its NGO partners; increased adoption of household health practices that favor improved child survival and health; better managed MOH and NGO health programs; greater community participation and empowerment; and increased use of selected MCH services provided by hospitals and clinics of the host-country's social security system. Midterm evaluations are also used to review performance to date and identify issues. An example is the August 2001 mid-term evaluation of Nicaragua's Diversification, Sustainability and Social marketing grant wherein program recommendations were outlined by an independent evaluator and carried out by the program partner.

Evidence: 1) CTO responsibilities as defined in ADS 300. 2) Brazil HIV/AIDS RFA, Guatemala G-CAP Performance-Based Statement of Work for its Health SO. 3) August 2001 mid-term evaluation of Nicaragua's Diversification, Sustainability and Social marketing grant. 4) Course description of CTO Certification Program.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: USAID funds are obligated in a timely manner and spent for the intended purpose. Obligations, among other financial data, are tracked via Phoenix, an Agency-wide accounting system. LAC/CSH funds are provided on an incremental, as-needed basis, with specific amounts determined by the budget and final Operating Year Budget setting processes. Once funds are obligated, OUs and the LAC Bureau monitor disbursements over the life of the program. Based on Agency policy and LAC Bureau's analysis, an OU's program may not receive its full increment of funds in a subsequent year if it is determined that an unacceptable level of funds is still unspent (undisbursed). According to Agency policy, a program can only have unspent funds sufficient to cover expected disbursements for the next 12-18 months. Finally, all OUs are required to prepare procurement plans that outline planned procurements and obligations and help ensure that funds are spent in a timely manner and for the intended purpose.

Evidence: 1) Phoenix Flash report. 2) ADS 602 also provides specific Agency guidance regarding forward funding of programs. 3) LAC Regional Procurement Plan.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: Mission Management Assessments (MMAs) recently undertaken by the LAC Bureau provide a systematic and comprehensive view of the Bureau's OUs as a basis for defining ways to improve operations and to help rationalize the allocation of scarce staff and operating expense (OE) resources. The results of the Mexico, Guatemala, Honduras and Nicaragua MMAs were taken into account during the development of the Central America and Mexico Regional strategy. Building upon favorable experience with regional hubs for service support, the Bureau has agreed to move toward one regional platform over time in Central America, which will be located in El Salvador. Consolidation of accounting functions in the Carribbean missions resulted in the reduction of five positions and a future annual cost reduction of approximately $60,000 in OE. The upgrade in Wide Area Renovation Plan communications capability througout the LAC region, coupled with Virtual Private Network technology, has facilitated secure and extremely rapid transmission of raw accounting and payment information from the serviced post to the regional center and back to the serviced post in the form of usable, almost real-time accounting information. As a result, El Salvador performs payment certification functions for Guatemala, resulting in the reduction of one position and a future annual cost savings of $70,000. Although the above examples demonstrate operational efficiencies, LAC does not have procedures to measure program efficiency in terms of program output or performance as required.

Evidence: 1) ADS 300 provides the overall guidance on procurement including competitive procurements (ADS 302 for Contracts and 303 for Grants). 2) FAR for ADS 302 -Contracts and 22 CFR 226 for ADS 303 - Grants. 3) LAC FY 2005/2006 Bureau Program and Budget Submissions (BPBS) (pg. 26-29 for Management Improvements/Operational Efficiencies).

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program does collaborate and coordinate effectively with related programs. Most notably, the Department of State and USAID have recently developed joint goals at the Department/Agency level via the Joint State-USAID FY 2004-2009 Strategic Plan which formalize the relationship highlighted in 1.3. Historically, State and USAID have planned and implemented programs collaboratively at the country level via the MPP process, which will now influence the joint State/USAID Strategic Plan. Weekly country team meetings are held among USG agencies and regular coordination meetings with other donors in a country. In addition, the LAC Bureau and its OUs collaborate and coordinate with related programs in a leadership or participant role through a variety of mechanisms: close working collaboration with U.N. health organizations and PAHO, working arrangements with other Federal agencies such as CDC, and alliances with the private sector. Finally, the budget request process is a joint effort between State and USAID and is reflected in the FY 2005 Bureau Program Budget Submission which will be the basis for an integrated budget that is submitted to the White House.

Evidence: 1) State-USAID Strategic Plan. 2) Examples of MPPs - El Salvador, Honduras and Bolivia. 3) Example of public-private partner alliance: GDA alliance with the Gates Foundation. 4) Example of grant with PAHO. 5) FY 2005/2006 Bureau Program Budget Submission. 6) Joint Memo dated 9/28/00 from STATE/USAID on guidance and recommendation to improve cooperation and coordination between the agencies.

Does the program use strong financial management practices?

Explanation: Through the Federal Managers Financial IntegrityAct (FMFIA)/Management Control Review Committee process, USAID ensures that resources are protected against fraud, waste and abuse and that they achieve the results for which funds were appropriated. The process requires each OU in the LAC Bureau to do a self-assessment of the adequacy of management controls in all areas of agency operations including program, administrative, and financial management. Each OU submits an FMFIA memo to the LAC Bureau, which in turn submits a Bureau memo to the USAID Administrator. LAC grants and contractors are also subject to audits. Currently, the Phoenix system is used for Washington accounting and the MACS system for field accounting, using two systems has created a disconnect between the two. USAID plans to roll out the Phonenix system to the field, which will resolve the outstanding problems. It is anticipated that a first LAC pilot Mission (USAID/Peru) may begin Phoenix operation in mid-FY 2004. Other pilot Missions will follow a few months later depending on the success of the first pilot. Once Phoneix is deployed to all missions, USAID anticipates getting a "yes" answer for this question.

Evidence: 1) Federal Manager's Financial Integrity Act (FMFIA) checklist. 2) OU completed FMFIA analysis, with Ecuador as an example. 3) LAC Bureau FMFIA memo to USAID Administrator dated October 29, 2002. 4) Report on Material Weaknesses - Guyana and Ecuador as examples. 5) Agency-wide guidelines for Recipient-contracted audits (RCA). The Agency's ADS 620 chapter provides the overarching accounting and financial reporting principles and standards for the Agency. 6) ADS 596 provides Agency guidance regarding the accountability and effectiveness of USAID's programs and operations by establishing, assessing, correcting, and reporting on management controls. These ADS chapters are based on regulations issued by Federal oversight agencies, including OMB, Treasury and GAO.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program has taken meaningful steps to address management deficiencies. For example, the FMFIA process requires each OU in the LAC Bureau to do a self-assessment of the adequacy of financial management controls in all areas of agency operations, including management of program assistance. Each OU submits an FMFIA memo to the LAC Bureau, which in turn submits a Bureau memo to the USAID Administrator. At the OU level, LAC missions address deficiencies or weaknesses that could appropriately be resolved at that level. Any significant deficiency or material weakness (whether new or unresolved from previous FMFIA reviews) is reported to the LAC Bureau and included in the Bureau FMFIA memo to the Administrator. Steps for resolving the weakness are identified and a date proposed by which the weakness or deficiency will be corrected. In 2002, missions were also specifically requested to complete an information system security checklist, as part of the FMFIA process. The LAC Bureau has also recently undertaken management assessments of its OUs. Six assessments have been completed since May 2002. The results of the Mexico, Guatemala, Honduras and Nicaragua management assessments were taken into account during the development of the Central America and Mexico (CAM) Regional strategy. The assessments have helped define ways to improve Bureau operations, such as consolidating accounting functions in regional hubs, resulting in significant savings. The El Salvador mission is poised to serve as the Central America regional hub and will provide support services for a number of missions in the region.

Evidence: 1) ADS 620, FMFIA checklist. 2) OU completed FMFIA analysis- with Ecuador as a sample. 3) LAC Bureau FMFIA memo to USAID Administrator, dated October 29, 2002. 4) Report on Material Weaknesses - Guyana and Ecuador as samples. 5) LAC FY 2005 BPBS (pg. 26-29 for Management Improvements/Operational Efficiencies).

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: To the maximum extent permitted by legislation and statute, LAC Bureau contracts and grants are awarded competitively, except where supported by documentation approved by officials with authority to approve non-competitive awards. For example, in FY 2002 $78.8 million of new assistance in LAC were awarded competitively including a qualified assessment of merit, out of a total of $104.7 million (about 75%). Decisions on grant and contract awards are properly documented (via selection memos, memos of negotiation, etc.). USAID Washington (via the Office of Procurement) reviews and provides oversight to ensure LAC adherence to Federal regulations and Agency guidance on competition.

Evidence: 1) ADS 303.5 for Grants and CBD notices for Contracts. 2) ADS 202.3.9 provides guidance on ensuring procurement integrity and ethics.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The LAC program has oversight practices that provide sufficient knowledge of grantee activities. For example, a Cognizant Technical Officer (CTO) is designated for each program to manage contractors and grantees. CTO responsibilities include, among others, technical review and approval of vouchers, monitoring contractor/grantee performance and financial pipelines, and overseeing sub-awards. The Activity Manager and the SO Team Leader, who may or may not be the CTOs, also provide oversight. In addition, mission management is kept informed of program performance, progress, and issues via periodic portfolio reviews, staff meetings, etc. The LAC Bureau, in turn, is informed via the Annual Review process. The OU Agreement Officer is the mandatory control point of record for all official communications and contacts with the recipient that may affect the award budget, the program description or any terms and conditions of the award. Audits per OMB Circular A-133 or recipient-contracted audits are also conducted as required by policy or regulation.

Evidence: 1) Cognizant Technical Officer (CTO) responsibilities as provided in ADS 303.3 for Grants and ADS 302.3 for Contracts. 2) Recipient-Contracted Audit guidelines. 3) USAID FY 2003 Performance and Accountability Report.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The LAC Bureau collects grantee performance data and makes it available to the public. In preparing USAID's reporting documents, the data supplied by the various grantees and contractors are utilized to support LAC Bureau's analysis and program descriptions. Disclosure to the public is done in a number of different ways. For example, the USAID FY 2003 Performance and Accountability Report discloses program information at the Agency level. The CBJ (available on the USAID web site) provides specific information to the public on LAC programs including information on SO performance and results, by OU. The CBJ also indicates the various grantees and contractors working in the different SOs. USAID's Center for Development Information and Evaluation (CDIE), via the Development Experience Clearinghouse (DEC), makes some evaluations of specific programs available to the public.

Evidence: 1) USAID FY 2003 Performance and Accountability Report. 2) CBJ 3. Program/Activity evaluations.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: At the regional level, the LAC Bureau has long-term performance measures (i.e., strategic objectives, strategic goals, and performance goals) and common regional indicators (contextual and performance) for CSH programs. The program is on-track to meet all of the long-term performance goals, including those which are joint goals/objectives with the Department of State, and the country-level OU indicators. For FY 2003, 93% of the strategic objectives USAID was required to report on met or exceeded their targets. USAID also met the target in the FY 2003 Annual Performance Plan of 90% of strategic objectives met or exceeded. The LAC Bureau review of ARs from OUs show that programs are generally meeting CSH targets. In the case of El Salvador, 4,400 latrines and 15 water systems were to be constructed over a three year period. The actual results were close to 5,000 latrines and 15 water systems. In this case, the SO team would consider revising their targets upward if targets over a two or three year period are consistently exceeded by 15% to 20%. The additional 600 latrines is less than 20% of the projected target over one year, therefore targets have not been adjusted.

Evidence: 1) Joint State/USAID Strategic Plan for Fiscal Year 2004 to 2009. 2) USAID FY 2005 Annual Performance Budget, March 25, 2004. 3) LAC Bureau Approved Regional Indicators, May 14, 2004. 4) Agency FY 2003 Performance and Accountability Report. 5) Examples of PMP and Annual Report: El Salvador PMP, Guatemala AR. 6) Draft 2005 Annual Reporting Guidance, Section I, Sub-section A, Item 5 (Page 6) and Guidance Annex VIII, Page 29.

Does the program (including program partners) achieve its annual performance goals?

Explanation: At the regional level, the LAC Bureau has specific, quantifiable (to the maximum extent possible) annual targets and common regional indicators (contextual and performance) for CSH programs. These measures are supplemented by country-level indicators which are contained in OU Performance Monitoring Plans (PMPs). The program is on-track to meet all of the annualized targets, including those which are joint targets with the Department of State and other program partners. At the country level, these measures do exist and are contained in OU PMPs. OUs report in their Annual Report whether targets are met, exceeded or not met. The PMP and Annual Report are the primary documents that show whether program performance is on track. These documents are, in large part, based on grantee and contractor (partner) information. In the case of CSH, targets at the OU level have been predominantly met.

Evidence: 1) Joint State/USAID Strategic Plan for Fiscal Year 2004 to 2009. 2) USAID FY 2005 Annual Performance Budget, March 25, 2004. 3) LAC Bureau Approved Regional Indicators, May 14, 2004. 4) Examples of PMP and Annual Report: El Salvador PMP, Guatemala AR. 5) USAID FY 2003 Performance and Accountability Report. 6) Draft 2005 Annual Reporting Guidance, Section I, Sub-section A, Item 5 (Page 6) and Guidance Annex VIII, Page 29.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The LAC program has demonstrated improved efficiencies and cost effectiveness in achieving program goals each year. For example, improvements in technology and consolidation of accounting functions have resulted in major savings. In the Caribbean, results include five less staff positions and an annual OE cost reduction of approximately $60,000. In Guatemala, a reduction of one position meant savings of $70,000 as El Salvador (already poised as a regional support center) assumed payment certification functions. The mission management assessments (MMAs) conducted by LAC since May 2002 provide a systematic and comprehensive view of the Bureau's overseas OUs as a basis for defining ways to improve Bureau operations, and to help rationalize the allocation of scarce staff and operating expense resources. The results of the Mexico, Guatemala, Honduras and Nicaragua MMAs have already been taken into account during the development of the Central America and Mexico Regional strategy.

Evidence: 1) LAC FY 2005/2006 BPBS (pg. 26-29 for Management Improvements/Operational Efficiencies). 2) CTO responsibilities as provided in ADS 303.3 for Grants and ADS 302.3 for Contracts. 3) Guyana and Ecuador FMFIA reports. 4) LAC Bureau's FMFIA memo dated October 29, 2002.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The performance of the LAC/CSH program compares favorably to other programs with similar purposes and goals. Because of its field presence and flexible programming, USAID is often cited by other donors and host governments for its ability to respond quickly, for both disaster assistance and standard international assistance, and for its leadership of country program coordination with other donors. For example, in 2001, the Government of Honduras prepared its Poverty Reduction Strategy Paper (PRSP) in response to the World Bank and the International Monetary Fund's highly indebted poor country (HIPC) initiative. USAID was widely credited for its leadership role among the donors. USAID programming decisions and experience also heavily influenced Nicaragua's PRSP and USAID's involvement facilitated public discussion and debate to a degree that would not have occurred in the past.

Evidence: 1) LAC consultations with the World Bank and IDB on Country Assistance Strategies for Honduras, Bolivia, Nicaragua, and Ecuador, May 2002. 2) Evaluation Brief Number 5, March 2003: USAID's Approach to Poverty Reduction - The Case of Honduras (pg. 7-8).

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Independent evaluations have shown that CSH programs are achieving results. The September 2002 final evaluation of Nicaragua's CSH project described it as a five-star project, which included among its results UNICEF's accreditation of the project-assisted health posts and centers and service-oriented leadership by the Ministry of Health. The August 2002 final evaluation of Bolivia's CSH project showed that all its goals have been achieved and the final results are marked improvements from the baseline data on diarrhea incidence in children, maternal health, child spacing, and STI and HIV/AIDS awareness. A perusal of Inspector General (IG) reports indicated that in the past three years, there is only one audit report which stated that results were not completely achieved - in large part due to a new bilateral partner.

Evidence: 1) LAC list of evaluations, as compiled by CDIE. 2) Examples: September 2002 final evaluation of Nicaragua's CSH Projects, and August 2002 final evaluation of Bolivia's CSH projects. 3) IG report on Mexico Program to Develop Institutional Capacity to Diagnose, Control, and Monitor Tuberculosis.