Detailed Information on the
Office of Transition Initiatives Assessment

Ensuring that country activities strive to be short-term in nature. In general, activities should be financed by the Agency's missions abroad with other funding or by other organizations or ended after 2-3 years.

Improve performance measures where possible to better track the effectiveness and sustainability of the Office's programs on advancing democracy and peace.

Improving coordination within USAID's related offices and programs and with the Department of State and other U.S. government agencies to ensure all program activities reflect current U.S. government policy, maximize synergy of assistance efforts, and avoid any duplication of efforts.

Measure: Percentage of OTI programs that demonstrate increased access to unbiased information by target population on key transition issues.

Explanation:This outcome measure tracks OTI's success at ensuring that target populations have access to unbiased information that helps them make informed decisions and facilitates the transition process. This percentage is determined through an assessment of active OTI programs and the specific activities they conduct that demonstrate increased access to objective, balanced information on peace processes, elections, or other national issues, such as printing/disseminating materials, holding press conferences, producing/airing public service announcements or shows on radio/television, financing billboards, supporting/developing Web sites, supporting the legal rights of media outlets, and producing media stories.

Measure: Percentage of OTI programs that have a sustainable handoff strategy (either to USAID Mission or local civil society groups) in place after 18 months of starting up a new country program.

Explanation:This outcome measure assesses OTI's success rate in arranging for a successful closeout/handoff to ensure longer term impact through collaboration with local partners and USAID Mission, international PVOs/UNOs or other donors.

Measure: Percentage of final evaluations that find that OTI had a significant impact in advancing democratic political transitions in priority conflict-prone countries.

Explanation:This outcome measure tracks OTI's success at achieving its overall goal (advancing political transitions in priority conflict-prone countries) through the strengthening of democratic institutions/ processes or an increase in momentum for peaceful resolution of conflict, depending on the specific country program's main objective. This percentage is determined through an analysis of OTI final evaluations and their responses to key questions all evaluators are asked to address, first among them: "Did the program support U.S. foreign policy objectives?"

Measure: Leveraging of additional non-TI funds to support OTI programs

Explanation:* Funds received for OTI programs in Iraq and Afghanistan are not included in these figures.

Is the program purpose clear?

Explanation: OTI's Mission Statement is: "OTI's mission is to help local partners advance peace and democracy in priority conflict-prone countries. Seizing critical windows of opportunity, USAID works on the ground to provide fast, flexible, short-term assistance targeted at key transition needs."

Evidence: OTI website: www.usaid.gov/our_work/humanitarian_assistance/transition_assistance; and 2001-2002 Report.

Does the program address a specific and existing problem, interest, or need?

Explanation: OTI was created in 1994 as a new tool to address the increasing number of complex crises that have appeared since the end of the cold war. OTI addresses these challenges by bridging the gap between traditional immediate-term relief and longer term development assistance.

Evidence: According to "Foreign Aid in the National Interest," there were 111 armed conflicts in 74 locations during the 1990's.

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: OTI is uniquely designed to provide fast, flexible, short-term assistance to conflict-prone countries. OTI 's work is distinct from that of the Office of Foreign Disaster Assistance (OFDA) because OTI focuses on key political transition needs, and its work is distinct from traditional development programs because of its short-term, transitional focus.

Evidence: OTI was established in 1994 to fill the gap between short-term relief assistance and longer-term development assistance, a gap that was not been being adequately addressed.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: For each country program, OTI prepares a detailed assessment that identifies the problem(s) being addressed, U.S. national interest, program areas and required budget. OTI has created an innovative contracting mechanism that allows it to support a range of short-term initiatives, quickly and efficiently. OTI revises/updates country strategic plans at least once a year in order to address new problems or unforeseen issues.

Evidence: New Country Assessment Guide, Checklist for New Country Assessments, Guide to Performance Management. SWIFT (Support with Implementing Fast Transition) Indefinite Quantity Contract (IQC) - competitive selection process, Congressional Budget Justification, OTI Database, Country Budget Estimates, Country Monitoring and Evaluation Quarterly Reports

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: OTI programs are individually designed to address a country's most pressing transitional needs, focusing attention on make or break issues that may decide a country's future. In developing its programs, regular consultations with local partners and experts and close monitoring ensure that funds are reaching the targeted beneficiaries.

Evidence: OTI Annual Report, Guide to Performance Management, New Country Assessment Guide, Checklist for New Country Assessments, Country evaluations and monitoring reports, OTI data base.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: OTI has recently developed two long-term performance measures, and OTI is working to develop a comprehensive long-term measure that will demonstrate OTI's post-close out impact in conflict prevention.

Evidence: See proposed measures.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: OTI is working to establish ambitious targets and timeframes for its long-term measures.

Evidence: Under development.

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: OTI has recently developed two annual performance measures that support the achievement of its goals.

Evidence: See proposed measures.

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: OTI is in the process of establishing baselines, and ambitious targets for its annual measures.

Evidence: Under development.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: OTI has developed an innovative small grants mechanism in which a range of partners have been pre-competed because of their demonstrated expertise in working on transitions in difficult operating environments. OTI's sub-grantees are selected because of their commitment to work on activities in support of the overall program objectives. All sub-grantees sign agreements that outline costs and responsibilities. Close monitoring by OTI staff ensure that programs are working towards meeting OTI's objectives. Strategic plans and program objectives are jointly agreed to by contractors and grantees.

Evidence: OTI web site, www.usaid.gov/our_work/humanitarian_assistance/transition_assistance/ (contains a section on highlights from the field (hot topics)), monthly reports from the field, final and mid-term evaluations, OTI 2001-2002 Report, SWIFT IQC. OTI holds strategic planning sessions with grantees/contractors to finalize country strategic plans.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: OTI conducts independent evaluations through independent consultants and contractors near the end of each country program to evaluate the program's effectiveness, identify lessons learned, and make recommendations for future programs. In addition to independent evaluations which OTI conracts for at the end of all of its country programs, USAID's Clearinghouse of Development Information and Exchange (CDIE) has also evaluated a number of OTI's programs using external contractors. OTI is currently working on improving its program evaluation process.

Evidence: OTI website: www.usaid.gov/hum_response/oti/pubs.html. OTI Final Evaluation Protocol (work in progress)

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: While OTI has just developed its annual and long-term measures, its annual congressional budget justification ties its resource needs to its assessment of current programs and clearly breaks out OTI's funding requests and estimates by country. OTI reviews the progress, budget, and operations support of its on-going programs on a quarterly basis. Before intervening in a new country OTI uses set criteria to determine whether a new country program would be worthwhile. These new country assessments include budget estimates.

Evidence: USAID Budget Justification to the Congress FY 2004

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: OTI has adopted or is working to adopt long-term and annual performance goals, as well as baselines, targets, and timeframes. OTI develops a set of lessons learned and best practices from its monitoring and evaluation processes and uses these to improve and strengthen its programs. OTI's Annual retreats are used to identify strategic planning and management weaknesses and recommend corrective actions.

Evidence: See proposed measures. Guide to Performance Management, Guide to Program Options in Conflict-Prone Settings, OTI's Annual reports, Final Evaluations. Strategic Plan - Strategic Principles

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: OTI develops a performance monitoring plan for each country program and establishes baseline data. OTI field staff routinely monitor programs and gather data from implementing partners. OTI holds semi-annual strategic planning sessions with its staff and implementing partners to revise/update the strategy and make necessary adjustments. OTI's grants database allows Washington staff to check a program's progress and create a variety of standardized reports.

Evidence: OTI has revised/updated country strategies to respond to changing situations/new opportunities (e.g., Burundi & DRC). OTI meets with implementing partners on a regular basis to discuss strategic and management issues and to make changes as appropriate. Lessons learned from evaluations are disseminated to all OTI staff and included on the OTI website. OTI annual report identifies problems encountered and changes made.

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: OTI appoints a cognizant technical officer (CTO) for each program to manage contractors/grantees (i.e. review/approve vouchers, financial pipelines, sub-grants, etc.) Contractors/grantees are routinely audited and are registered with USAID.

Evidence: OTI staff prepare annual contractor performance reports (CPRs) that identify both strengths and weaknesses. These are used in reviewing/rating new contract proposals. CTO's and Office of Procurement staff conduct contractor reviews on a regular basis.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: OTI's innovative contracting mechanism, SWIFT, allows it to obligates funds in a timely manner. OTI incrementally funds programs to hold contractors/grantees accountable. OTI prepares and updates monthly, quarterly and annually a procurement plan to schedule all procurements and obligations and ensure that funds are being spent for their intended purpose.

Evidence: OTI financial reports and budget tables, SWIFT IQC. OTI procurement planning documents and the database are accessible on every OTI computer and are updated on a regular basis.

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: OTI uses an innovative contracting mechanism where a range of partners have been pre-competed because of expertise in working on transitions in difficult operating environments. All sub-grantees sign agreements that outline costs and responsibilities. In-kind grants are provided when sub-grantees do not have approved financial management systems. OTI leverages funds from other sources/donors to improve cost efficiencies (e.g., Afghanistan, Indonesia, and Macedonia) OTI country representatives are given the authority to approve sub-grants up to $100,000 in the field. Larger amounts must be approved at the USAID Washington offices.

Evidence: OTI's new SWIFT II mechanism is now at the Office of Procurement for final selection and negotiation with implementing partners. New requirements call for all contractors to have a grants under contracts manual in place and approved by OP before any small grants can be funded. OTI database tracks leveraged funds from other donors/sources.

Does the program collaborate and coordinate effectively with related programs?

Explanation: During design and implementation, OTI routinely coordinates with the USAID Mission and Embassy as well as USAID's Washington staff. However, there is room for increased cooperation between OTI and the State Department's Bureau of Democracy, Human Rights and Labor, whose Human Rights and Democracy Fund programs could complement OTI's programs, particularly in the advancement of democracy. Friction with U.S. Embassies and USAID missions at the beginning stages of a new country program has sometimes hurt the success of some OTI programs. However, OTI has made significant progress in improving cooperation with USAID missions and Embassies during the design and implementation of its new country programs. OTI is also establishing closer linkages with DCHA/DG and DCHA/CMM in strategic planning.

Evidence: Guide to Performance Management, Country Final Evaluations, CDIE Country Evaluations. MOUs with USAID Missions..

Does the program use strong financial management practices?

Explanation: While USAID has not deployed the new accounting system to its overseas Missions, OTI programs are managed in Washington and are therefore linked into the Phoenix system. However, the Phoenix system is not yet compliant with federal financial management system requirements. Program managers carefully review payment vouchers submitted by contractors. Contracts must submit financial pipelines prior to receiving incremental funding. OTI's contracts are routinely audited after they are closed out. SWIFT II contractors now required to have an approved Grants under Contracts manual in place prior to disbursing any funds to local partners to ensure financial management capacity.

Evidence: OTI budget reports on accruals and disbursements. OTI budget database, country program budgets. SWIFT II Request for Proposal

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Through on-the-ground fiel monitoring, mid-term evaluations, performance monitoring plans, and regular meetings with implementing partners, OTI makes necessary changes to address any management deficiencies.

Evidence: OTI staff prepare written correspondence to implementing partners when specific action/changes are necessary to address management issues (e.g., Afghanistan and Burundi). OTI has held after-action (post-closeout) reviews with contractors to discuss overall performance (e.g., Serbia).

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The majority of OTI's programs are awarded competitively under its Support Which Implements Fast Transition (SWIFT) contracting mechanism. SWIFT partners have been pre-competed and only those with proven track records in complex emergencies are considered.

Evidence: Depending on the nature of the program, a country program may fund 200 or more sub-grants annually. Sub-grantees that have performed well as are often provided with additional sub-grants as appropriate. OTI has 4 SWIFT partners (contract ends on September 30, 2003), and OTI anticipates that it will have 6-7 SWIFT II partners (estimated award date is August 2003).

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: OTI prepares a performance monitoring plan and maintains a data base of all sub-grants. Sub-grantees must sign agreements with our implementing partners and OTI field staff must clear. These agreements identify purpose of grant, inputs and outputs, timeframe, etc. Database includes info on each sub-grant, including type, location, budget, sector focus, etc.

Evidence: Country level PMPs. County data bases (updated every 2 to 4 weeks by the field). Country level monthly reports (on website)

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: OTI prepares monthly reports on its programs and maintains a database on all sub-grants. OTI maintains a website which provides monthly reports, hot topics and other relevant information on program activities.

Evidence: OTI website: www.usaid.gov.hum_response/OTI/

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: While independent evaluations show that OTI has made progress in achieving its long term performance goals on an individual country level, OTI is working to develop performance targets to better measure progress towards these goals.

Evidence: OTI country evaluations, OTI 2001-2002 Report, Annual Report, 1999 Results Review

Does the program (including program partners) achieve its annual performance goals?

Explanation: OTI has recently developed two annual performance measures that support the achievement of its goals in order to better measure its success. OTI is in the process of establishing baselines, and ambitious targets for its annual measures.

Evidence: See proposed measures.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: OTI has part-time consultants/contractors that are on call to fill OTI's short term staffing gaps. A newly awarded Program Development Quickly (PDQ) IQC enables OTI to efficiently contract for range of short term services, such as strategic planning, assessments, evaluations, training, document preparation, etc. A new SWIFT II IQC to be awarded in FY 2003 expands the pool of pre-competed partners with the necessary expertise to implement a wide range of activities in the difficult operating environments where OTI manages programs.

Evidence: The new Program Development Quickly (PDQ) contracting mechanism and the new SWIFT II contracting mechanism are examples of significant improvements to OTI's operating efficiency and cost effectiveness. Use of part-time, on call senior field advisors also provide surge capacity. OTI's goal over the life of a program is to have 70% of program funds go to grants/sub-grants and 30% of program funds go to non-grant costs (local staff, field operations, and headquarter costs/overhead).

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: OTI receives very favorable remarks from organizations and think tanks who prepare reports/analyses on the type of work that OTI does with political transitions in conflict-prone areas. Although the State Department and other USAID Bureaus carry out similar programs, OTI is unique in its ability to quickly implement their programs in difficult working environments.

Evidence: Quotes from International Crisis Group, U.S. Institute for Peace, journalists. Quotes from National Security Studies Quarterly (Summer 2001), Foreign Affairs Journal (July.Aug 2001), The Harvard International Review (Fall 2000). Quotes from OTI 2001-2002 Report, 1999 Results Review.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: Final country evaluations have generally indicated that OTI programs have been effective in achieving significant measurable results and making a considerable impact in improving political and security conditions in conflict-pone areas. CDIE conducted an evaluation of USAID's experience with transitions, and four OTI programs were examined (East Timor, Nigeria, Kosovo, and Indonesia). OTI is currently working to standardize its evaluation process with a new Evaluation Protocol.

Evidence: Final country evaluations, CDIE evaluation reports. (See OTI website, Publications list), OTI Final Evaluation Protocol (work in progress). The main finding of the CDIE evaluation on USAID's experience with transitions was that OTI programs had a positiobe impact in East Timor, Nigeria. Kosovo, and Indonesia.