Detailed Information on the
Global Educational and Cultural Exchanges Assessment

Initiate independent evaluation of ECA Journalism programs.

Integrate evaluation function between ECA, IIP and R/PPR

Assess Key Youth Program

Review of Alumni Coordination

Measure: Ratio of Administrative Costs in relation to Program Costs (Administrative Efficiency)

Explanation:The efficiency measure assures that the highest percentage of funds go to direct program costs and program beneficiaries.

Measure: Percentage of exchange participants who report a more favorable view of the people of the United States within one year after their exchange experience.

Explanation:Favorable view of the United States is one indicator of increased trust and willingness to maintain relations with the United States.

Measure: Percentage of exchange participants who initiate or implement a positive change in their organization or community within five years of their exchange, based on knowledge gained from their exchange.

Explanation:Measure is an aggregate of multiple types of programs. Positive change includes items such as creating an NGO, revising or creating curriculum, drafting legislation, changing business practices, etc.

Measure: Percentage of participants who establish or continue professional collaborations more than five years after their exchange experience.

Explanation:Eighty percentage of participants will continue professional collaborations with people in the United States more than five years after their exchange experience.

Measure: The number of foreign exchange participants by region.

Explanation:Increase the number of foreign exchange participants in key regions, to reflect U.S. foreign policy objectives, commensurate with funding. Baseline AF;1,377; ERA/NIS 6,926; EAP 2,163; NEA 1,238;SA 706;WHA2,233; Total 25,183

Measure: Percentage of exchange participants who express satisfaction with the exchange experience based on an average of several program factors: 1) administration; 2) content quality; 3) range of experiences; and 4) relevance of program to professional or academic field.

Explanation:Ninety-five percent of exchange participants will remain satisfied with their exchange experience up to one year after the experience.

Measure: Percentage of participants who increased or changed their understanding of the United States immediately following their program.

Explanation:Ninety-two percent of participants will increase or change their understanding of the United States immediately following their program.

Measure: The number of foreign exchange participants specifically in the Near East Asia and South Asia region, reflecting current US foreign policy objectives commensurate with funding.

Explanation:This measure tracks the total number of participants in the Near East Asia and South Asia region. These participants are also included in the total number of participants measured above, but are broken out here to provide detail to a key region.

Is the program purpose clear?

Explanation: The purpose of educational and cultural exchange programs is to strengthen the international relations of the United States and develop peaceful, friendly and sympathetic relations between the people of the United States and other countries.

Evidence: 1. Fulbright-Hays Act; 2. Smith-Mundt Act; 3. Presidential Memorandum on International Education Policy dated April 19, 2000; 4. FY 2005 Performance Plan; 5. Partnerships for Peace and Freedom Brochure; 6. ECA Web Site www.exchanges.state.gov; 7. 2003 OIG inspection report BPP Evidence: A/S Statement: Goal Papers: PD.02 Goal Papers

Does the program address a specific and existing problem, interest or need?

Explanation: Exchanges are a critical component of U.S. diplomacy abroad directed at improving understanding of and support for U.S. policy, encouraging and empowering moderates, and discrediting intolerance. For example, exchanges offer a fuller picture of how religious, ethnic, and cultural tolerance are keys to a peaceful and prosperous community of nations, and open access to information that provides alternatives to learning extremism. Through ECA programs, men and women from all walks of life--journalists, students, religious leades, government officials--experience for themselves how democracy works and how they can be instruments of change within their own societies helping others realize their birthright of freedom. ECA exchanges create an expanding network of likeminded achievers around the world whose common bond is a belief in freedom and democracy.

Evidence: 4; 5; 8. public diplomacy website -www.publicdiplomacy.org -Multiple independent reports from variety of sources); 9. Pew Charitable Trust reports on attitudes toward the U.S. 10. Soft Power by Joseph Nye BPP Evidence: A/S Statement: Goal Papers: PD.02 Goal Papers; Access to Quality Education Goal Papers;

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: ECA is the only part of State that is charged to foster "mutual understanding" between the United States and other countries. ECA works primarily through the exchange of persons, not assistance programs. ECA is the only part of the Department mandated to represent the USG abroad in the fields of international education and culture. Studies by the Interagency Working Group on International Education and Training (IAWG), which includes representation by all government agencies, including State, who sponsor exchange programs, conclude there is no duplication with ECA exchanges by other Federal organizations.

Evidence: 1; 2; 11. Interagency Working Group on International Education and Training (IAWG) annual reports and specialized reports (www.iawg.gov). BPP Evidence: A/S Statement:

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Given limited funding and expanding resource requirements,ECA must ensure that all programs are efficient and meeting targets effectively in order to address all program goals and needs. In its '03 inspection report,the OIG stated,"...the United States gets high value from the effective way ECA is managing exchange programs."Through continuous grant monitoring and independent program evaluation,ECA ensures that its exchange activities are free of major flaws.For its new programming efforts in NEA, SA, EAP and AF, ECA tailored successful program models (FLEX to YES; FSA Undergrad to PLUS).To improve administrative efficiency,FSA and SEED programs were consolidated into the exchanges account In 03, ECA contracted with Economic Systems Inc.,to determine if the Traditional Public-Private Partnership Program (TPPP) still met non-competition status or if a new model was needed.The report concluded the existing model was the most effective.In 04,ECA is evaluating the direct "swap" model of the Fulbright Teacher Exchange to determine if it more or less effective than one way exchange programs.

Evidence: 7; 12. TPPP Competition Evaluation Report; 13. 2004 OMB Passback Documents; 14. Fulbright Teacher Exchange Evaluation Design BPP Evidence: A/S Statement:

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The ECE budget funds approximately 25,000 exchange participants per year. Target audiences are determined at the sub-program and project levels and participants are selected through a merit-based competition process or nomination and vetting process. ECA uses its efficiency measures to ensure that administrative costs are kept to a reasonable level to ensure majority of funding is used for participants and program costs. Generally, 75 to 80% of funds are used for direct program costs. Funding is provided to non-profit partner organizations responsible for supplying per diem, travel and other expenses directly for the participants. Grant organizations are audited to ensure proper fiduciary compliance with OMB standards. Funding levels are determined based on participant numbers and level of program activity. Funding and targeting data is maintained in Exchanges Statistical Management System database. Funds cannot be substituted for other activities.

Evidence: 15. ESMS database; 16. E-Grants database; 17. Grants process and procedures (grants handbook); 18. Grant agreements BPP Evidence: A/S Statement: Goal Papers:

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: ECA has partnered with a variety of performance measurement experts and adopted private sector practices to measure its long term goals. Measures include: a) percentage of participants who increase participation or responsibility in work, community or civil society within five years after the exchange experience; b) percent of participants who initiate or implement a positive change in their organization or community within five years of their exchange, based on knowledge gained from their exchange; c) percent who collaborate professionally with those met on their program more than five years after the exchange; and d) percent of participants who continue formal, sustainable institutional relationships (relationships would continue if participants left institutions) five or more years after the exchange. ECA measures goal attainment through independent program evaluation and US Embassy reporting.

Evidence: 4; 19. ECA logic model; 20. ECA Intermediate Outcomes List; 21. Standardized Questions List; 22. E-GOALS system documentation; 23. 2003 Performance and Accountability Report BPP Evidence: A/S Statement: Goal Papers: PD.02 Goal Papers

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: ECA has targets and timeframes set to established baselines and is improving its accuracy of measurement through online performance measurement.

Evidence: 4; 19; 20; 23 BPP Evidence: Goal Papers: PD.02 Goal Papers

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: ECA has annual perf. goals that demonstrate progress towards the overarching strategic goals.Annual perf. goals include:(1) the annual number of participants in key regions;(2)% of participants who increase or change their understanding of the host country immediately following their prog. experience measured through independent evaluation surveying;(3)% of private sector, foreign,&other USG funds generated for prog. use on an annual basis. Key program targeting (audience, program mix) happens at the sub-program level and by region. For example, youth and non-elites are the primary target for NEA, SA and EAP. Number of participants determines scope of program, and cost-sharing increases number of participants, size of program, quality of delivery. ECA measures goal attainment through grant reporting, on-line performance measurement system, reporting from US Embassies, and independent program evaluation. Annual data is also captured in Exchange Statistical Management System (ESMS).

Evidence: 4; 11; 15; 19; 20; 22; 23 BPP Evidence: A/S Statement: Goal Papers: PD.02 Goal Papers Other Goal Papers

Does the program have baselines and ambitious targets for its annual measures?

Explanation: ECA has targets and timeframes set to established baselines and is improving its accuracy of measurement through online performance measurement.

Evidence: 4; 11; 15; 19; 20; 22; 23 BPP Evidence: A/S Statement: Goal Papers:

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The grants process ensures the commitment of partners to the annual and LT goals of the prog.Prior to any solicitation, Bureau prog offices work with the reg. bureaus and PA Staff at US Embassies to determine the scope and need for progs/projs.These discussions are then codified in the RFGP.When an applicant submits a proposal in response to an RFGP, the applicant must demonstrate how it will attain the goals for the prog, if awarded funding.The applicant also must submit an eval plan to demo how it will measure its perf. and prog. toward the annual and LT goals.The proposals are reviewed by US embassy staff and reg.bureau staff as well as by the prog staff.PAS staff at US embassies submit comments about proposals to the prog office.Regional bureaus attend the grant review panel.The vetting process for grants also includes review by the Assistant Secretary and a 15-day review by Capitol Hill.The prog office monitors the implementation of the grant through weekly or monthly phone calls.

Evidence: 17; 18; 21; 24. RFGPs; 25. RFGP evaluation language; 26. Grantee reports

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: ECA uses independent professional evaluators, selected through a competitive process, to assess the impact and outcomes of its programs. A multi-year program evaluation plan is developed every three years and adjusted as necessary to ensure that each year an academic, professional, and citizen exchange program is evaluated.All major funding line items have been evaluated at least once and will be re-evaluated every five to seven years. Where feasible, ECA has used comparison groups to more accurately assess the impact of the particular program. ECA also requires grantee organizations to submit evaluation plans for each grant.Some grantee orgs hire independent evaluators as well.While the evaluations focus on the "brand name" program activity, all evaluations are done in country-specific context. Recent evaluations include: Community Connections, Freedom Support Act Undergrad, English Language Programs, and MEPI Student Leaders

Evidence: 23; 25; 27. ECA evaluation webpage; 28. ECA Paperwork Reducation Act Generic Clearance Request; 29. Evaluation Statement of Work; 30. Evaluation Schedule 31. Grantee evaluation proposal 32. Evaluation Reports (selected examples - US Fulbright Scholar, FSA Undergrad English Language Programs, ACYPL) BPP Evidence: A/S Statement: Goal Papers:

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget requests directly tie to the accomplishment of annual goals and measures, particularly participant numbers.Funding or policy changes directly affect goal accomplishment.For example, without receiving current services, the Bureau is loses approximately 1000 participants.Another example is the Bureau is trying to engage 50,000 alumni through its alumni web site by 2006, raised from the current number of 8,400.W/o funding for expansion of content development and outreach, the Bureau will not attain this goal. All costs are accounted for in ECA's budget presentation, which includes program costs and exchanges support.For Congressional purposes,the budget is presented based on sub-program line item;however,the budget can be broken out to show the full costs for achieving goals by region and strategic goal.The budget presentation also notes key indicators.

Evidence: 33. Congressional Justification Document BPP Evidence: A/S Statement:

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The OIG report notes the effective collaboration between ECA and posts in program planning as well as the use of evaluation data for strategic planning. ECA revised and sent planning guidance to the regional bureaus and posts for the MPP process and is collaborating through meetings and a resource request template to develop plans for 2006. Many of the changes result from ECA's application of the recommendations from OMB in its earlier PART for exchanges in NEA and SA to improve its strategic planning for all programs and regions. ECA is working with regional bureaus to develop a common list of critical institutions within regions for even more refined targeted programming.

Evidence: 4; 34. Resource request template; 35. 2006 MPP guidance cable; 36. Critical institution email BPP Evidence: A/S Statement: Goal Papers: Exchanges in NEA and SA (PART) Goal Paper - Milestones

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: ECA collects performance data through partner organization reporting, reports from U.S. embassies, and independent evaluations. Reporting from and surveying of participants and partners provides information for ECA managers to make program adjustments and improvements. Examples: International Vistors use a rating system of partner organizations, if organization is lowest on the list for two consecutive years, they can not bid on new grants. The IV Program used this data recently to work with a partner organization to improve its programming. The FSA Undergraduate evaluation demonstrated the need to change how the partners recruited and worked with community colleges as part of the program. Now the schools are screened differently and program quality and satisfaction have increased. Recommendations from evaluations and actions taken are reported annually in the Department's Performance Report.

Evidence: 23; 37. Grant reporting requirements survey matrix; Examples of the extensive performance information collection include: 38. International Visitor Program: National Program Agency Checklist; 39. International Visitor Program: Escort Interpreter Report; 40. International Visitor Program: End of Project Survey; 41. International Visitor Program: Results Cable; 42. Fulbright Program: Participant Final Report; 43. Academic Programs: Final Grant Organization Report; 44. ACIE: Alumni Discovery Newsletter; 45. Citizen Exchange: End of Project Report; 46. Youth Exchange: Survey; 47. Youth Exchange: Bradley Herald; 48. Embassy Program comments; 49. FLEX Reporting Document List BPP Evidence: Goal Papers: PD.02 Goal Papers

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Performance benchmarks for project/program managers were established in 1997. Managers must meet the benchmarks before being promoted. In the grant proposal review process, program managers must explain the performance of a program by the program partner in order to defend the proposal and secure new funding. All grant applications are reviewed for past performance or track record. Renewals are not allowed unless past performance has been demonstrated to be efficient and effective. In grant renewal cases, the proposal defense is put before a panel of senior bureau managers, who must determine as part of the decision-making process whether the program has encountered cost, schedule or performance results. All program partner organizations must submit program progress and financial statements. In order to provide a system of independent checking of grant performance, the program reports and the financial reports are sent both to the program manager and the grants/contracting staff

Evidence: 17; 24; 50. Program Manager benchmarks; 51. Grant Proposal Analysis Memo; 52. Panel Review Minutes; 53. Financial Statement Form; 54. Grant Closeout Document

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The Exchanges apporpriation is no-year money.The Bureau obligates 99% of the account annually for the intended purpose of its programs.Congress frequently directs the Bureau to expend unrestricted balances to the maximum extent possible on otherwise non-funded items in bills and conference reports. In terms of managing its funds to ensure that obligations are timely and spent for intended purposes, the Exchanges Bureau uses the A-133 audit process to audit grantees.ECA uses the HHS payment system in order to review funds before they are drawn down, and when there are questions about an organization's spending, the Bureau puts them on a reimbursable payment schedule.ECA's budget office distributes monthly status of funds reports to all middle and senior managers in the bureau.

Evidence: 53; 55. Status of funds report; 56. Grantee Audit Report

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: ECA uses efficiency measures in its review of grant proposals, including cost-per-participant, the ratio of administration to program funds, and the ratio of cost-sharing to public funds. The grants coordination office uses measures on how timely its processing of grants is and has demonstrated improvement over last year. ECA is establishing an IT strategy and plan. It is refocusing websites to further its long-term goals to promote mutual understanding, and where appropriate to use websites and Internet connectivity to create linkages between US and foreign individuals and institutions, which allows the bureau to reach more people. ECA is using technology to reach an increasing number of alumni to keep them engaged with the U.S., and is delivering Academic Advising through an interactive website. IV partner web site to share best practices. Ariba buying system used for all purchases & contracts.

Evidence: 24; 51; 57. Grants office newsletter; 58. IT Business Plan Phase 1 Report; 59. ECA Technology Prioritization Committee Documents; 60. AMIDEAST Virtual Advising Documents; 61. ECA website strategy; 62. IV Partner website documents; 63. Ariba screenshots/information BPP Evidence: Goal Papers: Management Goal Papers.

Does the program collaborate and coordinate effectively with related programs?

Explanation: Nearly 100 percent of ECA's work is conducted through collaboration with over 100 non-profit partner organizations and 200 colleges and universities. Collaboration is managed and documented through cooperative assistance agreements. In an evaluation of the Traditional Public-Private Partnership Program, an independent contractor researched new "market opportunities" and competitors for professional exchange programs. All the selected organizations that met the criteria were already grantee partners for ECA. No other Federal or private exchange program shares the same primary purposes to promote mutual understanding, or aim to influence foreign publics regarding U.S. policies and interests as ECA exchanges. ECA coordinates its efforts with other federal organizations through joint policy committees and the Interagency Working Group on U.S. Government-Sponsored International Exchanges and Training (IAWG).

Evidence: 11; 12; 18; 24; 31; 64. MEPI Project Scope Document; 65. MEPI Evaluation Report BPP Evidence: A/S Statement: Goal Papers:

Does the program use strong financial management practices?

Explanation: The Bureau is now part of the Department's systems and submitted itself to an independent review of its financial practices in 1999.The accountant identified no fundamental weaknesses in the Bureau's financial management practices. The Bureau utilizes a number of means to ensure strong financial management practices. It has developed its cuff record system to check obligations against commitments. In addition, the Bureau employs a financial analyst who checks line-by-line obligations and commitments to ensure quality payment control. The bureau now uses Ariba buying system to prepare, track, and report on purchase orders and contracts. The 2003 OIG report found no material weaknesses with respect to financial management.

Evidence: 7; 63; 66. Financial review report (submitted to OMB 2002); 67. CMFS screen shots

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The 2003 OIG report found no material weaknesses. With its merger into the State Department in 1999, ECA has addressed a number of deficiencies identified earlier, including incompatible financial tracking systems (ECA is now on the Department's system), and has expanded its Exchanges Database (EDS) to include all program elements. EDS tracks exchange projects and participants from beginning to end. ECA also has an integrated E-Grants database tracking solicitations, proposals and grant awards, which prevents duplicate tracking of programs. The Executive Information System, which is a reporting tool that covers the EDS, has a null data feature that allows managers to know where data is missing or incorrect. A report using this feature is printed out each month.

Evidence: 7; 16; 67; 68. EIS screen shots; 69. EDS architecture description BPP Evidence: Goal Papers: Management goal papers

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: One hundred percent of ECA's grant awards go through a merit-based review process. ECA annually reviews approximately 1,000 grant proposals and awards approximately 335 new grants. Approximately 10 percent are awarded to first time grantees. Competitions are announced publicly. ECA's E-Grants system is being considered as a model for the rest of the Department. ECA grants are also announced through the grants.gov .

Evidence: 16; 17; 24; 51; 52

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: ECA has well-established reporting and tracking features in place. For financial tracking, obligations and modifications to grants are tracked through CFMS. Grantee organizations submit regular financial reports, which are reviewed by the program office and grants coordination office. The Bureau requires detailed interim and final reports from all grantees and conducts site visits by program managers as well as regular email and telephone communications between program managers and grantees. One hundred percent of ECA IV participants on group projects are debriefed through a standardized questionnaire, 100% of English specialists submit online reports using a standardized form. Nearly 100% of Fulbrighters submit final reports which are standardized. Grantee organizations are also required through the grants process to conduct an evaluation of all of the grant activities and results and submit an analysis to ECA. Grantees are subject to A-133 audits.

Evidence: 16; 17; 18; 26; 37; 43; 45; 53; 54; 56; 67; 70. English Language Reporting Form

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: ECA regularly collects program and project performance data and makes it available to the public on its evaluation web page and via list serves as well as in its print and online newsletter, The Exchange. Grantee perf is contained within the eval reports.ECA encourages grantees to publicly post their results information.Most grantees including AMIDEAST and IREX post their results information on their website. IIE's web site (www.iie.org) contains"Notes from the Field" which allows Fulbrighters to post their experiences online. ECA also receives results information directly from participants via the In Their Own Words web site, where participants describe the results of their experience.

Evidence: 6; 22; 26; 27; 44; 47; 71. AMIDEAST web site: www.amideast.org,; 72. IREX web site: www.irex.org, 73. The Exchange Newsletter, 74. RESULTS database; 75. IIE website: www.iie.org; 76. In Their Own Words web site: speakout.state.gov; 77. CultureConnect website: cultureconnect.state.gov/

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Although it may take several years for a particular exchange experience to demonstrate results, ECA through its use of private sector methodology is able to demonstrate progress toward its long-term goals. Independent evaluations demonstrate that ECA meets its goals with its target audiences and within different regions. ECA's performance measurement demonstrates it meets its long-term goals and targets. The OIG report confirms that the American taxpayer gets high value from exchange programs and their effective management.

Evidence: 4; 6; 7; 27; BPP Evidence: A/S Statement: Goal Papers: PD.02 Goal Papers

Does the program (including program partners) achieve its annual performance goals?

Explanation: ECA cannot achieve its annual goals without its partners and has demonstrated that it has either met or exceeded its annual goals.

Evidence: 4; 6; 22; 26; 27; 44; 47; 71; 72; 73; 74; 75; 76; 77 BPP Evidence: A/S Statement: Goal Papers: PD.02 Goal Papers

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: ECA measures its efficiency at the sub-program level by reviewing Cost-per-Participant, Total Administrative to Program Ratio (Administrative Efficiency) and the time it takes to process its grants. ECA is meeting its efficiency goals and working to improve efficiency in these areas.

Evidence: 4; 16; 51; BPP Evidence: Goal Papers: PD.02 Goal Papers

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The Interagency Working Group for International Exchanges and Training (IAWG) in are report on performance measurement determined that it is too difficult to set common performance measures for exchange programs across the USG. The purposes of exchanges administered through private sources do not have the same goals. On the international front, ECA is conducting a joint evaluation with the British Council and Government to measure the effectiveness of teacher exchanges, which will be used to to look at comparability as well as feed into each organization's strategic planning. ECA is also consulting and advising Italian and Japanese exchange programs on how to measure effectiveness and has extended requests to Australia, Canada and Germany for like comparisons.

Evidence: 78. IAWG Performance Measures report; 79. Fulbright Teacher Exchange Evaluation SOW; 80. British Council Terms of Reference for Fulbright Teacher Exchange Evaluation

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: 21completed independent evaluations and 5 internal management reports of functional programs worldwide conducted by 15 external evaluation organizations have concluded that ECA's exchange programs are effectively meeting goals and administered well. ECA currently has four other evaluations in draft, 10 on-going projects, and four additional projects to be launched in 2004.

Evidence: 27; 30; 32; 81. Evaluation Project List BPP Evidence: A/S Statement: Goal Papers: PD.02 Goal Papers