Minutes of the U.S. Advisory Commission on Public Diplomacy January 2006 Official Meeting

Chairman Barbara M. Barrett
Commissioner Harold C. Pachios
Commissioner Jay T. Snyder
Commissioner Maria Sophia Aguirre
Commissioner Charles Evers
Commissioner Elizabeth Bagley
Commissioner Penne Korth Peacock
Athena Katsoulos, Executive Director
Leyla Strotkamp, Graduate Fellow

Subject: Progress of the New Public Diplomacy Evaluation Unit

Background: Rick A. Ruth, Director, Office of Policy and Evaluation, and Ted Kniker, Chief of Evaluation and Performance Measurement of the Bureau of Educational and Cultural Affairs, briefed the Commission on the progress made in evaluating public diplomacy programs, practices of program assessment, and measurement of the new public diplomacy evaluation unit.

Chairman Barrett brought the meeting to order.

Reminding the Commissioners of the four pillars of public diplomacy according to Under Secretary of Public Diplomacy and Public Affairs Karen Hughes (Engage, Exchange, Educate, and Empower), Mr. Ruth suggested that a "fifth E" might be added to that list: Evaluation. The Under Secretary’s commitment to performance measurement has taken shape in a new evaluation unit for public diplomacy. He noted that Under Secretary Hughes and Assistant Secretary of State for Educational and Cultural Affairs Dina Powell created this unit to consolidate evaluative functions of the public diplomacy bureaus—IIP, ECA, and PA—into one office. Performance measurement will become a standard, year-round process across all public diplomacy programs. This will assist senior management in making the best budgetary and programmatic decisions.

Mr. Ruth underscored that the current priorities of the new unit are twofold. First, it is working to take the best practices of performance measurement from the Bureau of Educational and Cultural Affairs and apply them to the evaluation of other public diplomacy offices. Second, the officers are building consensus on the guiding principles of how progress should be measured. At the moment, the emphasis is on developing tools for fast, actionable information.

Mr. Kniker noted that 95 percent of evaluative studies are contracted to outside agencies, with 20 different contractors from survey and marketing sector. Mapping studies evaluate the overall program model and document outcomes to determine where the model can be adjusted. Compass studies are more indicative of the general direction of a program by assessing it at different stages. The approach is applied to exchange programs through an online performance measurement system that surveys participants before, during, and after they complete their program. This is quantitatively assessed to determine the general impact of a program.

Chairman Barrett asked whether the surveys have found any trends regarding culture as a variable in how effective the programs are in a particular country. Mr. Kniker replied that because respondents are anonymous, that data is not available; however, the evaluations have seen that some participants struggle to implement what they have gained from the program once the return to the home country. Still, the studies show that over the long term, participants have a positive impression of the experience. The programs are overwhelmingly successful at meeting public diplomacy goals. Participants learn more about the United States, develop an appreciation for American life, and form enduring professional and personal relationships with Americans.

Commissioner Pachios wanted to know whether the surveys are of standard design and if they indicate trends in who is selected for exchange programs. Mr. Kniker said that the questions are similar, but addressed to the specific program under review. Mr. Ruth said that the exchange programs are highly merit-based, which creates a favorable impression of the United States as an equal opportunity nation.

In response to Commissioners’ inquiries, Mr. Ruth and Mr. Kniker discussed some findings on specific public diplomacy programs. Mr. Ruth cited the American Corners program as one that needs assessment because of its rapid expansion and high profile. The Arab-language magazine Hi! which suspended print publication in December but maintains an online presence, was generated under the specific circumstances of the post September 11 environment and, therefore, was launched without the usual preliminary market research, Ruth said. Subsequent evaluation has shown that, while it has appeal and a devoted online audience, Hi! must be targeted to a narrower demographic to have a significant impact. Undersecretary Hughes decided to suspend the print version until it can be further evaluated.

Mr. Kniker reported the positive effects of evaluations conducted on the Youth Exchange and Study (YES) and Partnerships For Learning—Undergraduate Studies (PLUS) programs, both of which bring foreign students with strong academic performance and leadership potential to the United States. In response to a first-year evaluation showing that participants were predominantly urban elites, the programs made a concerted effort to attract youth from other demographics.

Commissioner Snyder asked whether the evaluation unit had determined that any programs might need to be modified or removed. Mr. Ruth said that while they are not tasked with making such decisions, they have not yet encountered any obviously failed programs. However they do discover and note unexpected outcomes. To Commissioner Aguirre’s inquiries about the timing of reports, Mr. Ruth and Mr. Kniker responded that they address medium-and long-term goals by developing rapid response reports in addition to more long-ranging studies that are done every ten years. As an example, Mr. Ruth discussed the unit’s exploration of social network analysis to understand the impact of the Fulbright exchange program. This entails tracking a Fulbrighter’s foreign contacts to see the scope of his or her influence.

Chairman Barrett noted that Commission’s task to review public diplomacy programs is somewhat analogous to this new evaluation unit, and asked how the Commission could contribute to the latter’s efforts. Mr. Ruth suggested that Commissioners use the bully pulpit when undertaking fact-finding missions or speaking in public. By mentioning the importance of program assessment and accountability of these programs to the public interest, Commissioners can promote a culture of evaluation for public affairs and public diplomacy throughout the Foreign Service corps.