Detailed Information on the
Broadcasting in Arabic Assessment

Implementing a new long-term outcome performance measure that assesses the credibility of BBG radio and television programs in Arabic, since audiences will regularly tune in to broadcasts they can trust.

Designing and implementing more long-term performance measures that align w/BBGs mission to promote freedom and democracy by broadcasting objective news about the US and the world to Arabic-speaking audiences

Revising BBGs strategic plan to build on a market-based approach to broadcasting to reach audiences in their preferred media w/ programming tailored to their needs, and to support US strategic interests.

Measure: Audience Reach for Broadcasting in Arabic (millions of listeners/viewers). Note: the statistical methodology for computing the audience reach changed in 2007, resulting in a new baseline starting that year.

Explanation:A key indicator of whether or not BBG broadcasts are reaching their intended audience is the number of people who choose to listen to or view BBG programs. Audience reach, also called weekly audience, is a measure of the regular listening/viewing audience. Regular audience is defined as all adults in a given population listening/viewing at least once a week, as determined by an audience survey that has an adequately designed sample. The number is a reflection of several annual common measures, such as program quality and the audience's awareness of the program, used for all BBG language services broadcasting in Arabic. Annual measures of audience size, as well as annual performance goals for each individual service, factor into the overall audience reach for broadcasting in Arabic.

Measure: Audience Reach for Alhurra television (millions of viewers). Note: the statistical methodology for computing the audience reach changed in 2007, resulting in a new baseline starting that year.

Explanation:A key indicator of whether or not BBG broadcasts are reaching their intended audience is the number of people who choose to listen to or view BBG programs. Audience reach, also called weekly audience, is a measure of the regular listening/viewing audience. Regular audience is defined as all adults in a given population listening/viewing at least once a week, as determined by an audience survey that has an adequately designed sample. Annual measures of the number of people tuning in to BBG broadcasts in Arabic in their particular country or region are tracked, and directly support the long-term measure of growing the overall size of the audience. The data shown here is specific to Alhurra television.

Measure: Audience Rating of Broadcast Credibility for Radio Sawa (%)

Explanation:An important outcome measure of the external effect of BBG programs is whether or not the audience finds the broadcasts to be reliable and credible, and thereby regularly tunes in and hears the news and information BBG broadcasts. This measure is an industry research standard used to determine the audience's view of the effectiveness of media. This indicator is determined by the survey question about "trustworthiness of news and information" of those sampled respondents who listened at least once a week to each station. The answers are registered on a five-point scale -- very trustworthy, somewhat trustworthy, neither trustworthy nor untrustworthy, somewhat untrustworthy, or very untrustworthy. The credibility index is the percent of those answering the question in the survey (excluding those who did not respond or did not know) that endorsed very or somewhat trustworthy. As the percentage moves towards 100%, increases are more difficult to attain, thus targets are significantly ambitious when the measure is maintained or shows a marginal increase. The Broadcasting in Arabic long-term goal is that each language service broadcasting in Arabic sustains a credibility level above 60% and incrementally reaches a significant goal of 78-80%. The data shown here is for Radio Sawa.

Measure: Audience Rating of Broadcast Credibility for Alhurra Television (%). Note: In 2008, there was a slight decline from 2007, but Alhurra's credibility remains at a high level for the entire Arabic speaking region.

Explanation:An important outcome measure of the external effect of BBG programs is whether or not the audience finds the broadcasts to be reliable and credible, and thereby regularly tunes in and hears the news and information BBG broadcasts. This measure is an industry research standard used to determine the audience's view of the effectiveness of media. This indicator is determined by the survey question about "trustworthiness of news and information" of those sampled respondents who listened at least once a week to each station. The answers are registered on a five-point scale -- very trustworthy, somewhat trustworthy, neither trustworthy nor untrustworthy, somewhat untrustworthy, or very untrustworthy. The credibility index is the percent of those answering the question in the survey (excluding those who did not respond or did not know) that endorsed very or somewhat trustworthy. As the percentage moves towards 100%, increases are more difficult to attain, thus targets are significantly ambitious when the measure is maintained or shows a marginal increase. The Broadcasting in Arabic long-term goal is that each languages service broadcasting in Arabic sustains a credibility level above 60% and incrementally reaches a significant goal of 78-80%. The data shown here is for Alhurra Television.

Measure: Program quality for Broadcasting in Arabic to Iraq (Radio Free Iraq). Program quality is measured and scored on a range of 1 to 4; 4 is the highest score, 2.7 - 3.3 is considered "good".

Explanation:The program quality score is an outcome measure based in part on how the target population perceives the quality of the specific BBG language services. This measure is an industry standard used to determine the target audience's assessment of the quality of media. A monitoring panel of regular listeners from the target population rates the program's accuracy, reliability, authoritativeness, objectivity, comprehensiveness, and other variables reflecting distinct statutory, policy, and mission mandates for the different stations. Another component of the program quality score is the evaluation of IBB delivery systems (engineering and transmission), marketing and program placement efforts, and other broadcasting support functions. Program content and presentation are evaluated by both internal and external reviewers to determine the overall Program Quality Score. In total, the Program Quality Score is a comprehensive, weighted score, which ranges from 1-4, with 1.0-1.3 = poor; 1.4-1.6 = poor to fair; 1.7-2.3 = fair; 2.4-2.6 = fair to good; 2.7-3.3 = good; 3.4-3.6 = good to excellent; 3.7-4.0 = excellent. Within the range, small variations are common, but maintaining the rating or progressing towards the next rating is significantly ambitious. Annual measures of the perceived quality of BBG programming are tracked by each language service, and directly support the long-term measures of growing the size of the audience through broadcasting in Arabic, and increasing the overall perceived reliability of the programming. The data shown here is specific to RFE/RL's Radio Free Iraq.

Measure: Cost-per-viewer for Alhurra Television. Note: Alhurra began broadcasting in 2004, with start-up costs and building audiences.

Explanation:This efficiency measure tracks the cost per audience member of each language service. Cost per audience member is the total cost of programming and delivery divided by the unduplicated audience for all media, and is calculated for each language service. The ratio (measure) must be evaluated in context of the changes in either the cost or audience size. For example, a short-term capital cost increase to improve transmission may not increase audience size until after a lengthy installation. In some cases, audience size may be kept artificially low by restrictions imposed by the host government or increased competition, not necessarily poor programming or inefficient management. The data shown here is specific to Alhurra Television.

Is the program purpose clear?

Explanation: The program purpose is to promote freedom and democracy and to enhance understanding through multi-media communication of accurate, objective, and balanced news, information, and other programming about America and the world to audiences overseas -- in this case, Arabic-speaking audiences by the Middle East Broadcasting Networks (MBN) and Radio Free Europe/Radio Liberty (RFE/RL).

Evidence: BBG Strategic Plan (2008-2013); U.S. International Broadcasting Act of 1994; Foreign Affairs Reform and Restructuring Act of 1998; MBN and RFE/RL Mission Statements

Does the program address a specific and existing problem, interest, or need?

Explanation: The program addresses the need in the Middle East, North Africa, and Europe for accurate news and information in Arabic about the United States and events in the region. In addition to countries across Europe, the broadcast area includes 22 countries in North Africa, the Near East and Persian Gulf. Political instability, the dearth of free, unbiased media and the existence of strong anti-Americanism, particularly in the Middle East and North Africa, create a need for timely and accurate reporting of U.S. policies and actions as well as events in the region and the world at large. To address this need, in 2008 the BBG has three television streams (Alhurra, Alhurra-Iraq, and Alhurra Europe) and two radio services, Radio Sawa (which combines news and popular music on seven regional programming streams) and Radio Free Iraq.

Evidence: The 2007 Freedom House Press rating is "Not Free" for 16 countries in the Middle East and North Africa and "Partly Free" for 4 countries in the Middle East and North Africa; BBG FY 2009 Budget Justification to Congress

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The Broadcasting Board of Governors (BBG) is the only entity responsible for all U.S. government and government-sponsored, non-military international broadcasting. As such, its broadcasts are not duplicative of other Federal, state, or local contributions to the problems being addressed in the Middle East and North Africa. Although other media outlets broadcast to this region, U.S. international broadcasting (MBN and RFE/RL) provides access to information about U.S. interests and policies and can be relied upon to offer a distinctly American perspective on events. The mission of BBG broadcasting ("to promote freedom and democracy and to enhance understanding through multi-media communication of accurate, objective, and balanced news, information, and other programming about America and the world to audiences overseas") distinguishes it from commercial media organizations. The BBG is the sole non-military international broadcasting entity that serves as a critical arm of the U.S. public diplomacy mission.

Evidence: The U.S. International Broadcasting Act of 1994 and the Foreign Affairs Reform and Restructuring Act of 1998 state that U.S. international broadcasting should not duplicate the activities of private United States broadcasters or the activities of government-supported broadcasting entities of other democratic nations.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: There is no conclusive evidence that another approach would be more efficient or effective at achieving the program's goals. As detailed below, independent studies have identified areas for improvement, and BBG has worked to address them. However, none of these studies have stated that the programs are inherently or significantly flawed. As reported in the July 2003 GAO report "New Strategic Approach Focuses on Reaching Large Audiences but Lacks Measurable Program Objectives" (GAO-03-772), "The diversity of the BBG - organizations with different missions, different frameworks, and different constituencies - makes it hard to bring all the separate parts together in a more effective whole." The Broadcasting Board of Governors (BBG) also identified, during its own strategic planning process, the internal challenge of consolidating and rationalizing the overall enterprise. In follow-up testimony to Congress (GAO-04-711T) the GAO found that "The Broadcasting Board of Governors has responded to a disparate organizational structure by developing a new strategic approach to broadcasting" and "To streamline its operations, the Board has used its annual language service review process to address such issues as how resources should be allocated among language services on the basis of their priority and impact, what degree of overlap should exist among services, and whether services should be eliminated because they have fulfilled their broadcast mission." The GAO Report "Management of Middle East Broadcasting Services Could Be Improved" (GAO-06-762) found that "MBN has developed financial and administrative controls to manage and safeguard its financial resources," "MBN has procedures in place to help ensure its programming meets its journalistic standards," and "the BBG has developed several performance indicators and targets for MBN's Radio Sawa and Alhurra services, including measures of audience size and program credibility." It recommended several actions for the BBG and MBN to strengthen their strategic planning, internal control, adherence to journalistic standards, and statistical methodologies, all of which are currently underway. More recent GAO Testimony (GAO-07-795T ) reported on this progress, stating "The BBG subsequently created a single strategic goal to focus on the key objective of maximizing impact in priority areas of interest to the United States and made audience size a key performance measure," and "MBN has taken steps to improve its process of program review and evaluation." The BBG has developed measurable program objectives that are integrated with the budget by linking performance with resources and setting performance targets for requested enhancements. In addition, the BBG has annual and long-term performance measures that track the success of each individual language service in supporting the agency's overarching goal of promoting freedom and democracy and enhancing understanding about the United States and the world to audiences overseas. By providing objective and accurate news and information, the BBG plays an important part in the broad support of democracy and the open exchange of news and ideas in countries and regions of the world where, because of geographic, developmental, or political reasons, there is a dearth of free and open information. U.S. international broadcasting by the BBG provides an alternative voice in regions where free speech is stifled, and it empowers and educates oppressed populations about freedom, democracy, and the world.

Evidence: GAO Report: "New Strategic Approach Focuses on Reaching Large Audiences but Lacks Measurable Program Objectives" (GAO-03-772); GAO Testimony: "U.S. International Broadcasting: Challenges Facing the Broadcasting Board of Governors" (GAO-04-711T); GAO Report: "U.S. International Broadcasting: Management of Middle East Broadcasting Services Could Be Improved" (GAO-06-762); GAO Testimony: "Strategic Planning Efforts Have Improved, but Agencies Face Significant Implementation Challenges" (GAO-07-795T); FY 2006, 2007 and FY 2008 Congressional Budget Requests; Language Service and Program Reviews; Statistical Methodology update.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The BBG promotes freedom and democracy and enhances understanding through multi-media communication of accurate, objective, and balanced news, information, and other programming about America and the world to audiences overseas. To support this mission, the BBG must ensure that audiences are actually tuning in, are receiving the broadcasts in their preferred language and media, and are confident that the message is credible and reliable. The 2008-2013 BBG Strategic Plan includes implementation strategies that direct efforts and resources to priority areas of U.S. interests, including "Build on our reach and impact within the Muslim world" and "Help audiences in authoritarian countries understand the principles and practices of democratic, free, and just societies." To assure that the programs are designed effectively and that resources are directly supporting the mission, the BBG uses the annual Language Service Review and Program Review processes to conduct audience and market research that determines whether or not the target audiences are being reached and recommends actions to increase the effectiveness of the program when the research indicates a need to do so. This information supports the BBG Language Service Review process in determining the strategic priorities and resource allocations across all BBG audiences. The BBG's Language Service Review focuses on: 1) Choice of delivery medium: As part of the research conducted for Program Review, InterMedia (an independent research organization under contract to the BBG) conducts audience surveys on the type of media available to and preferred by target audiences. 2) Choice of language: BBG programming is broadcast in the local languages of the target areas. 3) Program Content: As part of the research conducted for Program Review, control panels and focus groups are used to determine the information needs of target populations and obtain feedback on several program attributes, such as the relevance and objectivity of content. Internal content analysis and production analysis conducted as part of Program Reviews also inform program content decisions. In addition, MBN and RFE/RL resources are granted directly to the broadcasting programs, and detailed financial information is reported during BBG budget reviews and the annual audit, which prevents unintended subsidies and ensures that funds are used for the intended purpose.

Evidence: BBG Strategic Plan 2008-2013; MBN Strategic Plan; InterMedia Research presentations during Arabic Language Services Program Reviews; State Department Background Notes for countries in the Middle East and North Africa; U.S. International Broadcasting Act of 1994; Foreign Affairs Reform and Restructuring Act of 1998; RFE/RL and MBN mission statements; Program Review Scoring Procedures; Language Service Review process; Statistical Methodology update; CFO's mid-year, 3rd quarter and year-end Reviews; monthly BBG status report; PAR

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The BBG's mission of objective journalism will build audiences in areas of U.S. strategic interests that trust U.S international broadcasting news and information and enhance understanding. Measuring the program's direct impact to promote or sustain freedom and democracy in the Middle East and North Africa is virtually impossible due to the numerous factors (political, geographic, historical, educational, economic, ethnographic, diplomatic, humanitarian, etc.), separate from the influence of BBG broadcasting efforts, that may also affect the situation in a given country or region. Therefore, having a population choose to tune in to BBG broadcasts, or the program's "audience reach," is the most quantifiable measure of the program's effectiveness. In addition, whether or not the audience finds the broadcasts to be reliable and credible, and thereby regularly tunes in and hears the information and ideas BBG broadcasts, is another important measure of the external effect of BBG programs. Similar to Nielsen ratings and research, these measures are industry research standards used to determine the audience's view of the effectiveness of media. The BBG currently has two long-term performance measures for this program and continues to explore additional or improved measures that address the quality and relevance of broadcasting. Audience reach, also called weekly audience, is an indicator based upon measurement of the regular listening/viewing audience. The Broadcasting in Arabic FY 2010 performance plan ties this measure to annual language service activities, the BBG overall strategic goal, and the annual performance objectives. Regular audience is defined as all adults in a given population choosing to listen/view at least once a week, as determined by an audience survey with an adequately designed sample. The long-term audience reach outcome measure is a culmination of a number of annual common measures used across all Arabic services (see question 2.3), as well as annual performance goals specific to each individual service. As detailed in the measures section, the overall BBG Arabic program has been achieving its long-term goals of expanding audience reach through individual targeted programs broadcast to 22 countries in the region and across Europe. In addition to the long-term outcome measure of whether or not the target population decides to tune in to BBG broadcasts, BBG has developed a long-term outcome measure of whether or not the target population considers the broadcasts to be credible/reliable. Having a growing audience tune in to BBG programs and find them to be credible/reliable sources of news and information means that BBG is achieving its mission and broadcasting its message, with the overarching result being the promotion of freedom and democracy and enhanced understanding of the U.S. and the world. The credibility score is determined by the survey question about "trustworthiness of news and information" of those sampled respondents who listened at least once a week. The answers are registered on a five-point scale -- very trustworthy, somewhat trustworthy, neither trustworthy nor untrustworthy, somewhat untrustworthy, or very untrustworthy. The credibility measure is the percent of those answering the question in the survey (excluding those who did not respond or did not know) that selected very or somewhat trustworthy.

Evidence: BBG Strategic Plan (2008-2013); Broadcasting in Arabic FY 2010 Performance Plan; FY 2010 performance plans for MBN and Radio Free Iraq; Statistical Methodology update.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: BBG's audience size for Arabic broadcasting was approximately 25.6 million in 2004 and grew by an estimated 20 million or 78% by 2006. In FY 2007, BBG implemented more robust statistical methodologies for calculating the audience estimates, therefore FY 2007 begins a new baseline of 42.9 million for BBG broadcasting in Arabic. Attracting significant new audiences to BBG's programming of objective news and information during the last several years was primarily due to the launch of Alhurra television in 2004 and the strengthening of the Radio Sawa and Radio Free Iraq brands. The satellite television market in the Middle East has expanded exponentially since 1990 and the market is saturated with international, regional, and local satellite channels. There are currently approximately 340 satellite channels available across the region and the market continues to expand - in March 2008, the British Broadcasting Corporation (BBC) launched an Arabic television channel broadcasting to the region. With this explosion of the satellite television market in the Middle East, MBN is strategically focusing to establish a niche by using research and capitalizing on Alhurra's unique American perspective. In FY 2008, the launch of a 3-hour daily signature show will redefine the station's flow and focus, and ultimately reposition Alhurra in the marketplace. MBN will continue to assess and redefine its original programming in the context of its mission to expand its reach and influence in the region. Radio Free Iraq (RFI) will build on its reputation as a station "for Iraqis, about Iraqis, and by Iraqis" to reach larger audiences. Radio Sawa will enhance its programming with customized streams as well as expanding its Internet presence. In this competitive media environment, these ambitious efforts are expected to expand the audience by 8 million people by 2013. This represents a more measured growth rate than the 2004-2006 period, which saw a large increase due to the introduction of a new BBG broadcast entity - Alhurra TV. But it is a sufficiently ambitious goal to attract new audience reach while retaining existing listeners/viewers, despite expanding competition and anti-American sentiment. The other BBG long-term measure, audience credibility , is the percent of those answering the question in the survey related to the "trustworthiness of news and information" (excluding those who did not respond or did not know) that endorsed the programming as very or somewhat trustworthy. Even in a region with strong anti-American sentiments, Broadcasting in Arabic has achieved strong credibility ratings (79% for Radio Sawa and 73% for Alhurra in FY 2007, 63% for RFI in FY 2004). The BBG has set ambitious goals to sustain scores above 60% with significant incremental increases to 78-85% by the individual language services over the next five years. The related annual goals and action steps will track the progress toward achieving the long-term goals. For example, an improved program quality score may be a contributing factor to attracting and retaining a regular audience. On a scale of 0 to 4, a program quality rating score of 2.7-3.3 is considered "good." All Arabic services achieved scores of good or higher and have annual goals to maintain or exceed that score.

Evidence: The long-term goals for broadcasting in Arabic are to increase the audience size by 8 million and sustain the credibility score above 60% to reach 78-85% over 5 years; BBG Strategic Plan (2008-2013); Broadcasting in Arabic FY 2010 Performance Plan; FY 2010 performance plans for MBN and Radio Free Iraq

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: In addition to the long-term measures described in 2.1 and 2.2, each Arabic broadcast service has a common annual outcome performance measure of program quality and an efficiency measure of cost per audience member. Audience reach is also tracked annually for each language service, and this data feeds directly into the long-term measure of audience reach in Arabic as a whole. Similar to Nielsen ratings and research, these measures are industry research standards in determining the audience's view of the effectiveness of media. The common measure of cost per audience member is an efficiency measure that tracks the cost per audience member of each broadcast service and is discussed further in 4.3. The program quality score is an outcome measure based in part on how the target population perceives the quality of the specific BBG language services. Listeners from the target population rate the program's accuracy, reliability, authoritativeness, objectivity, comprehensiveness and other variables reflecting its effectiveness. Another component of the program quality score is the evaluation of BBG delivery systems (engineering and transmission), marketing and program placement efforts, and other broadcasting support functions. Programming content and presentation are also evaluated by both internal and external reviewers. In total, the Program Quality Score is a comprehensive, weighted score, which ranges from 1-4, with 4 being the highest, that indicates whether the program is effective in meeting the over-arching goals and mission of the BBG. There are two other common measures that are tracked internally and support the outcome measures. Audience awareness is a measure that refers to the total of either prompted or unprompted acknowledgement of BBG station names in an audience survey that has an adequately designed sample. The statistic is expressed as a percent of the sampled population that knows or recognizes BBG stations, which is then translated to the percent of the entire population of the country or region of which the sample was representative. This measure of the extent to which a target population recognizes BBG is influenced by internal efforts taken by each broadcast service to advertise and market its programming and coordination among services. For example, Alhurra and Radio Sawa cross-promote and RFI is planning to begin collaborations with MBN. Signal strength is the average signal strength of monitored BBG cross border shortwave and medium wave transmissions measured during each half hour of scheduled programming. In addition, to support the goals reflected in these annual performance measures, each individual broadcast service or area has developed output-based performance objectives with annual milestones or measures tailored to the nature of the individual service's broadcasting and target population. Improved content and availability of the programs will contribute to achieving the annual outcome goals as measured by program quality. The individual language service improvements to program quality directly impact the long-term outcome goals of increasing the overall Arabic-speaking audience that chooses to tune in to BBG programs and improving the audience's perception of the broadcasts as credible/reliable.

Evidence: BBG Strategic Plan (2008-2013); Broadcasting in Arabic FY 2010 Performance Plan; FY 2010 performance plans for MBN and Radio Free Iraq

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The annual targets of the measures common to all Arabic services are sufficiently ambitious given the large population of the region, the political and press freedom restrictions in many countries, explosive growth in competitive broadcasters in diverse media, and strong anti-American sentiments in the region. For example, all broadcast services have set targets to maintain or increase their program quality score to "good or better." Baselines for each of the annual measures are based on the FY 2004 level, and annual targets have been set through FY 2010. Each language service has also developed tailored annual performance objectives that directly impact each service's progress toward achieving the targets set forth in the annual measures. For example, Alhurra is creating a new 3-hour daily signature show that will be broadcast live from Cairo, with auxiliary broadcast centers planned in Jerusalem, Beirut and Dubai. If successful, this performance objective will further define Alhurra and its niche, increase its impact throughout the region, and is expected to directly impact each of the broader common goals reflected in the annual measures. As in the example of the 3-hour daily show, the targets are set based on the expected effects of the annual performance objectives combined with a trend analysis of the measure.

Evidence: The baseline for all of the annual measures is the FY 2004 level. Annual targets are set through FY 2010. See the Measures tab of this report for examples of this program's annual goals; Broadcasting in Arabic FY 2010 Performance Plan (see tables of measures and targets); Language Service Review; Program Review (Source: Performance Review Scoring Procedures)

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Each of the BBG Arabic services (Alhurra, Radio Sawa, and Radio Free Iraq), including both agency and grantee broadcasting entities, contributed to establishing the BBG Strategic Plan (2008-2013) and the development of the Broadcasting in Arabic FY 2010 Performance Plan. In addition, the Broadcasting in Arabic program participates with all other BBG entities, including support elements, in the Language Service Reviews. RFE/RL conducts annual Program Reviews for each of its language services, including Radio Free Iraq. MBN is developing its own internal Program Review process for Alhurra TV and Radio Sawa; currently MBN's Program Reviews are performed by the BBG. Each Arabic broadcast service also develops an individual annual plan that includes annual performance goals that support the overall BBG long-term goals of the program. Senior managers within the program commit to and work toward the performance goals through the performance based appraisal system, which is specifically tied to program objectives and goals. Each year, managers review and include the most current objectives and goals in their annual appraisals. BBG Contract Officers monitor compliance with standard performance clauses that impact and support the goals of the program, including schedule, cost and technical requirements, as they manage contracts for the program.

Evidence: Language Service Review; Program Review; BBG Strategic Plan (2008-2013); Broadcasting in Arabic FY 2010 Performance Plan; FY 2010 performance plans for MBN and Radio Free Iraq; Performance Appraisal templates for BBG and MBN; excerpts from Articles 2 and 5 of RFE/RL freelance contract for bureau staff.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The BBG Broadcasting in Arabic program and each individual service within the program undergo annual comprehensive evaluations by professional and experienced broadcasters who are not part of the program, independent and unbiased focus groups randomly selected from audience members within the region, and based on research data obtained by an independent research firm (described below) that meet the scope and quality required. These Program Reviews include a rigorous assessment of each aspect of the program, including content, delivery systems (engineering and transmission), , and production content and presentation plus marketing and program placement efforts, and Internet development, as appropriate for the specific broadcast service. It examines the overall impact of the broadcast service and outlines a follow-up procedure to implement improvements. In addition to the comprehensive annual performance review, BBG's annual Language Service Review looks at BBG's overall broadcasting portfolio to assess two basic issues: (1) where should BBG broadcast and (2) how well is BBG broadcasting. Language Service Review is overseen by a Board Committee and is jointly staffed by the Office of the CFO and the Office of Strategic Planning, all of whom are positioned outside of the program being assessed, and relies on information and unbiased research data that is independent from the programs being evaluated. Through Language Service Review the importance of reaching each of the target audiences is evaluated and the highest priority audiences among all of those targeted are identified. Resource allocations or reallocations to reach specific target audiences are informed by the results of this review. In the case of broadcasting in Arabic, the broadcast services are included in the evaluation of BBG priorities throughout the world. Within each Arabic language service, performance measures are reviewed and effectiveness assessed, in concert with all other BBG broadcasting to the Middle East and North Africa. In addition to the comprehensive evaluations described above, BBG also regularly receives targeted independent reviews focused on the impact of the language services, as well as broader agency-wide evaluations focused on financial and other management issues. BBG contracts with InterMedia, an independent firm that conducts annual surveys for each language service and entity, evaluating a variety of factors: market profile, media access and use, and international broadcasting impact and trends. An independent annual audit by an external auditing firm has been conducted for the BBG since 2004, including the management of funds and evaluation of BBG performance, as reported in the Performance and Accountability Report (PAR). The BBG has received clean audit opinions for 2004, 2005, 2006, and 2007. Since 2000 the GAO has conducted several reports on BBG; the findings, recommendations and resulting actions by the BBG impacted all broadcast services, including those in Arabic. Follow-up testimony was provided in 2004. In 2006, the GAO Report "Management of Middle East Broadcasting Services Could Be Improved" found that "MBN has developed financial and administrative controls to manage and safeguard its financial resources," More recent GAO Testimony (GAO-07-795T) reported on this progress, stating "The BBG subsequently created a single strategic goal to focus on the key objective of maximizing impact in priority areas of interest to the United States and made audience size a key performance measure," and "MBN has taken steps to improve its process of program review and evaluation."

Evidence: Language Service Review; Program Review (Source: Performance Review Scoring Procedures and Analysis); InterMedia Research Presentations and research documents for Arabic broadcast services; PAR; GAO Report: "Strategic Planning and Performance Management System Could be Improved" (GAO/NSIAD-00-222); GAO Report: "New Strategic Approach Focuses on Reaching Large Audiences but Lacks Measurable Program Objectives" (GAO-03-772); GAO Report: "Enhanced Measure of Local Media Conditions Would Facilitate Decisions to Terminate Language Services" (GAO-04-374); GAO Testimony: "Challenges Facing the Broadcasting Board of Governors" (GAO-04-711T); GAO Report: "U.S. International Broadcasting: Management of Middle East Broadcasting Services Could Be Improved" (GAO-06-762); GAO Testimony: "Strategic Planning Efforts Have Improved, but Agencies Face Significant Implementation Challenges" (GAO-07-795T); Statistical Methodology update

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: BBG has excelled at clarifying its mission, establishing implementation strategies and performance objectives that support the mission, as well as establishing performance indicators. Actions taken in developing the plans each year since 2006 have enhanced the linkages between these strategic tools and increasingly show how they are supported and implemented by BBG's budget each year. In its Budget requests, the BBG explicitly linked all costs to six annual performance objectives that support BBG's mission, and will do so again in its FY 2010 budget request. The performance objectives include direct and indirect costs as appropriate. For example, the performance objective to "Reach the Arabic speaking world" includes all programming expenses and expenses for program delivery leases, power, and parts. In addition, each objective will link annual performance goals to the requested resources with performance measures that are monitored against the targets set for FY 2010. In FY 2010, the BBG will show how the performance objectives for the various broadcasting entities and support elements support the overall BBG strategic goal and annual performance goals. In FY 2010, the BBG will again include the agency's annual performance plan and performance measures in its budget requests.

Evidence: FY 2006, 2007, 2008, and 2009 Congressional Budget Requests; Language Service Review; Program Review; PAR

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The BBG developed a new strategic plan for 2008-2013. The plan reflects the experiences gained over the past several years and is a forward looking roadmap to achieve the goals of U.S. public diplomacy in reaching significant audiences in support of U.S. strategic interests. The Board increased entity involvement in the planning process (based on lessons learned regarding implementation of the prior plan) by enlisting all BBG entities in defining internal and external challenges, identifying organizational strengths and weaknesses, and drafting updates to strategies contained in the earlier plan. In addition to increased involvement by the broadcasting entities and grantees, the new plan also improved upon previous strategic planning efforts by considering other sources, such as global market and audience research findings, feedback from the Department of State, U.S. national security strategy documents, and think tank studies, in identifying challenges and deriving the implementation strategies to address them. BBG also developed a performance plan for Broadcasting in Arabic for FY 2010. The performance plan is based on extensive research and BBG strategic priorities, and is linked to the BBG mission. Each broadcast service also developed their performance plan with the performance goals specifically linked to the BBG implementation strategies. The annual performance goals link to the long-term goal of increasing the Arabic-speaking audience by specific objectives, such as website improvements that draw larger Internet audiences.

Evidence: BBG Strategic Plan (2008-2013); FY 2010 performance plans for MBN and Radio Free Iraq; research and results from Language Service Review and Program Review processes

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The BBG analyzes the performance data gathered through its annual Language Service Review and Program Reviews. VOA, Radio and TV Marti, and each of the BBG grantees (RFA, RFE/RL, MBN) undergo annual Language Service and Program Reviews - with MBN's Alhurra TV and Radio Sawa, and RFE/RL's Radio Free Iraq specifically relevant to this PART. The Language Service review comprehensively analyzes for each BBG Language Service: polling data gathered from InterMedia (from statistically valid surveys and focus groups), affiliate data from the Office of Marketing and Program Placement (OMPP), transmission data from Engineering, budget data from the Office of the CFO, current country conditions and information on US national interests. The Board uses the results of this review to adjust resources and set strategic priorities. In addition to survey data from InterMedia, Program Reviews also include content and production analysis that assist managers in improving their specific programming in such areas as music bridges, lively pace, interactive call-in shows, or younger voices to attract a larger target audience. Annual Performance Plans include long-term and annual goals for each Language Service, as well as relevant measures to track these goals. For the Broadcasting in Arabic Program, the baseline is 2004 actuals. The BBG also conducts follow-up meetings to ensure that programs have successfully made adjustments and changes that were recommended during the Language Service Reviews and Program Reviews. Recommendations may include management or performance issues and improvements. The Board uses this information to adjust resources and set strategic priorities.

Evidence: Language Service Review; Program Review

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: BBG entities evaluate senior managers from the BBG, the International Broadcasting Bureau (IBB), and the grantees through performance based appraisal systems that require critical elements and ratings that are tied to the objectives of the BBG with measurable goals and targets. Through the ongoing appraisal reviews and final annual rating, managers are held accountable for the performance results. Annual evaluations for senior managers are also tied to performance results. The annual Language Service Review holds managers and partners directly accountable for results with the same common outcome measures of audience reach, audience credibility, program quality and cost per audience member considered among other factors. The entities develop an annual financial plan that is reviewed in conjunction with the CFO's mid-year, 3rd quarter and year-end reviews of budget. The reviews hold managers accountable for their annual resource level against expected expenditures, schedules, including enhancements or major changes, and accomplishments. Capital projects are also monitored monthly for schedule and budget status. In addition, entities submit a monthly report to the Board on the status of resources, performance and key program initiatives. Contractors are accountable to the same program standards and the Contracting Officer monitors the quality of the deliverables. Many contractors work on a "pay-for-product" basis, with payment dependent upon the timely receipt of quality products. For example, many "stringers" receive payment only after delivering quality news stories that adhere to BBG journalistic standards.

Evidence: Performance based Employee Standards/Appraisals; Performance Appraisal templates for BBG and MBN; Language Service Review; CFO's mid-year, 3rd quarter and year-end Reviews; monthly BBG status reports; Capital Projects process; Program Review; excerpts from Articles 2 and 5 of RFE/RL freelance contract for bureau staff.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: All requisitions must include verification that funds are available, and the funds are obligated when a contract or agreement is signed or a purchase order is done. The mid-year, 3rd quarter and year-end reviews provide opportunities for managers to verify that unliquidated obligations are valid and that once an obligation has been fully expended, remaining balances are deobligated. Financial transactions are coded to both an organization and to a function or project. Given that BBG has a high percentage of fixed costs (salaries, transmission and infrastructure expenses, rent, etc.), funds must be obligated in a timely manner in order to continue operations. The BBG requires an annual financial plan and monthly status reports from the IBB and the grantees. The financial plans must detail anticipated costs for specific line items by each month of the fiscal year. BBG conducts a mid-year, 3rd quarter, and year-end review of each entity's financial plans and budgets to monitor funding and expenditures. BBG also requires a monthly status report of expenditures, disbursements and obligations, as well of the submission of the SF-169A Financial Status Report, and SF-172 Report of Federal Cash Transactions from the grantees. The reviews and the allotment control procedures enable the Agency to track obligations and awards to ensure that funds are used for the intended purpose by the appropriate Agency office. The BBG reports to the Federal Procurement Database System (FPDS-NG) and the Federal Assistance Award Database (FAADS) in compliance with the Federal Funding Accountability and Transparency Act .

Evidence: SF-169 Financial Status Report; SF-172 Report of Federal Cash Transactions; Monthly status reports on the IBB, MBN and RFE/RL on expenditures, disbursements, and obligations; Quarterly review reports; BBG's FPDS-NG and FAADS reporting

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The annual Language Service Review process includes reviewing each program's progress on an efficiency measure: cost per audience member. The Program Review process also assesses the efficiency and productivity of each language service in support of this efficiency measure. Cost per audience member is the total cost of programming and delivery divided by the unduplicated audience for all media, and is calculated for each language service. The baseline year for the annual efficiency measure is FY 2004 and annual targets are set through FY 2010. Annual targets are evaluated and established in context of anticipated changes in either the cost (based on pending budget requests) and/or projected audience size. For example, a capital cost increase to improve transmission may not increase audience size until after the capability is operational, thus increasing the cost per audience member target in the short term, but ultimately will result in a decrease in costs as the audience size begins to grow. In another example, anticipated audience gains from new programming initiatives that are projected to occur after an initial increase in funding or at no cost will decrease the cost per audience measure target. In addition to the efficiency measure described above and included in the PART measures section, BBG tracks efficiencies in a number of different ways. The CFO's reviews of budget and performance results hold managers accountable for their annual resource level against expected accomplishments. Enhancements to programs are tracked separately with specific performance targets based on the increased resources. Also, the BBG broadcasting entities work together to identify opportunities for collaboration and shared logistical arrangements that could result in cost savings to the agency and the U.S. government.

Evidence: Broadcasting in Arabic FY 2010 Performance Plan; FY 2010 performance plans for MBN and Radio Free Iraq; Language Service Review process; Program Review process; Federal Acquisitions Regulations 52.211-11, 52.211-12, 11.503

Does the program collaborate and coordinate effectively with related programs?

Explanation: In terms of broadcasting, BBG is solely responsible for all government and government sponsored non-military, U.S. international broadcasting. It is the only entity with the express objective of promoting freedom and democracy through the broadcasting of objective news and information about the U.S. and the world to audiences in Arabic. Although there are no related programs that share similar broadcasting goals and objectives, the BBG does collaborate with other entities, including the Department of State in order to produce and distribute its programming. For example, BBG worked closely with the U.S. Embassies in Morocco and Egypt to set up 24/7 FM transmission capabilities in those countries. BBG entities also cooperate among themselves. For example, MBN has worked closely with the Voice of America to coordinate on U.S. presidential election coverage to ensure potential efficiencies are realized to the extent possible. Representatives from the entities meet at least monthly to examine and establish shared logistical arrangements where cost savings to the government could be realized.

Evidence: U.S. International Broadcasting Act of 1994; Foreign Affairs Reform and Restructuring Act of 1998; U.S. International Broadcasting Act of 1994; Voice of America Charter; Broadcasting in Arabic FY 2010 Performance Plan; FY 2010 performance plans for MBN and Radio Free Iraq.

Does the program use strong financial management practices?

Explanation: While the BBG is currently cross-serviced by the State Department to provide financial services using the Broadcasting Financial Management System (BFMS) system, the BBG is converting to the National Business Center's Momentum financial management system in FY 2008. Internal controls are in place to provide for the separation of duties, minimize erroneous payments, ensure recording of receipts, and the accurate reporting to external entities. The BBG Administrative Officers obligate financial transactions accurately and reconcile their allotments on a monthly basis. In the CFO's Office, the Office of Financial Operations records and makes disbursements, reconciles the Fund Balances with Treasury, records and makes deposits, and prepares the monthly, quarterly and annual statement reports for the agency. FY 2004 was the first year the BBG prepared the Performance and Accountability Report (PAR) and underwent a financial statement audit. The PAR was submitted on time and the BBG received an unqualified opinion on its Balance Sheet. Again in 2005, 2006, and 2007, the BBG submitted an expanded PAR and received an unqualified opinion on its financial statements.

Evidence: BFMS System; PAR: An independent audit conducted in October 2004, October 2005, October 2006, and October 2007 by an independent auditor offered an unqualified opinion of the BBG financial statements

Has the program taken meaningful steps to address its management deficiencies?

Explanation: RFE/RL conducts annual Program Reviews for each of its language services. MBN is developing its own internal Program Review process; currently MBN's Program Reviews are performed by the BBG. The Program Review process examines both the content and presentation of BBG programming and sets out a plan to address any identified deficiencies within three months. Each Arabic language service also develops an individual annual plan that includes annual performance goals that support the overall BBG long-term goals of the program. Senior managers within the program commit to and work toward the goals through the BBG performance based appraisal system, which is specifically tied to program objectives and goals. Since 2000 the GAO has conducted several reports on BBG; the findings, recommendations and resulting actions by the BBG impacted all broadcast services, including those in Arabic. Follow-up testimony was provided in 2004. In 2006, the GAO Report "Management of Middle East Broadcasting Services Could Be Improved" (GAO-06-762) found that "MBN has developed financial and administrative controls to manage and safeguard its financial resources," "MBN has procedures in place to help ensure its programming meets its journalistic standards," and "the BBG has developed several performance indicators and targets for MBN's Radio Sawa and Alhurra services, including measures of audience size and program credibility." It recommended several actions for the BBG and MBN to strengthen their strategic planning, internal control, adherence to journalistic standards, and statistical methodologies, all of which are currently underway. More recent GAO Testimony (GAO-07-795T ) reported on this progress, stating "The BBG subsequently created a single strategic goal to focus on the key objective of maximizing impact in priority areas of interest to the United States and made audience size a key performance measure," and "MBN has taken steps to improve its process of program review and evaluation."

Evidence: Program Reviews; GAO Report: "Strategic Planning and Performance Management System Could be Improved" (GAO/NSIAD-00-222); GAO Report: "New Strategic Approach Focuses on Reaching Large Audiences but Lacks Measurable Program Objectives" (GAO-03-772); GAO Testimony: "Challenges Facing the Broadcasting Board of Governors" (GAO-04-711T); GAO Report: "U.S. International Broadcasting: Management of Middle East Broadcasting Services Could Be Improved" (GAO-06-762); GAO Testimony: "Strategic Planning Efforts Have Improved, but Agencies Face Significant Implementation Challenges" (GAO-07-795T); Statistical Methodology update; Performance Appraisal templates for BBG and MBN.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The number of people choosing to listen to or view BBG programming in Arabic, or the weekly audience reach, has grown significantly in recent years, from 25.6 million in 2004 to an estimated 45.6 million in FY 2006. In FY 2007, BBG implemented a more conservative statistical methodology for calculating the audience estimates; therefore FY 2007 begins a new baseline of 42.9 million for BBG broadcasting in Arabic. At the same time that audience reach expanded significantly, BBG maintained strong credibility scores. The credibility score is determined by the survey question about "trustworthiness of news and information" of those sampled respondents who listened at least once a week. The answers are registered on a five-point scale -- very trustworthy, somewhat trustworthy, neither trustworthy nor untrustworthy, somewhat untrustworthy, or very untrustworthy. The credibility measure is the percent of those answering the question in the survey (excluding those who did not respond or did not know) that endorsed very or somewhat trustworthy. New audience members who need to build trust in BBG programming and reactions to U.S. policy in the region have caused minor fluctuations in the annual credibility score, but scores for Radio Sawa, Alhurra Television and Radio Free Iraq remain over 60%, according to the most recent survey data. Over the next five years, Arabic broadcasting will sustain a credibility level above 60% and incrementally reach a significant goal of 78-85% for each of the three language services.

Evidence: U.S. International Broadcasting in Arabic FY 2010 Performance Plan

Does the program (including program partners) achieve its annual performance goals?

Explanation: In addition to the long-term weekly audience and credibility measures, there is a key annual outcome measure of program quality and an efficiency measure of cost per audience (addressed in question 4.3) for each Arabic language service. Data available from FY 2004 to FY 2007 for program quality shows that language services are achieving high levels. For example, on a scale of 0 to 4, a program quality rating score of 2.7-3.3 is considered "good" and all services achieved scores of good or higher and have annual goals to maintain or exceed that score. In addition to the annual outcome measures included in the PART Measures section, there are two annual output measures that support the outcome measures for each language service: credibility and signal strength. These are tracked by each program and are reported in the annual performance plan. The output performance goals are generally achieved; however fluctuations in annual scores can be caused by a number of outside factors. For example, increased competition and security concerns limiting advertising campaigns can impact audience awareness while signal strength can be impacted by solar activity and other environmental conditions, thus causing short-term fluctuations in these measures.

Evidence: U.S. International Broadcasting in Arabic FY 2010 Performance Plan

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The BBG has a common efficiency measure for all language services, cost per audience member, which tracks the cost effectiveness of broadcasting. This is the ratio of the total cost of programming and delivery divided by the unduplicated audience for all media, and is calculated for each broadcast service. The cost per audience member for each of the Arabic services has decreased since 2004. The cost per audience member for Radio Sawa in FY 2007 is $0.92, down 55% from the FY 2004 level of $2.06. For Radio Free Iraq, the cost per audience member in FY 2007 is $0.56 down 54% from the FY 2004 level of $1.22. For Alhurra television, the cost per audience member is $2.20 in FY 2007, down 75% from the FY 2004 level of $8.66. Costs may fluctuate from year-to-year, but overall, the cost per audience member for Arabic broadcasting is decreasing. Factors that may result in decreased costs annually could include established programming with no capital costs while factors that may result in increased costs annually could include construction or initial set-up costs. In addition, program activities to attract audiences, such as utilizing delivery methods that the audience prefers, additional programming in interest areas that appeal to the target audience and up-to-date programming production, will increase audiences regardless of costs; thus the effectiveness measure will improve. An example of both cost efficiency and program effectiveness for broadcasting in Arabic is the recent MBN change of vendors for the provision of overseas production services that will yield a net savings of approximately half a million dollars, which can be invested to improve production efficiency and quality in other areas.

Evidence: Broadcasting in Arabic FY 2010 Performance Plan; FY 2010 performance plans for MBN and Radio Free Iraq; Program Review

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: NA

Evidence: BBG is solely responsible for all U.S. government and government sponsored, non-military international broadcasting and is an important part of U.S. public diplomacy efforts overseas. No comparable programs exist. BBG promotes freedom and democracy overseas and presents the policies of the United States. Although there are other international broadcasters in the region, such as BBC, CNN, and Al Jazeera, the BBG has a unique mission -- promoting freedom and democracy and enhancing understanding through multi-media communication of accurate, objective, and balanced news, information, and other programming about America and the world to audiences overseas. BBG is the only international broadcasting entity that serves as a critical arm of the U.S. public diplomacy mission. U.S. International Broadcasting Act of 1994; Foreign Affairs Reform and Restructuring Act of 1998; MBN and RFE/RL Mission Statements.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: RFE/RL conducts annual Program Reviews for each of its language services. MBN is developing its own internal Program Review process; MBN's Program Reviews have been facilitated by the BBG. In all cases, the reviews include evaluations by professional and experienced broadcasters who are not part of the program, unbiased focus groups randomly selected from audience members within the broadcasting area, and research data obtained by an outside audience research firm (InterMedia) that is independent from this program and the agency. These Program Reviews assess each aspect of the program, including content, IBB delivery systems (engineering and transmission), marketing and program placement efforts, Internet development, and production content and presentation. The Program Reviews have shown good or better Program Quality scores, a weighted score that includes factors measuring all aspects of program quality, to indicate that the program is effective in meeting the over-arching goals and mission of the BBG. Each Arabic language service also develops an individual annual plan that includes annual performance goals that support the overall BBG long-term goals of the program. In addition to the comprehensive, independent Program Reviews described above, the BBG annual Language Service Review evaluates a wide range of performance measures against strategic priorities and indicates that Broadcasting in Arabic is an effective program and is achieving results. The Language Service Review is conducted by a committee of BBG Board members and includes input from the Department of State on public diplomacy strategic priorities, unbiased and independent evaluative data from the Freedom House Freedom Ratings, Freedom House Press Freedom Ratings, and Heritage Foundation Economic Freedom Ratings. Audience research conducted by InterMedia, an independent research organization, indicates that broadcasting services in the Middle East and North Africa reach target audiences with appropriate and effective material.

Evidence: InterMedia research including surveys, monitoring panels, and listener panels; Program Review schedules, results, action plans and independent evaluations; Language Service Review analysis and results.