Detailed Information on the
Foreign Service Institute Assessment

Analyze Department impact on providing distance learning training to Foreign Service Nationals, a historically underserved population.

Increase outreach to posts and bureaus to promote distance learning as a resource-efficient training option and increase use.

Review evaluation processes and tools to increase ensuring the integrity of responses, maximize response rates, and strengthen linkages to program results.

Review performance measures used to manage program and revise as necessary to develop the most useful and accurate indicators.

Measure: Percentage of students in Critical Needs Lanaguges Attaining Skill Objectives Within designed timeframes. (Note: state to provide prior year data)

Explanation:Expresses performance of FSI language training program (Critical Needs languages only) as a percentage of students who attain the intended proficiency level (as determined by Language Designated Position proficiency level) when they are enrolled for at least the recommended length of training.

Measure: Satisfaction with FSI Training Overall : Percent of responses indicating "Satisfied" or "Very Satisfied" on the annual FSI Training Survey.

Explanation:In an effort to ensure that training provided by FSI is appropriate and relevant for employees' job assignments, FSI surveys customers who have taken FSI training 1-5 years preceding the survey (i.e., after they have returned to and been on the job(s), tracking (on a five level Likert scale, i.e. "Very Dissatisfied", "Dissatisfied", "Neither Satisfied nor Dissatisfied", "Satisfied", or "Very Satisfied") those that rate how well prepared they were to perform their job duties as a result of training received as "Satisfied" or higher. The results are used to adjust training/curricula accordingly. Survey covers all FSI training areas (professional tradecraft, leadership & management, information technology, language studies, orientation, and security awareness)

Measure: FSI Language Training Satisfaction Rate: Level of satisfaction with foreign language competency/preparedness as measured by annual survey of employees and supervisors.

Explanation:In an effort to ensure that language training received was appropriate for the job assignment, SLS surveys employees after the employee has been on the job at post, tracking (on a four level scale) those that rate their assessment of how well they feel they were language-prepared as "Well" and higher. The results are used to adjust training/curricula accordingly. (Notes: 1) respondents' assessment of how well they were prepared can be an assessment of either or both the language training received or the language proficiency level designated for the position they were filling (i.e., that the proficiency level designated for their position is "too low"); 2) pre 2005 results were based on a 6 level methodology which was altered for 2005 to a 4 level scale; for those years only levels 3-6 are deemed "adequate or higher".)

Measure: DISTANCE LEARNING GROWTH: Increased preparedness, knowledge and skills of foreign affairs personnel as a result of expanded use of FSI's Learning Management System and distance learning.

Explanation:Increased use of distance learning provides capability to reach more students worldwide at less cost. FSI issues user passwords to its internet based LMS to provide access to online training. An increase in accounts represents more wide-spread use of distance learning and is an indicator that we are providing accessibility to the right training to world-wide customers at the right time. Distance learning represents a more cost-efficient method of training comparative to classroom based training, particularly in the Department of State context where the majority of employees are stationed in our 250 plus missions worldwide. FSI had not been able to provide training to most of the thousands of its Foreign Service Nationals and Locally Engaged Staff before. Through distance learning we are able to provide cost effective training opportunities which will have the outcome of a better prepared workforce. In addition to improved effectiveness in employee work performance that training overall can foster, increased distance learning can result in 1) potential cost avoidance of travel and per diem to bring overseas employees to FSI Washington, and 2) increasing training opportunities to overseas personnel who, realistically, would have little chance to come to FSI Washington for training due to budgetary constraints, time away from the office, etc.

Measure: Increase in Foreign Affairs distance Learning Products

Explanation:FSI produces distance learning products specifically greated to the foreign affairs community. Some, such as language training courses and products require multi-year efforts to indentify fonts, readers, linguists, obtain copyright permissions, etc. Increase in training products represents increased customer accessibility to training through an efficient and effective delivery platform.

Measure: FSI Cost Per Student Trained/enrollment etc.

Explanation:This measure is under development. It will optimally measure average cost of delivering (excluding student salaries) FSI domestic training per student enrollment, and seek to quantify the efficiency of increased utilization of more cost-efficient training methods, e.g. distance learning.

Measure: FSI IT Taining Satisfaction Rate: Employee and supervisor satisfaction with IT training recieved.

Explanation:In an effort to ensure that training provided to new IT employees was appropriate and relevant for their job assignment, FSI will regularly survey employees and their supervisors one year after the employee has completed training and been on the job at post, tracking (on a five level Likert scale, e.g. unsatisfactorily prepared, less than satisfactory, satisfactorily prepared, well prepared, or very well prepared) those that rate how well prepared they were to perform their job duties as a result of training received as "satisfactorily prepared" or higher. The results are used to adjust training/curricula accordingly.

Measure: Development of training curriculum to support the Office of the Coordinator for Reconstruction and Stabillization (Active Response Corps and Operational Readiness Reserve)

Explanation:With bi-partisan support in Congress and at the direction of the National Security Council, the Department of State in July 2004 created the Office of the Coordinator for Reconstruction and Stabilization (S/CRS). A Presidential Directive signed December 14, 2005 further defined the Secretary of State's responsibility to improve coordination, planning, and implementation for reconstruction and stabilization (R&S) and established that the Secretary shall coordinate and lead integrated U.S. Government efforts, involving all U.S. Departments and Agencies with relevant capabilities, to prepare, plan for, and conduct stabilization and reconstruction activities. The FSI has been charged with supporting S/CRS by working with them to develop and provide a curriculum of R&S related training to an interagency audience. As the Reconstruction and Stabilization Civilian Management Act of 2006 (S.3322) is currently working its way through the 109th Congress, there is also the potential of further expanded training audience.

Is the program purpose clear?

Explanation: The mission of the Foreign Service Institute (FSI) is to develop the men and women our nation requires to fulfill our leadership role in world affairs and to defend U.S. interests. Activities to support this mission and Department of State (DoS) strategic goals involve the provision of foreign affairs related training and services including: foreign languages; professional tradecraft (administrative management, consular, economic/commercial, political, public diplomacy); leadership & management; information management; overseas security awareness; and training services for eligible family members. FSI was created by the Foreign Service Act of 1946 directing that the Secretary of State create a training organization to promote career development with the Foreign Service and to provide necessary training and instruction in the field of foreign relations.

Evidence: FSI mission statement; the Foreign Service Act codified in 22 USC, Chapter 52, Subchapter VII, Volume 13 Foreign Affairs Manual; FSI annual Bureau Performance Plan. The Foreign Service Act, which was amended in 1980, specifies sub-areas including foreign language proficiency (22 USC 4022), and overall professional development, e.g. tradecraft (22 USC 4023); authorizes training for family of members of the Service (22 USC 4024(c)); and authorizes special hiring and contracting authority (22 USC 4024(a))) for the Foreign Service Institute. Section 2205(a)(3) of the Foreign Affairs Restructuring and Reform Act of 1998 (Public Law 105-277), as amended (codified in 22 USC 4021(d) and (e)) authorized FSI to train members of the legislative and judicial branches on a reimbursable basis, and provide some public sector training on a reimbursable basis. The goals, principles, scope and authority of the program are defined in the Foreign Affairs Manual used by DoS and other foreign affairs agencies.

Does the program address a specific and existing problem, interest, or need?

Explanation: FSI supports DoS's strategic goals by providing training to ensure the diplomatic readiness of foreign affairs personnel. There is no other existing U.S. Government training program geared directly to the provision of training in the area of foreign affairs for the conduct of diplomacy, and the Foreign Service Act advises other agencies to avoid duplicating the facilities and training provided by FSI. DoS personnel must receive job and career specific training before they are tenured and/or relevant job specific training prior to their assuming a new assignment as outlined in 13 FAM. In many cases, this training (which can result in consular commissions, warrant authorities, professional certifications, and similar) must be taken before the employee is allowed to report to post or (including for personnel of 40 plus U.S. agencies) required under Chief of Mission authority to be completed before an individual is given transit authorization (i.e., mandatory security training).

Evidence: 22 USC 4021(b); 13 FAM; FSI Annual Enrollment Report (showing number of enrollments from other agencies; Executive Secretary Memorandum for All Agencies dated 03/25/04. FSI responds quickly to emerging training needs and adapts to new requirements, ultimately captured in the goals and objectives outlined in its Bureau Performance Plan.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Yes. FSI is legislatively mandated to prepare/train individuals involved in the conduct of U.S. foreign affairs, including personnel from all branches of government and their eligible family members who will accompany them abroad, U.S. private sector individuals (on a limited basis) and, if the Secretary of State determines it necessary, to foreign government officials from emerging democracies. There is no private sector alternative for such training related to the conduct of U.S. foreign policy work -- diplomatic, consular and public diplomacy contexts specifically, which are inherently governmental. Some examples are: consular (e.g., visa issuance, consular related anti-terrorism , Interagency Cooperative Administrative Support Services (ICASS), post-specific administrative management (e.g., Central Financial Management System, Nonexpendable Property Application, Global Employment Management System, Integrated Logistics Management System), State specific orientation programs (including mandatory "Working in an Embassy" and new Ambassador orientation), IT training for post-specific systems and applications (e.g., Post Administrative Support System, CableXpress), post-specific crisis management, public diplomacy, Overseas Briefing Center information services. Other organizations may provide training that appears generically similar (e.g., there are commercial language schools, Defense and the Central Intelligence Agency have their own language schools), but do not instill knowledge and skills with the mission-specific context essential to perform U.S. diplomatic jobs. As example, while an individual could go to a private school to learn Spanish, there is no language school other than FSI which teaches Spanish in the context of a consular officer doing consular work; when other agencies with their own language schools have a higher level, mission specific need (e.g., to accomplish diplomatic negotiation) they send, on a reimbursable basis, those individuals to FSI for language training. FSI does not duplicate training that is available from other sources. When it is warranted, however, FSI obtains such training from other U.S. government and private vendors through its External Training Program (e.g., Human Resources training as it applies to the U.S. government competitive service, is purchased from the USDA Graduate School).In December 2005, FSI was approved by the Office of Personnel Management to be the fourth eGov, e-Training service provider since technology is an important methodology for delivering this training.

Evidence: 22 USC 4021(b); FSI Schedule of Courses; FSI Training Continua; DoS/FSI annual Training Plan; FSI-OPM e-Training service provider Memorandum of Understanding.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Yes, FSI's training program and curricula is free of major flaws and is effective in providing quality service. Consolidated foreign affairs training enables more than 40 federal agencies to leverage their resources to the mutual benefit of the U.S. foreign affairs community. FSI has a cadre of expert trainers and subject matter experts who are able to provide in-depth, specific training related to U.S. diplomacy. Many trainers are career Foreign Service personnel in rotational assignments who bring unique benefit to the program with their very specific, continually updated, real-life expertise and knowledge gained from working in embassies around the world doing jobs that are only done by State Department employees (i.e., Ambassadors, Deputy Chiefs of Mission, Charge d'Affaires, Political Counselors, Consular Officers, etc.). FSI continuously evaluates training to ensure ongoing effectiveness and efficiency by end of training test scores; post-training evaluations by students and their managers; industry standard tests (e.g., Sylvan, Microsoft); periodic training and customer satisfaction surveys; and annual bureau program reviews held before the Department's senior management. Independent evaluation is provided by State's Office of the Inspector General through inspections and audits, by external entities such as the General Accounting Office (GAO), and by tests such as those conducted by Sylvan.

Evidence: 2006 Training Satisfaction Survey results; end-of-training test scores; post-training evaluation results; OIG, GAO and Accountability Review Board reports.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: Yes, foreign affairs related training is FSI's mission. Training programs are done for and in collaboration with State's regional and functional bureaus, and are designed to ensure the right people get the right training at the right time by providing a variety of delivery methods including, Washington based classroom, computer and text based distance learning, regional offerings. FSI's training authorities (as codified in 22 USC 4021) and admissions policies are clearly defined to ensure that only those individuals allowed by law and with a legitimate requirement have access to FSI's programs in prioritized order. As pertains to non-State participants, FSI conducts its business in accordance with the Economy Act on a cost recovery basis, annually determines cost recovery tuition rates for all its programs, and receives reimbursements for training and testing services it provides. Some forty plus federal agencies (including the U.S. Agency for International Development, Commerce, Agriculture, Defense, Homeland Security, Health & Human Services, Energy, Transportation, the Environmental Protection Agency, Peace Corps, Justice, Interior) utilize FSI's training programs Non-State agencies and others participate on a reimbursable or advance of funds basis in FSI delivered training.

Evidence: 22 USC 4021; FSI Admissions Policy; FSI tuition rates; FSI Annual Enrollment Report. Over the past three years there has been on average around 300,000 annual hours of non-DoS persons in FSI training.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Yes, FSI utilizes the Department's performance planning process to develop and record specific performance goals and measures on an annual basis -- the present year's Performance Plan will project measures to FY 2009. For the PART, FSI has included four long-term outcome measures that capture 1) Overall Statisfaction with training recieved through FSI by the foreign affairs community; 2) Lanaguge Training Satifcation measure by how employees and supervisor rate preparedness of students on the job, following lanaguage training; 3) increase use of distance learning, which is also considered an efficiency measure for the PART; and 4) development of new curriculum to meet new Department, Administration, and Congressional initiatives. FSI's performance goals, both those included in the PART and in our Bureau Performance Plan feed into supporting the Department's Joint Performance Plan's Strategic Goal to "Ensure a High Quality Workforce Supported by Modern and Secure Infrastructure and Operational Capacities" through such means as providing training to provide foreign language proficient employees to fill Language Designated Positions, and to provide increasing distance learning opportunities to get the right training to the right people at the right time in resource efficient ways. In 2006 FSI conducted a training satisfaction survey, which will be repeated regularly. FSI's training programs also support the Department's hiring and assignments activities and plans, as well as priority training needs identified by Department leadership (e.g., Transformational Diplomacy, Reconstruction and Stabilization); its Career Transition Program supports the "up or out" Foreign Service system as well as the foreign affairs community's needs. Surveys of students and their supervisors (both at completion of training and post-training) assess the applicability and conduct of training programs in order for FSI to continually maintain and update curricula and training methods.

Evidence: FSI Bureau Performance Plan; DoS Joint Performance Plan; DoS Performance and Accountability Report; Department Hiring Plan; Language Designated Positions; State Training Plan; results of Training Satisfaction Survey.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Targets and measures referenced above are established in accordance with mission needs, as expressed Departmentally in terms of the agency's hiring and assignments plans. Diplomatic assignments are recurring and cyclical, 2-3 years in length. This requires a continuous stream of trained personnel with defined skills (e.g., consular, political, economic, public diplomacy, etc.) to replenish those persons cycling out to other onward assignments. Hiring and training are key factors in long-term planning to assure that a continuous stream of qualified staff for State and other agencies is available to staff 260 overseas missions and domestic offices. FSI must provide adequate number of courses geared to the hiring and rotation cycles, and to meet continually evolving job needs. One FSI efficiency/ long-term outcome measure is to track distance learning availability as a resource effective means to get the right training (including maintenance and refresher training) to the right people within DoS's worldwide employee population. If the training is not provided to assignment-bound individuals and knowledge and skills are not maintained, it could have an adverse impact on diplomatic readiness and on diplomatic and national security. For example, if a pending Consular Officer does not complete the appropriate basic and refresher consular training, they will not pass the test required for them to receive their consular commission and be unable to satisfactorily perform their job, or be impacted in their ability to detect visa fraud.

Evidence: FSI Bureau Performance Plan; DoS Joint Performance Plan; DoS Performance and Accountability Report; FSI Training Satisfaction survey

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Yes, the long-term measures link to key priorities in the Department and are reflected in indicators established in FSI's annual Bureau Performance Plan. All long-term measure in the PART are/will be evaluated on an annual basis, with annual targets. The PART also includes a specific annual goal to measure the attainment of critical lanaguge skills with the standard timeframe set for those key languages. FSI develops and delivers its training programs based on the needs identified by the Department, the Secretary, agency hiring plans, post openings, Accountability Review Board reports, and other similar. Examples include the mandatory leadership/management training initiative, development of Transformational Diplomacy training, and development of a Reconstruction and Stabilization curriculum. FSI gauges success by its ability to create and offer training sessions within defined timeframes. Success is measured by means such as student and supervisor surveys to measure satisfaction with training as it impacts on-the-job performance.

Evidence: FSI Bureau Performance Plan; DoS Joint Performance Plan; DoS Performance and Accountability Report; results of program specific post-training surveys.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Annual performance targets call for high levels of performance and continuing performance improvement. Baselines are established on recent performance, and targets have been established to routinely and continuously improve performance. FSI intends to review and update its performance measures annually, through the DoS performance planning process, to provide accountability for achieving ambitious goals. FSI annually reports (to OPM and others) its training hours and enrollment, and has extensive baseline data, which confirms historically (e.g., in comparing pre-Diplomatic Readiness Initiative (pre-FY 2000) data to the last reported year of FY 2005) the ambitiousness of the 100% target. For the PART measures, FSI has set aggressive targets to increase distance learning products and use, goals for improvement in 1) overall satisfaction; 2) statisfaction with IT training; 3) on the job preparedness of language students; 3) and cirriculum develop.

Evidence: FSI Bureau Performance Plan; DoS Joint Performance Plan; DoS Performance and Accountability Report; FSI Annual Report of Training; training survey results

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: FSI has many partners including our customer agencies, the Department's Bureau of Human Resources, the Bureau of Consular Affairs, the Bureau of Economic and Business Affairs, the Bureau of Information Resource Management, the Bureau of Diplomatic Security, the U.S. Agency for International Development, and the Office of Personnel Management. There is daily and periodic interaction with our clients and customers, including input and feedback from State's regional bureaus, embassies, consulates and missions abroad. Where FSI works with other bureaus of the Department to develop specific training programs in their area of expertise -- for example, distance learning training to support the Bureau of Administration's new Post Administrative Support System, or with the Office of the Legal Adviser to support implementing required ethics orientation for new hires -- formal Memoranda of Understanding have been created. Some funding for certain programs comes from the program bureau; examples include funding provided by the Bureau of Information Resource Management to support FSI IT training, or funding provided by the Bureau of Consular Affairs out of Machine Readable Visa (MRV) fees collected to support FSI consular training. FSI's annual Bureau Performance Plan also indicates partners on various goals and objectives.

Evidence: Joint Management Council meeting notes; FSI Bureau Performance Plan. Interagency meetings to interact with agency clients on common training goals.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Independent evaluations of the program can be described as direct and indirect. Direct independent evaluation is primarily provided by the Department of State's Office of the Inspector General (OIG) through its scheduled inspections. The Department conducts annual Senior Reviews in connection with the performance planning cycle. Indirect evaluation arises when an inspection, audit or investigation is conducted of another program and, as a byproduct, applicable FSI training that supports that program is evaluated. For example, the General Accounting Office (GAO) report "Border Security: Strengthened Visa Process Would Benefit from Improvements in Staffing and Information Sharing" (GAO-05-859) involved an examination of FSI consular training as its supports employee preparedness and execution of functions. Similarly, OIG inspections of other entities, such as of a domestic bureau or overseas post, can result in review of an FSI training program, though not the overall FSI training program as a whole. Other outside entities providing evaluative feedback include governmental commissions and reports (e.g., McKinsey "War for Talent" report), commercial entities (e.g., Sylvan) and other agencies (e.g., Office of Personnel Management through the Job Satisfaction Survey) and their personnel who train at FSI.

Evidence: OIG Inspection of FSI (ISP/I-99-16); related GAO reports, such as GAO-05-859; ARB reports; OPM Job Satisfaction Survey; Director General's Quality of Life Operational Progress Plan;

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: FSI's budget requests are relayed through the Department's strategic planning process and through Financial Plan requests or supplemental requests (e.g. Iraq) and are tied to specific training or assignments goals. New or shifting program requests are outlined in detail in the Bureau Performance Plan. Detailed planning for realignments such as post openings (Libya, Sudan), Global Repositioning, increased Language Enhancement Program, or to meet Federal Information Security Management Act training related requirements, for example, take place in partnership with Department leadership and our customer bureaus and agencies as appropriate. Tuition rates are analyzed and published annually.

Evidence: FSI Bureau Performance Plan; DoS Joint Performance Plan; DoS Performance and Accountability Report; FSI student training reports, test reports; FSI tuition rates

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: FSI utilizes its annual Bureau Performance Plan (BPP) as its strategic plan. Data flows from Mission Performance Plans (MPP) to BPPs. The senior review process at mission and bureau levels allow for re-calibration and re-programming as diplomatic priorities shift. When deficiencies occur, it is most often due to an unexpected, foreign affairs events outside the control of FSI. For example, the re-opening of posts in Sudan and Libya require training to prepare personnel for these posts (language, tradecraft, security). Another example is detecting terrorists -- training prior to the 9/11 Commission did not have specific modules related to the subject, but once the deficiency was identified the curriculum was adapted to include the subject.

Evidence: FSI Bureau Performance Plan

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Yes. FSI performance information includes end of training evaluations and post-training surveys to both students and their supervisors to gauge the impact of the training on the job, and annual surveys. Performance information and measures are collected the Department's annual performance planning exercises. FSI uses this feedback to improve and revise curriculum, determine new or outdated training topics, and/or make resource-related decisions such as adjusting the frequency of offerings of courses or the size of classes, or exploring distance learning options. Evaluations made by entities such as the Office of the Inspector General and the General Accounting Office can contain training-related recommendations, which result in specific actions/improvements made in training programs.

Evidence: FSI Bureau Performance Plan; DoS Joint Performance Plan; DoS Performance and Accountability Report; FSI world-wide training survey; end-of-training evaluations; post-training follow-up evaluations; FSI responses to OIG and GAO recommendations.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Through State's and FSI's performance planning process and reviews, managers and program partners are held accountable for meeting targets that support overall training performance measures. FSI is directly funded by State to provide training for State personnel and their eligible family members; other agency personnel share the costs of training through tuition fees paid to FSI. All FSI training users (State and non) adhere to written procedures and policies regarding FSI training management and billing/cost. In accordance with State Department performance evaluation regulations, each FSI program managers have required elements in their Job Performance Plans on supervision/management and management controls; training staff have specific elements relating to satisfactory delivery of training on which instructors are evaluated constantly. FSI's Bureau Performance Plan also identifies responsible elements for each performance indicator. Contracts awarded by FSI for the provision of training services clearly state requirements that must be completed in order for services to be deemed acceptable (and payable).

Evidence: FSI annual Bureau Performance Plan; contract documents; FSI tuition rates; 13 FAM; FSI Administrative Procedures

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: FSI programs are funded and obligated in a timely manner, and spent for intended purpose and accurately reported. FSI uses three integrated IT financial systems -- the Department's Integrated Logistics Management System (ILMS), the Central Financial Management System (CFMS), and the Financial Planning and Control System (FPAC) -- to manage its resources and produce detailed financial reports down to the course code level. In addition, routine monthly reviews of execution are performed in addition to comprehensive annual budget formulation, financial, and program reviews. From an internal control perspective, all obligations, recording/accounting and related financial management actions are transparent and performed consistently with Department guidance from 4 FAM, the Economy Act, and internal FSI Budget and Management Office procedures.

Evidence: Vol. 4 Foreign Affairs Manual; Vol. 4 Foreign Affairs Handbook 1, Vol. 4 Foreign Affairs Handbook 3; FSI Financial Plan; budget execution reports

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: FSI has procedures in place to measure and achieve efficiency and cost effectiveness. FSI has a decentralized budgeting model in which each program office develops, executes and tracks financial actions and expenditures, with access to the Department's financial systems to promote transparency and accountability at the most immediate level. The annual calculation of tuition rates identifies all costs down to each training division level, roughly capturing the monies spent against enrollment numbers. There are training management policies to control cost and efficiency, such as a minimum/maximum student number (course sessions not reaching the stated minimum number of students are cancelled). FSI has special authorities regarding the hiring of staff into time-limited positions so it can more quickly change when needed. FSI has a workforce that is about half staff and half contract. FSI has completed four streamlined competitions on 90 positions; with a cost avoidance in excess of $6.4 million over five years; all were in-house wins. FSI continually looks to improve efficiency through IT and technology. The Department's overall investment in upgrading world-wide IT infrastructure, coupled with investments in FSI product development, is increasing more cost-effective distance learning opportunities and other technology based enhancements such as paperless on-line training application. FSI has long-term goals to double the number of self-produced distance learning courses and the number of user accounts in its Learning Management System as a means to achieve efficiency in getting the right training to the right people at the right time.

Evidence: FSI Administrative Procedures Handbook (Section 4 on budget, and Section 8 on registration/enrollment); FSI Bureau Performance Plan; FSI Competitive Sourcing studies; number of FSI distance learning products available; number of Learning Management System user accounts established.

Does the program collaborate and coordinate effectively with related programs?

Explanation: FSI is providing cost-effective training by centralizing the training function across the Department. FSI provides standard training enterprise wide. FSI disseminates a common body of knowledge in different disciplines found only in the practice of foreign affairs (political/military, consular, attaches, etc). This common body of knowledge enables officers to work across cones and geographic differences. FSI is the coordinating body within the U.S. Department of State providing employees with the foreign affairs skills to advance U.S. interests worldwide. FSI uses subject matter experts from State and other agencies for the programs. FSI establishes Memoranda of Understanding/Agreement with State bureaus when developing distance learning products that directly support their programs (e.g., with the Office of the Legal Adviser with online Ethics Orientation, or with the Bureau of Administration with online PASS courses). In December 2005, the Office of Personnel Management designated FSI as the fourth authorized federal eGov, eTraining service provider. FSI collaborates with other bureaus and agencies on training and training management and several issues of mutual concern.

Evidence: 22 USC 4021(b); Vol. 13 Foreign Affairs Manual Sec. 110; Vol. 13 Foreign Affairs Manual Sec. 900; FSI-OPM e-Training Memorandum of Understanding (MOU); MOUs between FSI and other DoS bureaus re development of distance learning products. The Homeland Security Act of 2002 included language that DoS shall provide, on a space-available and reimbursable basis, necessary training to Department of Homeland Security employees on a priority basis after DoS but before other agencies (State-DHS MOU). As stated in the Foreign Service Act, the FSI is charged with providing foreign affairs training that is not to be duplicated by other agencies (22 USC 4021(b)). Vol. 13 Foreign Affairs Manual Sec. 111 states that FSI is responsible for developing training policy and facilitating necessary training for all DoS personnel; Vol. 13 Foreign Affairs Manual Sec. 900 requires all State bureaus to process all requests for external (non-FSI, vendor delivered) training through FSI, and Vol. 13 Foreign Affairs Manual Sec. 113.4 states that all proposed contracts and training programs with outside training contractors must be reviewed by FSI to ensuring training is appropriate and relevant and does not duplicate FSI and/or other Department training programs. All agencies, and others, that utilize FSI's training programs do so on a full-cost recovery tuition reimbursement basis; training enrollment forms (e.g., SF-182, DD-214) constitute a funded interagency MOU.

Does the program use strong financial management practices?

Explanation: The Department's financial management practices have been well documented and show clean financial audit opinions on its financial statements the past four years; FSI contributed to that accomplishment. FSI follows extensive Department level guidance and procedures, and these procedures are applied to FSI's program offices which undergo rigorous reviews three times a year. Financial management practices are also reviewed by the Office of the Deputy Chief Financial Officer and the Bureau of Resource Management's Domestic Financial Services Oversight Office . Another strategy to gauge FSI's financial management practice is done through the annual budget and financial planning process conducted by the Bureau of Resource Management. These reviews provide for transparency of FSI's programs and identify financial management practices that are free from material internal control weaknesses. FSI complies with all financial laws, regulations, appropriation language, and the OMB Circular A-11. The Federal Managers' Financial Integrity Act (FMFIA) requires agencies to establish management control and financial systems that provide reasonable assurance that the integrity of federal programs and operations are protected. It also requires that the head of the agency, based on an evaluation, provide an annual Statement of Assurance on whether the agency has met this requirement. The State Department evaluated its management control systems and financial management systems for the fiscal year ended September 30, 2005. This evaluation provided reasonable assurance that the objectives of the FMFIA were achieved in FY 2005, and formed the basis for the Secretary's Statement of Assurance.

Evidence: 4 FAM; 4 FAH-1, 4 FAH-3; FSI bureau Performance Plan; DoS Performance Accountability Report; FMFIA reviews and Annual Statement of Assurance; OMB Circular A-11.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: FSI fully participates in the Department's program and strategic planning process, including preparation of a Bureau Performance Plan which examines any problems on an annual basis and is subject to review by senior staff. FSI conducts regular customer surveys. There is constant curriculum review with State bureaus, which is part of FSI's total quality management process. FSI also utilizes its financial planning processes (which include preparation of an annual finplan by all program entities, mid-year reviews, and regular budget reviews/reconciliation) for ongoing monitoring of potential program management deficiencies. Program management deficiencies are also identified through Office of the Inspector General (OIG) and General Accounting Office (GAO) inspection and audit recommendations, to which FSI responds by taking appropriate actions to close out recommendations; the last full OIG inspection of FSI was completed in 1999. FSI also conducts assessments under the Federal Managers' Financial Integrity Act, and there have been no material weaknesses to report in the Annual Statement of Assurance to the Secretary of State.

Evidence: FSI Bureau Performance Plan; FSI financial plan; responses to OIG and GAO recommendations; FSI customer satisfaction survey; annual Statement of Assurance; FSI significantly helped the Department get to "green" on the eGov PMA initiative through being approved by OPM as an eTraining provider.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Goals stated in the annual Performance & Accountability Report and the Bureau Performance Plan indicate that FSI has demonstrated adequate progress in achieving long-term goals. For the PART measures, FSI has met goals in expanding distance learning products, and exceeded the annual goal for Language Attainment. A recent FSI Training Survey, polling employees and supervisors, has demonstrated an 83% overall satisfaction with training received over the past five years. Our efficiency/long-term measure for distance learning tracks FSI's capacity to increase resource effective means to provide training to the right people at the right time through increase in use (user accounts) of its Learning Management System (LMS), and shows LMS accounts increased 50%from FY 2004 to FY 2005.

Evidence: FSI Training Survey; PART performance measures; DoS Performance & Accountability Report; FSI annual BPP; training surveys; distance learning statistics.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Annual goals have been set in conjunction with long-term goals in the Department's and bureau's program and strategic plans and have displayed steady achievement. For example, the distance learning indicator used under one of the Department 's Strategic Goals in the 2005 Performance & Accountability Report was 70% above target, and FSI's provision of effective language training has consistently helped the Department meet targets on its indicator tracking the percent of employees assigned to language designated positions who meet the requirements of the position. While PART measures have only been defined now, in many cases prior year data exists to show prior achievement. For the PART, FSI has met goals in expanding distance learning products, and exceeded the annual goal for Language Attainment.

Evidence: FSI annual BPP; DoS Performance & Accountability Report; PART performance measures; training surveys; distance learning statistics.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: FSI has evidence and documentation related to programs, attendance, evaluation surveys, competitive sourcing studies, Office of Personnel Management designation of FSI as an eTraining program, and now its participation in PART. FSI has policies and procedures in use which promote resource-effiency, such as minimum class sizes and penalties for students who fail to show or complete a sufficient percentage of a class. The FSI PART has adopted one efficiency measures on decreasing per student enrollement costs. In addition, increases in LMS user accounts and increases in FSI produced distance learning products will also produce program efficiencies. LMS accounts increased 50% from FY 2004 to FY 2005. Increased use of technology-delivered training can achieve efficiencies by offsetting travel and per diem costs to FSI Washington, but also creates training opportunities for thousands of State employees (including 37,000-plus Foreign Service Nationals) who, for budget reasons, would not have ready access to FSI's programs in Washington.

Evidence: FSI Training Survey; PART performance measures; FSI BPP; SLS language training survey; LMS user account statistics; number of FSI DL products; DoS Performance and Accountability Report.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: FSI's mission-specific, foreign affairs training curricula is unique, and there is no U.S. government or private program that conducts training with purpose and goals similar to diplomatic readiness basic skills and advanced training to which it can be systematically compared. In many cases -- U.S. consular, mission-specific administrative services and programs (e.g., Interagency Cooperative Administrative Support Services, Post Administrative Support System, public diplomacy) -- there is no other source for training. Even in specific program areas where agencies conduct similar training -- for example, the Department of Defense's Defense Language Institute in Monterrey, California -- it should be noted that for individuals (such as Defense Attaches) requiring a higher level language proficiency that can be equated to State language-designated-position requirements, they are sent to FSI.

Evidence: 22 USC 4021; FSI Schedule of Courses; FSI-OPM eTraining service provider MOU citing FSI's unique foreign affairs curricula.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Under the President's Management Agenda eGovernment e Training initiative, an evaluation of our program led to the Office of Personnel Management designating FSI as the fourth authorized federal service provider of eTraining. FSI has been found to be effective in providing training. Outside customers routinely come back for training. Our authorities were expanded by request to train legislative and judicial employees; Homeland Security legislation specified that FSI give them first priority (after DoS only) for certain training. FSI has a formal agreement with Sylvan Testing Centers and has permanent test centers for industry standard IT professional certifications (Microsoft, Cisco, etc.). This on site testing validates the quality of instruction and knowledge and skills acquired by the students. Another indication of the quality of the program is that universities and colleges throughout the country routinely grant credits for individuals who have taken some of the longer FSI courses. This is an individual initiative and must be arranged by the individuals directly with their organization. Surveys to bureaus and posts related to training serve as needs assessments as well as evaluations on the scope and quality of the FSI program. These surveys are done by FSI and other bureaus (e.g. the Bureau of Consular Affairs' annual survey; the State ICAAS Service Provider Steering Committee 2005 survey of Principals/DCMs), employees and overseas training providers. In January 2006, an updated review by the outside firm McKinsey & Company included information about training. The 2005 "Best Places to Work in the Federal Government" survey issued by the non-profit Partnership for Public Service recently identified State as one of the top ten. FSI's last inspection (ISP/I-99-16 issued June 1999) found "FSI has done an impressive job in trying to meet the training and development needs of the foreign affairs community."

Evidence: MOU Designating FSI as 4th federal wide service provider; Section 428 of the Homeland Security Act of 2002 (P.L. 107-296); 22 USC 4021; selection by outside customers; Inspection and GAO reports; surveys by FSI and other bureaus or groups; OPM Quality of Life Survey; McKinsey 2006 Update Summary; 2005 Partnership for Public Service "Best Places to Work in the Federal Government" results; OIG Inspection of FSI (ISP/I-99-16)