Detailed Information on the
International Education Domestic Programs Assessment

Developing and implementing a strategy to use efficiency measures to improve cost effectiveness in achieving the program goals.

Completing the study of ED's graduate fellowship programs and utilizing the results to validate program performance measures and improve program performance.

Making grantee performance data available to the public in a transparent and meaningful manner.

Collecting baseline data for the newly developed annual and long-term performance measures.

Developing and implementing a strategy for conducting independent rigorous, periodic, meaningful evaluations of the outcomes and impact of the Domestic Programs.

Developing a list of less and least commonly taught languages from which the Secretary of Education would select certain languages as "critical need languages."

Developing a measure to track language skill changes through the use of reliable assessment tools.

Completing the study of ED's graduate fellowship programs and utilizing the results to validate program performance measures and improve program performance.

Continuing collaboration with other Federal and non-governmental entities in managing the programs to ensure that funding supports critical languages considered to be vital to the national interest.

Measure: Average competency score of FLAS fellowship program recipients at the end of one full year of instruction minus the average score at the beginning of the year.

Explanation:Overall change in the language competency self-assessment reflects a mix of different levels of improvement at all states (beginner, intermediate, advanced) of the three modalities of language acquisition for the assessment measures (reading, writing, speaking). Beginning language students may be expected to make larger advances over a given time period (and therefore have larger change scores) than more advanced students. A target value of 1.2 for change over the year reflects an ambitious overall goal for the program.

Measure: Federal cost per FLAS program fellow increasing average language competency by at least one level.

Explanation:Denominator: the count of FLAS fellows with an average difference of 1 point or more (averaged across all language areas (reading, writing, and speaking) from pre- to post-test. Numerator: the annual funding for the FLAS program.

Measure: Percentage of less and least commonly taught languages as defined by the Secretary of Education taught at Title VI National Resource Centers.

Explanation:Denominator: the total number of less and least commonly taught languages as defined by the Secretary of Education. Numerator: the total number of less and least commonly taught languages as defined by the Secretary of Education taught at NRCs in a given fiscal year. This measure focuses on Title VI's mission of maintaining national capacity in a broad range of less and least commonly taught foreign languages.

Measure: Percentage of critical need languages as defined by the Secretary of Education taught at National Resource Centers.

Explanation:Denominator: the total number of critical need languages as defined by the Secretary of Education. Numerator: the total number of critical need languages as defined by the Secretary of Education taught at NRCs in a given academic year. This measure proposes to capture how well our programs respond to new and unanticipated language challenges of the future.

Measure: Percentage of National Resource Center grants teaching intermediate or advanced courses in critical need languages as defined by the Secretary of Education.

Explanation:Denominator: total number of all NRC grants. Numerator: total number of NRC grants teaching intermediate or advanced courses in critical need languages as defined by the Secretary of Education. This measure proposes to capture how well our programs respond to new and unanticipated language challenges of the future.

Measure: Percentage of FLAS master's and doctoral graduates that studied critical need languages as defined by the Secretary of Education.

Explanation:Denominator: the total number of FLAS master's and doctoral graduates. Numerator: the number of FLAS master's and doctoral graduates reporting a post- and pre-score for any of the critical need languages as defined by the Secretary of Education. This measure proposes to capture how well our programs respond to new and unanticipated language challenges of the future.

Measure: Percentage of FLAS participants who report that they found employment that utilizes their language and area skills.

Explanation:Denominator: the number of FLAS participants who report that they found employment. Numerator: the number of FLAS participants who report that they found employment that utilizes their language and area skills.

Is the program purpose clear?

Explanation: The Department of Education's (ED) International Education and Foreign Language Studies (IEFLS) Domestic programs are designed to strengthen the capability and performance of American education in foreign languages and in area and international studies. The 9 IEFLS Domestic programs are (in order of largest to smallest, in terms of funding in 2007): National Resource Centers (NRC, $29 million), Foreign Language & Area Studies (FLAS) Fellowships ($29 million), Centers for International Business Education (CIBE, $11 million), International Research & Studies (IRS, $6 million), Language Resource Centers (LRC, $5 million), Undergraduate International Studies & Foreign Language (UISFL, $4 million), Business & International Education (BIE, $4 million), Technological Innovation & Cooperation for Foreign Information Access (TICFIA, $2 million), and American Overseas Research Centers (AORC, $1 million). The programs are authorized in Title VI of the Higher Education Act of 1965 (HEA), and referred to as Title VI Domestic programs throughout this assessment.

Evidence: Statutory purposes: Sections 601 and 611 of Title VI, HEA (as amended). Section 601--production of "increased numbers of trained personnel and research in foreign languages, area studies, and other international studies"; Section 611--"increasing and promoting the Nation's capacity for international understanding and economic enterprise through the provision of suitable international education and training for business personnel."

Does the program address a specific and existing problem, interest, or need?

Explanation: Foreign language development has been the major focus of Title VI since its inception. Today, these programs support projects in over 130 foreign languages. The increased complexity of the global economy, the events surrounding the September 2001 terrorist attacks on the U.S., and the war on terrorism underscore the importance of maintaining and expanding American students' foreign language, international, and area expertise. The Domestic Programs focus their resources on those areas of the world often neglected in the curricula of institutions of higher education, and on the foreign languages spoken in those areas; many of these languages, particularly the less commonly taught languages (LCTLs), would not be taught in the United States or at advanced levels without Title VI support. The Title VI programs are key to the teaching and learning of languages vital to the national interest. And the programs serve as a national resource that can be drawn on by the entire nation. Under the direction of the President, the Departments of Education, Defense, and State and the Office of the Director of National Intelligence have proposed to implement a comprehensive national plan to expand foreign language education beginning in early childhood and continuing through formal schooling and into the workforce. The National Security Language Initiative (NSLI) addresses weaknesses in our teaching and learning of foreign languages, especially critical need languages. Critical need languages are defined as foreign languages considered most-critical for national security.

Evidence: Congressional findings for Title VI of the Higher Education Act of 1965; Brecht/Rivers study "Language and National Security in the 21st Century: The Role of Title VI/Fulbright-Hays in Supporting National Language Capacity" (2000) and http://www.internationaled.org/BriefingBook/6.Building/6.e%20Language%20National%20Sec.doc (2001); briefing on the National Security Language Initiative (http://www.state.gov/r/pa/prs/ps/2006/58733.htm , January 5, 2006); National Research Council of the National Academies comprehensive review of International Education and Foreign Languages: Keys to Securing America's Future (Prepublication Copy March 2007).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The Title VI programs are the Federal Government's primary effort to develop and maintain a national infrastructure to produce expertise in critical and less-commonly-taught foreign languages (LCTLs), area studies, and other international studies, including international business. There are a handful of Federal programs with an interest in foreign language that complement, but do not supplant, ED's Title VI programs. These programs have key differences in mission, focus, and purpose. As an example, the Defense Department's National Security Education Program (NSEP) draws on Title VI-funded institutions to help produce experts in critical languages and the Foreign Service Institute (FSI) and the Defense Language Institute (DLI) utilize resources and instructional materials from Title VI-funded institutions.

Evidence: Other Federal programs take complementary approaches. An example of this complementary approach is a component of the Defense Department's NSEP called the National Flagship Language Program (NFLP) that awards financial support to IHEs recognized as leaders in the field of language education designed to help individuals achieve superior-level proficiency. The program currently focuses on a limited number of critical languages including Arabic, Mandarin Chinese, Korean, Russian, and Persian, and those of Central and South Asia. Unlike ED's programs, fellows in NSEP incur a service obligation to the U.S. government as a condition of their fellowship. Additional background on NSEP can be found at: http://www.ndu.edu/nsep/index.cfm?secID=309&pageID=167&type=section and www.casl.umd.edu/nflp. In addition, certain NSLI activities relating to establishing fully articulated K-16 programs of language study cannot be accomplished through the Title VI programs. Additional information on the NSLI can be found at: http://www.ed.gov/about/inits/ed/competitiveness/nsli/index.html.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program has no major design flaws and no evidence exists that a different approach would be more effective or efficient. In fact, the nine Title VI Domestic programs are part of a pipeline to create and strengthen academic, public, and private sector domestic expertise in language and area specialties. Previous Title VI program reauthorizations have responded nimbly to expressed needs for foreign language, area, and international studies while correcting major flaws within the programs. The current reauthorization of the HEA, which was originally scheduled for 2004, is not complete. Amendments currently proposed for Title VI Domestic programs include orienting the programs toward areas of national need; directing recipients into government service; creating an International Higher Education Advisory Board; ensuring a diversity of perspectives and general debate on world regions and international affairs; and authorizing use of program funds for activities relating to program evaluation, national outreach, and information dissemination.

Evidence: The National Defense Education Act of 1958 and the successor Title VI of the Higher Education Act of 1965, as amended, have changed the IEFLS programs over the years. Amendments have incorporated public comment as well as feedback from the international education and foreign language communities. Current Higher Education Act reauthorization bills under consideration (109th Congress): House passed H.R. 609 and the Senate introduced S. 1614 (www.thomas.gov).

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The Domestic programs focus their resources on those areas of the world and foreign languages that are typically neglected in the curricula of postsecondary institutions, but yet are critical for national security and global competitiveness. Data shows that NRCs train the majority of the nation's future international experts and professionals with proficiency in critical foreign languages. Graduates of these programs are successfully employed in fields where their expertise is needed. And, there is no evidence of unintended subsidies.

Evidence: A 2002 Modern Language survey, two sets of program data, and preliminary study results provide evidence of effectively targeted resources. (1) Data from the 2002 Modern Language Association (MLA) language enrollment survey funded under Title VI since 1958 indicates that Title VI-supported institutions enroll 55% of the graduate and 22% of the undergraduate enrollments in the LCTLs even though they account for less than 3% of all institutions of higher education. Data from the new survey will be published in fall 2007. (2) Data from the International Resource Information System (IRIS, http://www.ieps-iris.org/iris/ieps/irishome.cfm), a Web-based performance reporting system, continues to show that NRC graduates primarily select employment fields where their linguistic and/or area expertise is utilized. (3) Data from the e-LCTL Initiative, a collaborative effort to collect and analyze data on the teaching of LCTLs, show that in NRCs funded by Title VI, more than 30,000 students were enrolled in 126 LCTLs in 55 universities in 27 states in 2001-2002. These Centers enroll more than 60% of the students in the 10 languages deemed "critical" by the Department of Defense. During 2001-2004, 226 LCTLs were available through Title VI centers while Federal agencies offered 75 LCTLs. For the past decade, an estimated more than 80% of all instruction in LCTLs was in Title VI-supported centers. (http://elctl.msu.edu)(2005) (4) Finally, preliminary descriptive data from a study of ED's graduate fellowships programs, which includes the FLAS fellowship program, show that of the FLAS fellows receiving support between 1997 and 1999, nearly all FLAS fellows worked after completing their degrees, and most (71 percent) worked in jobs that involved the expertise they had gained through their FLAS-supported study. Nearly all fellows (96 percent) who reported working in a related job considered that job to be part of a career they were pursuing. The final report will not be ready for publication until later in 2007.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: ED has four long-term performance measures for the two largest Title VI Domestic programs -- National Resource Centers (NRC) and Foreign Language and Area Studies (FLAS) Fellowships. NRC has one set long-term measure and is considering two more, while FLAS has one long-term measure. The established NRC measure tracks the percentage of less and least commonly taught languages, as defined by the Secretary of Education, that are taught at our nation's Title VI NRCs. The NRCs measures being considered track the percentage of: critical need languages, as defined by the Secretary of Education, taught at NRCs; and, NRC grantees teaching intermediate or advanced courses in critical need languages, as defined by the Secretary of Education. ED dropped the NRC public sector employment measure because it is not a good measure of the results from NRC funding and the authorizing statute doesn't promote employment in a particular field. The one FLAS measure tracks the percentage of FLAS participants who report that they found employment that utilizes their language and area skills. These measures focus on Title VI's mission of maintaining national capacity in a broad range of foreign languages while at the same time capture how well our programs respond to new and unanticipated language challenges of the future.

Evidence: The basis for the percentage of less and least commonly taught languages measure is a statutorily mandated list of foreign languages that are determined to be critical to national security, economic, and scientific needs. The current list of 171 languages was developed in consultation with other Federal agencies in 1985 (see Section 631(a)(7) of the HEA which refers to the list of languages as documented in Federal Register; Vol. 50, Number 149 dated 8/2/1985.) To focus the program on national needs, the Secretary is developing a complementary list of less and least commonly taught languages from which certain languages will be deemed "critical need languages." Data for the remaining measures will be derived from IRIS.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Ambitious targets and timeframes will be established once final agreement on long-term measures is achieved.

Evidence: NA

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: ED will retain one of the established annual measures for these programs, and is considering five new annual measures. The established measure tracks the average language competency score of Title VI FLAS recipients at the end of one full year of instruction (post-test) minus the average competency score at the beginning of the year (pre-test), as determined by a self-assessment. In addition, ED is considering five new annual measures which track the percentage of: (1) less and least commonly taught languages, as defined by the Secretary of Education, taught at NRCs; (2) critical need languages, as defined by the Secretary of Education, taught at NRCs; (3) NRC grantees teaching intermediate or advanced courses in critical need languages, as defined by the Secretary of Education; (4) FLAS master's and doctoral graduates that studied critical need languages, as defined by the Secretary of Education; and (5) FLAS participants who report that they found employment that utilizes their language and area skills. These measures focus on Title VI's mission of maintaining national capacity in a broad range of foreign languages while at the same time capturing how well our programs respond to new and unanticipated language challenges of the future. ED dropped the NRC public sector employment measure because it is not a good measure of the results from NRC funding and the authorizing statute doesn't promote employment in a particular field.

Evidence: Data for all measures is derived from information reported by grantees in the International Resource Information System (IRIS, http://www.ieps-iris.org/iris/ieps/irishome.cfm), a Web-based performance reporting system. Currently, FLAS recipients' language proficiency is based on a self-assessment. The National Research Council of the National Academies concluded in its study that the language proficiency of FLAS students is not at present being adequately assessed and recommended that ED develop an alternative approach to measuring foreign language proficiency with demonstrated reliability and validity.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has established baselines and targets for one of its six measures -- the FLAS language competency annual measure. Baseline and targets for the remaining five annual measures are currently under development.

Evidence: Current measured value for the FLAS fellows' language competence pre-post score difference, as determined by a self-assessment, is consistently at or above the target value of 1.2 units.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: As part of the application process, all partners commit to and work toward the specified goals of the program. Program measures have been shared with grantees through application packages and closing date notices. Grant awards are based on the applicant's ability to describe specific and measurable ways by which the grantee will evaluate the success of the project. Annual performance reports are required for all grantees and their performance is measured on the basis of how well they meet program goals. In addition, all grantees complete final performance reports and program managers annually assess overall performance using the Project Evaluation Profile report (PEP).

Evidence: Annual and final reports collect data on graduate employment, languages taught at NRCs and taken by program participants (including languages deemed critical by the National Security Language Initiative, or NSLI), and participant language competency levels. In the FY 2007 competitions, ED used competitive priorities in four Title VI Domestic programs to target funds to certain languages or area studies. Three programs encouraged applicants to develop programs/materials in countries where the following NSLI critical languages are spoken: Arabic, Chinese, Japanese, Korean, Russian, as well as Indic, Iranian, and Turkic language families. The remaining program announced a competitive priority with a focus on the following world areas: the Middle East, South or East Asia, Russia, or Africa.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Several independent studies have been published that evaluated different elements of the Title VI Domestic programs, and one is underway. But most are not rigorous enough to fully determine impact. However, ED has used these implementation studies to support program improvements, confirm the validity of performance report data, and ascertain whether a program is achieving its goals. In order of the most recent publication date, the following describes the six most relevant studies for this assessment: (1) The National Research Council of the National Academies assessed the adequacy and effectiveness of Title VI programs in addressing their statutory missions and building the nation's international and foreign language expertise. The Council used extant evaluations; program monitoring data; select grant applications; historical funding data; written comments from experts and officials; commissioned papers and analyses; public meetings; meetings with Washington and officials from the U.S.; and site visits as a basis for their March 2007 report. (2) An independent comprehensive study of graduate fellowship programs in the Office of Postsecondary Education is underway, including the FLAS fellowship program. This study is expected to provide information on the effectiveness of the FLAS program with respect to employment outcomes and improvement in language competency. While preliminary descriptive data are available, the final report will not be ready for publication until later in 2007. (3) In 2002, the Modern Language Association "sent a questionnaire to the registrars of 2,781 two- and four-year institutions, soliciting information on credit-bearing enrollments for fall 2002 in all language courses other than English." The MLA had a 99.6% response rate, the highest rate among the 20 surveys the Association has administered in the 44 years it has conducted the survey. (4) A 2000 Brecht/Rivers independent study used literature reviews; surveys; and, opinions from policy-makers, experts, and consumers to assess the effectiveness of Title VI/Fulbright-Hays (Title VI/F-H) in meeting its statutory goals with regard to foreign language. (5) Schneider/Burn used background information from ED abstracts, site visits, and a questionnaire was developed and distributed to a sample comprised of 107 institutions funded in alternative years between 1982 and 1990 to evaluate and research the impact of Title VI's Undergraduate International Studies and Foreign Language (UISFL) program on the status of international and foreign language studies before and after grant funding, project objectives and strategies, course offerings, and project outcomes (both direct and indirect impacts). (6) In 1997, Pennock-Roman/Webb/Wooten used the Survey of Earned Doctorates to follow-up recipients of the FLAS program upon completion of their doctorates. A descriptive, statistical profile of FLAS award recipients was developed. The profiles on the FLAS fellows was compared with population profiles based on all doctorate recipients (non-FLAS) in the same fields for corresponding years.

Evidence: (1) National Research Council of the National Academies comprehensive review of International Education and Foreign Languages: Keys to Securing America's Future (Prepublication Copy March 2007). (2) ED's Draft Study of Four Federal Graduate Fellowship Programs: Education and Employment Outcomes of Participants (2007). (3) MLA Language Enrollment Survey (2002). http://www.adfl.org/resources/enrollments.pdf (4) Brecht/Rivers report "Language and National Security in the 21st Century: The Role of Title VI/Fulbright-Hays in Supporting National Language Capacity" (2000) (5) Schneider/Burn research report "Federal Funding for International Studies: Does it Help? Does It Matter? Long-Term Impacts of Federal Funding on International Studies and Foreign Language Programs (1999). (6) Pennock-Roman/Webb/Wooten report "Highlights in the Contrast Between FLAS Recipients Who Completed Their PhD and All U.S. Citizens Who Completed Their PhDs in the Same Fields, 1984-1994 (1997).

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget requests are not tied to the accomplishment of annual and long-term performance goals and resource needs are not presented in a complete and transparent manner in the program's budget.

Evidence: ED Congressional Budget Justifications for Fiscal Year 2008, http://www.ed.gov/about/overview/budget/budget08/justifications/r-highered.pdf

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program is implementing the recommendations of the previous PART assessment to: (1) complete the study of ED's graduate fellowship programs; (2) continue collaborating with other Federal and non-governmental entities in managing the programs to ensure that funding supports critical languages vital to the national interest; and (3) develop and implement a strategy to use efficiency measures. In addition, the program will pursue three strategic planning initiatives around picking outcome measures, developing an impact evaluation plan, and exploring standardized language proficiency assessments. First, ED has identified new long- and short-term measures to better assess performance outcomes related to national economic and security needs in critical languages, and has implemented and revised its Internet-based electronic data collection system (IRIS). ED continues to improve the IRIS system by reevaluating the utility of current data elements, reducing respondent burden, and ensuring the system can provide the most accurate possible information for the programs' performance measures. Second, the Department will consider ways to gauge impact of its Title VI programs, given the breadth and scope of the nine programs. And, finally, ED is committed to pursuing a better instrument to more accurately measure language proficiency. These last two issues directly stem from the recommendations made by the National Academies review of the programs.

Evidence: A description of ED's implementation of previous PART recommendations can be found in the FY 2008 Congressional Justifications at http://www.ed.gov/about/overview/budget/budget08/justifications/r-highered.pdf. ED has revised a number of its performance measures to better reflect how the Title VI Domestic IEFLS programs respond to the less and least CTL needs of the nation. And, in the FY 2007 competitions, ED used competitive priorities in the UISFL, IRS, and AORC programs to encourage applicants to develop programs/materials in countries where the following NSLI critical languages are spoken: Arabic, Chinese, Japanese, Korean, Russian, as well as Indic, Iranian, and Turkic language families. For BIE, a competitive priority was announced with a focus on the following world areas: the Middle East, South or East Asia, Russia, or Africa. IRIS has been implemented, updated, and revised, to collect, analyze, and report improved data and information on the performance of IEFLS programs. National Research Council of the National Academies comprehensive review of International Education and Foreign Languages: Keys to Securing America's Future (Prepublication Copy March 2007)

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Grantees are required to submit annual performance reports, interim reports, and final reports following completion of their grants. Continuation awards are dependent upon a determination of substantial progress being made, including progress toward performance goals. Program managers use this information to improve performance through the targeting of site visits, formulating the Directors' Meeting and Technical Assistance Workshop agenda, and planning initiatives to develop additional guidance for grantees. In addition, ED has developed the Project Evaluation Profile (PEP), a tool used by program officers to standardize their assessment of annual performance reports and other data, thus improving program monitoring and oversight.

Evidence: Performance data, information learned from technical assistance workshops, research studies, and discussions with experts in the field have resulted in program improvements. A Language Conference workshop was held in February 2007; the presentations have been posted to the program Website at http://www.ieps-iris.org/iris/ieps/Langworkshop.cfm. In addition, the PEP will be used to identify outstanding and problematic grantees, and as a basis for assessing common elements that can be used to identify best practices and areas requiring technical assistance.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ED's managers are subject to the Education Department Performance Appraisal System (EDPAS), which links employee performance to relevant Strategic Plan goals and action steps, and is designed to measure the degree to which a manager contributes to improving program performance. Program managers are identified and their EDPAS agreements are linked with the performance of projects or the program. Additionally, funding decisions for current grant recipients are based on prior performance.

Evidence: The EDPAS standards for Title VI Domestic program managers set forth requirements that managers establish strategies for implementing GPRA and Strategic Plan initiatives related to program goals. Project officers are held accountable for assessing project performance, estimating prior performance quality in considering continuation funding, and monitoring the progress of projects in achieving program goals and objectives. Grantees receive continuation funding on the basis of their reported progress in achieving program goals.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Federal funds are obligated within the timeframes set out by ED schedules (i.e., awards are made promptly and accurately before the end of the fiscal year) and used for the purposes intended. No internal or external audits have identified any deficiencies.

Evidence: Once funds have been obligated, program staff actively monitor grantee drawdown of Federal funds, examining reports on excessive drawdowns or inactivity in withdrawals. Annual performance reports and the Grants Administration and Payment (GAPS) financial management system are used to ensure funds are spent for intended purposes and managed effectively.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: ED's efficiency measures for all nine programs is annual cost per outcome, but no targets have been developed yet. However, ED has trend data for one of the largest Domestic programs, the Foreign Language and Area Studies (FLAS) fellowship program. In mid-2005, the Department undertook a pilot effort to implement efficiency measures for its higher education programs. The implementation strategy includes communicating with grantees, calculating efficiency data in a number of ways, publishing efficiency data, and setting targets for improved efficiency. The strategy is being implemented during FY 2007 with selected Title VI Domestic programs, beginning with the National Resource Centers (NRC) program, the other large Domestic program. Program managers will work with NRC projects to focus on methods and promising practices for improving efficiency over time.

Evidence: The annual cost per FLAS fellow increasing average language competency by at least one level was $17,439 in 2004 and $17,124 in 2005. Additional efficiency measures for the IEFLS programs are available at: http://www.ed.gov/about/reports/annual/2008plan/program.html.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The Department collaborates with other related programs at the official and staff levels. At the official level, coordination and interagency planning take place within the National Security Language Initiative (NSLI) among policy officials at the Departments of Education, Defense, and State, as well as the Office of the Director of National Intelligence. The different Departments have worked together to coordinate program goals, outreach materials, and legislative strategy. However, the National Research Council review reflects that more can be done. The Council concluded that the Department of Education does not have strategic coordination of foreign language and international programs within the Department or with other federal agencies. They recommended creating a high-level position within the Department to better coordinate programs throughout the federal government. At the staff level, ED staff actively participate as members of several boards relating to international education issues: the National Security Education Program Board, Interagency Language Roundtable, Japan-U.S. Friendship Commission, and Interagency Working Group on Government-Sponsored International Exchanges and Training. In addition, program managers share information with and plan events with other offices, and agencies, in and outside of ED, provide contacts between grantees and government entities both domestic and foreign, program managers from different agencies serve as peer reviewers for Department of Education applications, and products of ED-funded grants are utilized by other Federal entities involved in language training.

Evidence: Collaboration and coordination between ED and the Departments of Defense and State, and the Office of the Director of National Intelligence for NSLI and other international education-related purposes. National Research Council of the National Academies comprehensive review of International Education and Foreign Languages: Keys to Securing America's Future (Prepublication Copy March 2007). In addition, http://www.ndu.edu/nsep/; http://www.govtilr.org/; http://www.jusfc.gov/members.asp; http://www.iawg.gov/member_orgs/; http://www.ed.gov/about/inits/ed/internationaled/interagency.html.

Does the program use strong financial management practices?

Explanation: No internal or external audits have identified any deficiencies. Program staff monitor grant withdrawals to identify excessive drawdowns or periods of inactivity. ED continuously monitors grantee management and administration of Federal funds using GAPS and institutional financial reports.

Evidence: The annual performance reports and the Grant Administration and Payment (GAPS) financial management system are used to ensure funds are spent for intended purposes and managed effectively.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Since 2002, ED has used a Web-based electronic data collection system, initially known as EELIAS, which streamlined grantee reporting, improved data quality, and enhanced the usefulness of program performance data for program managers. The system, now renamed IRIS, has been enhanced during 2006 to improve data collection, provide management reporting tools, and reduce user burden.

Evidence: IRIS enables Title VI program grantee institutions and individual fellows to submit electronic performance report information and data, including project abstracts, project status, performance data, and budget information that are more reliable, comprehensive, and comparable. http://www.ieps-iris.org/iris/ieps/irishome.cfm

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: External peer review panels consisting of reviewers from approved lists of randomly selected international education scholars are used to evaluate, score, and rank applications.

Evidence: Program funds are used to pay for the peer review process. 100% of grants are subject to review.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program staff have a close oversight relationship with grantees as well as a high level of understanding of grantee activities. Program staff have increased monitoring of grantee activities. As a result, grantees continue to demonstrate improvement in project management. Program staff conduct routine financial monitoring to determine if funds are managed according to program and financial requirements. In some, program officers have reduced continuation funding to grantees that were making insufficient progress toward achieving performance goals.

Evidence: Program oversight includes review of annual performance reports, documentation of grantees' use of funds, site visits, email communications, and project director's meetings. This review process has resulted in the adjustment of grantee work plans that were not allowed within the scope of the program. ED's recently-developed Project Evaluation Profile (PEP) will be used to identify outstanding and problematic grantees, and as a basis to assess common elements that can be used to identify best practices and areas requiring technical assistance.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Although the Department collects data on an annual basis, it does not make it readily available to the public or on the Internet at this time. ED has developed an information strategy which includes communicating with grantees and, specifically, publishing grantee-level data on the program's Web site. The Department will make grantee-level data dating back to 2000 available on the web for many Title VI Domestic programs, and continually update the information as warranted.

Evidence: Annual performance reports, interim and final reports in the IRIS system, assessing annual progress for continued funding; program manager PEP assessments.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: ED has established long-term performance measures, but has not collected data yet to determine long-term performance goals.

Evidence: A web-based data collection system, IRIS, will be used to collect regular information on grantee performance.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The nine Title VI Domestic programs have six annual outcome measures, but only the measure for the Foreign Language and Area Studies (FLAS) fellowships has targets and actual date to judge performance against. The FLAS measures tracks the average change in language competency level (pre-test to post-test, as determined by a self-assessment), and has met or exceeded target values for 2005 and 2006.

Evidence: Average language competency change scores for FLAS fellows were 1.2 for 2005 and 1.22 for 2006, meeting or exceeding the target values of 1.2 for each year. See measures section.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: An efficiency measure has been developed: the Federal cost per FLAS fellow that increased their pre-post difference score by at least one point. ED is developing efficiency measure targets.

Evidence: The average cost per successful outcome was $17,439 in FY 2004 and $17,124 in FY 2005. A successful annual outcome is defined as a FLAS fellow increasing their proficiency by at least one point from pre- to post-test.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Unlike other Federal programs, the Title VI Domestic programs focus on strengthening the capability and performance of American education in foreign languages (including languages critical for economic and national security) and in area and international studies. Other Federal programs with a foreign language interest are smaller in size and have different scopes and missions. Given the unique mission and goals of the Title VI program, no comparable program was identified. Under current law, the Title VI programs were created to maintain a broad skill base and a long-term national capacity in language and area studies in about every region of the world, with a focus on the less commonly taught languages.

Evidence: The Department looked at other federal sources for comparable programs. The closest comparable program is the Department of Defense's National Security Education Program's (NSEP) research grant program. However, the appropriation for this program is smaller, at $7 million, than that for the Title VI Domestic programs. Further, its scope and focus are not the same as the Title VI program. NSEP focuses on a narrow set of critical languages. In FY 2002, the three institutional grants awarded by NSEP's Flagship program went to existing Title VI Domestic grantees; and in FY 2001, six out of seven funded projects were connected to Title VI grants. Over 70% of NSEP research grants have been awarded to previous Title VI Domestic grantees.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The programs received a no in question 2.6 based on the National Academies finding that, in general, the programs lacked well-designed evaluations that demonstrated impact. However the six studies described in 2.6 do highlight encouraging trends. (1) Congress directed the National Research Council at the National Academies to conduct a comprehensive review of all Title VI/Fulbright-Hays programs. Despite its many recommendations for improvement, the Committee recognizes that "the Title VI/FH programs have served as a foundation in the internationalization of higher education and should continue to do so." Further, "although the programs have demonstrated some success in achieving the objectives??more meaningful performance measures and well-designed research and evaluation are needed to assess program accomplishments." (2) The Draft Study of Graduate Fellowship Programs, including the FLAS fellowships program that distributed fellowships to institutions between 1997 and 1999, indicated that nearly all FLAS fellows worked after completing their degrees, and most (71 percent) worked in jobs that involved the expertise they had gained through their FLAS-supported study. Nearly all fellows (96 percent) who reported working in a related job considered that job to be part of a career they were pursuing. FLAS fellows reported that their oral and written language skills improved over the course of their FLAS-supported study. Nearly all FLAS fellows viewed their experiences as extremely positive, indicating gratitude for the program, its value to their professional skill development and careers, and the kinds of work they have pursued with these skills. The final report will not be ready for publication until later in 2007. (3) In 2002, a Modern Language Association (MLA) language enrollment survey revealed that while Title VI-supported institutions account for less than 3% of all higher education institutions, they enroll 55% of the graduate enrolled students and 22% of the undergraduate enrollment in LCTLs. (4) The support of Title VI/F-H has been critical in sustaining the nation's capacity in LCTLs and crucial in the training of foreign language experts in critical-need languages. (5) A study of the UISFL program found that the program's impact has been strong and long lasting. Respondents to the study stated that there would not be international education in the U.S. without the Title VI programs. (6) A study of the FLAS program looked at approximately 2,000 FLAS fellows who received their doctorates between 1984-1994. Data suggest that language majors who receive FLAS fellowships chose the more difficult, rarer languages more often than non-FLAS recipients and FLAS Fellows account for a growing proportion of Ph.D.s in the less (32%) and least (60%) commonly-taught languages.

Evidence: (1) National Research Council of the National Academies comprehensive review of International Education and Foreign Languages: Keys to Securing America's Future (Prepublication Copy March 2007). (2) ED's Draft Study of Four Federal Graduate Fellowship Programs: Education and Employment Outcomes of Participants (2007). (3) MLA Language Enrollment Survey (2002). (4) Brecht/Rivers report "Language and National Security in the 21st Century: The Role of Title VI/Fulbright-Hays in Supporting National Language Capacity" (2000) (5) Schneider/Burn research report "Federal Funding for International Studies: Does it Help? Does It Matter? Long-Term Impacts of Federal Funding on International Studies and Foreign Language Programs (1999). (6) Pennock-Roman/Webb/Wooten report "Highlights in the Contrast Between FLAS Recipients Who Completed Their PhD and All U.S. Citizens Who Completed Their PhDs in the Same Fields, 1984-1994 (1997).