Detailed Information on the
Teacher Quality Enhancement Assessment

Assess the results of the recently completed evaluation and utilize those findings to evaluate program performance and modify program management where appropriate.

Collect baseline data, and establish appropriate targets for all of the programs performance measures.

Review and revise the program's performance measures in light of the program changes made by the Higher Education Opportunity Act.

Develop and implement a strategy for making program performance data accessible to the public in a transparent way.

Measure: The percentage of preservice teachers passing subject matter competency tests as part of state licensure requirements.

Explanation:Of the States who receive grant funds to address subject matter competency, the percentage of program completers who pass subject matter competency tests as part of state licensure requirements.

Measure: The percentage of program completers who are highly qualified teachers.

Explanation:The percentage of program completers from Partnership grants who are highly qualified teachers at the time of program completion.

Measure: The percentage of program participants who are retained in the teaching profession three years after program completion.

Explanation:The three-year teacher retention rate for teachers completing teacher preparation programs with Teacher Quality Enhancement Partnership grants.

Measure: Cost per successful outcome: the federal cost per highly qualified teacher.

Explanation:The Federal cost of producing one highly qualified teacher through the program.

Measure: Cost per successful outcome: the federal cost per teacher retained for three years or more.

Explanation:The Federal cost to retain a teacher for three years or more.

Is the program purpose clear?

Explanation: The program's purpose is to improve teacher preparation and professional development programs for current and prospective teachers with the ultimate goal of improving the quality of the Nation's elementary and secondary school teachers and consequently improving student achievement.

Evidence: Sec. 201 [20 U.S.C. 1201] Title II, Part A of the Higher Education Act, as amended: "The purposes of this title are to (1) improve student achievement; (2) improve the quality of the current and future teaching force by improving teacher preparation of prospective teachers and enhancing professional development activities; (3) hold institutions of higher education accountable for preparing teachers who have the necessary teaching skills??; and (4) recruit highly qualified individuals, including individuals from other occupations, into the teaching force."

Does the program address a specific and existing problem, interest, or need?

Explanation: Researchers and policymakers agree that teacher quality is key to improving student achievement. However, much of the research in this area says that teacher preparation programs are not adequately training new teachers and that these new teachers do not receive enough support in their early years of teaching. As a result, many students are taught by underprepared teachers. In addition, research has found that one third of new teachers leave the profession within five years. High levels of attrition are most severe in the highest need areas, where one half of new teachers leave within their first five years.

Evidence: Scheerens/Bosker report 'The Foundations of Educational Effectiveness' (1997); Sanders/Rivers report, 'Cumulative and Residual Effects of Teachers on Future Student Academic Achievement' (1996); National Center for Educational Statistics report 'The Condition of Education' (2001); Office of Postsecondary Education report 'A Highly Qualified Teacher in Every Classroom: The Secretary's Fifth Annual Report on Teacher Quality' (2006); Levine report 'Educating School Teachers' (2006).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: All of the activities allowable under the Teacher Quality Enhancement program can be carried out under other existing Department of Education programs. These programs include the Improving Teacher Quality State Grants program, the Transition to Teaching program, and the Teacher Incentive Fund. In addition, funds are available for institutions of higher education to improve teacher training through other Federal programs, including Improving Teacher Quality State Grants, Transition to Teaching, Office of English Language Acquisition Professional Development Program, Indian Education Professional Development Program, and IDEA Personnel Preparation.

Evidence: Program information for each of the programs listed above may be found at www.ed.gov.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Implementation of the program has not revealed any major flaws in the actual program model that would greatly limit the program's effectiveness or efficiency. However, implementation has revealed several flaws in the authorizing statute: 1) The authorizing statute mandates that funds appropriated for the program are divided between the State, Partnership and Recruitment parts of the program according to a 45:45:10 ratio. In the past, the statutorily-mandated ratio did not reflect the level of demand for program funds, and the Department lapsed $655,000 and $1,416,000 under the State and Recruitment programs respectively in FY 2002. In 2005, this statutory requirement was waived, and funds have since been appropriated as needed among the three parts of the program. While funds can now be distributed as needed, a Government Accountability Office (GAO) report has recommended that eligibility and allocation requirements be clarified further when Congress reauthorizes the Higher Education Act. 2) The authorizing statute creates redundancies within the program as all of the activities of the Recruitment program can be carried out under the State and Partnership programs. 3) States and eligible territories can only receive one grant under the TQE State program. At this time, 52 states and territories have received funding under the program and, as there have already been five competitions for the program, it is not clear whether the seven remaining eligible entities are interested in receiving funding under the program. 4) Although the State, Partnership, and Recruitment parts of the program are able to use grant funds to award scholarships, no funds may be used to administer the scholarship component, which includes a provision that the scholarship must be repaid if a required service commitment is not completed. The Department carries the burden associated with administering the scholarship component and uses up to three staff plus a part-time contractor annually for this purpose.

Evidence: Higher Education Act, Title II, Sections 201-210. Teacher Quality: Approaches, Implementation, and Evaluation of Key Federal Efforts, 2007, United States Government Accountability Office (GAO-07-861T).

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The program's focus on institutions of higher education and States ensures that both prospective and veteran teachers have the content knowledge and teaching skills they need to help all students reach high academic standards. The focus on institutions of higher education is integral to improving teacher preparation, strengthening teacher quality and, ultimately, raising student achievement because the vast majority of teachers are trained by these institutions. Since States oversee the teacher certification process and establish student achievement standards, it is appropriate that the State program supports better alignment of teacher certification with student achievement standards. In addition, authorizing language for the Partnership program mandates that institutions of higher education partner with high-need schools or school districts in developing and implementing their programs, ensuring that the program resources are targeted to achieve the maximum benefit.

Evidence: Higher Education Act, Title II, Sections 201-204. Partnerships for Reform: Changing Teacher Preparation Through The Title II HEA Partnership: Final Evaluation Report, 2006.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The Department has developed four long-term performance measures for the Partnership and State grants participants that reflect the program's purpose, which is to improve teacher preparation and professional development with the ultimate goal of improving student achievement. Because many schools and districts are not yet able to link individual teachers to the performance of their students, the program uses two measures as proxies for teacher quality which look at teacher characteristics. The first measures the percentage of preservice teachers in funded states passing subject matter competency tests as part of state licensure requirements. The second measures the percentage of program completers who are highly qualified teachers (according to the NCLB definition). In addition, the program has adopted a third measure looking at teacher retention as a proxy for quality of teacher preparation. The third performance measure is the three-year teacher retention rate for teachers completing teacher preparation programs with Teacher Quality Enhancement Partnership grants. In 2008, the program adopted a fourth performance measure focused on program outcomes: the percentage of students enrolled in schools and/or districts participating in Teacher Quality Enhancement partnerships who score proficient or better on the States' assessments in reading/ language arts or mathematics.

Evidence: ED's FY 2008 Performance Plan; FY 2008 TQE Program Performance Plan; FY 2007 Program Performance Report; Higher Education Act Title II State Reporting System at http://title2.ed.gov.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has four long term performance measures. The performance measures address subject matter competency, highly qualified teachers, teacher retention, and student achievement. The primary measure for the State program relates to State subject matter competency tests. The long term target for this measure is for 98 percent of program completers statewide to pass subject matter competency tests in 2012, as part of State licensure requirements. It would be a considerable achievement for teacher preparation programs in the relevant States to achieve rates near 100 percent considering the diverse mix of teacher preparation programs, often addressing diverse student populations. The percentage of program completers passing varies by State and will reflect the specific States that receive State grants in any given year. The distribution of pass rates for 2003-04 was as follows: 97% or above: 18 states; 95-96%: 12 states; 90-94%: 7 states; and 89% and below: 7 states The average value was 95 percent. [Not all states tested program completers in 2003-04.] For the Partnerships program, the first measure is the percentage of program completers from Partnership grants who are highly qualified teachers at program completion. The long term target for this measure is 100 percent, meaning that from 2008 forward the target is for every graduate of a Partnership teacher preparation program to be highly qualified. This goal is consistent with the No Child Left Behind Act, which has had the goal of having all classrooms staffed with highly qualified teachers since 2006. The Partnership program has recently established a second measure relating to teacher retention. The initial long-term target for this measure is a 72 percent three-year retention rate. This target has been established based on national teacher retention data from the National Center for Education Statistics' Schools and Staffing Survey (SASS). National data from re-analyses of the 2004-05 SASS one-year teacher follow-up survey show a 72 percent three-year retention rate. These data are from a nationally representative sample of schools. Since the Teacher Quality Enhancement program targets high-need schools that generally do significantly worse than the national average with regard to retention, the program's goal of bringing the teacher retention rates in participating schools up to the national average initially is ambitious. Once baseline and trend data are collected, program staff will revise targets as appropriate. The Department is working to establish baselines and targets for the student achievement measure.

Evidence: ED's FY 2008 Performance Plan; FY 2008 TQE Program Performance Plan; FY 2007 Program Performance Report; Higher Education Act Title II State Reporting System at http://title2.ed.gov.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The Department has developed four annual performance measures for the Partnership and State grants participants that reflect the program's purpose, as well as two annual efficiency measures for the program. The annual performance measures are the same as the long term performance measures and track annual performance toward the program's long term goals. Because many schools and districts are not yet able to link individual teachers to the performance of their students, the program uses two measures as proxies for teacher quality which look at teacher characteristics. The first measures the percentage of preservice teachers in funded states passing subject matter competency tests as part of state licensure requirements. The second measures the percentage of program completers who are highly qualified teachers (according to the NCLB definition). In addition, the program has adopted a third measure looking at teacher retention as a proxy for quality of teacher preparation. The third performance measure is the three-year teacher retention rate for teachers completing teacher preparation programs with Teacher Quality Enhancement Partnership grants. In 2008, the program adopted a fourth performance measure focused on program outcomes: the percentage of students enrolled in schools and/or districts participating in Teacher Quality Enhancement partnerships who score proficient or better on the States' assessments in reading/ language arts or mathematics. Additionally, ED has developed two efficiency measures for the program: 1) the Federal cost per Teacher Quality Enhancement program completer who is highly qualified, and 2) the Federal cost per Teacher Quality Enhancement program completer who remains in teaching for 3 years after program completion. Data from these measures help the program assess the cost effectiveness of each Partnership project on an annual basis.

Evidence: ED's FY 2008 Performance Plan; FY 2008 TQE Program Performance Plan; FY 2007 Program Performance Report; Higher Education Act Title II State Reporting System at http://title2.ed.gov.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has four annual performance measures and two annual efficiency measures. The performance measures address subject matter competency, highly qualified teachers, and teacher retention. The primary measure for the State program relates to State subject matter competency tests. The 2008 target for this measure is for 96 percent of program completers statewide to pass subject matter competency tests as part of state licensure requirements. Annual targets increase by one percent every other year, tracking the program's progress towards its long term goal of 98 percent. For the Partnerships program, the first measure is the percentage of program completers from Partnership grants who are highly qualified teachers at program completion. The 2008 target for this measure is 100 percent, consistent with the No Child Left Behind Act, which has had the goal of having all classrooms staffed with highly qualified teachers since 2006. The Partnership program has recently established a second measure relating to teacher retention. The initial annual target for 2008 for this measure is a 67 percent 3-year retention rate. The program's annual targets will increase by one percentage point each year until they reach the long term target of 72 percent. Once baseline and trend data are collected, program staff will revise targets as appropriate. The Department is working to establish baselines and targets for the student achievement measure. The program has two efficiency measures, one relating to the cost of program completers and the other relating to the cost of retention. The target for the program completer efficiency measure assumes a five percent annual increase in the number of program completers, with no increase in program cost. We believe a sustained five percent annual increase in the overall number of program completers represents a challenging rate of growth for this program, given cyclical fluctuations in the numbers of program completers produced (i.e., few or no program completers are available at the start of a new grant cycle, and numbers increase over the life of a grant). This efficiency measure is based on the number of program completers, which itself is closely linked to the first two performance measures: increasing the number of program completers while maintaining high pass and retention rates is an ambitious goal. The retention efficiency measure is closely tied to the teacher retention performance measure. While the performance measure assesses whether the partnership programs have improved their ability to produce highly qualified teachers who remain in teaching for at least three years, the efficiency measure looks at whether these improvements are made with the same or similar resources. Targets will be established once baseline data are available for this measure.

Evidence: ED's FY 2008 Performance Plan; FY 2008 TQE Program Performance Plan; FY 2007 Program Performance Report; Higher Education Act Title II State Reporting System at http://title2.ed.gov.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All program partners commit to and work toward the goals of the program. The Department has shared the existing program performance measures with grantees through pre-application workshops, application packages, closing date notices and Project Directors Meetings. As part of the application process, all partners commit to measuring their performance using these program performance measures. Program officers review grantees' progress toward these goals annually; grantee performance reports show continuing progress in these areas. With the recent development of new annual and long-term performance measures on teacher retention and student achievement in Partnership Grant schools and/ or districts, partners have not yet been able to commit to these new measures. The program plans outreach to its grantees to communicate the new measures and integrate them into each grantee's work plan. The program will revise its Annual Performance Reports to gather the necessary data for the new measures.

Evidence: ED's FY 2008 Performance Plan; TQE's 2008 Program Performance Plans and FY 2007 Programs' Performance Reports; A Guide to States and Partnerships for the Preparation of Applications for: TQE Grants.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: A four-year national evaluation of the Partnership program conducted by American Institutes for Research and SRI International was initiated in FY 2001 and was published in FY 2006. The evaluation examined implementation of the Partnership grants. The evaluation's goal was to learn about the collaborative activities taking place in partnerships and the ways projects changed over the course of the grant. The study was also designed to examine approaches to preparing new and veteran teachers and to assess the sustainability of project activities after the grant ends. Specifically, the evaluation was designed to answer four key research questions: 1. Did partnerships fulfill their program mandate to encourage colleges and universities to partner with and address the needs of high-need districts and schools? 2. Did partnerships undertake activities designed to improve the academic content knowledge of new or veteran teachers? 3. Were changes to the student internship component associated with partnership efforts to improve teacher preparation? 4. Did partnership initiatives address the accountability concerns about teacher preparation? The evaluation focused on the 25 grantees in the 1999 cohort of the Title II Partnership grant program. A diverse cohort, these grantees consisting of 65 colleges and universities, 28 community colleges, 178 school districts, and 821 elementary schools in 25 different states, received a total of more than $171 million over the 1999-2004 period. Over four years, the partnership evaluation surveyed nearly 300 institutions of higher education (IHEs) and district project representatives and more than 500 principals at the start of implementation (2000-01; 2002-03) and during the third year (2003-04). The study also included secondary data analyses using publicly available data on school characteristics, school-level achievement data, and pass rates on teacher assessments reported as part of the Title II HEA reporting requirements. In order to augment the quantitative data, the study also included qualitative data from case studies of five diverse projects, that included two consecutive week-long visits.

Evidence: Partnerships for Reform: Changing Teacher Preparation Through the Title II HEA Partnership Program, 2006. http://www.ed.gov/rschstat/eval/teaching/title2hea/changing-teacher-prep-final.pdf

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Program performance changes are not identified with changes in funding levels.

Evidence: N/A

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program has identified strategic planning deficiencies and taken meaningful steps to address these deficiencies. The program now shares program performance goals with grantees through pre-application workshops, application packages, closing date notices, and Project Directors' Meetings. The program has focused a significant portion of its annual meetings on questions related to evaluation and performance measurement. For example, in FY 2004 a workshop was held in Washington, D.C. for grantees which focused exclusively on strategic planning and project evaluation. Also in 2004, the Department began to calculate the program's efficiency measures at the grantee level as part of an Office of Postsecondary Education (OPE)-wide strategy to improve efficiency data and performance. In addition, in 2008 the Department worked with OMB to develop two new long term/ annual performance measures for the program.

Evidence: ED's FY 2008 Performance Plan; TQE's 2008 Program Performance Plans and FY 2007 Programs' Performance Reports; A Guide to States and Partnerships for the Preparation of Applications for TQE Grants.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Grant recipients are required to submit Annual Performance Reports and a Final Performance Report at the end of each project. Grantees are also expected to maintain a sustained and substantive dialogue with the Department about the progress they are making toward their performance goals. In addition, student achievement data are collected by the Department annually. In 2006 and 2007, student achievement data, along with data from the program's performance measures were used in the creation of grantee-level analyses. These data were used by program managers to better focus monitoring and technical assistance resources where they were most needed, as well as to highlight exemplary grantees. For example, data from the grantee-level analyses helped identify several partnership grantees who may require monitoring and technical assistance or consultation during the remainder of their grants: three grantees had fewer than ten program completers in both 2005-06 and 2006-07; three other grantees had sharply decreasing numbers of program completers; and five grantees had partner districts where student achievement decreased. On the other side, these data also highlighted a number of grantees that had partner districts whose student acheivement levels improved. Additionally, in response to data from the Annual Performance Reports highlighting key grantee concerns, the program has focused a significant portion of the program's annual project directors' meetings on questions related to evaluation and performance measurement. Along the same lines, in 2004, a workshop was held in Washington for grantees that focused exclusively on strategic planning and project evaluation.

Evidence: FY 2006 and 2007 Grantee Level Analyses; FY 2006 and 2007 Annual Performance Reports; FY 2005 TQE Recruitment Application Package.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The Department's managers are subject to EDPAS, a system which links employee performance to relevant Strategic Plan goals and is designed to measure the degree to which a manager contributes to improving program performance. The EDPAS agreements of all of the program managers are linked with the performance of projects or the program. The EDPAS standards for Teacher and Student Development Programs Service (TSDPS) program managers require program managers to set forth strategies for implementing GPRA and Strategic Plan initiatives related to TSDPS. TSDPS program officers are held accountable for assessing project performance and monitoring the progress of projects in achieving program goals and objectives. Grantees and program partners are held accountable for performance results. Performance goals and objectives are incorporated into program application packets and notices. Applicants are required to discuss the processes that they intend to implement to achieve these goals and objectives, as well as the data that they will track in order to measure their progress to achieving these goals and objectives, as part of the application process. Successful applicants are required to submit data tracking their progress to achieving these goals and objectives as part of their annual performance reports. Annual meetings give ED program managers a forum in which they can discuss expectations about cost, schedule, and performance results with grantees.

Evidence: Education Department Performance Appraisal System (EDPAS).

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: At the Federal level, all funds are obligated according to an annual spending schedule that is established at the beginning of the fiscal year. At the partner level, grantees are obligating funds at a reasonable rate. Further, to ensure that funds are properly managed, the Department monitors grantees' financial records, including quarterly reports monitored by the Office of the Assistant Secretary for Office of Postsecondary Education to alert staff of any grantees' excessive drawdown or inactivity of Federal funds. In addition, to ensure that funds are properly managed, the Department has strengthened its financial management practices by implementing the Grantee Electronic Monitoring System (GEMS), a new software tool that enables grant program staff to better monitor their grants and helps to prevent potential fraud, waste, and abuse.

Evidence: At the start of each fiscal year, the program establishes an Annual Spending Plan that governs the timing of all obligations and ensures that funds are spent for the intended purposes. The Department has not lapsed TQE funds due to a lack of quality applications because, beginning in 2005, Congress has enacted appropriations language overriding the statutory requirement that funds be allocated among the three parts of the program according to a 45:45:10 ratio. TQE grantees have obligated funds at approximately the same rate as grantees in other ED higher education programs (GAPS, GEMS).

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The Department has developed two efficiency measures for the program: 1) the cost per student who completes his or her program of study and becomes a highly qualified teacher (as defined by No Child Left Behind, Title IX, Sec. 9101) and 2) the cost per program completer who remains in teaching for three years after program completion. Data from these measures help the program assess the cost effectiveness of each Partnership project on an annual basis. The first efficiency measure is calculated as the allocation for partnership grants divided by the number of highly qualified teacher candidates graduating from grantee postsecondary institutions. The second efficiency measure is calculated as the allocation for partnership grants divided by the number of program completers who remain in teaching for three years after program completion. In 2004, the Department began to calculate the program's first efficiency measure at the grantee level as part of an Office of Postsecondary Education-wide strategy to improve efficiency data and set targets for improving efficiency. This effort emphasizes increased communication with grantees regarding performance data. The Department has other procedures in place to achieve efficiencies and cost effectiveness. For example, the Department relies on competitive sourcing to provide technical and other administrative support. Contractual support provides for electronic submission of applications, as well as electronic submission of performance reporting, in addition to expert administrative and technical support. The program is using a contractor to help support the scholarship initiative. Activities include support of an ACCESS database to track funds awarded and students' fulfillment of their teaching obligation. This usage of private industry resources increases the cost effectiveness of the programs. Additionally, project-specific outcome and efficiency analyses were completed in March 2008 and these analyses are currently being used to inform the development of appropriate efficiency targets and program changes that could be implemented to boost cost effectiveness.

Evidence: Data for FY 2004 show that the Federal cost per program completer is $2,932 (only a small number of grantees were able to report on highly qualified teachers; therefore, results for that year may not necessarily have been representative of a full set of grantee institutions). Data for FY 2005 show that the Federal cost per program completer is $4,728 (FY 2005 appropriation for TQE grantees $15,106,625 divided by 3,195 program completers who were certified as highly qualified teachers). Data for FY 2006 show that the Federal cost per program completer is $4,427 (FY 2006 appropriation for TQE grantees $28,356,277 divided by 6,406 program completers who were certified as highly qualified teachers). Data for FY 2007 show that the Federal cost per program completer is $3,459 (FY 2007 appropriation for TQE grantees $35,270,166 divided by 10,196 program completers who were certified as highly qualified teachers). Multiple contracts provide database, technical assistance, and reporting support. Electronic applications and annual and interim performance reports also create efficiencies in data collection, analysis, and reporting and reduce burden on grantees.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The Department has implemented strategies to encourage collaboration and coordination between the Teacher Quality Enhancement program and other programs addressing teacher quality issues, both within the Office of Postsecondary Education and across the Department as a whole. For example, the program has maintained relationships with the Transition to Teaching program. The team leader for Transition to Teaching program has attended and presented at Teacher Quality Enhancement project directors meetings to share information about their program. The team leader for Teacher Quality Enhancement has done the same at Transition to Teaching meetings. In addition, program officers are often able to meet with Teacher Quality Enhancement grantees who are attending the Transition to Teaching meeting, thus providing an opportunity for further interaction with Teacher Quality Enhancement grantees. At Transition to Teaching's request, Teacher Quality Enhancement program officers also provided an extensive briefing on how Teacher Quality Enhancement administers the program's scholarship component. Within the Office of Postsecondary Education, the two programs that most directly address teacher quality (Teacher Quality Enhancement and Teachers for a Competitive Tomorrow) have been placed under the leadership of a single Federal manager, in order to further encourage collaboration and coordination.

Evidence: Teacher Quality Enhancement Project Directors' Meeting Agenda, 2006; Transition to Teaching Project Directors' Meeting Agenda, 2006; Teacher Quality Enhancement Scholarship Briefing Materials, 2007; Office of Postsecondary Education Organizational Chart, 2008.

Does the program use strong financial management practices?

Explanation: No internal control weaknesses have been reported by auditors. The Department continuously monitors grantee management and administration of federal funds using its grants management system and institutional financial reports. In 2004, the Office of Postsecondary Education within the Department implemented an e-Monitoring system enabling program staff to better mitigate potential fraud, waste, and abuse. In late 2007, the Department further enhanced program monitoring by rolling out the Grant Electronic Monitoring System (GEMS), which allows staff to classify each grant into categories based on grantee performance, and maintain electronic records of phone, correspondence, and e-mail communications with grantees. The Department's systems enable program officers to track all funds applications made by an institution. Also, these systems are used to identify institutions determined not to have applied funds appropriately and to withhold funding, if necessary.

Evidence: Teacher Quality: Approaches, Implementation, and Evaluation of Key Federal Efforts (GAO-07-861T). All TQE funded grantee institutions are documented in GAPS and GEMS. In FY 2007, approximately $3.6 million was withheld in continuation awards, with funds diverted to funding new projects off of the FY 2005 Recruitment slate. Funds were withheld in two instances due to failure to expend at an appropriate pace; in the third instance funds were withheld when a preliminary IG report indicated that the grantee could not meet its match.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: A 2003 Government Accountability Office (GAO) report on the program identified important management deficiencies, including the lack of an effective system for communicating program information to grantees. In response, the Department has improved communications efforts in a number of areas, including hosting national grantee meetings for program participants. In addition to the annual project directors meeting, the program organized a mandatory one-day workshop for all grantees who award scholarships. The purpose of this workshop was to improve grantee management of scholarship programs and ensure that program regulations governing scholarships were understood. When FY 2007 Recruitment grants were awarded, all new grantees were invited to a one-day meeting to be briefed on grant management and compliance issues as well as on scholarship administration.

Evidence: GAO report GAO-03-6.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Independent peer review panels are used to score and rank all applications.

Evidence: Program funds are used to pay for the peer review process. 100 percent of grants are subject to peer review; Section 205(b) of Title II of the Higher Education Act.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Program officers provide continual oversight and coordination of activities of grantees. The program staff review annual and interim performance reports to ensure that the data provided are of sufficiently high quality to provide evidence of the individual project's progress in achieving the program's objectives and goals. In addition, the program office has significantly enhanced its monitoring activities for grantees by conducting more site visits, placing grantees on "route payment" (which involves a heightened scrutiny of individual grant payments) when serious performance and/or fiscal issues arise, withholding continuation awards, requesting audits, and requiring grantees to conduct external audits using non-federal funds. The Department has also developed new procedures for improving the monitoring of expenditures in response to Investigator General concerns with several Office of Postsecondary Education programs. Two systems are currently used to ensure that all related grant financial management practices are followed. In addition, the Department has added new features to these systems to significantly improve the quality of monitoring. Some of these features currently include (1) a Monitoring Index which identifies postsecondary institutions that may be at risk, allowing program managers to focus their efforts toward likely problems; (2) more institutional level data, enabling closer monitoring of grants from the same institution in other programs; and (3) program level reports for managers to alert service area directors to issues in their programs. Future system upgrades, scheduled for 2008 implementation, include (1) incorporation of accreditation adverse actions, (2) links to audit findings, (3) capability of tracking grants by population served, academic discipline, or intervention employed, and (4) data on site visits including responses to findings. Additional important oversight tools used in managing the program are the Audit Accountability and Resolution Tracking System (AARTS) and PRE-AARTS systems. These systems help the Office of Postsecondary Education to better manage compliance with Government Accountability Office and Office of the Inspector General audits. Finally, Office of Postsecondary Education Strategic Planning staff have been able to provide support to program staff by preparing analyses of performance and efficiency measures at the grantee level and by providing program improvement suggestions through the identification of outliers. As issues are identified, program staff address them through telephone interactions and email; if issues persist or cannot be addressed remotely, site visits are undertaken. Because of the Department's increased emphasis on fiscal monitoring, these reviews have been triggered in most cases by concerns over drawdown (which does link to project implementation), match, and other fiscal issues. However, the program office always devotes a significant portion of any visit to a program review, and grantees receive technical assistance on all areas related to the project.

Evidence: GEMS, the OPE grant monitoring database, tracks financial and performance issues throughout a project's lifetime; excessive drawdowns, declines in expenditures, changes in leadership, are all tracked through GEMS. In addition, all communications with grantees can be recorded in GEMS. GEMS can also be used to broadcast 'blast' emails to all or a subset of project directors, which provides a way to address problems that may have broad applicability as soon as they are identified. AARTS is the Department's single database system used for tracking, monitoring, and reporting on the follow-up status of the audits. Pre-AARTS helps to ensure that action items are successfully completed and accurately documented. The system also generates reminders to key players to complete their audit responsibilities on time and stores dated communications and other supporting documentation for corrective actions. Program staff track grant progress through examination of grantee annual and final performance reports (APRs, FPRs). Analyses of performance and efficiency measures at the grantee level are available on the Web. Program oversight includes documentation of grantees' use of funds, email communications, and project directors' meetings. Grant folders, as well as the GEMS and AARTS systems, are reviewed for compliance.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: Performance data are now reported in several formats and through a variety of means, including on the Department's website. Basic award information and grant amounts are also available on the Department's website. The program office collects and compiles data from performance reports into a program profile report, which includes grantee level data analysis. This report is available on the program's website and makes performance data available to the public at both an aggregated and disaggregated level. Performance and efficiency measure information are provided at the grantee level through these reports. The FY 2006 and 2007 grantee performance analyses have been posted on the Department's website.

Evidence: Grantee performance and efficiency analyses for 2006 and 2007 can be accessed on the performance section of ED's website at http://www.ed.gov/programs/heatqp/performance.html beginning.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Data are available for two of the four long-term performance measures for the program. The percentage of preservice teachers passing subject matter competency tests as part of state licensure requirements for two of the three most recent years (2005: 96%, 2006: 100%) suggest that the program is on track to meet its long-term targets, despite the drop in performance in 2007. The percentage of program completers who are highly qualified teachers for each of the three most recent years (2005: 95%, 2006: 97%, 2007: 99%) demonstrate that the program is on track to exceeding the long-term target of 96% originally set for 2010-12. Because of the changes in program implementation over time, the Department has found it necessary to develop two new long-term performance measures for the program. It is expected that data for these new measures will be available in FY 2010.

Evidence: FY 2007 Annual Performance Reports; 2008 Program Performance Plan; Higher Education Act Title II data.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Data are available for two of the four annual performance measures for the program. The program has exceeded annual performance goals in two of the three most recent years for the measure of the percentage of preservice teachers passing subject matter competency tests as part of state licensure requirements and exceeded annual performance goals in each of the three most recent years for the measure of the percentage of program completers who are highly qualified teachers. Because of the changes in program implementation over time, the Department has found it necessary to develop two new long-term performance measures for the program. It is expected that data for these new measures will be available in FY 2010.

Evidence: FY 2007 Annual Performance Reports; 2008 Program Performance Plan; Higher Education Act Title II data.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program currently has performance data for one of the program's two efficiency measures: the annual cost per program completer who is a highly qualified teacher. These data helps the program assess the cost effectiveness of each Partnership project on an annual basis. The annual cost per successful outcome for the Partnership program was $2,932 in 2005, $4,728 in 2005, $4,427 in 2006 and $3,459 in 2007, suggesting small gains in efficiency since 2005. Data for the second efficiency measure are not yet available.

Evidence: FY 2009 Draft Program Performance Plan for TQE.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Comparable data are not yet available to compare the Teacher Quality Enhancement with other Federal teacher quality programs. However, the Department will be able to make some comparisons between teacher quality programs in future years as the new performance measures are implemented.

Evidence: Other programs that focus on teacher preparation (for example, English Language Acquisition National Development, Indian Education Professional Development, and Transition to Teaching) all have measures that are similar to those used in the TQE program.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: An independent evaluation of the Partnership program was concluded in 2006. The evaluation found that the grantees had formed meaningful partnerships in which collaboration was pervasive and sustained and enjoyed support at both the staff and leadership levels, and that the Partnerships were targeting high need schools and school districts. The evaluation also found that the Partnerships had substantive impacts on teacher preparation programs, noting particularly the impact the grant had on improving the alignment of course work with State standards and in increasing the amount of field experience to which pre-service teachers were exposed. The research also noted positive changes in the supervision of student teachers and the fact that many of the Partnerships followed the Professional Development School reform model, where education faculty are in residence at schools where their students are doing their student teaching. These are all attributes that previous research has found to be associated with improvements in teacher quality. Overall, the research found that the implementation of Partnership grants was largely in line with the legislative expectations for the program. Because of limited evaluation funding, program evaluations are not being carried out on the State and Recruitment programs. A 2003 Government Accountability Office report also noted that Teacher Quality Enhancment grantees have formed meaningful partnerships that appear to have had positive impact in improving the quality of teacher preparation among partner postsecondary institutions. The results of these evaluative efforts are currently being used to improve program management and evaluate effectiveness.

Evidence: Partnerships for Reform: Changing Teacher Preparation Through the Title II HEA Partnership Program, 2006. http://www.ed.gov/rschstat/eval/teaching/title2hea/changing-teacher-prep-final.pdf. GAO report GAO-03-6.