Detailed Information on the
Magnet Schools Assessment

Make program performance data and evaluation findings available to the public in a transparent and timely manner.

Work with the subset of MSAP grantees that are conducting rigorous evaluation to provide assistance in developing sound performance measures, ensuring treatment fidelity, and improving data collection and reporting. (First phase (evaluation analysis) completed by spring 2009.)

Continue to develop strategies to evaluate the educational achievement and desegregation impacts of the Magnet Schools program by implementing a focused evaluation, with the first round of site visits and data collection (phase 1) completed by late spring of 2009.

Collect data on annual and long-term performance measures and use these data to develop performance targets as well as to inform technical assistance and funding recommendations.

Develop strategies to evaluate the educational achievement and desegregation impacts of the program, with the first phase (feasibility study) completed in early 2008.

Measure: Percentage of magnet schools whose student applicant pool reduces, prevents, or eliminates minority group isolation.

Explanation:

Measure: Percentage of magnet schools whose students from major racial and ethnic groups meet or exceed State's annual progress standards in reading.

Explanation:The measure and established performance targets address the statutory goal of all students meeting 100 percent proficiency in reading by SY 2013-2014.

Measure: Percentage of magnet schools whose students from major racial and ethnic groups meet or exceed State's annual progress standards in mathematics.

Explanation:The measure and the established performance targets address the statutory goal of all students meeting 100 percent proficiency in mathematics by SY 2013-2014.

Measure: Percentage of magnet schools that received assistance that are still operating magnet school programs 3 years after Federal funding ends.

Explanation:

Measure: Percentage of magnet schools that received assistance that meet State standards at least 3 years after Federal funding ends.

Explanation:

Measure: Cost per Student in a Magnet School

Explanation:Data will be collected through annual grantee performance reports.

Is the program purpose clear?

Explanation: The purpose of the program is to establish and operate magnet schools under a court-ordered or federally approved desegregation plan. Magnet programs aim to eliminate, reduce, or prevent minority group isolation in elementary and secondary schools while strengthening students' knowledge of academic subjects.

Evidence: Section 5301 (b): "The purpose of this part is to assist in the desegregation of schools served by local educational agencies by providing financial assistance to eligible local educational agencies for- (1) the elimination, reduction, or prevention of minority group isolation...(2) ... achieving systemic reforms and providing all students the opportunity to meet challenging State academic content standards and student academic achievement standards..."

Does the program address a specific and existing problem, interest or need?

Explanation: Minority group isolation in elementary and secondary schools continues to remain a problem. In addition, the program promotes educational choice, a major priority of No Child Left Behind and the Administration.

Evidence: Studies document continuing issues with educational access and racial segregation. Enrollment data from the NCES Common Core of Data (2000-2001) compiled in "Race in American Public Schools: Rapidly Resegregating School Districts" (Harvard Civil Rights Project) found that almost all schools with enrollments greater than 25,000 students show lower levels of inter-racial exposure since the mid-eighties.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: This is the only Federal program that provides grants focusing on school desegregation. Only a small number of States offer school desegregation assistance and, unlike the Federal program, often support costs (such as transportation costs) but do not focus exclusively on planning and implementing new or significantly revised magnet schools.

Evidence:  

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: There is no evidence indicating that program has fundamental design flaws.

Evidence:  

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The program is effectively targeted.

Evidence: All LEAs must have an Office of Civil Rights approved desegregation plan in order to receive an MSAP grant. In addition, the priorites established in section 5306 of the ESEA provide for targeting to applicants that demonstrate the greatest need for assistance.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program recently developed long-term performance measures that focus on outcomes and reflect the purpose of the program.

Evidence:  

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Baseline data are not yet available. Targets will be established once there data are collected.

Evidence:  

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The Department has a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals.

Evidence: The previous annual measure for the program, which is almost identical to the measure being baselined in 2005, demonstrated progress toward the long-term goal of reducing minority group isolation.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program recently revised the annual measures; baseline data have not yet been collected.

Evidence:  

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Grantees commit to and work toward the annual goals of the program. Annual measures are included in the application package and grantees are required to submit annual progress reports that address the measures.

Evidence:  

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The Department, contracting with the American Institute of Research has conducted a series of independent program evaluations. Also, ED is conducting a feasibility study to examine the viability of a study of student acheivement in magnet schools using random assignment.

Evidence:  

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: ED has not satisfied the first part of the question because program performance changes are not identified with changes in funding levels. However, ED has satisfied the second part of this question in that ED's budget submissions show the full cost of the program (including S&E).

Evidence:  

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The Department recently revised annual indicators and developed long term indicators for this program.

Evidence:  

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Department collects annual performance reports to oversee grantee performance. In addition, information is collected for a series of independent evaluations.

Evidence: The program has used information to help manage the program and improve performance. When grantee reports identify weaknesses, the program works with the grantee to improve performance. For example, if grantee data does not show progress toward reaching desegregation objectives, the Department works with grantees to make sure a corrective action plan is put in place. Additionally, assessing the information provided by grantees on the performance indicators has led to refining the indicators to more appropriately measure school isolation and academic achievement.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Currently, ED cannot demonstrate how federal managers and program partners are held accountable for program goals. ??However, the Department has initiated several efforts to improve accountability in its programs. ??First, ED is in the process of ensuring that EDPAS plans -- which link employee performance to relevant Strategic Plan goals and action steps ' hold Department employees accountable for specific actions tied to improving program performance. ??ED is also revising performance agreements for its SES staff to link performance appraisals to specific actions tied to program performance.?? Finally, ED is reviewing its grant policies and regulations to see how grantees can be held more accountable for program results.

Evidence: The President's Management Agenda scorecard (Human Capital and Budget & Performance Integration initiatives) notes ED's efforts to improve accountability. ??The Department's Discretionary Grants Improvement Team (DiGIT) recommendations indicate that ED is reviewing its grant policies and recommendations.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Funds are obligated within the timeframes set out by Department schedules and used for the purposes intended.

Evidence: Program staff monitor to make sure that grantees are drawing down funds at an acceptable rate.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: To date, the Department has not established procedures for this program to measure and achieve efficiencies in program operations. ??However, ED is in the process of developing its competitive sourcing Green Plan, and is working to improve the efficiency of its grantmaking activities. ??The Department has also established a strengthened Investment Review Board to review and approve information technology purchases agency-wide. ??

Evidence: Department Investment Review Board materials. ??ED's Discretionary Grants Improvement Team (DiGIT) recommendations.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The MSAP program coordinates with the Office of Civil Rights and with the Training and Advisory Services program.

Evidence: The MSAP coordinates with the Office of Civil Rights to ensure that proposed desegregation plans upon which applications are based meet current legal standards. In addition, the program coordinates with Training and Advisory Services by providing centers with up-to-date information and, when possible, conducting joint technical assistance activities.

Does the program use strong financial management practices?

Explanation: Recent agency-wide audits have not identified deficiencies in the financial management of this program.

Evidence:  

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Major internal management deficiencies have not been identified for this program.

Evidence:  

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Independent peer review panels are used to score and rank all applications.

Evidence:  

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The Department maintains information on grantee activities through annual reports, meetings with grantees, and other technical assistance activities.

Evidence: The program's monitoring efforts focus on the review of annual performance reports as well as meetings and regular telephone contact with grantees. Program staff also perform on-site visits.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: While the program collects grantee performance data on an annual basis, the information has not been made available to the public. Education is developing a department-wide approach to improve the way programs provide performance information to the public. In 2004, Education will conduct pilots with selected programs to assess effective and efficient strategies to share meaningful and transparent information.

Evidence: The program is working on plans to make performance data available on the magnet schools web site.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Department recently developed long-term performance goals for this program and will be collecting baseline data within the next year.

Evidence:  

Does the program (including program partners) achieve its annual performance goals?

Explanation: According to a 2004 study, Evaluation of the Magnet Schools Assistance Program- 1998 Grantees, the program was moderately successful at preventing, eliminating, or reducing minority group isolation in MSAP schools. Adjusting for district wide demographic trends, about 57 percent of the desegregation-targeted schools succeeding in preventing, eliminating, or reducing minority group isolation.

Evidence:  

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Department has not yet developed appropriate efficiency measures for this program.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The MSAP program is structured to institutionalize improvements by providing competitive grants and funding for a sustained period of time. In contrast, most State programs are targeted to either court ordered desegregation plans or are earmarked for specific schools and provide a single year of funding at a time. However, no data are available on which to base comparisons of program performance.

Evidence:  

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Independent evaluations indicate that the program is moderately effective and achieving results.

Evidence: A 2004 study, Evaluation of the Magnet Schools Assistance Program- 1998 Grantees, indicated that about 57 percent of the desegregation-targeted schools succeeding in preventing, eliminating, or reducing minority group isolation. Two studies of student achievement were conducted by AIR. Once concluded that the MSAP-supported schools were most successful in meeting or making progress toward their student achievement goals they had set for the first year of magnet program operation, but continued improvement over longer time periods proved more difficult. The other study, an analysis of statewide test data, found that MSAP-supported elementary magnet schools made noticeable progress in reading and mathematics during the grant period. However, when the analysis controlled for changes in the demographic composition of the schools, the gains were not significantly different from those exhibited by non-MSAP schools with similar characteristics.