Detailed Information on the
21st Century Community Learning Centers Assessment

Hold States accountable for meeting program performance goals during fiscal years 2008 and 2009.

During fiscal year 2008, support research on after-school models that effectively improve student achievement.

Develop a meaningful efficiency measure for the program, collecting performance information, and setting performance targets for the measure.

Reverse last year's budget policy of reducing funding based on the preliminary findings from a rigorous evaluation of the antecedent program. Instead, level funding of $999 million is proposed for FY 2005.

Ensure that the program has a data collection and evaluation system that will allow Education to analyze whether State and school district performance goals are being met.

Implement a technical assistance strategy to identify and disseminate promising and proven instructional practices in academic areas.

Measure: The percentage of all 21st Century regular program participants whose English grades improved from fall to spring.

Explanation:

Measure: The percentage of all 21st Century regular program participants whose mathematics grades improved from fall to spring.

Explanation:

Measure: Percentage of students with teacher-reported improvements in student behavior.

Explanation:The method of data collection for this measure changed beginning in 2004. Therefore, data from 2004 are not comparable to data from prior years.

Measure: Percentage of students with teacher-reported improvements in student behavior.

Explanation:The method of data collection for this measure changed beginning in 2004. Therefore, data from 2004 are not comparable to data from prior years.

Measure: The percentage of SEAs that submit complete and accuarate data on 21st Century Community Learning Centers performance measures in a timely manner will increase.

Explanation:

Measure: The time it takes SEAs to draw funds down to reimburse grantees will decrease.

Explanation:

Measure: The percentage of State Educational Agencies that submit complete data on 21st Century program performance measures by the deadline.

Explanation:This measure will track the program's success in obtaining data in a timely manner.

Measure: The average number of days it takes the Department to submit the final monitoring report to an SEA after the conclusion of a site visit.

Explanation:

Measure: The average number of weeks a State takes to resolve compliance findings in a monitoring visit report.

Explanation: 

Is the program purpose clear?

Explanation: The purpose is to establish community learning centers that help students in high-poverty, low-performing schools meet academic achievement standards; offer a broad array of additional services designed to complement the regular academic program; and offer families of students opportunities for educational enrichment.

Evidence: Section 4201 of the Elementary and Secondary Education Act, as amended by the No Child Left Behind Act of 2001. The antecedent program did not have a clear focus on academic achievement. The reauthorized program does.

Does the program address a specific and existing problem, interest, or need?

Explanation: The program supports local communities in providing students, particularly students who attend schools that have been identified as in need of improvement under Title I, with opportunities for academic enrichment that will reinforce classroom learning. Also, the program provides a safe haven for youth, supervised activities, and services focused on crime, violence, and substance abuse prevention.

Evidence: A 2000 study conducted by the Urban Institute found that 5 percent of 6- to 9-year-olds and 24 percent of 10- to 12-year-old children have self-care as their primary child care arrangement in the after-school hours. Also, both the current and antecedent program consistently receive 10 times the number of applications than can be funded. A majority of the applications were to fund centers focusing on improving participants' academic achievement.

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: This is the only program that focuses on providing Federal support to create infrastructure for extended-learning programs (as opposed to providing per-capita funds for student care during non-school hours) with an emphasis on improving academic achievement of students who attend schools that have been identified as in need of improvement under Title I. This is also the only program that supports such a wide range of activities within its centers.

Evidence: Other Federal programs that support the care of students during non-school hours (but do not fund the creation of extended-learning program infrastructure) include CCDF and ESEA Title I.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The reauthorized ESEA, as amended by NCLB, corrected what was perceived as flawed in the original structure of the antecedent program by improving the targeting of funds, converting the program to a State-administered grant, and requiring centers to provide academic enrichment activities to students.

Evidence: Sections 4201-4206 of the Elementary and Secondary Education Act, as amended by the No Child Left Behind Act of 2001

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The reauthorized program requires States to make awards that will primarily serve students who attend schools with concentrations of poor students. It also requires States to give priority to applications that propose to target services to students who attend schools identified for improvement under Title I and applications that are submitted jointly by at least one LEA that receives funds under Part A of Title I and at least one community-based organization or other public or private entity.

Evidence: Sections 4201-4205 of the Elementary and Secondary Education Act, as amended by the No Child Left Behind Act of 2001

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has developed performance measures that reflect the program goals regarding student achievement and behavior. In addition,to ensure that centers operate high-quality programs, the statute requires local grantees to develop programs that meet specified principles of effectiveness focusing on improving the number of students that meet State proficiency measures.

Evidence: The data for the program effectiveness measures are being collected through annual performance reports submitted by local grantees. Data regarding the long-term efficiency goals has not yet been collected. Baseline data for the program effectiveness measures, collected through annual performance reports submitted by local grantees, should be available beginning in 2005.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Long-term performance measures aim for 100 percent of participants showing improvements in academic, social, and behavioral areas by 2012.

Evidence: These measures are included in the Department's Planning & Performance Management Database. Targets for 3 of the 4 long term measures are under development.

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: The Department collects and reports on data about the program's performance measures on an annual basis. Measures address improvement in academic achievement, and improvement in behavioral issues such as homework completion and class participation.

Evidence: The grantee database and individual districts' annual performance reports are available upon request to the Department.

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: The Department has baseline data collected in 2000 for the antecedent program and has established ambitious targets in each area. The program assesses its progress towards its long-term goals through the annual data collection process.

Evidence: Baseline data have been collected and some annual targets have been set for the antecedent program. However, some targets need revision because the original targets have been met. Therefore, the new measures will be higher than the original targets, providing more ambitious goals.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All States will report on their progress toward the performance targets on an annual basis. The information reported will be posted on the internet.

Evidence: For the reauthorized program, all State RFPs were analyzed by ED staff to be sure that they are consistent with the mission and purpose of the program. Through this work with the States and through the program guidance all States have State-level regulations for their subgrantees. The program's website provides links to all State 21st Century program websites regulations, guidelines, and performance reports can be found.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Current evaluations include a multi-year national evaluation ' using rigorous methodologies ' to answer questions relating to program implementation, student access and participation, and student outcomes and impacts. The study is examining both in-school and out-of-school outcomes such as achievement, high school completion, crime, and drug use. In addition, the Institute for Education Sciences' National Center for Evaluation is developing two after-school interventions (one each in reading and math) and will rigorously test their effectiveness through experimental studies. The Department also plans to begin funding a new evaluation of the State-administered program.

Evidence: An interim report from the national evaluation was released in early 2003. A final report is expected in the summer of 2004. Reports from IES's study and the evaluation of State implementation will be released at a later date.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The budget materials for this program show both the full cost of administering it and the cost of specific outputs related to the annual and long-term goals. In addition, the program was proposed for a cut in 2004 due to poor performance.

Evidence:  

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: No strategic planning deficiencies have been identified. This program has an internal strategic plan as well as a National Activities that were created since No Child Left Behind in order to specifically address the weaknesses of the previous program.

Evidence:  

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program has implemented web-based collection of data from states through annual performance reports, an innovation that is being considered for use in other programs. In addition, the States' annual consolidated reports and Title I State Report Cards also provides data that the Department uses to improve the management of the program.

Evidence: In order to draw information from State-funded programs, the Department is conducting a study that examines how States and communities are implementing the reauthorized program. Supported by National Activities funds, the study focuses on how, and to what extent, funds support high-quality programs that emphasize academic content. The study also examines project activities to improve academic outcomes and maintain student engagement in programs, and how they link with State and Federal education goals.

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: As part of the President's Management Agenda, the Department has implemented an agency-wide system -- EDPAS -- that links employee performance to progress on strategic planning goals. As one of its program reforms, ED will monitor grantee performance on an annual basis through review and approval of annual budget plans, compliance reviews, audits, and site visits. If this system is implemented effectively, we would expect this answer to convert to a "yes."

Evidence: States applications indicate that grantees will be required to make substantial progress each year toward the achievement of performance results and that, absent those results, States will not provide continued funding to poorly performing subgrantees. The Department will provide assistance, through a contract, to help States obtain the data they need to hold the grantees accountable for these results.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Funds are obligated within the timeframes set out by Department schedules and used for the purposes intended. The Department reserves some funds for national activities including evaluation, which are obligated based on an approved national activities spending plan and evaluation plan.

Evidence: States appear to be drawing down funds at an acceptable rate. This evidence comes from recent reports of current drawdown awards to States (reports that are checked every quarter). To date, every State has made at least one round of subgrant awards.

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: This program has established a partnership with a private Foundation (the Mott Foundation) of $100 million over 7 years to offset all technical assistance and grantee training for infrastructure development and sustainability. This partnership is so successful it won the Public Service Excellence Award. The Mott funding provided biannual training for all grantees designed to: 1) help grantees build collaborative partnerships, 2) provide comprehensive services to participants, and 3) diversify the sources of support. For the first 99 grants that received this training, approximately two-thirds are still providing services even though Federal funding ended two years ago.

Evidence: Within the next three years, Education will be analyzing the major business functions of all of its program offices. Once that analysis is complete, we will re-evaluate the extent to which this program is implementing those efficiency improvements.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program has conducted joint training forums with the Department of Health and Human Services; coordinates with State program offices; partners with the Mott Foundation for technical assistance, sustainability, and infrastructure development; works with Title I offices regarding supplemental services; and collaborates with other agencies such as NASA and the NEA for content area support.

Evidence: The Department cosponsored a meeting with all the 21st Century Community Learning Centers State coordinators, all the State HHS coordinators, HHS administrators, and TANF coordinators to various Federal efforts to support after-school programs. The Mott Foundation also funded a Finance Project to create a series of handbooks that show how to use funds across Federal agencies to support after-school programming.

Does the program use strong financial management practices?

Explanation: Recent agency-wide audits have not identified deficiencies in the financial management of this program.

Evidence: There have been no audits of the reauthorized program.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Material internal management deficiencies within the Department have not been identified for this program.

Evidence: Program staff monitor excessive draw downs of funds to prevent high-risk situations. Past technical assistance efforts have worked toward creating sustainable funding sources and now the technical assistance is focused on improving program quality.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The Department maintains information on grantee activities through consolidated annual reports, site visits and compliance monitoring, and technical assistance activities.

Evidence: The National Evaluation is one of the Department's oversight measures. The fact that the evaluation focused on early implementation made it possible for the report to identify the issues that the Department is working on with States and grantees to prevent and remedy.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The performance reports are annual and will be widely disseminated.

Evidence: The public can access the Department's evaluation on the program's website (www.ed.gov/offices/OESE/21stcclc/). The grantee database and individual districts' annual performance reports are available upon request to the Department.

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: Recent evaluations suggest that the program is not on track to meet most of its long-term goals regarding student achievement or student behavior. (see below for details)

Evidence: When Schools Stay Open Late: The National Evaluation of the 21st Century Community Learning Centers Program." (www.ed.gov/pubs/21cent/firstyear/)

Does the program (including program partners) achieve its annual performance goals?

Explanation: Recent evaluations suggest that consistent attendance by students tends to drop off in these programs over the year - in part because students felt the activities did not meet their needs or were too much like the regular school day. The National Evaluation of the 21st Century program also indicates that the academic component of these programs is often inadequate. However, there were small academic gains reported for certain subgroups (African American and Hispanic students), and the program was associated with increased involvement of middle school parents.

Evidence: When Schools Stay Open Late: The National Evaluation of the 21st Century Community Learning Centers Program." (www.ed.gov/pubs/21cent/firstyear/) Evidence of academic gains for African American and Hispanic students can be found on pages xii and 70.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: Under the antecedent program directly administered by the Department, grant costs decreased each year while the number of grantees increased. This means that the program is spending less money per participant while increasing the number being served and increasing the academic focus of the program. In addition, the partnership with the Mott Foundation has improved this program's cost-effectiveness by utilizing private funds to support this Federal program.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: No evaluations of similar rigor have been conducted on other extended-learning programs.

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: Recent evaluations suggest that consistent attendance by students tends to drop off in these programs over the year - in part because students felt the activities did not meet their needs or were too much like the regular school day. The National Evaluation of the 21st Century program also indicates that the academic component of these programs is often inadequate.

Evidence: When Schools Stay Open Late: The National Evaluation of the 21st Century Community Learning Centers Program." (www.ed.gov/pubs/21cent/firstyear/)