Download the full report: PDF (1.12M)

Executive Summary

Officials from federal agencies with education programs aimed at improving America's competitiveness in science, technology, engineering, and mathematics (STEM) engaged in a yearlong endeavor to assess their programs' success and to identify areas for improvement for current and future programs. This effort, carried out by the Academic Competitiveness Council (ACC) and led by Secretary of Education Margaret Spellings, lays the groundwork for sustained collaboration among STEM education programs across federal agencies that will greatly strengthen America's competitiveness.

The Deficit Reduction Act of 2005 (P.L. 109-171) (the act) established the ACC. The statute charged the ACC to:

• Identify all federal programs with a mathematics or science education focus;
• Identify the effectiveness of those programs;
• Determine areas of overlap or duplication among those programs;
• Identify target populations served by such programs; and,
• Recommend processes to efficiently integrate and coordinate those programs.

In addition to assuming those responsibilities delineated in the act, the ACC set milestones to guide its mandate:

• Delineate the goals of the programs;
• Determine the extent to which the programs have undergone independent, external evaluation based on sound, scientific principles;
• Ascertain the extent to which the programs have quantitative evidence of achieving their stated goals;
• Establish standards for measuring and evaluating these programs, including common measures as appropriate; and
• Formulate recommendations for administrative or legislative action that, if carried out, would more efficiently integrate and coordinate federal spending on STEM education programs.

Agencies' catalogued programs focused on STEM education. The resulting inventories were then aggregated into an overall inventory for the federal government comprising 105 STEM education programs, with approximately $3.12 billion in total funding for Fiscal Year (FY) 2006. The programs cover kindergarten through postgraduate education and outreach as follows:

• 24 elementary and secondary school (K–12) programs, which received approximately $574 million, or 18.4 percent of total funding;
• 70 undergraduate, graduate and postgraduate programs, which received more than $2.4 billion, or 77.2 percent of total funding; and
• 11 informal education and outreach programs, which received close to $137 million, or 4.4 percent of total funding.

To perform its work, the ACC formed three working groups, each focused on an education category: K–12 Education, Postsecondary Education, and Informal Education and Outreach. Each working group developed common goals and measures for its programs: national goals to reflect the overarching education outcomes for a given group; national metrics to measure the nation's progress toward those outcomes; common goals and metrics to measure the impact of individual programs and projects; and plans to integrate these metrics into program and project operations and evaluations.

The K–12 Education working group adopted goals and metrics at both the national level and the program and project level that focus primarily on improving student achievement, teacher quality, and student engagement. The Postsecondary Education working group identified one overarching national goal for undergraduate programs: to increase the number of undergraduates who enroll in and complete STEM degree programs. Further, the group identified program-level goals and metrics in three areas: improving the STEM workforce; encouraging development of collaborative communities among the education, government, industry, and professional sectors; and strengthening higher education institutional capacity. The Informal Education and Outreach working group identified two national and program-level goals: increasing awareness, interest, engagement, and understanding of STEM concepts, processes, and careers by the general public and other targeted populations; and improving practice and building professional and institutional capacity by funding efforts that generate, develop, and apply innovative ideas and models for informal science education.

There are 27 graduate and postgraduate programs that represent nearly $1.46 billion or 47 percent of total FY 2006 funding for STEM education programs in the ACC program inventory. These programs are highly individualized, with the primary mission of strengthening the nation's research capacity rather than broadly improving the nation's education system. Notwithstanding the contributions of graduate and postgraduate programs to the fields of science and engineering, the ACC chose not to assign a national education goal to these programs. However, the Postsecondary Education working group developed metrics to help agencies assess the educational aspects of graduate and postsecondary programs. These metrics include the percentage of fellowship recipients who complete their degree program or are subsequently employed in a STEM field.

The ACC employed a methodological framework to address the requirement to determine STEM program effectiveness. The ACC used the framework not only to review existing evaluations of a project's effectiveness, but also to serve as a guide for designing such evaluations in the future. This Hierarchy of Study Designs (fig. 2) encompasses only those study designs intended to estimate a project's impact on educational outcomes, such as student math and science achievement, or Ph.D. completion. (These are sometimes called "impact" studies.) For such study designs, the hierarchy provides guidance on designs most likely to produce valid estimates of a project's true impact.

The ACC solicited support from the Coalition for Evidence-Based Policy (coalition), a nonprofit, nonpartisan organization sponsored by the Council for Excellence in Government, to assess evaluations. Each agency submitted its best evaluations for this review. Of the 115 total evaluations, the coalition found 10 impact evaluations that were scientifically rigorous, four of which concluded that the educational activity evaluated had a meaningful positive impact. Three of them had results published in academic journals. Based on the 115 evaluations, the ACC's review revealed that, despite decades of significant federal investment in science and math education, there is a general dearth of evidence of effective practices and activities in STEM education. Even the 10 well-designed studies would require additional replication and validation to be used as the basis for decisions about education policy or classroom practice.

The act also charged the ACC with identifying areas of overlap and duplication. Many of the programs in the ACC's inventory share similar goals. While such duplication is not inherently bad, the ACC found coordination among agencies could be improved. For example, grants to some projects that supported similar interventions appeared uninformed by the results of earlier projects. Agencies with similar STEM programs sometimes do not share information about the work they fund.

Based on its analysis, the ACC makes the following recommendations:

Recommendation 1: The ACC program inventory and goals and metrics should be living resources, updated regularly and used to facilitate stronger interagency coordination.

Recommendation 2: Agencies and the federal government at large should foster knowledge of effective practices through improved evaluation and-or implementation of proven-effective, researchbased instructional materials and methods.

Recommendation 3: Federal agencies should improve the coordination of their K–12 STEM education programs with states and local school systems.

Recommendation 4: Federal agencies should adjust program designs and operations so that programs can be assessed and measurable results can be achieved, consistent with the programs' goals.

Recommendation 5: Funding for federal STEM education programs designed to improve STEM education outcomes should not increase unless a plan for rigorous, independent evaluation is in place, appropriate to the types of activities funded.

Recommendation 6: Agencies with STEM education programs should collaborate on implementing ACC recommendations under the auspices of the National Science and Technology Council (NSTC).

When these recommendations are implemented, the body of evidence of the effectiveness of STEM practices will grow and the impact of federal programs on STEM goals will improve. American students will be the beneficiaries and the nation's overall competitiveness will be strengthened.