- Final Report:
PDF (9 MB) | Word (4.5 MB)
For many years, policymakers have been concerned by the relatively low levels of academic achievement by economically disadvantaged K-12 students in math and science, by the underrepresentation of disadvantaged college students in math and science majors, and by the underrepresentation of people from disadvantaged groups in math and science careers. While racial gaps in math and science test scores narrowed somewhat in the 1970s and 1980s, substantial gaps persisted through the 1990s to the present.
To help address these disparities, the U.S. Department of Education (ED) established a math and science initiative in 1990 within Upward Bound, a federal grant program designed to provide disadvantaged high school students with skills and experiences that will prepare them for college success. The initiative, referred to as Upward Bound Math-Science (UBMS), awards grants to institutions—largely colleges and universities—to operate UBMS projects. These projects were designed to differ from "regular" Upward Bound projects in several respects. To ensure that participants receive an intensive math and science precollege experience, UBMS projects provide instruction that includes hands-on experience in laboratories, computer facilities, and at field sites. Opportunities are also provided to learn from mathematicians and scientists employed at the host institution or engaged in research or applied science in other institutions in the community. A six-week summer program providing intensive instruction in laboratory science and mathematics through precalculus is also offered.
Initially, ED funded 30 UBMS projects. by FY 2004, there were 127 UBMS projects serving 6,845 students at a total cost of $32.8 million. Therefore, the annual cost per student—approximately $4,800—is comparable in cost to regular Upward Bound but much more expensive than other federally funded precollege programs. More than 80 percent of UBMS projects are hosted by four-year colleges and universities; most of the rest are hosted by two-year colleges (Curtin and Cahalan 2004).
Participants in UBMS must meet the same eligibility requirements as regular Upward Bound students: students must (1) belong to families classified as low-income (taxable income of no greater than 150 percent of the poverty line), or (2) be a potential first-generation college student (neither parent has a bachelor's degree). Some students who participate in UBMS summer programs are referred from regular Upward Bound programs and then return to those programs during the academic year. However, as would be expected, UBMS projects are more likely to consider students' interests in math and science when reviewing applications than are most regular Upward Bound projects (Moore 1997b). While 25 percent of participants are white, most program participants are from underrepresented minority groups: about 60 percent of participants are African American or Hispanic (Curtin and Cahalan 2004).
Evaluation of Upward Bound Math-Science
Since 1991, Mathematica Policy Research, Inc., (MPR) has been conducting the National Evaluation of Upward Bound for ED. The centerpiece of this evaluation has been a random assignment evaluation of regular Upward Bound. In 1997, ED added a new component to the evaluation that is focused on UBMS. In 1998, MPR selected a random sample of the students who participated in UBMS between 1993 and 1995 at projects that were still operating at that time. This report constitutes the first of two evaluation reports on UBMS, and it is based on participant surveys and student transcripts collected for this sample between 1998 and 1999 and again between 2001 and 2002. The second report is scheduled for completion in 2006 and will be based on data collected between 2003 and 2005.
The evaluation of UBMS has two components: a descriptive analysis and an impact analysis. The descriptive analysis relies primarily on a survey of project directors to describe the resources available to UBMS projects; the types of institutions that host them; the credentials and demographic characteristics of project staff; recruitment, eligibility, and enrollment of students; student characteristics; and program offerings. The impact analysis is designed to measure the effects of UBMS on (1) performance in high school, especially in math and science courses; (2) postsecondary attendance, persistence and completion; and (3) the likelihood of completing a postsecondary degree in mathematics or a scientific field.
The impact analysis is based on a comparison of UBMS participants with a sample of students that (1) applied to enroll in regular Upward Bound programs in the early 1990s, (2) never participated in UBMS and (3) have been tracked by MPR as part of the national evaluation. This comparison group was selected to ensure that it had similar characteristics to the sample of UBMS participants, and we controlled statistically for the small remaining differences in these characteristics between UBMS participants and the comparison group.
If UBMS participants are more interested or skilled in math and science than the students in the comparison group, the estimated effects of the program may be subject to "selection bias" and may overstate the true effects of participating in UBMS. However, the comparison group we selected was probably the best available short of a randomized control group because the students in the comparison group exhibited the motivation to pursue Upward Bound services, and our analysis shows that the participant and comparison groups are similar in other ways as well. In addition, we implemented a data collection and analysis plan designed to minimize selection bias (see Chapter III for more details). While a control group from a randomized experiment would have prevented selection bias, the comparison group we selected greatly reduced the cost of the evaluation because we were already collecting data for this group as part of the national evaluation.
Note that the descriptive findings and impact estimates presented in this report describe the operations and effects of the Upward Bound Math-Science Program as it operated in the mid-1990s. At that time, it was a relatively new program, and some changes have occurred in how UBMS projects operate. In Chapter II, we mention some of these changes as they are reflected in information provided to us by UBMS project directors in a survey of grantees. It is certainly possible that some of the changes in the program since the mid-1990s have influenced the effectiveness of UBMS projects, and the evaluation does not attempt to measure any changes in effectiveness since that time. In this report, we measure the effects of the program on people who participated between 1993 and 1995 and describe the operations of the program at that time.
Report Findings
From our descriptive analysis, we found that UBMS projects:
Provide a large quantity of academic instruction. in the summer, the average UBMS project provided a total of 240 hours of academic instruction, and participation in the program is roughly full-time for a six-week period.
Are most active during the summers. UBMS projects typically provide services, such as tutoring or study sessions, during the school year, but they provide most of their services during summer residential programs at the colleges or universities hosting the program.
Provide academic enrichment in math and science subjects. Many UBMS projects offer courses in algebra II, geometry, precalculus, biology, chemistry, physics and computer software; in contrast, few offer courses in Social Studies (though many offer English courses in addition to their math and science offerings). At most projects, the course work is designed to provide academic enrichment instead of academic remediation.
Provide instruction through a combination of single-subject courses and interdisciplinary instruction. While other instructional techniques were used, three out of four projects provided instruction primarily through single-subject academic courses or the combination of these courses with interdisciplinary instruction.
Given the academic services provided by UBMS, it is natural to ask whether participating in UBMS affects the educational outcomes of the students that participate. From our impact analysis, we found that UBMS:
Improved high school grades in math and science and overall. UBMS had a positive effect on high school grades, increasing the average GPA in math courses from 2.7 to 2.8, the average GPA in science courses from 2.7 to 2.9 and the average GPA overall.
Increased the likelihood of taking chemistry and physics in high school. UBMS increased the likelihood that participants took upper-level science courses in high school, raising the percentage of students taking chemistry from 78 percent to 88 percent and raising the percentage of students taking physics from 43 percent to 58 percent. in contrast, UBMS did not affect coursetaking in advanced math subjects (see Chapter III, Exhibit III.3).
Increased the likelihood of enrolling in more selective four-year institutions. UBMS increased the percentage of students that attended four-year colleges and universities from 71 percent to 82 percent. The increase in four-year attendance is particularly pronounced for more selective schools (those rated as "most selective", "highly selective" or "very selective" by the Barron's Guide): UBMS increased the percentage of students that attended more selective four-year colleges from 23 percent to 33 percent (see Chapter III, Exhibit III.4).
Increased the likelihood of majoring in math and science. UBMS affected students' choice of major, increasing the percentage majoring (or planning to major) in math or science from 23 percent to 33 percent and decreasing the percentage majoring in a field outside of math or science and the social sciences from 51 to 42 percent. UBMS also seems to increase the percentage of participants majoring in the social sciences (see Chapter III, Exhibit III.6).
Increased the likelihood of completing a four-year degree in math and science. UBMS increased the percentage of students that earned a bachelor's degree in a math and science field from 6 percent to 12 percent and decreased the percentage that earned a bachelor's degree outside of math, science, and the social sciences from 20 to 14 percent (see Chapter III, Exhibit III.6). Because 47 percent of participants in our sample were still in college when we interviewed them in 2002, findings related to degree completion should be treated as preliminary, and a final assessment will be presented in a subsequent report.
In addition, we computed separate impact estimates for subgroups defined by sex, race and ethnicity, and prior participation in regular Upward Bound. For some outcomes, we found differences in subgroup impacts that were statistically significant. For example, the effect of UBMS on four-year college attendance was larger for women than for men. However, the number of significant differences between subgroups was relatively small, and there was no obvious pattern to the findings suggesting that particular groups benefited more from UBMS than other groups. Therefore, it is not clear whether the significant subgroup differences are due to chance or to systematic differences in the effects of UBMS on different groups of participants.
To summarize the report's findings, UBMS provides intensive academic instruction in math and science, and our impact estimates suggest that it improves several student outcomes in high school and college. In addition, and consistent with the objectives of the program, preliminary estimates suggest that UBMS participation increases the odds of majoring in math or science. In the next report, we will reexamine the effects on college completion, examine the effects on labor market outcomes, such as employment in the sciences, and weigh the benefits of the program against the costs.
It is tempting to compare the estimated impacts of UBMS to the estimated impacts of regular Upward Bound presented in earlier reports. However, it is important to recognize that the two studies used different methods: while the evaluation of regular Upward Bound is based on an experimental design, the "gold standard" in evaluation research, the evaluation of UBMS is based on nonexperimental methods that may suffer from selection bias, as described earlier. If the estimated effects of UBMS are inflated due to selection bias, then the impression based on our findings that UBMS is more effective than regular Upward Bound might be attributable to differences in the methods used to estimate the impacts instead of differences in the effectiveness of the two programs.