While each of the previous sections of this guide has included examples of evaluation of that specific component of program operations, both providers and districts need to look beyond process to outcomes.^* The bottom line in measuring course success is how well students do. While partnering districts and schools must track the progress and success of their individual students, online providers will want to track how the broad range of their online students are performing as an indicator of their own success. All data can then be used to make program improvements.

VHS, for example, tracks and reports several measures of student success, each of which tells an important part of the story. For AP courses, in particular, VHS tracks student enrollment, course completion rates, the number of students taking AP exams, and the how they do on those exams. As part of schools' partnership agreements with VHS, their students are required to share their AP exam scores. Site coordinators receive student scores from the College Board and submit a copy to VHS. To protect students' privacy, when VHS leaders receive score reports, they record them using a random tracking number, and the copies are destroyed.

Figure 6 shows a dramatic increase in the number of AP course enrollments through VHS over the past five years. As of 2007, VHS reported an 85 percent completion rate for AP courses. ⁴¹ Figure 7 shows how students are doing on AP exams. As enrollment has increased, so has the number of exams taken — up to 350 in 2005–06. The bottom part of each bar shows the number of exams taken but not passed; the top part shows the number of exams taken and passed. The percentage of students passing the exam has fluctuated some over the years, starting quite high when only a few students took the exams and dropping as more students enrolled, ⁴² but has always exceeded the average pass rate for the U.S. as a whole, which was 59 percent in each of the last two years. ⁴³

Of course, the goal is for all students to master the material well enough to take and pass the exam. But, as noted earlier, some research shows that taking an AP course is valuable even if a student does not take or does not pass the exam. ⁴⁴ And there are likely to be some transition steps as more students are encouraged to take AP courses. Some students may be more comfortable taking the course without having to take the exam, so allowing this option may encourage them to try taking an AP course. Likewise, students may not do well on a first exam, but may do better on any subsequent exams they take. By tracking all four measures, a program and a district or school will gain valuable data that shed light on areas to investigate in an effort to improve outcomes over time.

The providers profiled in this guide also track a variety of other outcome measures. COL contracts with a private consulting firm, the Public Good, to conduct an especially comprehensive evaluation of program outcomes that includes measures of enrollment, course completion, grades, and student perceptions of what they have learned. Course completion rates at COL have increased each of the last four years, from 78 percent in 2003 to 92 percent in 2006. ⁴⁵ When students themselves reflected on what they had learned, many mentioned 21st century skills that were enhanced by the online learning process; that is, skills recognized as important for success as citizens and workers in the 21st century, like self-direction and communication. For example, one student reported, "I think that students should take online courses because it teaches them how to be independent in their studies." Another said, "This course taught me more self-discipline in one semester than in all of my other years combined." ⁴⁶

Figure 6. Virtual High School's Advanced Placement Enrollment, 2001–02 to 2005–06 School Years

Source: Liz Pape of Virtual High School, 2007, "VHS Quality Benchmark Indicators," PowerPoint presentation, slide 14.

COL also looks closely at the characteristics of the students being served. Charged with serving schools with high academic needs and those with high poverty rates, COL can report that more than half the districts enrolling students in 2006 can be considered high-need, high-poverty districts, up from an average of 34 percent in the preceding years. This makes the high course completion rate of 92 percent all the more impressive.

In addition to collecting quantitative data (e.g., enrollment numbers, passing rates), these providers and their partner schools or districts also find it useful to collect qualitative data to help them improve their program. Their evaluations include satisfaction surveys of key stakeholders, including parents and students, as well as administrators, teachers, and site coordinators. A number of providers ask students to complete end-of-course evaluations (see fig. 8 for an example). CTY's survey asks about quality of instruction, the student's level of interest in course content prior to and after taking the course, ease of technology use, and the student's academic plans. COL also asks those who withdraw from an online course to complete an "early exit" survey, via e-mail. All providers indicated an interest in learning more from students who dropped an online course prior to completion about their reasons for doing so. VHS and FLVS also use their surveys to probe about unmet course needs. FLVS increased the number of electives it offers based on requests delivered through its customer satisfaction survey.

Figure 7. Virtual High School Students' Performance on Advanced Placement Examinations, 2001–02 to 2005–06 School Years

Source: Liz Pape of Virtual High School, 2007, "VHS Quality Benchmark Indicators," PowerPoint presentation, slide 13.

Note: The Virtual High School pass rate on AP examinations has always exceeded the average pass rate for the U.S. as a whole, which was 59 percent in each of the last two years.

* An upcoming guide in this Innovations in Education series will focus exclusively on how to evaluate online learning programs.