Online learning is a relatively new development in K-12 education but is rapidly expanding in both number of programs and participants. According to a report by the North American Council for Online Learning (NACOL), "As of September 2007, 42 states [had] significant supplemental online learning programs (in which students enrolled in physical schools take one or two courses online), or significant full-time programs (in which students take most or all of their courses online), or both."² In addition, the Internet houses an ever-expanding number of Web sites with a broad range of education resources for students, parents, and teachers. Given this expansion and a dearth of existing research on the topic, it is critical to conduct rigorous evaluations of online learning in K-12 settings to ensure that it does what people hope it will do: help improve student learning.

However, those undertaking such evaluations may well encounter a number of technical and methodological issues that can make this type of research difficult to execute. For example, the scant research literature on K-12 online learning evaluation provides few existing frameworks to help evaluators describe and analyze programs, or tools, such as surveys or rubrics, they can use to collect data or assess program quality. Another common challenge when students are studying online is the difficulty of examining what is happening in multiple, geographically distant learning sites. And multifaceted education resources—such as vast Web sites offering a wide range of features or virtual schools that offer courses from multiple vendors—are also hard to evaluate, as are programs that utilize technologies and instructional models that are new to users.

Furthermore, evaluations of online learning often occur in the context of a politically loaded debate about whether such programs are worth the investment and how much funding is needed to run a high-quality program; about whether online learning really provides students with high-quality learning opportunities; and about how to compare online and traditional approaches. Understandably, funders and policymakers—not to mention students and their parents—want data that show whether online learning can be as effective as traditional educational approaches and which online models are the best. These stakeholders may or may not think about evaluation in technical terms, but all of them are interested in how students perform in these new programs. At the same time, many online program leaders have multiple goals in mind, such as increased student engagement or increased student access to high-quality courses and teachers. They argue that test scores alone are an inadequate measure for capturing important differences between traditional and online learning settings. And, like educators in any setting—traditional or online—they may feel a natural trepidation about inviting evaluators to take a critical look at their program, fearing that it will hamper the progress of their program, rather than strengthen it.

This guide will discuss how evaluators have tried to compare traditional and online learning approaches, what challenges they have encountered, and what lessons they have learned.