Seek to determine if a comparison study is appropriate, and, if it is, whether there is a viable way to compare program participants with others. Consider the online program's goals, the student population being served, and the program's structure. For example, online programs dedicated to credit recovery would not want to compare their student outcomes with those of the general student population.

Clearly articulate the purpose of the comparison. Is the evaluation seeking to find out if the online program is just as effective as the traditional one or more effective? For example, "just as effective" findings are desirable and appropriate when online programs are being used to expand education access.

If considering a quasi-experimental design (see Glossary of Common Evaluation Terms, p. 65), plan carefully for what classes will be used as control groups, and thoroughly assess the ways they are similar to and different from the treatment classes. What kinds of students does each class serve? Did the students (or teachers) choose to participate in the class or were they assigned to it randomly? Are students taking the treatment and control classes for similar reasons (e.g., credit recovery, advanced learning)? Are the students at a similar achievement level? Where feasible, use individual student record data to match treatment and comparison students or to hold constant differences in prior learning characteristics across the two groups.

If considering a randomized controlled trial (see Glossary of Common Evaluation Terms, p. 65), determine how feasible it is for the particular program. Is it possible to assign students randomly either to receive the treatment or be in the control group? Other practical considerations: Can the control group students receive the treatment at a future date? What other incentives can be offered to encourage them to participate? Will control and treatment students be in the same classroom or school? If so, might this cause "contamination" between treatment and control groups?

In the event a randomized controlled trial or quasi-experimental study is planned, plan to offer meaningful incentives to participating individuals and schools. When deciding on appropriate incentives, consider the total time commitment that will be asked of study participants.

Study sites with no vested financial interest are more likely to withdraw or to fail to carry through with study requirements that differ from typical school practices. If compliance appears unlikely, do not attempt an experimental or quasi-experimental study, unless it is an explicit requirement of a mandated evaluation.

When designing data management systems, keep in mind the possibility of comparisons with traditional settings. Collect and organize data in a way that makes such comparisons possible.