The research approach underlying this guide is a combination of case study methodology and benchmarking of best practices. Used in businesses worldwide as they seek to continuously improve their operations, benchmarking has more recently been applied to education for identifying promising practices. Benchmarking is a structured, efficient process that targets key operations and identifies promising practices in relationship to traditional practice, previous practice at the selected sites (lessons learned), and local outcome data. The methodology is further explained in a background document, ⁵¹ which lays out the justification for identifying promising practices based on four sources of rigor in the approach:

• Theory and research base;
• Expert review;
• Site evidence of effectiveness; and
• Systematic field research and cross-site analysis.

The steps of the research process were defining a study scope, seeking input from experts to refine the scope and inform site selection criteria, screening potential sites, selecting sites to study, conducting site visits, collecting and analyzing data to write case reports, and writing a user-friendly guide.

Site Selection Process

Guided by a review of the literature, recommendations from an advisory group, and review of advanced course offerings and student participation in online courses, researchers initially considered 35 online course providers managed by districts, states, universities, or education consortia. Based on a requirement that providers be able to provide at least two years of student performance data, that initial list was reduced to 27 providers.

Seven online course providers were selected from this larger list as case study sites,* based on the compiled information and criteria ratings in a screening matrix. (See next section for more on criteria.) The type of entity creating or operating the course provider (e.g., a state, a consortium of schools or districts, a university), geographic location, student demographic variation, and a range of promising practices were all considered in this final site selection.

Site Selection Criteria

Programs were selected for case study based on a set of features identified during the review of literature and prioritized by the advisory group as key issues for consideration. Programs were evaluated based on their ability to provide evidence that they:

• Assess for student readiness to take online courses, especially advanced course work online;

• Provide a ladder for student success, with frequent and varied supports for students including teacher interaction and on-site mentors;

• Design and deliver highly engaging courses using online media;

• Serve a diverse student population;

• Offer opportunities and support, such as supplemental funding, to encourage disadvantaged students to participate; and

• Track program cost data.

Study Framework and Data Collection

A conceptual framework was developed to guide the study of the selected programs. The framework was derived from the research literature on students taking advanced course work, on online learning, and on organizational or program effectiveness. The major categories in the framework were program goals and performance evaluation, funding models and sources, program promotion and student recruitment, course creation and delivery, and staffing. The framework focused on practices that are implemented to increase opportunities for students to participate in advanced course work and to support their efforts in completing courses successfully. During the case studies, researchers also heard from program, district, and school leaders and from parents and students about what they view as key factors for student success when studying advanced content online.

To gather information for this guide, researchers conducted interviews both on-site and by telephone. Each one-day site visit was supplemented by telephone interviews with school site coordinators, principals, central office administrators, parents, and students. All interviews followed a protocol based on the study framework and adapted to each role group. That is, separate but overlapping sets of questions were developed for program leaders, school administrators, parents, students, and others. All of the interviews were recorded, with key interviews later transcribed for more detailed analysis.

Documentation from each program served as additional sources of information. Collected during the site visits, these documents included such items as course descriptions, recruiting and marketing materials, online screen shots, instructor training materials, school site facilitator guides, parent-student agreements for participation, and program evaluations.

Analysis and Reporting

A case report was written about each program and reviewed by program leaders for accuracy. Drawing from these case reports, program documentation, and interview transcripts, the project team identified common themes that contributed to success across the programs and districts and schools that participated in one of the programs. This cross-site analysis built on both the research literature, as reflected in the study scope, and on emerging patterns in the data.

This descriptive research process suggests promising practices—ways to do things that others have found helpful, that is, lessons they have learned—and offers practical how-to guidance. This is not the kind of experimental research that can yield valid causal claims about what works. Readers should judge for themselves the merits of these practices, based on their understanding of why they should work, how they fit the local context, and what happens when they actually try them. Also, readers should understand that these descriptions do not constitute an endorsement of specific practices or products.

Using the Guide

Ultimately, readers of this guide will need to select, adapt, and implement practices that meet their individual needs and contexts. District, school, and online learning program leaders coming together in learning communities may continue to study the issues identified in this guide, using the ideas and practices from these sites as a springboard for their own action research. In this way, a pool of promising practices will grow, and educators can support each other in implementation and learning.

* One of the selected programs declined to participate in the study.