The project methodology is an adaptation of the four-phase benchmarking process used by the American Productivity and Quality Center, ^* along with case descriptions of individual districts and cross-site analysis of key findings. Although classic benchmarking looks for best or promising practices, using quantitative measures and comparisons among organizations, the practice of implementing supplemental educational services programs is too new to fully support this methodology. A brief description of this project's adapted methodology follows.

Plan

First, a conceptual framework was developed from an analysis of research on out-of-school programs and organizational management as well as an examination of what districts need to do to meet the No Child Left Behind Act's supplemental educational services requirements. Researchers, providers, and district administrators recruited to serve on an external advisory panel (see page 43) provided feedback to refine this framework and set priorities for issues to investigate. The resulting study scope guided all aspects of the study (see figure 16).

Site selection was a multistep process to ensure that the guide would feature an array of practices covering the elements of the framework and would represent a variety of geographic locations and contexts with which district administrators could identify. A list of 36 potential supplemental educational services sites was compiled through primary and secondary research by WestEd with suggestions from the expert advisory panel. A screening template was developed to systematically analyze the weighted criteria for site selection identified by the advisors, including the presence of clearly articulated strategic plans for administering SES, outreach and communications strategies for both parents and providers, and explicit contracting and recordkeeping procedures The template was completed for the candidate districts based on public documents such as brochures, reports, and district Web sites, supplemented by targeted phone interviews with district staff. The five districts that were selected had relatively high ratings on the template for preliminary evidence that promising practices were in place. No site was uniformly excellent, but each had developed practices in several areas from which others might learn.

Collect Data

Collecting detailed descriptive information from project participants was key to understanding the district's practices, the outcomes or impact achieved, and lessons learned in implementation from which others could. benefit. The major steps to this phase were finalizing the site-visit interview guide and arranging and conducting site visits to the innovation sites.

Each of the five innovation sites hosted a two-day site visit that included interviews with providers, principals, and parents, as well as district staff, to obtain multiple perspectives on the questions in the site-visit discussion guide. In addition, artifacts from the sites, such as letters to parents or brochures or contracts, were collected to provide concrete examples of district practices. The study team collated the information collected during the site visits and developed a case study for each site.

Analyze and Report

Once all the information was collected, the project team analyzed the data to understand the promising practices uncovered throughout the benchmarking project, both within and across sites. Thirteen key findings discussed in the final report emerged from the cross-site analysis.

Two products resulted from this research: a report of findings and this practitioner's guide. The report of findings provides an analysis of key findings across sites, a detailed profile of each site, a collection of artifacts, and key project documents. The practitioner's guide is a shorter excerpt and summary that is intended for broad distribution. The guide and report are also accessible online at http://www.ed.gov/nclb/choice.

Adapt

Ultimately, readers of this guide will need to select, adapt, and implement practices that meet their individual needs and contexts. Dissemination will take place through a variety of channels. The guide will be broadly distributed around the country through presentations at national and regional conferences, as well as through national associations and networks.

Districts coming together in learning communities may continue the study, using the ideas and practices from these sites as a springboard for their own action research. In this way, a pool of promising practices will grow, and districts can support each other in implementation and learning.

^*American Productivity and Quality Center. (2001). Benchmarking in education: Pure and simple. Houston, TX: Author.