Where the rubber meets the road is in providing academic instruction that boosts student achievement. SES service providers have met state requirements that include evidence of effectiveness. But how does that translate into instruction tailored to each student's specific needs that will complement and extend classroom instruction and contribute to reaching standards on state assessments?

NCLB regulations give states the responsibility of ensuring that SES services are high quality and effective in raising students' academic achievement. Each state must identify methods of measuring each provider's impact on students' academic learning, and it may enlist the assistance of local districts to collect data and supply input. Many states are just now gearing up to meet this requirement, which will inform decisions about which providers remain on the list. Although states must take the initiative to evaluate providers, they may solicit districts to collect student participation data, feedback from parents, and other information from district contracts with providers. According to the U.S. Department of Education's non-regulatory guidance on SES, "The State may want to request assistance from its LEAs or may want to handle this monitoring [of continued effectiveness by the provider] at the State level."⁷

Although each state has chief responsibility for ensuring that its SES-eligible students overall receive high quality, effective SES services, the law has embedded certain mechanisms at the local level that are also intended to help ensure the success of SES services for individual students. Most important are its requirements for individual student learning plans, for services to be aligned with state standards and consistent with district curriculum and instruction, and for progress reports. It falls to the district to make sure these elements are in place.

First Steps

Use Detailed and Specific Student Learning Plan Forms

According to NCLB's SES provisions, a district's contract with a provider must require detailed achievement goals for the individual student, a timeline for meeting the goals, a method to measure student improvement, and a schedule for informing parents and teachers about student progress. The provider must develop student goals in concert with parents. To streamline the communication and contracting process, many districts have created a learning plan template as an independent, yet integral, part of their contract with providers; this ensures that learning plans, although tailored to the needs of each student, are also consistent across providers and students in what they address. Districts have discretion over how these plans are formatted and what, if any, additional content will be included, resulting in a range of templates and a variety of data being collected across districts. But by and large, the templates ensure two things: that unique goals are established for each student and that providers indicate the evaluation tools they will use to document student improvement.

In Los Angeles, the student learning plan calls for providers to work with parents to identify achievement goals, services a student will receive based on those goals, and the measurement and assessment tools to be used to track progress. Such tools may include a pre- and post-test, a daily written or oral diagnostic, or other assessment measures determined or developed by the provider. See figure 12 for an example of a student learning plan that has been completed. The goals are generally drawn from the provider's understanding of the state standards that are applicable to each student's needs.

Rochester has created a slightly different Supplemental Education Plan template for providers to use in developing a learning plan for each client. Providers do not simply select a set of mathematics or reading goals from a list of state or district standards. Instead, they select an area of service (e.g., English language arts, mathematics, IEP, 504, or LEP) and then, in consultation with parents, determine the unique skills to target for each student. Providers also indicate how student progress on these skills will be measured and how often parents will receive reports on student achievement as required in the SES provisions.

Share Data and Set Student Goals Aligned with the District's Instructional Program

An important step in developing student learning plans is ensuring that, as required in the law, SES services are consistent with the district's curriculum and instruction and are aligned with the state's academic standards. In approving providers, states must first ensure that a pro-vider's services are "consistent with the instructional program of the LEA [local education agency] and with State academic content and achievement standards."⁸ This does not mean, however, that there is only one acceptable service approach. Far from it: SES tutoring might amplify and reinforce the student's classroom instruction, giving more practice and perhaps a different explanation or perspective; or, a student might have a weakness in some very specific skill areas and need concentrated practice in those areas. Whichever strategy is pursued, it should be deliberate and informed by a thorough understanding of the standards the student needs to master and by as much diagnostic information as possible, with input from parents and classroom teachers.

To facilitate development of the student learning plan and help ensure that it addresses the student's individual needs, districts may need to seek parent permission to share student data with the SES provider. In addition to sharing assessment data, they may also try to link the provider to the classroom teacher so that the classroom teacher can share additional information-again, with parent permission. Close coordination at this level also helps ensure that SES complements, rather than duplicates or conflicts with, district efforts. Once students are receiving services, districts can encourage continued communication between teachers and SES tutors about student progress. Such efforts are critical in supporting the work both of providers and classroom teachers. Not incidentally, they also help build greater school-level commitment to the SES program.

In Forsyth, data-driven decision-making is a key component of the district's long-term strategic planning, of schools' day-to-day operations, and of teachers' instructional planning. So, not surprisingly, Forsyth has encouraged its SES provider to contact students' classroom teachers and solicit additional achievement data beyond what the district supplied. The provider developed a Teacher-Tutor Data Collection Sheet (see figure 13) that was used to determine students' initial skill set and to provide feedback to teachers and the district about student progress. After obtaining parent permission, provider staff would contact students' regular classroom teachers and request input on student performance in reading, writing, and mathematics. Tutors also had an opportunity to provide their own input on the same indicators after meeting with and assessing students. This sheet also included the results of the diagnostic test conducted by the provider, who used the data to develop student learning plans and report to the district on students' progress in the program. Teachers also had access to this data to inform their work with students. By creating a feedback loop between teachers and tutors, Forsyth is able to ensure instructional goals are being met.

In several of the districts studied, the state department of education required them to maintain a learning plan for all students at risk of not meeting state grade-level standards. In addition, the districts' own strategic goals called for a learning plan for such students. To avoid piling one learning plan on top of another, the districts are trying to devise their own systems to coordinate the information in all of the learning plans.

San Diego and Rochester are ahead of the curve here because they have chosen to use previously required student progress plans as a base for the SES plan. Under Rochester's Academic Intervention Services (AIS) system, mentioned earlier, instructors are required to monitor and document student progress on state learning standards in an AIS plan. Because the indicators on the AIS progress report are identical to those that need to be measured in a Supplemental Education Plan, the AIS report now serves double duty as the basis of the district's Supplemental Education Plan. In San Diego, the Learning Contract is a key component of the district's Blueprint for Student Success. A Learning Contract must be developed for every student at risk of not meeting grade-level standards in reading or math. Because SES is one of many interventions supporting students at academic risk, San Diego has made students' SES plan a component of their learning contract rather than a separate stand-alone document.

San Diego and Rochester, along with Los Angeles, are also in the process of modifying their student achievement data systems to coordinate each student's learning plan information into a coherent record that will be a meaningful and useful guide to academic intervention.

Day-to-day tracking efforts, like reviewing student attendance, can enable districts to get an early handle on the benefit a student receives from SES. A student's attendance record suggests whether he or she is responding to an SES program. Using attendance as a metric encourages providers to maintain communication and positive relationships with parents, ensuring students' consistent participation. It can give an early warning of the need for extra attention.

Los Angeles is one of several districts that track student attendance as a measure of SES program effectiveness. Rather than using separate forms to collect the data, Los Angeles asks that providers include the information as part of the invoice they submit for payment.

Forsyth has also found an efficient way to monitor student attendance in SES: SES tutors report attendance on the Teacher-Tutor Data Collection Sheet. Making attendance reporting a natural part of the work providers do with students-as opposed to requiring a separate report-can make the process easier for both the district and SES providers.

Set up and Use Regular Progress Reports to Parents and Teachers

Providers must be accountable to parents by giving them regular reports on student progress. As required by NCLB, classroom teachers must also receive progress reports to help ensure that "students are improving their academic achievement and that instructional goals are being met."⁹ Each district in this group has progress reporting in place, with slightly different features and timelines.

To facilitate the billing and reporting components of its provider contract, Los Angeles requires all providers to send progress reports to a student's parents and teachers after every 15 hours of service. Providers can report more frequently if they choose to or if parents request it. But this length of time gives providers a chance to work with students to meet initial goals and gives parents a chance to revise learning goals after talking with teachers or reflecting on student progress.

Forsyth's contract calls for providers to supply parents and teachers with the results of pre- and post-testing, as well as biweekly progress reports (see figure 14). In addition, the district's provider — a one-on-one, in-home tutoring service — boasts that its tutors meet with parents at the end of each tutoring session to discuss the work they did and homework expectations for students. This provider encourages parents to monitor students' work and even to replicate some of the teaching behaviors used by tutors. All of this is intended to make sure students receive the greatest benefit from the SES program.

Going Deeper

Evaluate Student Progress on District Assessments

Many districts are in the midst of strengthening their assessment systems and database capacity so that data on student performance and other measures can be better used to guide decisions. Benchmark assessments throughout the school year, perhaps paced by units in the curriculum, are increasingly common. These assessment tools can be used to judge the contribution of different instructional programs, including SES.

In 2003, Rochester was in conversation with Tungsten Learning-a division of Edison Schools-and Scantron to contract to use their systems, which provide benchmark assessments aligned with state standards. The Tungsten system assesses students monthly, while Scantron would do so every nine weeks. District leadership approved a pilot of both programs at various grade levels in a limited number of schools. Rochester intends to use one of these systems to measure provider impact on student achievement over the next few years. Most providers are already using their own pre- and post-test assessments to evaluate student progress; however, lacking early evaluation data from the state, the district is trying to establish a way to supply parents with objective information that can be compared across providers.

Figure 15. Rochester Parent Survey

SURVEY PARENTS

Parents are a key source of information to evaluate SES. Some districts have begun surveying parents about their experiences with SES providers. Like student attendance, the degree of parent satisfaction with a provider is an important early indicator of effectiveness. Los Angeles and Toledo use parent comments and evaluations as a means of determining a provider's effectiveness. Parents' satisfaction with the progress their children are making as a result of the services they are receiving is a fundamental indicator of how well providers are serving students' needs.

The staff overseeing Rochester's SES implementation are committed to finding meaningful ways of evaluating both SES providers and district SES implementation. The district conducted a survey of parents whose children received supplemental services during 2002-03. This survey asked parents to rate both the provider and the district's NCLB Office (see figure 15). Staff have discussed reporting the results as part of the letter or brochure to parents of eligible students. At the very least, the district will use parent feedback in reviewing its SES policies and compare results over the duration of SES implementation as part of its evaluation efforts.

Reflect on Implementation and Adjust Efforts Based on Formative Feedback

Evaluating SES is not just about student performance. It's also about districts themselves learning how best to implement the program. Many of the previous sections of this guide highlighted practices that emerged when districts were trying to improve upon initial practice. Ongoing reflection informed by a range of indicators is the hallmark of a learning organization.

In Toledo, for example, when the early mailing to parents failed to generate much response, staff considered this feedback and quickly changed tack. They switched from a largely central office outreach effort to site-based outreach, recruiting parents and teachers at participating schools to make phone calls to other parents of SES-eligible students. The district also conducted outreach during parent-teacher conferences, parent meetings, and other school-based events. This kind of responsiveness resulted in both a higher yield of eligible student participants and a school community more actively engaged around implementing SES.