The following posting is from Larry Condelli, Managing Director in the Education and Human Development Division of the American Institues for Research. Larry leads the adult education team and is project director for the National Reporting System. He has been involved in projects on ESL research, the professional development of adult educators and the development of content standards for adult education. He has worked in adult education since 1990.

At the end of his posting, Larry poses several questions he hopes you'll respond to. I, too, hope you will take this opportunity to discuss the National Reporting System and how NRS data can be used for program improvement.

Sandy Strunk, PLI List Moderator

********************************************************


	Several years ago, I started a new job and my first assignment was a very small project to help the Division of Adult Education and Literacy (DAEL) assist states to improve their ability to evaluate the quality of local adult education programs.  Little did I suspect that this work would eventually evolve into developing the national accountability system for the adult education program, and what we now know as the National Reporting System (NRS).  Before this, I had little knowledge of adult education and accountability, but life has a way of bringing about the unexpected. 
	Most people probably don’t know that the NRS had its origin in 1996, when a group of state directors of adult education were the first to move forward the idea for a national accountability system.  At that time there were proposals in Congress to roll adult education into a national work force development block grant system.  It was widely feared that block grants, giving governors discretion on how to set up the system in their states, would mean the end of the adult education program.  At the same time, demands for accountability increased for all programs, at both the federal and state levels.  To preserve meet these demands and adult education, state directors and others realized we needed a way to collect valid and reliable data to show what adult education is, the type of students who enrolled and the outcomes students achieved – and that these outcomes went beyond employment.  DAEL responded in 1997 with a two-year project, which I directed, to develop an outcome-based accountability system in collaboration with state education directors and other adult educators.  
	One goal in developing this system was to produce a set of valid and reliable data that could demonstrate to legislators and other policymakers the value of adult education.  But we also wanted the system to do more than that.  We recognized the value of data as a program improvement tool and wanted the system to be a source of information for states and local programs to use for program management and improvement. Therefore, we tried to build state flexibility into the data requirements.

Based on the system we developed though this project in 1997-1998, the NRS was born and eventually incorporated into the Workforce Investment Act (WIA) in 1998.  WIA made the system mandatory (we had considered it voluntary until then) and added a requirement for states to set performance standards for outcome measures and an incentive award program to motivate performance.  Our focus then shifted from system development to implementation.

Since 1998, we have engaged in a series of training and technical assistance activities that we hoped would build on each other to meet our dual goals of having an effective accountability system and a source of data for local and state program improvement efforts.  We planned these activities in three phases.  First, we needed to get the NRS understood and working in each state, which we accomplished through national and regional training, development of policy and procedural documents, web sites and individual technical assistance. 
In the second phase we focused on improving the quality of data and improving the uniformity of how states and programs collected the date.  In 2002, we produced a data quality guide for local programs that described ways to collect data in valid and reliable ways and reinforced this information with regional trainings of state staff.  As these data quality improvement efforts continue we moved into the third phase of assistance, promoting data use for program management and improvement among states and local programs.  We again developed a “how-to” guide, accompanied by regional state trainings in 2003.  Most recently, we are just now completing a third series of workshops on using data for program monitoring to assess local program performance and identify areas for change.

Promoting NRS data use for program improvement continues to be one of our top priorities.  We see using data as the key to improving the quality of local programs and as a way to get state and local staff to understand the value of data – to see data not as an administrative burden, but a valuable tool.  I think we are beginning to see some successes in this area.  We used to have to spend some time at our trainings to answer questions about the NRS and defend its purpose and value.  Now we not only hear more acceptances, but a genuine desire to learn to use data and really improve program quality—our original goal all along.  For example, programs are using data to make recruitment and enrollment decisions and schedule class times and instructional approaches.  States are using data to identify high performing and low performing programs, to target technical assistance for program improvement and make funding decisions.   Many states NRS data systems now have reports built in that help local programs understand their student attendance, enrollment and outcomes and make informed decisions on program improvement efforts.

While this is heartening to know, we also know there is a lot more work to do, which brings us to our topic, using NRS data for program improvement.   In traveling the country and doing training and technical assistance on this topic I recognize some of the issues involved in using data and have heard several of the questions that can serve as a basis for our discussion here.

What kinds of decisions do you make using data?  What kinds of data do you use to steer local improvement?

How accessible are these data to you?

What successes have you had in using data?

What data do you need that NRS does not collect?

What are barriers to using data?  What other systems and structures do states and local programs need to put into place to supplement NRS data?

Larry Condelli
American Institutes for Research