Data collection is an important aspect of any kind of program because it
allows us to focus our attention on improving the quality of service to our
students and provides a mechanism to acknowledge accomplishments and set
goals.  In Maryland, we have a very accessible data system.  All programs
have access to their data and to the statewide data anytime they need it.
The data helps decision making at local and state levels.  Some of those
decisions center around the accuracy of the data collected.  Other decisions
involve local program improvement plans and setting state targets for
performance.  The data also holds the local and state programs accountable
for meeting the outcomes addressed in the grant application.  At a state and
local level, the data can help programs determine the types of state
leadership activities that need to be developed and implemented to improve
program quality. In addition, the data can be used to inform the state
legislators about the needs in our state for additional funding.  An example
of success in using the data was in analyzing the diploma success rate. It
was determined that goal setting was an area that needed to be addressed at
the local areas and that a data quality plan needed to be completed by some
programs.  Those targeted programs received technical assistance on these
issues and the diploma retention rate has improved.  There are weaknesses to
the data collection process.  The NRS system does not collect all the data
on a student who is enrolled in more than one class.  Therefore, learner
gains are missed.  Teachers complain about the extra time it takes to
collect the information and that the information does not capture all the
accomplishments a student may make through the year.  So there are some
financial and logistical barriers to collecting the NRS information.

Karen Gianninoto   

-----Original Message-----
From: Sandy Strunk [mailto:sandy_strunk at iu13.org]
Sent: Monday, May 10, 2004 10:59 AM
To: NIFL -PLI List
Subject: [NIFL-PLI] Using NRS Data for Program Improvement


The following posting is from Larry Condelli, Managing Director in the
Education and Human Development Division of the American Institues for
Research. Larry leads the adult education team and is project director for
the National Reporting System. He has been involved in projects on ESL
research, the professional development of adult educators and the
development of content standards for adult education. He has worked in adult
education since 1990.

At the end of his posting, Larry poses several questions he hopes you'll
respond to. I, too, hope you will take this opportunity to discuss the
National Reporting System and how NRS data can be used for program
improvement.

Sandy Strunk, PLI List Moderator

********************************************************


	Several years ago, I started a new job and my first assignment was a
very small project to help the Division of Adult Education and Literacy
(DAEL) assist states to improve their ability to evaluate the quality of
local adult education programs.  Little did I suspect that this work would
eventually evolve into developing the national accountability system for the
adult education program, and what we now know as the National Reporting
System (NRS).  Before this, I had little knowledge of adult education and
accountability, but life has a way of bringing about the unexpected. 
	Most people probably don't know that the NRS had its origin in 1996,
when a group of state directors of adult education were the first to move
forward the idea for a national accountability system.  At that time there
were proposals in Congress to roll adult education into a national work
force development block grant system.  It was widely feared that block
grants, giving governors discretion on how to set up the system in their
states, would mean the end of the adult education program.  At the same
time, demands for accountability increased for all programs, at both the
federal and state levels.  To preserve meet these demands and adult
education, state directors and others realized we needed a way to collect
valid and reliable data to show what adult education is, the type of
students who enrolled and the outcomes students achieved - and that these
outcomes went beyond employment.  DAEL responded in 1997 with a two-year
project, which I directed, to develop an outcome-based accountability system
in collaboration with state education directors and other adult educators.  
	One goal in developing this system was to produce a set of valid and
reliable data that could demonstrate to legislators and other policymakers
the value of adult education.  But we also wanted the system to do more than
that.  We recognized the value of data as a program improvement tool and
wanted the system to be a source of information for states and local
programs to use for program management and improvement. Therefore, we tried
to build state flexibility into the data requirements.

Based on the system we developed though this project in 1997-1998, the NRS
was born and eventually incorporated into the Workforce Investment Act (WIA)
in 1998.  WIA made the system mandatory (we had considered it voluntary
until then) and added a requirement for states to set performance standards
for outcome measures and an incentive award program to motivate performance.
Our focus then shifted from system development to implementation.

Since 1998, we have engaged in a series of training and technical assistance
activities that we hoped would build on each other to meet our dual goals of
having an effective accountability system and a source of data for local and
state program improvement efforts.  We planned these activities in three
phases.  First, we needed to get the NRS understood and working in each
state, which we accomplished through national and regional training,
development of policy and procedural documents, web sites and individual
technical assistance. 
In the second phase we focused on improving the quality of data and
improving the uniformity of how states and programs collected the date.  In
2002, we produced a data quality guide for local programs that described
ways to collect data in valid and reliable ways and reinforced this
information with regional trainings of state staff.  As these data quality
improvement efforts continue we moved into the third phase of assistance,
promoting data use for program management and improvement among states and
local programs.  We again developed a "how-to" guide, accompanied by
regional state trainings in 2003.  Most recently, we are just now completing
a third series of workshops on using data for program monitoring to assess
local program performance and identify areas for change.

Promoting NRS data use for program improvement continues to be one of our
top priorities.  We see using data as the key to improving the quality of
local programs and as a way to get state and local staff to understand the
value of data - to see data not as an administrative burden, but a valuable
tool.  I think we are beginning to see some successes in this area.  We used
to have to spend some time at our trainings to answer questions about the
NRS and defend its purpose and value.  Now we not only hear more
acceptances, but a genuine desire to learn to use data and really improve
program quality-our original goal all along.  For example, programs are
using data to make recruitment and enrollment decisions and schedule class
times and instructional approaches.  States are using data to identify high
performing and low performing programs, to target technical assistance for
program improvement and make funding decisions.   Many states NRS data
systems now have reports built in that help local programs understand their
student attendance, enrollment and outcomes and make informed decisions on
program improvement efforts.

While this is heartening to know, we also know there is a lot more work to
do, which brings us to our topic, using NRS data for program improvement.
In traveling the country and doing training and technical assistance on this
topic I recognize some of the issues involved in using data and have heard
several of the questions that can serve as a basis for our discussion here.

What kinds of decisions do you make using data?  What kinds of data do you
use to steer local improvement?

How accessible are these data to you?

What successes have you had in using data?

What data do you need that NRS does not collect?

What are barriers to using data?  What other systems and structures do
states and local programs need to put into place to supplement NRS data?

Larry Condelli
American Institutes for Research

_______________________________________________
NIFL-PLI mailing list
NIFL-PLI at literacy.nifl.gov
To unsubscribe or change your subscription settings, please go to 
http://www.nifl.gov/mailman/listinfo/nifl-pli