Dan, Karen, Steve and David,
 
You all have raised the issue of changing the NRS reporting period from
one year to multiple years.  While this is off the topic of using data,
I will give a quick response.
 
First, the mandate is to have an annual reporting system so some
information is required each year top report to Congress.  Beyond this,
this topic has come up and been considered multiple times and there is
some flexibility with ED to make some changes to the reporting period,
if there is a compelling reason that can be demonstrated.  Our analyses
of several states' data (not NRS reported data but individual student
data from over several years), however, including some very large
states, is that there are proportionally very few students who continue
year to year (on the order of 5 percent or less in some states) and it
does not appear at this time that it would make a difference in
performance data at the national level, as Dan Wann suggested.
 
NRS is a national system so with some local programs (such as Karen's)
or other states, there may be large numbers of students who continue
year to year and in those instances it might be advisable to look at and
report multi-year data.  To bring us back to our topic of using data,
this would be a good analysis a state or local program to pursue-- to
look at returning and continuing students and see how they differ in
outcomes and other factors from students who stay a short time.  We also
can rely on research, such as Steve Reder's study to look at long-term
relationships, which if compelling, could result in a change to the
reporting period in the future.  
 
________________________________

From: assessment-bounces at nifl.gov [mailto:assessment-bounces at nifl.gov]
On Behalf Of Karen Mundie
Sent: Thursday, April 19, 2007 11:51 AM
To: The Assessment Discussion List
Subject: [Assessment 761] Re: Using Data


Dan, I know that's the perception, but I also know that we roll over
about half of our students from one year to the next. . . and some of
those students had rolled over the previous year as well. We've actually
had to put a three year limit on some students (especially ESL). 


I'm having our data person look this up as well as we can.
Unfortunately, our data tends to be divided, as David indicated, in
discrete yearly "lumps." We can get the information, but it's
time-consuming because the data bases are designed for accountability
over a contract year.  


We certainly do have a lot of students who come in with short term goals
and leave when these are accomplished. We also have a lot of stop out
students, who have to put goals on the back burner while they work out
other issues. I think, however, we do keep a significant number of
students over time. I think for my own little research project, I'm
going to investigate gains over multiple years. 






Karen Mundie
Associate Director
Greater Pittsburgh Literacy Council
100 Sheridan Square, 4th Floor
Pittsburgh, PA  15206
412 661-7323 (ext 101)
kmundie at gplc.org


GPLC - Celebrating 25 years of literacy, 1982-2007




This e-mail is intended solely for the use of the named addressees and
is not meant for general distribution.  If you are not the intended
recipient, please report the error to the originator and delete the
contents.



On Apr 19, 2007, at 9:47 AM, Dan Wann wrote:


	I wonder if there is enough data to even show that adult basic
and ESL students stay with a program in large enough numbers to track
over a longer period?   The conventional wisdom of those outside of the
adult basic skills network is that basic skills programs have little
impact because students do not stay long to make a difference.  Do we
have any evidence that shows we work with the same students more than
one year and that we work with a high enough number of students more
than one year to make a significant difference? 

	

	

	Dan Wann

	Professional Development Consultant 

	IN Adult Education Professional Development Project 

	

	dlwann at comcast.net  

	

	
________________________________


	From: assessment-bounces at nifl.gov
[mailto:assessment-bounces at nifl.gov] On Behalf Of David J. Rosen
	Sent: Thursday, April 19, 2007 2:05 AM
	To: The Assessment Discussion List
	Subject: [Assessment 752] Re: Using Data

	

	Larry, and others,
	
	Tina, and many other program administrators have observed
patterns like this that suggest that a one-year time frame, a funding
year, may not be the best unit of time in which to measure learner
gains, except for those who are doing basic skills brush-up or who have
very short-term goals like preparing for a drivers license test. I
wonder if there is a possibility that the NRS might be adjusted, perhaps
in a pilot at first, so that a longer period of learning, say three
years, might be used to demonstrate learner gains.  Of course, there
would need to be intermediate measures, but accountability -- for
programs and states -- might be based on a longer period of time.  
	
	It seems to me that the one-year time frame within to measure
learning gains or goals accomplished comes not from K-12 or higher
education, but rather Congressional expectations for job skills
training. Would you agree?
	
	Also I wonder if you or others have some examples of programs
that track and report learner outcomes over several years, and use the
data for program improvement.
	
	David J. Rosen
	djrosen at comcast.net
	
	
	Tina_Luffman at yc.edu wrote: 

	Hi Luanne,

	

	I find it interesting that what you are finding in data seems to
be consistent with what we see in our GED classes here in Arizona. Often
the last group who enter in March are the least likely to stay with the
program until posttesting, and the August group seem to have the highest
posttesting and retention rate.

	

	Tina

	
	
	
	
	Tina Luffman
	Coordinator, Developmental Education
	Verde Valley Campus
	928-634-6544
	tina_luffman at yc.edu

	

	-----assessment-bounces at nifl.gov wrote: -----

	To: <assessment at nifl.gov> <mailto:assessment at nifl.gov> 
	From: "Luanne Teller" <lteller at massasoit.mass.edu>
<mailto:lteller at massasoit.mass.edu> 
	Sent by: assessment-bounces at nifl.gov
	Date: 04/18/2007 10:56AM
	Subject: [Assessment 746] Using Data
	
	
	

	Hi all: 

	  

	I wanted to chime in about our program's use of data since this
is the focus of our discussion.  Coincidentally, I am in the process of
writing our proposal for next year, so I am knee-deep in data even as we
speak! 

	  

	The use of data takes many forms in our program.  We look at
what most people consider the "hard data" -- the raw numbers with regard
to attendance, learner gains, retention, goal attainment, etc.  We
believe; however, that the numbers alone provide an incomplete picture
of what is happening, so we use the numbers as a basis for discussion,
not decision making.  After analyzing the numbers, we begin to look at
additional sources of data that we find essential in informing our
planning---meetings with staff, classes, our student advisory board, and
focus groups. 

	  

	Here's an example we're currently working on---we did a two year
analysis of learner retention, and began to document why students did
not persist.  We found that the retention for students who enrolled
after January 1 (our programs runs on a school calendar year from
September to June) was significantly lower than the retention for
students who began in September.  Even more compelling, we learned that
the retention for students who began after March 1 was 0%. 

	  

	We met with staff and students, and did some research around
student retention issues.  After a year-long process, we decided to
pilot a "managed enrollment" approach.  In Massachusetts , our grantor
(MA DOE) allows us to "over-enroll" our classes by 20%, so we enroll 20%
more students in the fall.  When students leave, we "drop" the
overenrolled students into funded slots.  This allows us to keep the
seats filled even with the typical attrition that occurs. 

	  

	In January, when we do our mid-point assessments; we move
students to the higher level who are ready to progress....that typically
leaves several openings in the beginner levels and we begin students in
February as a cohort.  This year, we implemented new orientation
programs including a requirement that new students observe a class
before enrolling.  

	  

	While it is still too early to tell if these new procedures will
have a positive impact, we are hopeful and we know anecdotally that the
transition seems to be easier for some of these students.  We are eager
to look at the data at the end of the year to analyze the effectiveness
of this plan.  

	  

	As we begin to look at our data, we are finding that there seem
to be a unique set of issues for our beginner ESOL students.  We suspect
that the lack of effective English communication skills to advocate for
themselves with employers is influencing their attendance and
persistence.  This is an issue that we are beginning to tackle in terms
of policy.  Do we need to have a more flexible, lenient policy for
beginner students?  Is there a way to support students in addressing
these employment issues?  How can we empower students more quickly?  Are
there other issues for these beginner level students that affect their
participation?  As we enter these discussions, the numbers will provide
a basis for developing strategies, but the students themselves with be
our greatest source of valuable data. 

	  

	Luanne Teller 

	  

	Luanne Teller 

	-------------------------------
	National Institute for Literacy
	Assessment mailing list
	Assessment at nifl.gov
	To unsubscribe or change your subscription settings, please go
to http://www.nifl.gov/mailman/listinfo/assessment

	

	 
	
________________________________



	 
	-------------------------------
	National Institute for Literacy
	Assessment mailing list
	Assessment at nifl.gov
	To unsubscribe or change your subscription settings, please go
to http://www.nifl.gov/mailman/listinfo/assessment
	  

	

	-------------------------------
	National Institute for Literacy
	Assessment mailing list
	Assessment at nifl.gov
	To unsubscribe or change your subscription settings, please go
to http://www.nifl.gov/mailman/listinfo/assessment


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.nifl.gov/pipermail/assessment/attachments/20070419/a64f637a/attachment.html