Hello Venu,

I don't think a longitudinal study, or a random sample of programs,  
would meet the need that individual programs or students have for  
measuring learner progress, or programs' interest in knowing learner  
outcomes or impact on learners.  These need to be addressed in other  
ways.  But since you have asked, here -- in broad brush strokes -- is  
what we need to do:

• Programs need formative and summative evaluation measures. A good  
formative system should have as one of its goals no surprises for the  
student and teacher when they see the results of a summative assessment.
• Standardized tests could be part of program evaluation if there is  
a high degree of validity between what is taught and tested.
• Multiple measures are important.  In addition to standardized  
tests, teacher evaluation, student self-evaluation, performance-based  
measures, and other measures could be part of the mix.
• In adult education it is especially important to measure progress  
toward and attainment of student goals and objectives. A superficial  
process of asking students to check off their goals when they enroll  
is not sufficient.  Student goals need to regularly be re-visited,  
and broken down into short-term objectives.  Students need an  
opportunity to regularly reflect on their progress, perhaps with a  
teacher or tutor, or with other students. Their goals may change.   
indeed changing goals may be one measure of progress!

Does all of this sound like it would take more resources than most  
adult education programs have? Yes.  High quality in education often  
requires more resources.  If we believe in high quality, then we  
should advocate for it, and for the resources needed to achieve it.

David J. Rosen
djrosen at comcast.net

On Oct 19, 2007, at 6:55 PM, Venu Thelakkat wrote:


> David,

>

>

>

> An independent study of adult education students over a longer  

> period of time sounds like a great way to evaluate the adult  

> education system from a national or state perspective, even though,  

> like you said, there are some methodological challenges.  It will  

> certainly provide more valid and reliable data than we get from  

> NRS.  However, it still leaves unresolved the issue of evaluating  

> individual programs.  Like Diane said, states have been using NRS  

> data to allocate funding and target technical assistance.  How  

> would a longitudinal study meet this need?

>

>

>

> Venu Thelakkat

>

> Director of ASISTS/Data Analysis

>

> Literacy Assistance Center

>

> 32 Broadway, 10th floor

>

> New York, NY 10004

>

> (212) 803-3370

>

> venut at lacnyc.org

>

> www.lacnyc.org

>

> From: assessment-bounces at nifl.gov [mailto:assessment- 

> bounces at nifl.gov] On Behalf Of David J. Rosen

> Sent: Thursday, October 18, 2007 8:38 PM

> To: The Assessment Discussion List

> Subject: [Assessment 984] Re: level movement

>

>

>

> Assessment Colleagues,

>

>

>

> On Oct 18, 2007, at 7:00 PM, Venu Thelakkat asked:

>

>> I am told that standardized assessments, especially those used in  

>> adult literacy, have many problems.  But what is the alternative?   

>> Policy makers and funders (private or public) want accountability  

>> for the money they disperse and there is very little else that has  

>> been proposed.  Even in NRS, educational gain is the only measure  

>> that has some validity as a program outcome.  The other measures  

>> such as getting a job or entering post secondary education are  

>> very unreliable given the wide variety of methods used to collect  

>> and report the data.

>>

>

>

> Great question, Venu.

>

> One alternative that makes sense has been proposed, but I am told  

> that Congress so far has been unwilling to accept it. Instead of  

> reporting outcomes for every student in every federally-funded  

> program, OVAE should each year pick a random sample of federally- 

> funded programs and measure attainment of student goals over a  

> multi-year period, that is, measure impact.  Some would argue that  

> measuring impact is not possible, that adult education students are  

> too hard to track over time, but the Longitudinal Study of Adult  

> Literacy, now in its eighth year in Portland, Oregon, has shown  

> that it is not impossible.

>

>

>

> Isn't impact more useful to policy makers than outcomes? Does  

> anyone in Congress really care how many level gains students make?  

> Some policy makers may care about how many program participants --  

> especially over time -- have improved their employment status, can  

> read to their children, can use  a computer, now have a diploma or  

> GED, and have succeeded in their education beyond secondary level.   

> Of these goals, only the GED and high school diploma (for now at  

> least) require a standardized test.

>

>

>

> David J. Rosen

>

> djrosen at comcast.net

>

> -------------------------------

> National Institute for Literacy

> Assessment mailing list

> Assessment at nifl.gov

> To unsubscribe or change your subscription settings, please go to  

> http://www.nifl.gov/mailman/listinfo/assessment

> Email delivered to djrosen at comcast.net