Maybe aggregate figures for large groups can give some indication are doing
better or worse than they used to. Maybe.

But NRS gives surely gives meaningless results for small groups like ours.
(Total 20-25 students with 12 or more hours per year) One individual can
vary the group results far too drastically.

My first posting on the 16th asked a question about the statistical validity
of the figures for a five member level containing only 3 or 4 post-tested
individuals. If these figures has real meaning for our group, would someone
please explain? (I'm not a statistician, but I have had some college math
classes, including calculus.)

The standard for a level with only one member could be 2% or 98%, it still
amounts to pass/fail. At level 4 (high intermediate ABE)  the student could
improve more than 10 CASAS points and the whole level, and therefore our
group, would not be an NRS success!

Yes, I know, the state will take our smallness into consideration when
looking at our figures, but the rules are not written that way.

It seems like it would be useful for small groups to have their results
bundled for comparison to the big guys. Maybe invent a standard for a whole
organization, if it is big enough. If level-by-level analysis of small
groups is as nonsensical as I think it is, why burden us with it? Let us
just report.

And you are right, JTA, the whole NRS system might be useless in the long
run. Maybe even harmful.

Mary G. Beheler
Tri-State Literacy
455 Ninth Street
Huntington, WV 25701
304 528-5700, ext 156
  -----Original Message-----
  From: assessment-bounces at nifl.gov [mailto:assessment-bounces at nifl.gov]On
Behalf Of JTA
  Sent: Wednesday, April 18, 2007 11:49 AM
  To: The Assessment Discussion List
  Subject: [Assessment 743] Data quality and usefulness of NRS



  Folks,


  I hope that this is not a duplicative posting, but one implicit assumption
being made here is that NRS is itself valid.  Many of the postings assume
that it is, and work from there to elaborate usage for various
accountability and program improvement purposes.


  I work in Ohio and have noted the requirements promulgated by DAEL that
other systems must align to NRS (most recently last fall with the Federal
register outline of the periodic review process) and in various letters from
the AIR psychometrician to states or vendors.

  When a national system is above all of the state systems, I would think
that we should strive to keep it on the quality control line as well as all
of the underlings.


  In my reading of the history of NRS, I have not seen the sorts of
construct validity studies that seem to be requested / required for state
tests and vendor tests.  Consider the width of the levels and the statements
that comprise them.

  There was a thread of work coming out of the CRESST organization a few
years ago (Eva Baker's name comes to mind) called Standards for Educational
Accountability (SEA) which might be useful if folks wanted to think about
validation of  NRS.

  Another useful framework is the Data Quality Campaign (DQC) that Achieve,
Inc. and other organizations are using to improve the seamlessness of
transitions in P-16 systems.  In Ohio, for example, we have ABLELink for
adult basic education, EMIS for K-12, and other systems for One-Stops and
post-secondary.  Many of these systems end up in front of the "tower of
babel" when they try to communicate, as noted in some of our "Data Match" to
unemployment wage records.



  JTA
  CoreComm Webmail.
  http://home.core.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.nifl.gov/pipermail/assessment/attachments/20070418/8639794f/attachment.html