Return-Path: <nifl-povracelit@literacy.nifl.gov> Received: from literacy (localhost [127.0.0.1]) by literacy.nifl.gov (8.10.2/8.10.2) with SMTP id f6PGgff05268; Wed, 25 Jul 2001 12:42:42 -0400 (EDT) Date: Wed, 25 Jul 2001 12:42:42 -0400 (EDT) Message-Id: <20010725.123943.9286.1.GDEMETRION@juno.com> Errors-To: listowner@literacy.nifl.gov Reply-To: nifl-povracelit@literacy.nifl.gov Originator: nifl-povracelit@literacy.nifl.gov Sender: nifl-povracelit@literacy.nifl.gov Precedence: bulk From: "GEORGE E. DEMETRION" <gdemetrion@juno.com> To: Multiple recipients of list <nifl-povracelit@literacy.nifl.gov> Subject: [NIFL-POVRACELIT:538] Re: Washington Post Article on the NALS X-Listprocessor-Version: 6.0c -- ListProcessor by Anastasios Kotsikonas X-Mailer: Juno 1.49 Status: O Content-Length: 7206 Lines: 165 I had originally written the message below, this morning and had trouble sending it. When I finally did, I saw Irwin Kirsch's response. Not that I've had time to read and certainly not to assimilate it, though I am pleased that there is a response and a discussion on these matters. Also I misattributed the first quoted paragraph to the Washington Post article, which was actually part of Tom Sticht's July 10th NLA message. In any event, the genie is out of the bottle. The only ything now is to go forward with a full discussion of these matters. Significant issues have been raised and the matter needs a thorough exploration. On the run. George Demetrion _______________________________________________________ Colleagues: With Debbie and others, I look forward to NIFL's letter to the Washington Post. Along with that I believe close public analysis of the issues in the Post article is warranted. In the meantime, I would like to offer my thoughts and encourage others to share their views, question, and concerns. Many thanks to Tom Sticht for posting this article and for vigorously pursuing the logic of his research to the conclusions he reaches. He tackles the tough issue of calling them as he sees them and then holding the policy and research communities to account. There is no equivocation in what he says (Refer also to Tom's July 10th NLA posting). If sound pedagogy is going to determine policy over assessment, then Tom has and continues to make a substantial contribution to this effort. If Tom's analysis is accurate, will the policy community work with him and others to create an assessment system that is intellectually sound? Without that there can only be chaos, and ultimately a form of insantity that will permeate the land of literacy, if it hasn't already. Does "realism" require us to embrace such insanity that an assessment system *not* based on sound pedagogy logically leads to? The report to which Tom refers as well as his own analysis seems rather technical and perhaps too arcane for practitioners to follow. But there are some critical points readily accessible in the Washington Post article which may be worthy of our most careful scrutiny. I highlight one sentence: "Chapter 14, written by Andrew Kolstad, the original project director for the NALS at NCES, systematically undermines the entire test and questions its construct validity, that is, the question of just what it is that the test measures, its standards validity, that is, the validity of the 80 percent probability standard that was used to assign people to the five literacy levels, and the use validity, that is, the validity of the NALS for accurately identifying adults at risk for poor literacy skills." The key point in the Post article is that the 80% response rate was too high a figure by which to base the problematic category of the NALS "levels" that assign the adult population to Levels 1-5 in their capacity to mediate print in the areas of mastering prose, documents and quanitative information (if I have that last NALS category right). The extent to which the concept of "levels" (itself a metaphor) adequately depicts the ways in which individuals process information is itself a problematic issue that we have seen in various, largely unanswered critiques againt National Reporting System levels. Tom has written incisively on the notion that there are no levels out there that can be used to adequately address the highly contextual manner in which adults appropriate literacy for their own purposes--a concept he applies both to NALS and the NRS. Perhaps that's part of the issue as to why there is a wide disparity between what the NALS experts identify as the 90 million adults with "at risk" literacy skills and what adults themselves say, who don't, on those proportions identify themselves as having a literacy "problem." I think of my brother, a high school drop out, who would probably rate in the NALS 2 level, at least based on the current 80% response rate that is used to determine the levels. He is quite competent (more so than I) in the arenas out of which he chooses to live out his life. I'm not sure he would understand the terminology of "deficiency," though I think he would get the jist and categorically reject any attribution of it to himself based on how he might do on a test score. Until the recent NLA message and the Post article, I thought the major problem with NALS was mainly epistemological (that is, the positivistic theoy of knowledge upon which I assume it is based). I took for granted that from a technical perspective that it was accurate based upon the paradigmatic assumptions of its construction. I work out of a different set of paradigmatic assumptions, those grounded out of qualitative research and a pragmatic epistemological tradition, which assumes that knowledge is constructed out of the process of living and in seeking to resolve problems and issues that perplex/stimulate people. Consequently, I place a lot of credence on a critical exploration of rich narrative descriptions to explore the development or "growth" of such learning. In short, my research paradigm may be defined as anthropological, experiencial, and process-oriented. Though now, based on the Post article, one begins to wonder about the validity of the NALS study stemming from the empirical research tradition upon which it seems to have been based. While I am clearly in no position to adequately assess the technical issues raised in the Post article, if in fact, the 80% response rate was too high to base the (some would say, problematical) concept of literacy levels, and that the 50% figure would have provided the more accurate figure, the simple and logical question would be, why wasn't the 50% figure used in the first place? Perhaps we could have some responses to that. Perhaps why that lower figure wan't selected is a telling matter in itself. This may seem like a highly technical question, and in no small measure, it is. Yet as Tom puts it, it has profound implications for the field, since the NALS figures have been used authoritatively throughout the field and practitioners have taken their accuracy on faith in making their cases to various publics upon it. As put at the end of the Post article: "This new technical report by NCES calls into question the entire validity and hence the meaning of the NALS (and other tests and studies based on the NALS methodology, such as the International Adult Literacy Survey). That this major assessment can throw little light on how many adults are at risk for their literacy skills should command a major dialogue and investigation into national adult literacy assessments. This dialogue should happen before another 10 to 15 million dollars are spent in a follow-up survey that may be equally invalid and lead to further defamatory practices by reporters or government officials here and abroad." I would suggest that this is no small matter. My thanks to Tom, for his willingness to raise these issues in a very visible manner. The viability of democracy demands no less. What do others think? George Demetrion Gdemetrion@msn.com
This archive was generated by hypermail 2b30 : Fri Jan 18 2002 - 11:33:07 EST