![]() |
![]() |
[Assessment 929] Is Adult Literacy Assessment Healthy?tsticht at znet.com tsticht at znet.comSun Sep 23 19:53:36 EDT 2007
September 23, 2007 The Assessment of Health Literacy: Is it in Good Health? Tom Sticht International Consultant in Adult Literacy I recently arrived back in the office from travel and took a quick look at what was going on on various discussion lists on the internet. Of particular interest was a conversation that had just ended on the Health Literacy discussion list of the National Institute for Literacy (NIFL) which was concerned with the results of the Health Literacy test results of the 2003 National Assessment of Adult Literacy (NAAL). The NAAL general findings were released late in 2003 while the Health Literacy results were released in 2007. For me, the discussion once again illustrated the severe problem of knowing what is actually being measured with the NAAL and similar assessments, such as the International Adult Literacy and Life Skills (ALL) survey, which like the NAAL was also released in 2006 but in January, well before the release of the first of the NAAL reports. In the Health Literacy assessment, items that were part of the overall NAAL survey were pulled out for separate analysis. In the NAAL there were three scales of literacy: Prose, Document, and Quantitative. But in forming the Health Literacy scale, these three distinctions were ignored and instead items from these different scales were used to create one scale of Health Literacy. The discussion on the NIFL Health Literacy list raised some important questions regarding the construct validity, that is, the question of what is being measured, of both the general NAAL, which included the Health Literacy items, and the Health Literacy assessment pulled from the general NAAL. One of the strands of discussion dealt with just what was the difference between general literacy and health literacy. This question was never answered in any definitive manner, thus raising the issue of the construct validity of both the general NAAL and the specific Health part of the NAAL. The issue is: what are these two assessments assessing? Apparently, all general literacy includes health literacy, because the items comprising the latter Health Literacy scale were included in the general literacy scales, but Health Literacy does not necessarily include all aspects of general literacy. Just what these "general" aspects are was not stated. Discussions of the meaning of "health literacy" lead to such a wide ranging set of proposals as to render questionable the very construct of "health literacy," rendering it as near meaningless as the construct of "general literacy" purportedly measured in the total NAAL. A second question of the validity of the NAAL was raised when it was asked why the NAAL used a probability of .67% for saying someone was proficient at a given level of either general or health literacy. The answer given by a principal in the development of the NAAL was, "The 67% is a parameter in a statistical theory of item performance that must be understood in connection with all the other parameters." However this does not explain how the .67% probability of performance came to be the standard for assigning skill levels to people and people to task difficulty levels. In the International ALL survey released in early 2006, which included data for adult literacy in the United States, the parameter for proficiency was a .80% probability of getting items correct. But a National Academy of Sciences panel determined that this was too high and it was an unreasonable standard, so they recommended dropping the standard to .67%. All this standard setting was the discussion of an earlier meeting of experts in psychometrics in which it was acknowledged that a 50 percent standard was the most defensible technical standard because it kept the probability of making mistakes about peoples skills equal in terms of overstating or understating such skills. The fact is that setting the standards of proficiency, even though given much forethought, remains an arbitrary action and illustrates the point that far from measuring some "levels' of skills that adults possess, it is instead a creation of a distribution or distributions of skills that can be made to make a population look good or bad depending on the decisions taken. As things stand today, when asked, most adults in the national literacy surveys do not think they have much of a problem with literacy. This raises validity issues again. The question is, which scale of measurement provides the most valid indicator of people's actual ability to perform literacy tasks in the "real world" in which they live: people's self reports or the creations of distributions on literacy tests by psychometricians? Unlike geological strata, there are no literacy levels out there to be discovered, they must be created. How many adults are "at risk" because of low literacy skills? No one knows. NOTE: I have refrained from identifying particular discussants on the Health Literacy list without their permission. The discussion itself may be found online by going to www.nifl.gov and clicking on the Discussion Lists pages for Health Literacy. It is an interesting discussion raising questions about the "health" of the assessment of adult literacy in general. For additional comments on the assessment of adult literacy google Sticht assessment adult literacy. Thomas G. Sticht International Consultant in Adult Education 2062 Valley View Blvd. El Cajon, CA 92019-2059 Tel/fax: (619) 444-9133 Email: tsticht at aznet.net
More information about the Assessment mailing list |