September 23, 2007

The Assessment of Health Literacy: Is it in Good Health?

Tom Sticht
International Consultant in Adult Literacy

I recently arrived back in the office from travel and took a quick look at
what was going on on various discussion lists on the internet. Of
particular interest was a conversation that had just ended on the Health
Literacy discussion list of the National Institute for Literacy (NIFL)
which was concerned with the results of the Health Literacy test results of
the 2003 National Assessment of Adult Literacy (NAAL). The NAAL general
findings were released late in 2003 while the Health Literacy results were
released in 2007.  For me, the discussion once again illustrated the severe
problem of knowing what is actually being measured with the NAAL and similar
assessments, such as the International Adult Literacy and Life Skills (ALL)
survey, which like the NAAL was also released in 2006 but in January, well
before the release of the first of the NAAL reports.

In the Health Literacy assessment, items that were part of the overall NAAL
survey were pulled out for separate analysis. In the NAAL there were three
scales of literacy: Prose, Document, and Quantitative. But in forming the
Health Literacy scale, these three distinctions were ignored and instead
items from these different scales were used to create one scale of Health
Literacy.

The discussion on the NIFL Health Literacy list raised some important
questions regarding the construct validity, that is, the question of what
is being measured,  of both the general NAAL, which included the Health
Literacy items, and the Health Literacy assessment pulled from the general
NAAL. One of the strands of discussion dealt with just what was the
difference between general literacy and health literacy. This question was
never answered in any definitive manner, thus raising the issue of the
construct validity of both the general NAAL and the specific Health part of
the NAAL. The issue is: what are these two assessments assessing?
Apparently, all general literacy includes health literacy, because the
items comprising the latter Health Literacy scale were included in the
general literacy scales, but Health Literacy does not necessarily include
all aspects of general literacy. Just what these "general" aspects are was
not stated. Discussions of the meaning of "health literacy" lead to such a
wide ranging set of proposals as to render questionable the very construct
of "health literacy," rendering it as near meaningless as the construct of
"general literacy" purportedly measured in the total NAAL.

A second question of the validity of the NAAL was raised when it was asked
why the NAAL used a probability of .67% for saying someone was proficient
at a given level of either general or health literacy. The answer given by
a principal in the development of the NAAL was, "The 67% is a parameter in
a statistical theory of item performance that must be understood in
connection with all the other parameters." However this does not explain
how the .67% probability of performance came to be the standard for
assigning skill levels to people and people to task difficulty levels. In
the International ALL survey released in early 2006, which included data
for adult literacy in the United States, the parameter for proficiency was 
a .80% probability of getting items correct. But a National Academy of
Sciences panel determined that this was too high and it was an unreasonable
standard, so they recommended dropping the standard to .67%. All this
standard setting was the discussion of an earlier meeting of experts in
psychometrics in which it was acknowledged that a 50 percent standard was
the most defensible technical standard because it kept the probability of
making mistakes about peoples skills equal in terms of overstating or
understating such skills.

The fact is that setting the standards of proficiency, even though given
much forethought, remains an arbitrary action and illustrates the point
that far from measuring some "levels' of skills that adults possess, it is
instead a creation of a distribution or distributions of skills that can be
made to make a population look good or bad depending on the decisions taken.

As things stand today, when asked, most adults in the national literacy
surveys do not think they have much of a problem with literacy. This raises
validity issues again. The question is, which scale of measurement provides
the most valid indicator of people's actual ability to perform literacy
tasks in the "real world" in which they live: people's self reports or the
creations of distributions on literacy tests by psychometricians?

Unlike geological strata, there are no literacy levels out there to be
discovered, they must be created. How many adults are "at risk" because of
low literacy skills? No one knows.

NOTE: I have refrained from identifying particular discussants on the Health
Literacy list without their permission. The discussion itself may be found
online by going to www.nifl.gov  and clicking on the Discussion Lists pages
for Health Literacy. It is an interesting discussion raising questions about
the "health" of the assessment of adult literacy in general. For additional
comments on the assessment of adult literacy google Sticht assessment adult
literacy.

Thomas G. Sticht
International Consultant in Adult Education
2062 Valley View Blvd.
El Cajon, CA 92019-2059
Tel/fax: (619) 444-9133
Email: tsticht at aznet.net