SciPICH:Interactive Health Communication, Health Care Providers and Purchasers and Evaluation of IHC Applications

Health Care Providers and Purchasers and Evaluation of Interactive Health Communication Applications

Authors: Holly Jimison, Linda Adler, Molly Joel Coye, Albert Mulley, Jr., and Thomas R. Eng for the Science Panel on Interactive Communication and Health.

Citation: Health Care Providers and Purchasers and Evaluation of Interactive Health Communication Applications. American Journal of Preventive Medicine, January 1999; Vol.16 No.1:16-22

Medical Subject Headings (MeSH): health care providers, purchasers, computers, health communication, evaluation, computer communication networks.

[PDF] [References]

Abstract

Health care providers and purchasers of health services have an opportunity to improve patient care and potentially save costs through the wise purchase of interactive health communication applications for patients and employees. Purchasing decisions based on evaluation and evidence should drive the design and development of new systems. The cycle of evaluation includes a needs assessment before system development, usability testing during development, and studies of use and outcomes in natural settings. This type of evidence is critical to our understanding of how best to provide health information and decision assistance to patients, employees, and others.

Introduction

In recent years there has been increasing interest and progress in the development, implementation, and research regarding telecommunication and computer applications designed for consumers to access information on a wide variety of health care topics. This technology, both hardware and software, is part of a growing trend toward self-advocacy, empowering consumers to take a more active role in their own health care and providing the necessary information to enhance their decision making. More than ever, consumers are using interactive health communication (IHC) applications to augment the information and support provided by health care professionals in the course of clinical encounters. Consumers may access databases of health information, educational tools, and communicate with health care professionals or other patients through e-mail or online support groups. A variety of IHC applications now exist,[1,2] with topics on health concerns such as self-care, informed consent,[3] coping skills,[4,5] and treatment decision-making skills.[4,6]

Health care providers and purchasers have a vested interest in offering patients and consumers reliable and useful health information.[7] Providing access to quality consumer health information has the potential to improve the quality of medical care and lower costs. Interactive health communication technologies are a promising venue for the effective dissemination of health information, social support, and skill-building techniques for changing health behaviors or addressing health problems.[8] The challenge for health care providers and purchasers is to identify systems that are effective and warrant additional investment of resources, time, and an adaptation of current medical practice. Some IHC applications seem to show clear benefit for patient care. However, most remain untested, and there is a growing interest about whether they will provide benefit or result in potential harm. As potential purchasers and distributors of these systems, it is imperative that physician groups, medical centers, and health plans make wise and informed selections based on evidence. However, at this point in time there are few evaluations of commercially available IHC applications. Health care providers and purchasers need to promote and require meaningful evaluations of these systems.

From the provider and purchaser point of view, the importance of supporting the evaluation of IHC applications is based on quality-of-care and business criteria. Evaluation information can be used to identify more effective products, and the quality of health care will be improved if patients are using more effective systems for their health decision-making. Additionally, evaluation feedback on patients' utilization of health care services and satisfaction provides valuable information for purchasing decisions based on cost effectiveness and cost savings.

However, there are several barriers to acquiring comparable evaluation information on IHC applications. Developers are pressed to produce high-quality applications in an extremely competitive environment with limited resources. The emphasis is on getting products out the door as quickly as possible, often with fewer features than originally anticipated and minimal testing. To be competitive, developers cannot afford to take the time and resources to evaluate a system before bringing it to market. Without purchaser demand for evidence, there is no incentive for developers to evaluate their systems. System purchasers must create the demand and promote concern for the quality and cost of medical care to motivate the evaluation of IHC applications.

Motivation for Evaluating IHC Applications

There are several practical issues that are important to health care providers and managed care organizations when considering the purchase or use of an IHC application. A developer who has integrated evaluation throughout the product development cycle will be able to provide clear answers to these concerns. Evaluation of a system is a process that should occur throughout the development cycle. As outlined in the introduction to this article,[8] it includes a user needs assessment, a clear definition of the goals and desired outcomes prior to development, pretesting and quality control of content, iterative usability testing throughout development, the evaluation of system performance, accuracy, security, pilot testing on potential system users, and finally, outcome evaluation of actual system use, measuring outcomes of importance to users and purchasers of the system. Attention to the evaluation process throughout the product development cycle will prepare developers to respond to the type of questions a health care provider or organization will have when making a careful purchase.

The "Evaluation Reporting Template for Interactive Health Communication Applications" proposed by the Science Panel on Interactive Communication and Health, which appears in a recent article,[7] attempts to facilitate the disclosure of information about the evaluation methods used by developers in a standard format. While the proposed template is not intended to function as a purchasing guide, per se, many of the questions that health plans, large employers, and other organizations should consider before buying an IHC application, as presented in Table 1 , are addressed by the reporting elements described in the template. Purchasers should keep in mind, however, that many individuals are typically interested in outcomes related to improvements in health, service, convenience, social support, and general ease of use, rather than cost-effectiveness and market share. Therefore, patient satisfaction, system use, health status outcomes, and health behaviors should also be important criteria for organizations purchasing these systems.

Table 1 [refer to PDF file, p.18]

Outcome Measures of Interest to Providers and Purchasers

From the health care provider or purchaser perspective, it is important that evaluations of IHC applications address outcomes related to quality-of-care and cost effectiveness. One outcome to consider may be the potential for market growth with the use of IHC applications. Use of such systems may be likely to engender client satisfaction and loyalty, as well as encourage new enrollments. Cost savings may be expected from systems that facilitate disease management, self-care, and self-triage. However, as with many medical interventions, it is important to measure whether the desired effect of a system is actually obtained in routine use. The measured outcomes should be relevant to a medical care organization's decision on whether or not to purchase and use such a system. Product evaluations of IHC applications must target the outcomes of interest to the potential purchasers of these systems and address these outcomes from a meaningful and coherent perspective.

Targeted outcomes related to cost and quality of care from the purchaser's perspective include: (1) cost—all expenditures potentially influenced by the use of the system; and (2) quality—including provider satisfaction, patient satisfaction, knowledge gains, health behaviors, health outcomes, access, process control, more appropriate utilization of health care services, and concordance between utilization and expressed preferences (decision quality).

In addition, potential purchasers must consider the strength of evidence when interpreting an evaluation of a system.[7] The results should show internal validity (higher with randomized controlled trials, lower with observational studies), statistical significance (higher with more samples), and clinical or system significance (sufficiently large effect). Finally, providers and medical organizations need to consider how applicable the evidence from an evaluation is to their own organization. To determine whether findings are generalizable to a different organization and situation, one should consider the characteristics of the patient population (demographics) and the characteristics of providers (culture, incentives, willingness to change or adapt).

When is it Worth Collaborating with Developers in an Evaluation?

There are many ways that health care providers and purchasers, as potential purchasers of IHC applications, can promote and participate in evaluations that will produce relevant information for purchasing decisions. The most basic way is for the purchaser to demand that the systems under consideration be (or have been) evaluated. However, the purchaser must understand the added value of evaluation information and be willing to pay a higher price.

Similarly, for systems that have not yet been evaluated in their type of organization, a purchaser should consider funding in-house evaluations, on systems under consideration and those that may be currently in use. It may also be in the interest of providers and purchasers to collaborate with developers in evaluating new systems. This may be of particular value to an organization in unusual clinical settings. The benefits to the purchaser in collaborating with a developer on evaluation include:

Being perceived as a leader in the field with a new product,
Reduced system costs (in return for participating in the evaluation),
Opportunity to influence the experimental design to obtain outcomes of interest, and
Ability to measure effects in the actual population of interest.

The collaboration can be worthwhile for both parties because the risks and rewards of the innovation are shared.

How to Judge the Design of an IHC Application

Providers and purchasers should have some familiarity of the underlying scientific basis for the methods commonly used by IHC applications to effect behavior change or improve decision-making. Having this knowledge is especially important with regards to IHC applications because, in some cases, "showy" technology may distract and even obscure consideration of the content or methodologies used in application design. In this section, we briefly review the concepts of empowerment and self-efficacy, individual preferences, use of the computer as a health information medium, and influence of individual characteristics on usability of IHC applications.

Potential purchasers of these products need to consider whether accepted principles have been applied to the product in question. From previous research, we know that access to health information can enable patients to be more active participants in their care and lead to better medical outcomes.[9-12] Patients report that they want to be informed about their medical condition,[13,14] and the process of sharing information enhances the doctor-patient relationship.

Empowerment and Self-Efficacy

Involvement in one's own medical care also involves the closely linked concepts of patient empowerment and self-efficacy. In general, empowerment can be thought of as the process that enables people to exert control over their lives and their destiny.[15,16] It is closely related to health outcomes in that powerlessness has been shown to be a broad-based risk factor for disease. Studies demonstrate that people who feel "in control" in a health situation have better outcomes than those who feel "powerless."[17,18]

Similarly, self-efficacy is a person's level of confidence that they can perform a specific task or health behavior in the future.[19-21] Clinical studies show that self-efficacy can be most predictive of improvements in patients' functional status.[22,23] Perceived self-efficacy was shown to play a significant role in smoking cessation relapse rate, pain management, control of eating and weight, success of recovery from myocardial infarction, and adherence to preventive health programs.[24-28]

An important measure of success for many IHC applications is how well they promote empowerment and self-efficacy for patients. Empowerment can be enhanced, for instance, by online support groups that allow patients to feel "connected" to others with a similar medical problem. This has been demonstrated in women with breast cancer and patients with AIDS.[4,5,29]

Individual Preferences

The concept of individual preferences is important for IHC applications that focus on health decision-making.[30] While patients need information about the quality of life associated with the medical outcomes of possible decisions, reliable assessment of individual preferences and risk attitudes for clinical outcomes are probably the weakest links in clinical decision-making. Recent efforts to explore the use of computers in communication about health outcomes and in assessing patients' preferences for various health outcomes have started to address these issues.[31-33] Information on patient preferences is important for tailoring information to patients and for providing decision support.[34] Tailored information has been found to be more effective in providing consumer information[35-38] and is preferred by patients.[39] In addition to differences in preferences for health outcomes, patients differ in the degree to which they choose to be involved in decision-making. Research confirms that age (younger), gender (females more than males), and education level (better educated) are strong predictors of the desire to be involved in medical decisions. There is also a greater desire to be involved in medical decisions that appear to require less medical expertise, such as knee injury as opposed to medical decisions for cancer.[40]

The Computer as a Health Information Medium

There has been an increase in research devoted to testing the effectiveness of various formats and types of media for conveying health information to consumers.[41-44] These studies generally show that video and slides are educationally more effective than books and audiotapes. Computer-based approaches offer interactivity, provide feedback in the learning process, and can tailor information to the individual, but more research is needed on the effectiveness of computer-based approaches. Developers will need to be sensitive to human-computer interface issues and implement specifications that meet the needs of a wide variety of users.

Influence of Individual Characteristics on Usability

Factors that influence the health-information-seeking behavior of people include age, gender, disability, race and ethnicity, and socioeconomic status.[45] Research indicates that these variables can predict differences in the amount and type of health-related information that individuals want. While some do not seek much information, others encounter serious barriers to the use of IHC applications.[46]

A lack of reading ability is a functional barrier affecting use of these systems. Numerous studies on literacy and readability confirm the widespread problem of low literacy skills.[47-49] Approximately 1 out of 5 Americans is functionally illiterate, reading at or below the fifth-grade level. Studies show that only one half of people examined are able to comprehend written health education materials and that people's reading levels were well below what is required to understand standard health brochures.[47,50-52] In developing health information, one cannot assume that a person who has completed a certain grade level can read at that level. Health materials should be written at least three grade levels lower than the average educational level of the target population.[53] Text characteristics and organization and clarity also play important roles in comprehension and retention of material.[48] Multimedia techniques can be used to facilitate comprehension by conveying information through video, audio, and graphics, in lieu of text. Additionally, computer approaches that represent material in multiple languages allow efficient tailoring to the user's language of choice, thereby increasing comprehension. Developers should also consider cultural issues associated with health-information-seeking behavior and willingness to use health information technology when designing IHC applications.

Issues Related to Access

As the demand for more health information and decision support grows, the need for wider availability of these systems becomes even more important. Today, these systems can be found in a variety of settings and forms. The most common locations to access these systems are physician waiting rooms, hospitals, health resource libraries, public libraries, worksites, schools, community centers, and of course on personal home computers. Different systems may require quite different physical locations. For instance, many people are uncomfortable exploring sensitive health information in a public space.

Finally, the question of who will pay for the access and use of technologies for consumer health information is still unresolved. Educational and socioeconomic factors still determine access to computers and information technologies. Younger, more affluent, and well-educated patients are more likely to have access to home computers, diagnostic software, and the Internet than disadvantaged populations. Special effort is required to ensure universal access to health information and support.[46] Overcoming these barriers will go a long way toward ensuring access for the people who have the greatest need for these resources.

Elements of a Good Evaluation

Ideally, evaluation should be designed at the conception of a system. Consumer needs and the desired effects of a system should be clearly specified prior to system implementation, so purchasers should ask to see this information. These desired effects should help define the outcomes of interest and evaluation design to carefully measure those outcomes. As presented elsewhere, initial stages of evaluation include specifying a problem or need of a particular target audience through needs assessment.[8] The results of this analysis are used to define the specifications for a product to address those needs.

Evaluations during product development include iterative usability testing to ensure that the product meets the needs of potential users with regard to usability and the facilitation of workflow or tasks. Component testing ensures that all aspects of the system perform accurately and meet design specifications. The final stage of evaluation is to actually measure outcomes during system use. However, a preliminary first step usually involves a pilot evaluation to work out the implementation details of the evaluation and assessment tools. Quite often there are obvious misunderstandings of terms or unanticipated barriers that can be corrected before beginning the larger, more complete study. A more detailed description of evaluation methods and additional references can be found in Rossi and Freeman's book on evaluation.[54]

Role of Evaluation in Driving Quality of Care

Developers of commercial systems have pushed the field of interactive health communication forward with many innovative systems. However, to achieve significant improvements in quality-of-care and health outcomes, we need to acquire evidence on the effectiveness of these systems. Additionally, researchers and system developers must focus on integrating the knowledge gained from these evaluations into the design of new systems. This is a new and emerging field with significant innovations in the commercial sector. Research in several areas is needed to move the field forward in providing real benefits to individual's health outcomes and in showing the effectiveness of the systems to purchasers of health care. Careful needs assessment before system development, usability testing during development, controlled clinical trials, and studies of use and outcomes in natural settings are all critical to our understanding of how best to provide health information and decision assistance to patients.

Advances in communications and information technology will change the way in which medicine is practiced, and it will also change the way in which patients receive information and interact with the health care system.[7] The future holds great promise for consumers becoming empowered and active participants in their medical care decisions. Because the vast majority of health-related decisions are made by the people outside of the formal medical setting,[55] the input of health care providers and purchasers into the IHC application development and evaluation processes is critical.

Acknowledgements

Panel members and staff: Farrokh Alemi, PhD, Cleveland State University, Cleveland, OH; David Ansley, Consumer Reports, Yonkers, NY; Patricia Flatley Brennan, RN, PhD, FAAN, University of Wisconsin-Madison, Madison, WI; Mary Jo Deering, PhD, Office of Disease Prevention and Health Promotion, U.S. Department of Health and Human Services, Washington, DC; David Gustafson, University of Wisconsin, Madison, WI; Joseph Henderson, MD, Dartmouth Medical School, Dartmouth, NH; John Noell, PhD, Oregon Center for Applied Science and Oregon Research Institute, Eugene, OR; Kevin Patrick, MD, San Diego State University and University of California San Diego, San Diego, CA; Thomas Reeves, University of Georgia, Athens, GA; Thomas N. Robinson, MD, MPH, Stanford University School of Medicine, Palo Alto, CA; and Victor Strecher, PhD, MPH, University of Michigan Comprehensive Cancer Center, Ann Arbor, MI.

The authors are grateful to Paul Kim; Andy Maxfield, PhD; Anne Restino, MA; and John Studach, MA, for their contributions to the panel's work, and to Linda Friedman for assistance with copyediting. In addition, the authors thank the liaisons to the Science Panel on Interactive Communication and Health, especially the following persons who offered valuable suggestions for improving this manuscript: Loren Buhle, PhD; David Cochran, MD; Connie Dresser, RDPH, LN; Alex Jadad, MD; Craig Locatis, PhD; Ed Madara; Georgia Moore; Kent Murphy, MD; Scott Ratzan, MD, MPA, MA; Helga Rippen, MD, PhD; and Christobel Selecky.

Address correspondence to: Thomas R. Eng, Office of Disease Prevention and Health Promotion, HHS, 200 Independence Avenue, SW, Room 738G, Washington, DC 20201.

Address reprint requests to: Mary Jo Deering, PhD, Office of Disease Prevention and Health Promotion, U.S. Department of Health and Human Services, 200 Independence Avenue, SW, Room 738G, Washington, DC 20201.

References

[1]: Kieschnick T, Adler LJ, Jimison HB. 1996 Health informatics directory. Baltimore, MD: Williams & Wilkins; 1996.
[2]: Hersey JC, Matheson J, Lohr KN. Consumer health informatics and patient decision-making. Rockville, MD: Agency for Health Care Policy and Research, 1997.
[3]: Llewellyn-Thomas HA, Thiel EC, Sem FWC, et al. Presenting clinical trial information: a comparison of methods. Patient Education and Counseling 1995; 25:97-107.
[4]: Gustafson DH, Bosworth K, Hawkins RP, et al. CHESS: a computer-based support system for providing information, referrals, decision support and social support to people facing medical and other health-related crises. 16th Annual Symposium on Computer Applications in Medical Care, 1992:161-5.
[5]: Pingree S., Hawkins RP, Gustafson DH, et al. Will HIV-positive people use an interactive computer system for information and support? A study of CHESS in two communities. 16th Annual Symposium on Computer Applications in Medical Care 1993:22-26.
[6]: Kasper JF, Mulley, AF Jr., Wennberg JE. Developing shared decision making programs to improve the quality of health care. Quality Review Bulletin 1992; 18:183-190.
[7]: Robinson TN, Patrick K, Eng TR, Gustafson D, for the Science Panel on Interactive Communication and Health. An evidence-based approach to interactive health communication: a challenge to medicine in the Information Age. JAMA 1998; 280:1264-1269.
[8]: Eng TR, Gustafson D, Henderson J, Jimison H, Patrick K, for the Science Panel on Interactive Communication and Health. Introduction to evaluation of interactive health communication applications. Am J Prev Med 1999; 16:10-15.
[9]: Brody DS, Miller SM, Lerman CE, Smith DG, Caputo GC. Patient perception of involvement in medical care: relationship to illness attitudes and outcomes. J Gen Intern Med 1989; 4(6):506-511.
[10]: Greenfield S, Kaplan S, Ware, J Jr. Expanding patient involvement in care: effects on patient outcomes. Ann Intern Med 1985; 102:520-528.
[11]: Korsch BM. What do patients and parents want to know? What do they need to know? Pediatrics 1984; 74:917-919.
[12]: Mahler HI, Kulik JA. Preferences for health care involvement, perceived control and surgical recovery: a prospective study. Soc Sci Med 1990; 31:743-751.
[13]: Ende J, Kazis L, Ash A., et al. Measuring patients' desire for autonomy: decision making and information-seeking preferences among medical patients. J Gen Intern Med 1989; 4(1):23-30.
[14]: Waitzkin H. Doctor-patient communication: clinical implications of social scientific research. JAMA 1984; 252: 2441-2446.
[15]: Peterson C, Stunkard AJ. Personal control and health promotion. Soc Sci Med 1989; 28:819-828.
[16]: Feste C, Anderson RM. Empowerment: from philosophy to practice. Patient Education and Counseling 1995; 26: 139-144.
[17]: Israel BA, Sherman SJ. Social support, control and the stress process, In: Health behavior and health education. Glanz K, Lewis FM, Rimer B, eds. San Francisco, CA: Jossey-Bass, 1990.
[18]: Anderson RM, Funnell MM, Butler PM, et al. Patient empowerment. Results of a randomized trial. Diabetes Care 1995; 18(7):943-949.
[19]: Lorig K, Chastain RL, Ung E, et al. Development and evaluation of a scale to measure perceived self-efficacy in people with arthritis. Arthritis Rheum 1989; 32(1):37-44.
[20]: Holman H, Lorig K. Patient education in the rheumatic diseases-pros and cons. Bull Rheum Dis 1987; 37(5):1-8.
[21]: Bandura A. Self-efficacy towards a unifying theory of behavioral change. Psychol Rev 1977; 84(2):191-215.
[22]: Cunningham AJ, Lockwood GA, Cunningham JA. A relationship between perceived self-efficacy and quality of life in cancer patients. Patient Education and Counseling 1991; 17(1):71-78.
[23]: O'Leary A. Self-efficacy and health. Behav Res Ther 1985; 23(4):437-451.
[24]: Strecher VJ, DeVellis BM, Becker MH, Rosenstock IM. The role of self-efficacy in achieving health behavior change, Health Educ Q 1986; 13:73-91.
[25]: Mullen PD, Laville EA, Biddle AK, et al. Efficacy of psychoeducational interventions on pain, depression, and disability in people with arthritis: a meta-analysis. J Rheum 1987; 14(Suppl. 15):33-39.
[26]: Maibach E, Flora J, Nass C. Changes in self-efficacy and health behavior in response to a minimal contact community health campaign. Health Communication 1991; 3(2):1-15.
[27]: O'Leary A, Shoor S, Lorig K, et al. A cognitive-behavioral treatment for rheumatoid arthritis. Health Psychol 1988; 7(6):527-544.
[28]: Allen JK, Becker DM, Swank RT. Factors related to functional status after coronary artery bypass surgery. Heart Lung 1990; 19:337-43. Comment in: Heart Lung 1991; 20:210.
[29]: Gustafson DH, Hawkins RP, Boberg EW, et al. The impact of computer support on HIV infected individuals. Final report to the Agency for Health Care Policy and Research, 1994.
[30]: Barry MJ, Fowler FJ, Mulley AG, Henderson J Jr, Wennberg JE. Patient reactions to a program designed to facilitate patient participation in treatment decisions for benign prostatic hyperplasia. Medical Care 1995; 33:771-782.
[31]: Jimison HB and Henrion M. Hierarchical preference models for patients with chronic disease. Med Decis Making 1992; 7:351.
[32]: Goldstein MK, Clarke AE, Michelson D, et al. Developing and testing a multimedia presentation of a health-state description. Med Decis Making 1994; 14(4):336-344.
[33]: Nease, RF Jr. Risk attitudes in gambles involving length of life: aspirations, variations, and ruminations. Med Decis Making 1994; 14(2):201-203.
[34]: Jimison HB. Patient-specific interfaces to health and decision-making information. In: Health promotion and interactive technology: theoretical applications and future directions. Street R, Gold M, Manning T, eds. Mahwah, NJ: Lawrence Erlbaum Associates, 1997.
[35]: Mullen PD, Green LW, Persinger GS. Clinical trials of patient education for chronic conditions: a comparative meta-analysis of intervention types. Prev Med 1985; 14:753–81.
[36]: Strecher VJ, Kreuter M, Den Boer DJ, Kobrin S, Hospers HJ, Skinner CS. The effects of computer-tailored smoking cessation messages in family practice settings. J Fam Pract 1994; 39(3):262–70.
[37]: Skinner CS, Siegfried JC, Kegler MC, Strecher VJ. The potential of computers in patient education. Patient Education and Counseling 1993; 22(1):27–34.
[38]: Skinner CS, Strecher VJ, Hospers H. Physicians’ recommendations for mammography: do tailored messages make a difference? Am J Public Health 1994; 84(1):43–9.
[39]: Jimison HB, Fagan LM, Shachter RD, Shortliffe EH. Patient-specific explanation in models of chronic disease. Artif Intell Med 1992; 4(3):191–205.
[40]: Thompson SC, Pittes JS, Schwankovsky L. Preferences for involvement in medical decision-making: situational and demographic influences. Patient Education and Counseling 1993; 22:133–40.
[41]: Funnell MM, Donnelly MB, Anderson RM, et al. Perceived effectiveness, cost, and availability of patient education methods and materials. Diabetes Educator 1992; 18(2):139 –45.
[42]: Alterman AI, Baughman TG. Videotape versus computer interactive edu-cation in alcoholic and nonalcoholic controls. Alcoholism 1991; 15:39 –44.
[43]: Gillispie MA, Ellis LBM. Computer-based patient education revisited. J Med Syst 1993; 17:119 –25.
[44]: Consoli SM, Ben Said M, Jean J, et al. Benefits of a computer-assisted education program for hypertensive patients compared with standard education tools. Patient Education and Counseling 1995; 26:343–7.
[45]: Harris J. National assessment of consumer health information demand and delivery. Presented at Partnerships for Networked Health Information for the Public, Rancho Mirage, CA. May 14–16, 1995.
[46]: Eng TR, Maxfield A, Patrick K, Deering MJ, Ratzan S, Gustafson D. Access to health information and support: a public highway or a private road? JAMA 1998;280:1371–5.
[47]: Baker DW, Parker RM, Williams MV, et al. The health care experience of patients with low literacy. Arch Fam Med 1996; 5:329 –34.
[48]: Reid JC, Klachko DM, Kardash CAM, et al. Why people don’t learn from diabetes literature: Influence of text and reader characteristics. Patient Education and Counseling 1995; 25:31– 8.
[49]: Holt GA, Hallon JD, Hughes SE, et al. OTC lables: can consumers read and understand them? American Pharmacologist 1990; 30:51– 4.
[50]: Davis TC, Mayeaux EJ. Reading ability of parents compared with reading level of pediatric patient education materials. Pediatrics 1994; 93(3):460–8.
[51]: Morgan PP. Illiteracy can have major impact on patients’ understanding of health care information. Can Med Assoc J 1993; 148(7):1196–7.
[52]: Feldman SR, Quinlivan A. Illiteracy and the readability of patient education materials. A look at Health Watch. North Carolina Medical Journal 1994; 55(7):290 –2.
[53]: Jubelirer SJ, Linton JC. Reading versus comprehension: implications for patient education and consent in an outpatient oncology clinic. J Cancer Educ 1994; 9(1):26 –9.
[54]: Rossi PH, Freeman HE. Evaluation: a systematic approach. New York: SAGE Publications, 1993.
[55]: Sobel DS. Self-care in health: information to empower people. In: Levy AH, Williams B, eds. Proceedings of the American Association for Medical Systems and Informatics, Congress 87. San Francisco, CA, 1987:12-125.

Top of page

Comments: SciPICH@nhic.org Updated: 05/01/08