Introduction: Evaluation of IHC
Health Care Providers, Purchasers & IHC
Health Care Providers and Purchasers and Evaluation of Interactive Health Communication Applications
Authors: Holly Jimison, Linda Adler, Molly Joel Coye, Albert Mulley, Jr., and Thomas R. Eng for the Science Panel on Interactive Communication and Health.
Citation: Health Care Providers and Purchasers and Evaluation of Interactive Health Communication Applications. American Journal of Preventive Medicine, January 1999; Vol.16 No.1:16-22
Medical Subject Headings (MeSH): health care providers, purchasers, computers, health communication, evaluation, computer communication networks.
In recent years there has been increasing interest and progress in the development, implementation, and research regarding telecommunication and computer applications designed for consumers to access information on a wide variety of health care topics. This technology, both hardware and software, is part of a growing trend toward self-advocacy, empowering consumers to take a more active role in their own health care and providing the necessary information to enhance their decision making. More than ever, consumers are using interactive health communication (IHC) applications to augment the information and support provided by health care professionals in the course of clinical encounters. Consumers may access databases of health information, educational tools, and communicate with health care professionals or other patients through e-mail or online support groups. A variety of IHC applications now exist,[1,2] with topics on health concerns such as self-care, informed consent, coping skills,[4,5] and treatment decision-making skills.[4,6]
Health care providers and purchasers have a vested interest in offering patients and consumers reliable and useful health information. Providing access to quality consumer health information has the potential to improve the quality of medical care and lower costs. Interactive health communication technologies are a promising venue for the effective dissemination of health information, social support, and skill-building techniques for changing health behaviors or addressing health problems. The challenge for health care providers and purchasers is to identify systems that are effective and warrant additional investment of resources, time, and an adaptation of current medical practice. Some IHC applications seem to show clear benefit for patient care. However, most remain untested, and there is a growing interest about whether they will provide benefit or result in potential harm. As potential purchasers and distributors of these systems, it is imperative that physician groups, medical centers, and health plans make wise and informed selections based on evidence. However, at this point in time there are few evaluations of commercially available IHC applications. Health care providers and purchasers need to promote and require meaningful evaluations of these systems.
From the provider and purchaser point of view, the importance of supporting the evaluation of IHC applications is based on quality-of-care and business criteria. Evaluation information can be used to identify more effective products, and the quality of health care will be improved if patients are using more effective systems for their health decision-making. Additionally, evaluation feedback on patients' utilization of health care services and satisfaction provides valuable information for purchasing decisions based on cost effectiveness and cost savings.
However, there are several barriers to acquiring comparable evaluation information on IHC applications. Developers are pressed to produce high-quality applications in an extremely competitive environment with limited resources. The emphasis is on getting products out the door as quickly as possible, often with fewer features than originally anticipated and minimal testing. To be competitive, developers cannot afford to take the time and resources to evaluate a system before bringing it to market. Without purchaser demand for evidence, there is no incentive for developers to evaluate their systems. System purchasers must create the demand and promote concern for the quality and cost of medical care to motivate the evaluation of IHC applications.
There are several practical issues that are important to health care providers and managed care organizations when considering the purchase or use of an IHC application. A developer who has integrated evaluation throughout the product development cycle will be able to provide clear answers to these concerns. Evaluation of a system is a process that should occur throughout the development cycle. As outlined in the introduction to this article, it includes a user needs assessment, a clear definition of the goals and desired outcomes prior to development, pretesting and quality control of content, iterative usability testing throughout development, the evaluation of system performance, accuracy, security, pilot testing on potential system users, and finally, outcome evaluation of actual system use, measuring outcomes of importance to users and purchasers of the system. Attention to the evaluation process throughout the product development cycle will prepare developers to respond to the type of questions a health care provider or organization will have when making a careful purchase.
The "Evaluation Reporting Template for Interactive Health Communication Applications" proposed by the Science Panel on Interactive Communication and Health, which appears in a recent article, attempts to facilitate the disclosure of information about the evaluation methods used by developers in a standard format. While the proposed template is not intended to function as a purchasing guide, per se, many of the questions that health plans, large employers, and other organizations should consider before buying an IHC application, as presented in Table 1 , are addressed by the reporting elements described in the template. Purchasers should keep in mind, however, that many individuals are typically interested in outcomes related to improvements in health, service, convenience, social support, and general ease of use, rather than cost-effectiveness and market share. Therefore, patient satisfaction, system use, health status outcomes, and health behaviors should also be important criteria for organizations purchasing these systems.
Table 1 [refer to PDF file, p.18]
From the health care provider or purchaser perspective, it is important that evaluations of IHC applications address outcomes related to quality-of-care and cost effectiveness. One outcome to consider may be the potential for market growth with the use of IHC applications. Use of such systems may be likely to engender client satisfaction and loyalty, as well as encourage new enrollments. Cost savings may be expected from systems that facilitate disease management, self-care, and self-triage. However, as with many medical interventions, it is important to measure whether the desired effect of a system is actually obtained in routine use. The measured outcomes should be relevant to a medical care organization's decision on whether or not to purchase and use such a system. Product evaluations of IHC applications must target the outcomes of interest to the potential purchasers of these systems and address these outcomes from a meaningful and coherent perspective.
Targeted outcomes related to cost and quality of care from the purchaser's perspective include: (1) costall expenditures potentially influenced by the use of the system; and (2) qualityincluding provider satisfaction, patient satisfaction, knowledge gains, health behaviors, health outcomes, access, process control, more appropriate utilization of health care services, and concordance between utilization and expressed preferences (decision quality).
In addition, potential purchasers must consider the strength of evidence when interpreting an evaluation of a system. The results should show internal validity (higher with randomized controlled trials, lower with observational studies), statistical significance (higher with more samples), and clinical or system significance (sufficiently large effect). Finally, providers and medical organizations need to consider how applicable the evidence from an evaluation is to their own organization. To determine whether findings are generalizable to a different organization and situation, one should consider the characteristics of the patient population (demographics) and the characteristics of providers (culture, incentives, willingness to change or adapt).
There are many ways that health care providers and purchasers, as potential purchasers of IHC applications, can promote and participate in evaluations that will produce relevant information for purchasing decisions. The most basic way is for the purchaser to demand that the systems under consideration be (or have been) evaluated. However, the purchaser must understand the added value of evaluation information and be willing to pay a higher price.
Similarly, for systems that have not yet been evaluated in their type of organization, a purchaser should consider funding in-house evaluations, on systems under consideration and those that may be currently in use. It may also be in the interest of providers and purchasers to collaborate with developers in evaluating new systems. This may be of particular value to an organization in unusual clinical settings. The benefits to the purchaser in collaborating with a developer on evaluation include:
The collaboration can be worthwhile for both parties because the risks and rewards of the innovation are shared.
Providers and purchasers should have some familiarity of the underlying scientific basis for the methods commonly used by IHC applications to effect behavior change or improve decision-making. Having this knowledge is especially important with regards to IHC applications because, in some cases, "showy" technology may distract and even obscure consideration of the content or methodologies used in application design. In this section, we briefly review the concepts of empowerment and self-efficacy, individual preferences, use of the computer as a health information medium, and influence of individual characteristics on usability of IHC applications.
Potential purchasers of these products need to consider whether accepted principles have been applied to the product in question. From previous research, we know that access to health information can enable patients to be more active participants in their care and lead to better medical outcomes.[9-12] Patients report that they want to be informed about their medical condition,[13,14] and the process of sharing information enhances the doctor-patient relationship.
Involvement in one's own medical care also involves the closely linked concepts of patient empowerment and self-efficacy. In general, empowerment can be thought of as the process that enables people to exert control over their lives and their destiny.[15,16] It is closely related to health outcomes in that powerlessness has been shown to be a broad-based risk factor for disease. Studies demonstrate that people who feel "in control" in a health situation have better outcomes than those who feel "powerless."[17,18]
Similarly, self-efficacy is a person's level of confidence that they can perform a specific task or health behavior in the future.[19-21] Clinical studies show that self-efficacy can be most predictive of improvements in patients' functional status.[22,23] Perceived self-efficacy was shown to play a significant role in smoking cessation relapse rate, pain management, control of eating and weight, success of recovery from myocardial infarction, and adherence to preventive health programs.[24-28]
An important measure of success for many IHC applications is how well they promote empowerment and self-efficacy for patients. Empowerment can be enhanced, for instance, by online support groups that allow patients to feel "connected" to others with a similar medical problem. This has been demonstrated in women with breast cancer and patients with AIDS.[4,5,29]
The concept of individual preferences is important for IHC applications that focus on health decision-making. While patients need information about the quality of life associated with the medical outcomes of possible decisions, reliable assessment of individual preferences and risk attitudes for clinical outcomes are probably the weakest links in clinical decision-making. Recent efforts to explore the use of computers in communication about health outcomes and in assessing patients' preferences for various health outcomes have started to address these issues.[31-33] Information on patient preferences is important for tailoring information to patients and for providing decision support. Tailored information has been found to be more effective in providing consumer information[35-38] and is preferred by patients. In addition to differences in preferences for health outcomes, patients differ in the degree to which they choose to be involved in decision-making. Research confirms that age (younger), gender (females more than males), and education level (better educated) are strong predictors of the desire to be involved in medical decisions. There is also a greater desire to be involved in medical decisions that appear to require less medical expertise, such as knee injury as opposed to medical decisions for cancer.
There has been an increase in research devoted to testing the effectiveness of various formats and types of media for conveying health information to consumers.[41-44] These studies generally show that video and slides are educationally more effective than books and audiotapes. Computer-based approaches offer interactivity, provide feedback in the learning process, and can tailor information to the individual, but more research is needed on the effectiveness of computer-based approaches. Developers will need to be sensitive to human-computer interface issues and implement specifications that meet the needs of a wide variety of users.
Factors that influence the health-information-seeking behavior of people include age, gender, disability, race and ethnicity, and socioeconomic status. Research indicates that these variables can predict differences in the amount and type of health-related information that individuals want. While some do not seek much information, others encounter serious barriers to the use of IHC applications.
A lack of reading ability is a functional barrier affecting use of these systems. Numerous studies on literacy and readability confirm the widespread problem of low literacy skills.[47-49] Approximately 1 out of 5 Americans is functionally illiterate, reading at or below the fifth-grade level. Studies show that only one half of people examined are able to comprehend written health education materials and that people's reading levels were well below what is required to understand standard health brochures.[47,50-52] In developing health information, one cannot assume that a person who has completed a certain grade level can read at that level. Health materials should be written at least three grade levels lower than the average educational level of the target population. Text characteristics and organization and clarity also play important roles in comprehension and retention of material. Multimedia techniques can be used to facilitate comprehension by conveying information through video, audio, and graphics, in lieu of text. Additionally, computer approaches that represent material in multiple languages allow efficient tailoring to the user's language of choice, thereby increasing comprehension. Developers should also consider cultural issues associated with health-information-seeking behavior and willingness to use health information technology when designing IHC applications.
As the demand for more health information and decision support grows, the need for wider availability of these systems becomes even more important. Today, these systems can be found in a variety of settings and forms. The most common locations to access these systems are physician waiting rooms, hospitals, health resource libraries, public libraries, worksites, schools, community centers, and of course on personal home computers. Different systems may require quite different physical locations. For instance, many people are uncomfortable exploring sensitive health information in a public space.
Finally, the question of who will pay for the access and use of technologies for consumer health information is still unresolved. Educational and socioeconomic factors still determine access to computers and information technologies. Younger, more affluent, and well-educated patients are more likely to have access to home computers, diagnostic software, and the Internet than disadvantaged populations. Special effort is required to ensure universal access to health information and support. Overcoming these barriers will go a long way toward ensuring access for the people who have the greatest need for these resources.
Ideally, evaluation should be designed at the conception of a system. Consumer needs and the desired effects of a system should be clearly specified prior to system implementation, so purchasers should ask to see this information. These desired effects should help define the outcomes of interest and evaluation design to carefully measure those outcomes. As presented elsewhere, initial stages of evaluation include specifying a problem or need of a particular target audience through needs assessment. The results of this analysis are used to define the specifications for a product to address those needs.
Evaluations during product development include iterative usability testing to ensure that the product meets the needs of potential users with regard to usability and the facilitation of workflow or tasks. Component testing ensures that all aspects of the system perform accurately and meet design specifications. The final stage of evaluation is to actually measure outcomes during system use. However, a preliminary first step usually involves a pilot evaluation to work out the implementation details of the evaluation and assessment tools. Quite often there are obvious misunderstandings of terms or unanticipated barriers that can be corrected before beginning the larger, more complete study. A more detailed description of evaluation methods and additional references can be found in Rossi and Freeman's book on evaluation.
Developers of commercial systems have pushed the field of interactive health communication forward with many innovative systems. However, to achieve significant improvements in quality-of-care and health outcomes, we need to acquire evidence on the effectiveness of these systems. Additionally, researchers and system developers must focus on integrating the knowledge gained from these evaluations into the design of new systems. This is a new and emerging field with significant innovations in the commercial sector. Research in several areas is needed to move the field forward in providing real benefits to individual's health outcomes and in showing the effectiveness of the systems to purchasers of health care. Careful needs assessment before system development, usability testing during development, controlled clinical trials, and studies of use and outcomes in natural settings are all critical to our understanding of how best to provide health information and decision assistance to patients.
Advances in communications and information technology will change the way in which medicine is practiced, and it will also change the way in which patients receive information and interact with the health care system. The future holds great promise for consumers becoming empowered and active participants in their medical care decisions. Because the vast majority of health-related decisions are made by the people outside of the formal medical setting, the input of health care providers and purchasers into the IHC application development and evaluation processes is critical.
Panel members and staff: Farrokh Alemi, PhD, Cleveland State University, Cleveland, OH; David Ansley, Consumer Reports, Yonkers, NY; Patricia Flatley Brennan, RN, PhD, FAAN, University of Wisconsin-Madison, Madison, WI; Mary Jo Deering, PhD, Office of Disease Prevention and Health Promotion, U.S. Department of Health and Human Services, Washington, DC; David Gustafson, University of Wisconsin, Madison, WI; Joseph Henderson, MD, Dartmouth Medical School, Dartmouth, NH; John Noell, PhD, Oregon Center for Applied Science and Oregon Research Institute, Eugene, OR; Kevin Patrick, MD, San Diego State University and University of California San Diego, San Diego, CA; Thomas Reeves, University of Georgia, Athens, GA; Thomas N. Robinson, MD, MPH, Stanford University School of Medicine, Palo Alto, CA; and Victor Strecher, PhD, MPH, University of Michigan Comprehensive Cancer Center, Ann Arbor, MI.
The authors are grateful to Paul Kim; Andy Maxfield, PhD;
Anne Restino, MA; and John Studach, MA, for their contributions to the panel's work, and
to Linda Friedman for assistance with copyediting. In addition, the authors thank the
liaisons to the Science Panel on Interactive Communication and Health, especially the
following persons who offered valuable suggestions for improving this manuscript: Loren
Buhle, PhD; David Cochran, MD; Connie Dresser, RDPH, LN; Alex Jadad, MD; Craig Locatis,
PhD; Ed Madara; Georgia Moore; Kent Murphy, MD; Scott Ratzan, MD, MPA, MA; Helga Rippen,
MD, PhD; and Christobel Selecky.
Address correspondence to: Thomas R. Eng, Office of Disease Prevention and Health Promotion, HHS, 200 Independence Avenue, SW, Room 738G, Washington, DC 20201.
Address reprint requests to: Mary Jo Deering, PhD, Office of Disease Prevention and Health Promotion, U.S. Department of Health and Human Services, 200 Independence Avenue, SW, Room 738G, Washington, DC 20201.
Copyright © 1999 American Journal of Preventive
Comments: SciPICH@nhic.org Updated: 05/01/08