Introduction to Evaluation of Interactive Health Communication Applications
Citation: Introduction to Evaluation of Interactive Health Communication Applications. American Journal of Preventive Medicine January 1999; Vol.16 No.1:10-15
Medical Subject Headings (MeSH): health communication, computers, information technology, evaluation, computer communication networks.
Virtually all aspects of society have been altered in some way by advances in computer and communication technologies. In 1997, the information technology industry was the single largest industry in the United States in terms of sales and accounted for 33% of the growth in GDP in 1996.[1,2] An estimated 41.5 million U.S. adults were active users of the Internet in 1997, and more than 43% of Internet users have used it to research health information . At the same time that these new technologies have emerged, consumers seem to be demanding increasing access to a wide range of informationb, including health information, and social support as a vehicle for recovering from illness.
Consumer demand for health information and the availability of new media technologies have spurred substantial interest in interactive health communication (IHC), the interaction of an individual consumer, patient, caregiver, or professional--with or through an electronic device or communication technology to access or transmit health information or receive guidance and support on a health-related issue. Using this definition, IHC encompasses technology-mediated health communication and does not include direct communication such as face-to-face clinician-patient counseling. The panel chose the term IHC because it focuses on the content rather than on the technology that facilitates IHC. The panel uses the term IHC applications to refer to the operational software programs or modules that interface with the end user. This includes health information and support Web sites and clinical decision-support and risk assessment software (which may or may not be online), but does not include applications that focus exclusively on administrative, financial, or clinical data, such as electronic medical records, dedicated clinical telemedicine applications, or expert clinical decision-support systems for providers. Some of these latter applications, however, are integrated with health communication functions. The panel uses the term IHC technologies to refer to the hardware and infrastructure technologies that run or disseminate IHC applications, such as networks, computers, telecommunications equipment and the like.
IHC applications are increasingly accessible to the public through the Internetc and non-networked technologies, such as stand-alone computers and kiosks.[6,7] Their major functions are to: (1) relay information, (2) enable informed decision-making, (3) promote healthy behaviors, (4) promote peer information exchange and emotional support, (5) promote self-care, and (6) manage demand for health services.d
IHC has the potential to fundamentally change the way consumers and health professionals communicate and may enhance prevention efforts and clinical care. With access to IHC applications, consumers gain greater control of influences over their health, and health professionals may become more effective and efficient providers of care, health information, and support. Gains in community and individual health status and reduced health care costs may result as access to health information and support increases, and patients and others become more knowledgeable and empowered health care consumers. Sharing ideas and experiences with others through online health support groups may have health benefits (e.g., improved emotional well being) as well as non-health-related benefits (e.g., community building and advocacy and activism). Few other health-related interventions have the potential that IHC does to simultaneously improve health outcomes, decrease health care costs, and increase consumer satisfaction.
Whether the benefits of IHC applications are realized, however, rests largely on their quality and effectiveness. Cost will be a major factor in how widespread they are used. The rapid growth of IHC raises serious concerns about the accuracy, quality, and health impact of these programs.[5,8] The Science Panel on Interactive Communication and Healthe believes that promoting evaluation of these applications should be a central strategy for improving their quality and effectiveness. The panel proposes a level of evaluation that is sufficient to support the intended purposes of the application and the resources it consumes. The level of evidence of safety and effectiveness for a specific IHC application should increase as its potential risk for harm and/or requirements for investment of resources increase.
The panel has identified four stakeholder groups that must participate if meaningful evolution and quality improvement of IHC is to occur--consumers (including patients, families, and caregivers), health care professionals and purchasersf, IHC developers, and policy-makers. Consumers are the intended users of most IHC applications. Health care professionals often mediate the use of these applications with consumers, and are often involved in the development of IHC applications. Potential health care purchasers, including health plans and employers, determine whether IHC applications are implemented for their plan members or employees. Developers of IHC applications have ultimate control of quality assurance aspects of these interventions, and are clearly influenced by the needs of the purchasers. Policy-makers can influence the climate in which the other stakeholders make decisions about the development, use, or purchase of IHC applications. The four articles, which follow, focus on evaluation and quality improvement issues that are relevant for each stakeholder group. As background for these articles, we provide a brief overview of the benefits, types, and challenges of evaluation as it relates to IHC.
Inaccurate or inappropriate health information and poorly designed applications may result in harmful outcomes, such as receiving inappropriate treatment or delaying necessary health care-seeking behavior.[9-11] Most applications are being marketed without formal evaluation of effectiveness or health impact. As with other health care technologies, health care expenditures may rise and resources may be squandered if such technologies are ineffective or harmful.
The Science Panel on Interactive Communication and Health recommends an evidence-based approach to IHC development and use as a way of addressing these issues. From the panel's perspective, evaluation of IHC applications may:
High-quality evaluations, regardless of whether the evaluation results are positive or negative, are valuable in advancing the field of IHC. That is, negative results also promote development of effective products by reducing resources and time wasted on ineffective approaches.
There are many approaches to evaluation of health interventions like IHC applications. All approaches share one purposeto systematically obtain information that can be used to improve the design, implementation, adoption, use, redesign, and overall quality of an intervention or program. The design and implementation of an evaluation typically depends on the purpose of the evaluation, the stage that the intervention is in, and the type of decision the evaluation is intended to address. Formative evaluation may be used in the early stages of development to assess the nature of the problem and the needs of the target audience(s), with a focus on informing and improving program design and ensuring accuracy of content. During the developmental and implementation phases, process evaluation may be used to monitor the administrative, organizational, or other operational characteristics of the intervention or application. Outcome evaluation may be used to examine an intervention's ability to achieve its intended effect under ideal conditions (i.e., efficacy) or under real-world circumstances (i.e., effectiveness) and its ability to produce benefits in relation to costs (i.e., efficiency or cost-effectiveness). Active and flexible models of evaluation may be best for IHC applications because traditional evaluation models, while useful for more static and traditional interventions, will not adapt easily to the rapidly changing nature of IHC application design, implementation, and need for continuous quality improvement.
The Science Panel on Interactive Communication and Health has adapted a model commonly used to guide the design, implementation, and assessment of health communication programs to the evaluation of IHC applications (Table 1).  This approach may be helpful in understanding how evaluation activities relate to the development process from application conceptualization and design through implementation, assessment, and refinement. At each of these steps, critical information and data are required to inform key decisions that ultimately impact on application quality and effectiveness.
Table 1.Evaluation activities in the interactive health communication application development cycle [refer to PDF file, p13].
In considering the issues related to evaluating IHC applications, it may be helpful to examine the differences between the evaluation of IHC applications and other health interventions. The following example illustrates the need to develop evaluation models that are appropriate for IHC applications.
In the case of drugs and medical devices, the U.S. Food and Drug Administration (FDA) requires that all drugs and medical devices sold in the United States be shown to be "safe and effective" before approval. Drugs and devices that are deemed to have a potential to cause serious harm must typically undergo rigorous controlled clinical trials that often span several years. IHC applications are not physiologically harmful to the body in the same sense as drugs or devices because any negative consequences usually result from inappropriate health decisions of users rather than as a direct effect of the application. Pharmaceutical and medical device companies invest heavily in research and evaluation of potential products before committing to product development. The approved products may remain in use relatively unchanged for many years. In evaluating drugs and devices, the outcome of the intervention is often clear and measurable (e.g., improvement in a biological measure, such as reduction in blood pressure or absence of infection). In the case of IHC applications, the outcomes of interest may be less clear and measurable (e.g., improvement in knowledge, attitudes, practices, and well-being; improved ability to make appropriate clinical decisions).
A well-conducted evaluation of a health intervention or program is a task that requires careful planning and a systematic approach. The evaluation of IHC applications is particularly challenging based on several factors:
The panel supports the following principles regarding the evaluation of IHC applications:
As the subsequent four articles in this issue suggest, no single stakeholder will be able to substantially improve the quality of IHC applications by themselves. Additional development of approaches to IHC quality improvement that are appropriate and practical are neededg. Consumers will need to assess applications before and while using them and avoid using unevaluated ones, developers will need to implement evaluation methods throughout the development process, health professionals will need to become involved in development of quality applications and refer their patients to evaluated products, health care purchasers will need to demand evaluated products, and policymakers will need to implement policies supportive of these actions.
b. For example, less than one year after free Medline searches became available on the Web, the number of searches increased 10-fold, and 30% of users were members of the general public (Testimony of Dr. Donald A.B. Lindberg, Director, National Library of Medicine to the House Appropriations Sub-Committee on Labor, HHS and Education, March 18, 1998. Accessed on April 6, 1998. Available from: URL: http://www.nlm.nih.gov/pubs/staffpubs/od/budget99.html
c. For example, a search for the keyword "health" on the World Wide Web using common search engines yielded more than 16 thousand indexed Web sites (www.yahoo.com) and 20 million matching Web pages (altavista.digital.com) on October 28, 1998.
d. Other functions include professional education and skills building, which are beyond the scope of this article.
e. The Office of Disease Prevention and Health Promotion of the U.S. Department of Health and Human Services established the Science Panel on Interactive Communication and Health (SciPICH) to accelerate the appropriate development, adoption, and evaluation of IHC applications, and to develop a framework for the evaluation of these technologies. This consensus panel consists of 14 national multidisciplinary experts in the areas of medicine and public health, human-computer interaction, communication sciences, educational technology, health promotion, and consumer informatics.
f. For the purposes of these articles, issues for health care professionals and purchasers of health care are discussed together because both intermediaries to end-users of IHC applications, and, in some cases, such as in large medical groups, health professionals also may make purchasing decisions.
g. In addition to these and other journal articles, the panel has developed a Web site [http://www.health.gov/scipich] to serve as a resource for those interested in the evaluation of IHC applications.
a. Panel members and staff: Linda Adler, MPH, MA, National Member Technology Group, Kaiser Permanente, Oakland, CA; Farrokh Alemi, PhD, Cleveland State University, Cleveland, OH; David Ansley, Consumer Reports, Yonkers, NY; Patricia Flatley Brennan, RN, PhD, FAAN, School of Nursing and College of Engineering, University of Wisconsin-Madison, Madison, WI; Molly Joel Coye, MD, MPH, The Lewin Group, San Francisco, CA; Mary Jo Deering, PhD, Office of Disease Prevention and Health Promotion, U.S. Department of Health and Human Services, Washington, DC; Albert Mulley Jr, MD, MPP, Massachusetts General Hospital, Boston, MA; John Noell, PhD, Oregon Center for Applied Science, Inc. and Oregon Research Institute, Eugene, OR; Thomas C. Reeves, PhD, University of Georgia, Athens, GA; Thomas N. Robinson, MD, MPH, Stanford University School of Medicine, Palo Alto, CA; and Victor Strecher, PhD, MPH, University of Michigan Comprehensive Cancer Center, Ann Arbor, MI.
The authors are grateful to Paul Kim; Andy Maxfield, PhD; Anne Restino, MA; and John
Studach, MA, for their contributions to the panel's work, and to Linda Friedman for
assistance with copyediting. In addition, the authors thank the liaisons to the Science
Panel on Interactive Communication and Health, especially the following persons who
offered valuable suggestions for improving this manuscript: Loren Buhle, PhD; David
Cochran, MD; Connie Dresser, RDPH, LN; Alex Jadad, MD; Craig Locatis, PhD; Ed Madara;
Georgia Moore; Kent Murphy, MD; Scott Ratzan, MD, MPA, MA; Helga Rippen, MD, PhD; and
Address correspondence to: Thomas R. Eng, VMD, MPH, Office of Disease Prevention and Health Promotion, HHS, 200 Independence Avenue, SW, Room 738G, Washington, DC 20201.
Address reprint requests to: Mary Jo Deering, PhD, Office of Disease Prevention and Health Promotion, U.S. Department of Health and Human Services, 200 Independence Avenue, SW, Room 738G, Washington, DC 20201.
1. American Electronics Association. Cybernation. Accessed January 28, 1998. Available from: URL: http://sc-notes1.aeanet.org/www/ResearchStatistics.nsf/homepage/index.html
2. National Telecommunications and Information Administration. NTIA Information. Accessed February 21, 1998. Available from: URL: http://www.ntia.gov/nitahome/about97.htm
3. Cyber Dialogue, Inc. American Internet User Survey finds more than 41.5 million U.S. adults are actively using the Internet. Accessed March 5, 1998. Available from: URL: http://www.cyberdialogue.com/frame.html
4. FIND/SVP, Inc. The 1997 American Internet User Survey. Accessed February 18, 1998. Available from: URL: http://etrg.findsvp.com/inter-net/ findf.html
5. Robinson TN, Patrick K, Eng TR, Gustafson D, for the Science Panel on Interactive Communication and Health. An evidence-based approach to interactive health communication: A challenge to medicine in the information age. JAMA 1998;280:12649.
6. Wingerson L, Simon K, Northup L, Restino A, eds. Patient resources on the Internet. 1997 guide for health care professionals. New York: Faulkner & Gray, Inc., 1997.
7. Kieschnick T, Adler L, Jimison H. 1996 Health informatics directory. Baltimore, MD: Williams & Wilkins Press, 1996.
8. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet. Caveant lector et vieworlet the reader and buyer beware. JAMA 1997; 277:12445.
9. Weisbord SD, Soule JB, Kimmel PL. Poison on lineacute renal failure caused by oil of wormwood purchased through the Internet. N Engl J Med 1997;337:8257.
10. Food and Drug Administration. FDA warns consumers on dangerous products promoted on the Internet. FDA Talk Paper T97-26, June 17, 1997.
11. Keoun B. Cancer patients find quackery on the Web, J Natl Cancer Inst. 1996; 88: 1263-1265.
12 Rossi P.H. and Freeman H.E., Evaluation. A systematic approach, 5th ed, (Sage Publications, Newbury Park, CA, 1993)
13. National Cancer Institute. Making health communication programs work. Bethesda, MD: National Institutes of Health, US Department of Health and Human Services. NIH Publication no. 89-1493, April 1989.
Copyright © 1999 American Journal of Preventive Medicine
Comments: SciPICH@nhic.org Updated: 05/01/08