An Evidence-Based Approach

Introduction: Evaluation of IHC

Consumers & IHC Evaluation

Developers & IHC Evaluation

Policy Issues Relevant to IHC

Health Care Providers, Purchasers & IHC

SciPICH Final Report

Introduction to Evaluation of Interactive Health Communication Applications

Authors: Thomas R. Eng, David H. Gustafson, Joseph Henderson, Holly Jimison, and Kevin Patrick for the Science Panel on Interactive Communication and Health^a

Citation: Introduction to Evaluation of Interactive Health Communication Applications. American Journal of Preventive Medicine January 1999; Vol.16 No.1:10-15

Medical Subject Headings (MeSH): health communication, computers, information technology, evaluation, computer communication networks.

[PDF] [References]

Introduction

Virtually all aspects of society have been altered in some way by advances in computer and communication technologies. In 1997, the information technology industry was the single largest industry in the United States in terms of sales and accounted for 33% of the growth in GDP in 1996.[1,2] An estimated 41.5 million U.S. adults were active users of the Internet in 1997,[3] and more than 43% of Internet users have used it to research health information [4]. At the same time that these new technologies have emerged, consumers seem to be demanding increasing access to a wide range of information^b, including health information, and social support as a vehicle for recovering from illness.

Consumer demand for health information and the availability of new media technologies have spurred substantial interest in interactive health communication (IHC), the interaction of an individual consumer, patient, caregiver, or professional--with or through an electronic device or communication technology to access or transmit health information or receive guidance and support on a health-related issue.[5] Using this definition, IHC encompasses technology-mediated health communication and does not include direct communication such as face-to-face clinician-patient counseling. The panel chose the term IHC because it focuses on the content rather than on the technology that facilitates IHC. The panel uses the term IHC applications to refer to the operational software programs or modules that interface with the end user. This includes health information and support Web sites and clinical decision-support and risk assessment software (which may or may not be online), but does not include applications that focus exclusively on administrative, financial, or clinical data, such as electronic medical records, dedicated clinical telemedicine applications, or expert clinical decision-support systems for providers. Some of these latter applications, however, are integrated with health communication functions. The panel uses the term IHC technologies to refer to the hardware and infrastructure technologies that run or disseminate IHC applications, such as networks, computers, telecommunications equipment and the like.

IHC applications are increasingly accessible to the public through the Internet^c and non-networked technologies, such as stand-alone computers and kiosks.[6,7] Their major functions are to: (1) relay information, (2) enable informed decision-making, (3) promote healthy behaviors, (4) promote peer information exchange and emotional support, (5) promote self-care, and (6) manage demand for health services.[5]^d

IHC has the potential to fundamentally change the way consumers and health professionals communicate and may enhance prevention efforts and clinical care.[5] With access to IHC applications, consumers gain greater control of influences over their health, and health professionals may become more effective and efficient providers of care, health information, and support. Gains in community and individual health status and reduced health care costs may result as access to health information and support increases, and patients and others become more knowledgeable and empowered health care consumers. Sharing ideas and experiences with others through online health support groups may have health benefits (e.g., improved emotional well being) as well as non-health-related benefits (e.g., community building and advocacy and activism). Few other health-related interventions have the potential that IHC does to simultaneously improve health outcomes, decrease health care costs, and increase consumer satisfaction.

Whether the benefits of IHC applications are realized, however, rests largely on their quality and effectiveness. Cost will be a major factor in how widespread they are used. The rapid growth of IHC raises serious concerns about the accuracy, quality, and health impact of these programs.[5,8] The Science Panel on Interactive Communication and Health^e believes that promoting evaluation of these applications should be a central strategy for improving their quality and effectiveness. The panel proposes a level of evaluation that is sufficient to support the intended purposes of the application and the resources it consumes.[5] The level of evidence of safety and effectiveness for a specific IHC application should increase as its potential risk for harm and/or requirements for investment of resources increase.

The panel has identified four stakeholder groups that must participate if meaningful evolution and quality improvement of IHC is to occur--consumers (including patients, families, and caregivers), health care professionals and purchasers^f, IHC developers, and policy-makers. Consumers are the intended users of most IHC applications. Health care professionals often mediate the use of these applications with consumers, and are often involved in the development of IHC applications. Potential health care purchasers, including health plans and employers, determine whether IHC applications are implemented for their plan members or employees. Developers of IHC applications have ultimate control of quality assurance aspects of these interventions, and are clearly influenced by the needs of the purchasers. Policy-makers can influence the climate in which the other stakeholders make decisions about the development, use, or purchase of IHC applications. The four articles, which follow, focus on evaluation and quality improvement issues that are relevant for each stakeholder group. As background for these articles, we provide a brief overview of the benefits, types, and challenges of evaluation as it relates to IHC.

Benefits of Evaluation

Inaccurate or inappropriate health information and poorly designed applications may result in harmful outcomes, such as receiving inappropriate treatment or delaying necessary health care-seeking behavior.[9-11] Most applications are being marketed without formal evaluation of effectiveness or health impact. As with other health care technologies, health care expenditures may rise and resources may be squandered if such technologies are ineffective or harmful.

The Science Panel on Interactive Communication and Health recommends an evidence-based approach to IHC development and use as a way of addressing these issues.[5] From the panel's perspective, evaluation of IHC applications may:

• Improve quality, utility, and effectiveness. Evaluation allows for the identification of potential problems and provides valuable feedback for application development and quality improvement.
• Minimize the likelihood of harm. Evaluation of health impact may identify and reduce the use of IHC applications likely to have unexpected harmful effects.
• Promote innovation. Evaluation can encourage innovation in application design by identifying promising approaches for additional development and, if done correctly, reduce "time-to-market."
• Conserve resources. By informing purchasing and implementation decisions, evaluation can avert the investment of resources on ineffective applications.
• Encourage participation of stakeholders in the development and implementation process. Appropriate evaluation necessitates engaging end-users and others early in application development. This, in turn, can increase the probability of a favorable impact on health and quality outcomes.
• Promote confidence among end users. The results of evaluation can help consumers and other users of applications make informed choices about IHC applications. And,
• Promote a positive public image of the industry. Without an industry norm of product evaluation, potentially harmful products will be released. This could tarnish the perception of all companies and organizations involved in IHC application development.

High-quality evaluations, regardless of whether the evaluation results are positive or negative, are valuable in advancing the field of IHC. That is, negative results also promote development of effective products by reducing resources and time wasted on ineffective approaches.

Types of Evaluation

There are many approaches to evaluation of health interventions like IHC applications. All approaches share one purpose—to systematically obtain information that can be used to improve the design, implementation, adoption, use, redesign, and overall quality of an intervention or program. The design and implementation of an evaluation typically depends on the purpose of the evaluation, the stage that the intervention is in, and the type of decision the evaluation is intended to address.[12] Formative evaluation may be used in the early stages of development to assess the nature of the problem and the needs of the target audience(s), with a focus on informing and improving program design and ensuring accuracy of content. During the developmental and implementation phases, process evaluation may be used to monitor the administrative, organizational, or other operational characteristics of the intervention or application. Outcome evaluation may be used to examine an intervention's ability to achieve its intended effect under ideal conditions (i.e., efficacy) or under real-world circumstances (i.e., effectiveness) and its ability to produce benefits in relation to costs (i.e., efficiency or cost-effectiveness). Active and flexible models of evaluation may be best for IHC applications because traditional evaluation models, while useful for more static and traditional interventions, will not adapt easily to the rapidly changing nature of IHC application design, implementation, and need for continuous quality improvement.

The Science Panel on Interactive Communication and Health has adapted a model commonly used to guide the design, implementation, and assessment of health communication programs to the evaluation of IHC applications (Table 1). [13] This approach may be helpful in understanding how evaluation activities relate to the development process from application conceptualization and design through implementation, assessment, and refinement. At each of these steps, critical information and data are required to inform key decisions that ultimately impact on application quality and effectiveness.

Table 1. Evaluation activities in the interactive health communication application development cycle [refer to PDF file, p13].

In considering the issues related to evaluating IHC applications, it may be helpful to examine the differences between the evaluation of IHC applications and other health interventions. The following example illustrates the need to develop evaluation models that are appropriate for IHC applications.

In the case of drugs and medical devices, the U.S. Food and Drug Administration (FDA) requires that all drugs and medical devices sold in the United States be shown to be "safe and effective" before approval. Drugs and devices that are deemed to have a potential to cause serious harm must typically undergo rigorous controlled clinical trials that often span several years. IHC applications are not physiologically harmful to the body in the same sense as drugs or devices because any negative consequences usually result from inappropriate health decisions of users rather than as a direct effect of the application. Pharmaceutical and medical device companies invest heavily in research and evaluation of potential products before committing to product development. The approved products may remain in use relatively unchanged for many years. In evaluating drugs and devices, the outcome of the intervention is often clear and measurable (e.g., improvement in a biological measure, such as reduction in blood pressure or absence of infection). In the case of IHC applications, the outcomes of interest may be less clear and measurable (e.g., improvement in knowledge, attitudes, practices, and well-being; improved ability to make appropriate clinical decisions).

Challenges of Evaluating IHC Applications

A well-conducted evaluation of a health intervention or program is a task that requires careful planning and a systematic approach. The evaluation of IHC applications is particularly challenging based on several factors:

• The dynamic nature of IHC technologies and health information content. Changes and revisions to IHC applications are common because of changes and upgrades in information technology and advances in biomedical and public health research. This raises the need for methods of evaluating applications over time. Evaluation of IHC applications differs from evaluation of print materials such as books and journal articles because the content of print materials typically has been reviewed and "vetted" before and after publication. Therefore, there is often a trade-off between accuracy and currency of information in IHC applications in that, over time, new health information often becomes more refined and its relevancy better understood. In addition, the advent of "smart agents" that automatically update information within IHC applications is another challenge to evaluation.
• The wide spectrum of applications and vehicles for dissemination. The variety of methods for dissemination of IHC applications may influence program effectiveness and complicate assessments of utility. For example, a health risk appraisal program disseminated through an anonymous Web site may be more widely used and more accurate in assessing risk behaviors than the same application accessed through a health plan's computer network because of user confidentiality concerns.
• The complex nature of IHC technologies. Any or all components or attributes of an application may impact on its utility or effectiveness. In assessing applications, it may be difficult to delineate accurately the relative effects of program content, design, user interface, method of dissemination, and user-specific characteristics. Making this problem worse is the need to account for potential nonintervention-related factors, including the myriad of other media influences that may influence health outcomes.
• Lack of practical approaches and tools. Practical evaluation approaches and tools appropriate for various stakeholders are limited. An evidence-based approach should be used, but it must be sufficiently flexible for a heterogeneous field and recognize the constraints of developers and users.
• Perceptions that evaluation will constrain development, increase development costs, and be unimportant for marketing purposes. Concerns that the evaluation process may delay product release or increase development costs may be problematic because this is evolving to be a highly competitive field where time-to-market and development costs are important. Some developers believe that product marketing, rather than product evaluation results, is the key determinant of sales. In addition, some developers perceive that purchasers are unwilling to pay for the costs of product evaluation.

Key Principles for Evaluation of IHC

The panel supports the following principles regarding the evaluation of IHC applications:

• Evaluation should be practical. Evaluation methods should reflect real-world considerations. Persons with limited resources, experience, or training in evaluation methodologies should be able to participate fully in the evaluation process.
• Evaluation should be proactive. Evaluation should seek to prevent problems and help create an environment where it is in everyone's best interest to create high- quality products, rather than relying solely on after-market evaluations to weed out ineffective applications.
• Evaluation should have a clear purpose. Evaluators should have a clear vision of how their results will be used to improve the design, implementation, or use of the application, rather than conduct evaluations for the sake of evaluation.
• Evaluation should be a shared responsibility. Developers, providers, purchasers, consumers, and policy-makers should all share responsibility for evaluation. And,
• Evaluation should be ubiquitous in product development.Evaluation methods should be woven throughout the conceptualization, design, implementation, and dissemination phases of product development.

As the subsequent four articles in this issue suggest, no single stakeholder will be able to substantially improve the quality of IHC applications by themselves. Additional development of approaches to IHC quality improvement that are appropriate and practical are needed^g. Consumers will need to assess applications before and while using them and avoid using unevaluated ones, developers will need to implement evaluation methods throughout the development process, health professionals will need to become involved in development of quality applications and refer their patients to evaluated products, health care purchasers will need to demand evaluated products, and policymakers will need to implement policies supportive of these actions.

^b. For example, less than one year after free Medline searches became available on the Web, the number of searches increased 10-fold, and 30% of users were members of the general public (Testimony of Dr. Donald A.B. Lindberg, Director, National Library of Medicine to the House Appropriations Sub-Committee on Labor, HHS and Education, March 18, 1998. Accessed on April 6, 1998. Available from: URL: http://www.nlm.nih.gov/pubs/staffpubs/od/budget99.html

^c. For example, a search for the keyword "health" on the World Wide Web using common search engines yielded more than 16 thousand indexed Web sites (www.yahoo.com) and 20 million matching Web pages (altavista.digital.com) on October 28, 1998.

^d. Other functions include professional education and skills building, which are beyond the scope of this article.

^e. The Office of Disease Prevention and Health Promotion of the U.S. Department of Health and Human Services established the Science Panel on Interactive Communication and Health (SciPICH) to accelerate the appropriate development, adoption, and evaluation of IHC applications, and to develop a framework for the evaluation of these technologies. This consensus panel consists of 14 national multidisciplinary experts in the areas of medicine and public health, human-computer interaction, communication sciences, educational technology, health promotion, and consumer informatics.

^f. For the purposes of these articles, issues for health care professionals and purchasers of health care are discussed together because both intermediaries to end-users of IHC applications, and, in some cases, such as in large medical groups, health professionals also may make purchasing decisions.

^g. In addition to these and other journal articles, the panel has developed a Web site [http://www.health.gov/scipich] to serve as a resource for those interested in the evaluation of IHC applications.

Acknowledgements

^a. Panel members and staff: Linda Adler, MPH, MA, National Member Technology Group, Kaiser Permanente, Oakland, CA; Farrokh Alemi, PhD, Cleveland State University, Cleveland, OH; David Ansley, Consumer Reports, Yonkers, NY; Patricia Flatley Brennan, RN, PhD, FAAN, School of Nursing and College of Engineering, University of Wisconsin-Madison, Madison, WI; Molly Joel Coye, MD, MPH, The Lewin Group, San Francisco, CA; Mary Jo Deering, PhD, Office of Disease Prevention and Health Promotion, U.S. Department of Health and Human Services, Washington, DC; Albert Mulley Jr, MD, MPP, Massachusetts General Hospital, Boston, MA; John Noell, PhD, Oregon Center for Applied Science, Inc. and Oregon Research Institute, Eugene, OR; Thomas C. Reeves, PhD, University of Georgia, Athens, GA; Thomas N. Robinson, MD, MPH, Stanford University School of Medicine, Palo Alto, CA; and Victor Strecher, PhD, MPH, University of Michigan Comprehensive Cancer Center, Ann Arbor, MI.

The authors are grateful to Paul Kim; Andy Maxfield, PhD; Anne Restino, MA; and John Studach, MA, for their contributions to the panel's work, and to Linda Friedman for assistance with copyediting. In addition, the authors thank the liaisons to the Science Panel on Interactive Communication and Health, especially the following persons who offered valuable suggestions for improving this manuscript: Loren Buhle, PhD; David Cochran, MD; Connie Dresser, RDPH, LN; Alex Jadad, MD; Craig Locatis, PhD; Ed Madara; Georgia Moore; Kent Murphy, MD; Scott Ratzan, MD, MPA, MA; Helga Rippen, MD, PhD; and Christobel Selecky.

Address correspondence to: Thomas R. Eng, VMD, MPH, Office of Disease Prevention and Health Promotion, HHS, 200 Independence Avenue, SW, Room 738G, Washington, DC 20201.

Address reprint requests to: Mary Jo Deering, PhD, Office of Disease Prevention and Health Promotion, U.S. Department of Health and Human Services, 200 Independence Avenue, SW, Room 738G, Washington, DC 20201.

References

9. Weisbord SD, Soule JB, Kimmel PL. Poison on line—acute renal failure caused by oil of wormwood purchased through the Internet. N Engl J Med 1997;337:825–7.

10. Food and Drug Administration. FDA warns consumers on dangerous products promoted on the Internet. FDA Talk Paper T97-26, June 17, 1997.

11. Keoun B. Cancer patients find quackery on the Web, J Natl Cancer Inst. 1996; 88: 1263-1265.

12 Rossi P.H. and Freeman H.E., Evaluation. A systematic approach, 5th ed, (Sage Publications, Newbury Park, CA, 1993)

13. National Cancer Institute. Making health communication programs work. Bethesda, MD: National Institutes of Health, US Department of Health and Human Services. NIH Publication no. 89-1493, April 1989.

Top of page

Copyright © 1999 American Journal of Preventive Medicine
Published by Elsevier Science Inc.

Comments: SciPICH@nhic.org   Updated: 05/01/08