

An Evidence-Based
Approach
Introduction:
Evaluation of IHC
Consumers & IHC
Evaluation
Developers &
IHC Evaluation
Policy Issues Relevant
to IHC
Health Care
Providers, Purchasers & IHC
SciPICH Final
Report

|
|
Introduction to Evaluation of Interactive
Health Communication Applications
Authors: Thomas R. Eng, David H. Gustafson, Joseph Henderson, Holly Jimison, and Kevin Patrick for the Science
Panel on Interactive Communication and Healtha
Citation: Introduction to Evaluation of Interactive
Health Communication Applications. American Journal of Preventive Medicine January 1999;
Vol.16 No.1:10-15
Medical Subject Headings (MeSH): health
communication, computers, information technology, evaluation, computer communication
networks.
[PDF] [References]
Introduction
Virtually all aspects of society have been altered in some
way by advances in computer and communication technologies. In 1997, the information
technology industry was the single largest industry in the United States in terms of sales
and accounted for 33% of the growth in GDP in 1996.[1,2] An estimated 41.5 million
U.S. adults were active users of the Internet in 1997,[3] and more than
43% of Internet users have used it to research health information [4].
At the same time that these new technologies have emerged, consumers seem to be demanding
increasing access to a wide range of informationb,
including health information, and social support as a vehicle for recovering from illness.
Consumer demand for health information and the availability
of new media technologies have spurred substantial interest in interactive health
communication (IHC), the interaction of an individual consumer, patient, caregiver, or
professional--with or through an electronic device or communication technology to access
or transmit health information or receive guidance and support on a health-related issue.[5] Using this definition, IHC encompasses technology-mediated health
communication and does not include direct communication such as face-to-face
clinician-patient counseling. The panel chose the term IHC because it focuses on the
content rather than on the technology that facilitates IHC. The panel uses the term IHC
applications to refer to the operational software programs or modules that interface
with the end user. This includes health information and support Web sites and clinical
decision-support and risk assessment software (which may or may not be online), but does
not include applications that focus exclusively on administrative, financial, or clinical
data, such as electronic medical records, dedicated clinical telemedicine applications, or
expert clinical decision-support systems for providers. Some of these latter applications,
however, are integrated with health communication functions. The panel uses the term IHC
technologies to refer to the hardware and infrastructure technologies that run or
disseminate IHC applications, such as networks, computers, telecommunications equipment
and the like.
IHC applications are increasingly accessible to the public
through the Internetc and non-networked
technologies, such as stand-alone computers and kiosks.[6,7] Their major
functions are to: (1) relay information, (2) enable informed decision-making, (3) promote
healthy behaviors, (4) promote peer information exchange and emotional support, (5)
promote self-care, and (6) manage demand for health services.[5]d
IHC has the potential to fundamentally change the way
consumers and health professionals communicate and may enhance prevention efforts and
clinical care.[5] With access to IHC applications, consumers gain
greater control of influences over their health, and health professionals may become more
effective and efficient providers of care, health information, and support. Gains in
community and individual health status and reduced health care costs may result as access
to health information and support increases, and patients and others become more
knowledgeable and empowered health care consumers. Sharing ideas and experiences with
others through online health support groups may have health benefits (e.g., improved
emotional well being) as well as non-health-related benefits (e.g., community building and
advocacy and activism). Few other health-related interventions have the potential that IHC
does to simultaneously improve health outcomes, decrease health care costs, and increase
consumer satisfaction.
Whether the benefits of IHC applications are realized,
however, rests largely on their quality and effectiveness. Cost will be a major factor in
how widespread they are used. The rapid growth of IHC raises serious concerns about the
accuracy, quality, and health impact of these programs.[5,8]
The Science Panel on Interactive Communication and Healthe
believes that promoting evaluation of these applications should be a central strategy for
improving their quality and effectiveness. The panel proposes a level of evaluation that
is sufficient to support the intended purposes of the application and the resources it
consumes.[5] The level of evidence of safety and effectiveness for a
specific IHC application should increase as its potential risk for harm and/or
requirements for investment of resources increase.
The panel has identified four stakeholder groups that must
participate if meaningful evolution and quality improvement of IHC is to occur--consumers
(including patients, families, and caregivers), health care professionals and purchasersf, IHC developers, and policy-makers. Consumers are the
intended users of most IHC applications. Health care professionals often mediate the use
of these applications with consumers, and are often involved in the development of IHC
applications. Potential health care purchasers, including health plans and employers,
determine whether IHC applications are implemented for their plan members or employees.
Developers of IHC applications have ultimate control of quality assurance aspects of these
interventions, and are clearly influenced by the needs of the purchasers. Policy-makers
can influence the climate in which the other stakeholders make decisions about the
development, use, or purchase of IHC applications. The four articles, which follow, focus
on evaluation and quality improvement issues that are relevant for each stakeholder group.
As background for these articles, we provide a brief overview of the benefits, types, and
challenges of evaluation as it relates to IHC.
Benefits of Evaluation
Inaccurate or inappropriate health information and poorly
designed applications may result in harmful outcomes, such as receiving inappropriate
treatment or delaying necessary health care-seeking behavior.[9-11] Most
applications are being marketed without formal evaluation of effectiveness or health
impact. As with other health care technologies, health care expenditures may rise and
resources may be squandered if such technologies are ineffective or harmful.
The Science Panel on Interactive Communication and Health
recommends an evidence-based approach to IHC development and use as a way of addressing
these issues.[5] From the panel's perspective, evaluation of IHC
applications may:
- Improve quality, utility, and effectiveness.
Evaluation allows for the identification of potential problems and provides valuable
feedback for application development and quality improvement.
- Minimize the likelihood of harm. Evaluation of health
impact may identify and reduce the use of IHC applications likely to have unexpected
harmful effects.
- Promote innovation. Evaluation can encourage
innovation in application design by identifying promising approaches for additional
development and, if done correctly, reduce "time-to-market."
- Conserve resources. By informing purchasing and
implementation decisions, evaluation can avert the investment of resources on ineffective
applications.
- Encourage participation of stakeholders in the
development and implementation process. Appropriate evaluation necessitates engaging
end-users and others early in application development. This, in turn, can increase the
probability of a favorable impact on health and quality outcomes.
- Promote confidence among end users. The results of
evaluation can help consumers and other users of applications make informed choices about
IHC applications. And,
- Promote a positive public image of the industry.
Without an industry norm of product evaluation, potentially harmful products will be
released. This could tarnish the perception of all companies and organizations involved in
IHC application development.
High-quality evaluations, regardless of whether the
evaluation results are positive or negative, are valuable in advancing the field of IHC.
That is, negative results also promote development of effective products by reducing
resources and time wasted on ineffective approaches.
Types of Evaluation
There are many approaches to evaluation of health
interventions like IHC applications. All approaches share one purposeto
systematically obtain information that can be used to improve the design, implementation,
adoption, use, redesign, and overall quality of an intervention or program. The design and
implementation of an evaluation typically depends on the purpose of the evaluation, the
stage that the intervention is in, and the type of decision the evaluation is intended to
address.[12] Formative evaluation may be used in the early stages
of development to assess the nature of the problem and the needs of the target
audience(s), with a focus on informing and improving program design and ensuring accuracy
of content. During the developmental and implementation phases, process evaluation
may be used to monitor the administrative, organizational, or other operational
characteristics of the intervention or application. Outcome evaluation may be used
to examine an intervention's ability to achieve its intended effect under ideal conditions
(i.e., efficacy) or under real-world circumstances (i.e., effectiveness) and its ability
to produce benefits in relation to costs (i.e., efficiency or cost-effectiveness). Active
and flexible models of evaluation may be best for IHC applications because traditional
evaluation models, while useful for more static and traditional interventions, will not
adapt easily to the rapidly changing nature of IHC application design, implementation, and
need for continuous quality improvement.
The Science Panel on Interactive Communication and Health
has adapted a model commonly used to guide the design, implementation, and assessment of
health communication programs to the evaluation of IHC applications (Table
1). [13] This approach may be helpful in understanding how
evaluation activities relate to the development process from application conceptualization
and design through implementation, assessment, and refinement. At each of these steps,
critical information and data are required to inform key decisions that ultimately impact
on application quality and effectiveness.
Table 1.
Evaluation activities in the interactive health communication application development
cycle [refer to PDF file, p13].
In considering the issues related to evaluating IHC
applications, it may be helpful to examine the differences between the evaluation of IHC
applications and other health interventions. The following example illustrates the need to
develop evaluation models that are appropriate for IHC applications.
In the case of drugs and medical devices, the U.S. Food and
Drug Administration (FDA) requires that all drugs and medical devices sold in the United
States be shown to be "safe and effective" before approval. Drugs and devices
that are deemed to have a potential to cause serious harm must typically undergo rigorous
controlled clinical trials that often span several years. IHC applications are not
physiologically harmful to the body in the same sense as drugs or devices because any
negative consequences usually result from inappropriate health decisions of users rather
than as a direct effect of the application. Pharmaceutical and medical device companies
invest heavily in research and evaluation of potential products before committing to
product development. The approved products may remain in use relatively unchanged for many
years. In evaluating drugs and devices, the outcome of the intervention is often clear and
measurable (e.g., improvement in a biological measure, such as reduction in blood pressure
or absence of infection). In the case of IHC applications, the outcomes of interest may be
less clear and measurable (e.g., improvement in knowledge, attitudes, practices, and
well-being; improved ability to make appropriate clinical decisions).
Challenges of Evaluating IHC
Applications
A well-conducted evaluation of a health intervention or
program is a task that requires careful planning and a systematic approach. The evaluation
of IHC applications is particularly challenging based on several factors:
- The dynamic nature of IHC technologies and health
information content. Changes and revisions to IHC applications are common because of
changes and upgrades in information technology and advances in biomedical and public
health research. This raises the need for methods of evaluating applications over time.
Evaluation of IHC applications differs from evaluation of print materials such as books
and journal articles because the content of print materials typically has been reviewed
and "vetted" before and after publication. Therefore, there is often a trade-off
between accuracy and currency of information in IHC applications in that, over time, new
health information often becomes more refined and its relevancy better understood. In
addition, the advent of "smart agents" that automatically update information
within IHC applications is another challenge to evaluation.
- The wide spectrum of applications and vehicles for
dissemination. The variety of methods for dissemination of IHC applications may
influence program effectiveness and complicate assessments of utility. For example, a
health risk appraisal program disseminated through an anonymous Web site may be more
widely used and more accurate in assessing risk behaviors than the same application
accessed through a health plan's computer network because of user confidentiality
concerns.
- The complex nature of IHC technologies. Any or all
components or attributes of an application may impact on its utility or effectiveness. In
assessing applications, it may be difficult to delineate accurately the relative effects
of program content, design, user interface, method of dissemination, and user-specific
characteristics. Making this problem worse is the need to account for potential
nonintervention-related factors, including the myriad of other media influences that may
influence health outcomes.
- Lack of practical approaches and tools. Practical
evaluation approaches and tools appropriate for various stakeholders are limited. An
evidence-based approach should be used, but it must be sufficiently flexible for a
heterogeneous field and recognize the constraints of developers and users.
- Perceptions that evaluation will constrain development,
increase development costs, and be unimportant for marketing purposes. Concerns that
the evaluation process may delay product release or increase development costs may be
problematic because this is evolving to be a highly competitive field where time-to-market
and development costs are important. Some developers believe that product marketing,
rather than product evaluation results, is the key determinant of sales. In addition, some
developers perceive that purchasers are unwilling to pay for the costs of product
evaluation.
Key Principles for Evaluation of IHC
The panel supports the following principles regarding the
evaluation of IHC applications:
- Evaluation should be practical. Evaluation methods
should reflect real-world considerations. Persons with limited resources, experience, or
training in evaluation methodologies should be able to participate fully in the evaluation
process.
- Evaluation should be proactive. Evaluation should
seek to prevent problems and help create an environment where it is in everyone's best
interest to create high- quality products, rather than relying solely on after-market
evaluations to weed out ineffective applications.
- Evaluation should have a clear purpose. Evaluators
should have a clear vision of how their results will be used to improve the design,
implementation, or use of the application, rather than conduct evaluations for the sake of
evaluation.
- Evaluation should be a shared responsibility.
Developers, providers, purchasers, consumers, and policy-makers should all share
responsibility for evaluation. And,
- Evaluation should be ubiquitous in product development.Evaluation
methods should be woven throughout the conceptualization, design, implementation, and
dissemination phases of product development.
As the subsequent four articles in this issue suggest, no
single stakeholder will be able to substantially improve the quality of IHC applications
by themselves. Additional development of approaches to IHC quality improvement that are
appropriate and practical are neededg.
Consumers will need to assess applications before and while using them and avoid using
unevaluated ones, developers will need to implement evaluation methods throughout the
development process, health professionals will need to become involved in development of
quality applications and refer their patients to evaluated products, health care
purchasers will need to demand evaluated products, and policymakers will need to implement
policies supportive of these actions.
b. For example, less than one
year after free Medline searches became available on the Web, the number of searches
increased 10-fold, and 30% of users were members of the general public (Testimony of Dr.
Donald A.B. Lindberg, Director, National Library of Medicine to the House Appropriations
Sub-Committee on Labor, HHS and Education, March 18, 1998. Accessed on April 6, 1998.
Available from: URL: http://www.nlm.nih.gov/pubs/staffpubs/od/budget99.html
c. For example, a search
for the keyword "health" on the World Wide Web using common search engines
yielded more than 16 thousand indexed Web sites (www.yahoo.com) and 20 million matching
Web pages (altavista.digital.com) on October 28, 1998.
d. Other functions
include professional education and skills building, which are beyond the scope of this
article.
e. The Office of Disease
Prevention and Health Promotion of the U.S. Department of Health and Human Services
established the Science Panel on Interactive Communication and Health (SciPICH) to
accelerate the appropriate development, adoption, and evaluation of IHC applications, and
to develop a framework for the evaluation of these technologies. This consensus panel
consists of 14 national multidisciplinary experts in the areas of medicine and public
health, human-computer interaction, communication sciences, educational technology, health
promotion, and consumer informatics.
f. For the purposes of
these articles, issues for health care professionals and purchasers of health care are
discussed together because both intermediaries to end-users of IHC applications, and, in
some cases, such as in large medical groups, health professionals also may make purchasing
decisions.
g. In addition to these
and other journal articles, the panel has developed a Web site [http://www.health.gov/scipich] to serve as a resource for those
interested in the evaluation of IHC applications.
Acknowledgements
a. Panel members and staff:
Linda Adler, MPH, MA, National Member Technology Group, Kaiser Permanente, Oakland, CA;
Farrokh Alemi, PhD, Cleveland State University, Cleveland, OH; David Ansley, Consumer
Reports, Yonkers, NY; Patricia Flatley Brennan, RN, PhD, FAAN, School of Nursing and
College of Engineering, University of Wisconsin-Madison, Madison, WI; Molly Joel Coye, MD,
MPH, The Lewin Group, San Francisco, CA; Mary Jo Deering, PhD, Office of Disease
Prevention and Health Promotion, U.S. Department of Health and Human Services, Washington,
DC; Albert Mulley Jr, MD, MPP, Massachusetts General Hospital, Boston, MA; John Noell,
PhD, Oregon Center for Applied Science, Inc. and Oregon Research Institute, Eugene, OR;
Thomas C. Reeves, PhD, University of Georgia, Athens, GA; Thomas N. Robinson, MD, MPH,
Stanford University School of Medicine, Palo Alto, CA; and Victor Strecher, PhD, MPH,
University of Michigan Comprehensive Cancer Center, Ann Arbor, MI.
The authors are grateful to Paul Kim; Andy Maxfield, PhD; Anne Restino, MA; and John
Studach, MA, for their contributions to the panel's work, and to Linda Friedman for
assistance with copyediting. In addition, the authors thank the liaisons to the Science
Panel on Interactive Communication and Health, especially the following persons who
offered valuable suggestions for improving this manuscript: Loren Buhle, PhD; David
Cochran, MD; Connie Dresser, RDPH, LN; Alex Jadad, MD; Craig Locatis, PhD; Ed Madara;
Georgia Moore; Kent Murphy, MD; Scott Ratzan, MD, MPA, MA; Helga Rippen, MD, PhD; and
Christobel Selecky.
Address correspondence to: Thomas R. Eng, VMD, MPH,
Office of Disease Prevention and Health Promotion, HHS, 200 Independence Avenue, SW, Room
738G, Washington, DC 20201.
Address reprint requests to: Mary Jo Deering,
PhD, Office of Disease Prevention and Health Promotion, U.S. Department of Health and
Human Services, 200 Independence Avenue, SW, Room 738G, Washington, DC 20201.
References
1. American Electronics Association.
Cybernation. Accessed January 28, 1998. Available from: URL:
http://sc-notes1.aeanet.org/www/ResearchStatistics.nsf/homepage/index.html
2. National Telecommunications and Information Administration. NTIA
Information. Accessed February 21, 1998. Available from: URL:
http://www.ntia.gov/nitahome/about97.htm
3. Cyber Dialogue, Inc. American Internet User Survey finds more than
41.5 million U.S. adults are actively using the Internet. Accessed March 5, 1998.
Available from: URL: http://www.cyberdialogue.com/frame.html
4. FIND/SVP, Inc. The 1997 American Internet User Survey. Accessed
February 18, 1998. Available from: URL: http://etrg.findsvp.com/inter-net/ findf.html
5. Robinson TN, Patrick K, Eng TR, Gustafson D, for the Science Panel
on Interactive Communication and Health. An evidence-based approach to interactive health
communication: A challenge to medicine in the information age. JAMA 1998;280:12649.
6. Wingerson L, Simon K, Northup L, Restino A, eds. Patient resources
on the Internet. 1997 guide for health care professionals. New York: Faulkner & Gray,
Inc., 1997.
7. Kieschnick T, Adler L, Jimison H. 1996 Health informatics directory.
Baltimore, MD: Williams & Wilkins Press, 1996.
8. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and
assuring the quality of medical information on the Internet. Caveant lector et
vieworlet the reader and buyer beware. JAMA 1997; 277:12445.
9. Weisbord SD, Soule JB, Kimmel PL. Poison on lineacute renal
failure caused by oil of wormwood purchased through the Internet. N Engl J Med
1997;337:8257.
10. Food and Drug Administration. FDA
warns consumers on dangerous products promoted on the Internet. FDA Talk Paper T97-26,
June 17, 1997.
11. Keoun B. Cancer patients find quackery
on the Web, J Natl Cancer Inst. 1996; 88: 1263-1265.
12 Rossi P.H. and Freeman H.E., Evaluation.
A systematic approach, 5th ed, (Sage Publications, Newbury Park, CA, 1993)
13. National Cancer Institute. Making
health communication programs work. Bethesda, MD: National Institutes of Health, US
Department of Health and Human Services. NIH Publication no. 89-1493, April 1989.
Top of page
Copyright © 1999 American Journal of Preventive Medicine
Published by Elsevier Science Inc.

Comments: SciPICH@nhic.org
Updated: 05/01/08 |