SciPICH Logo

SciPICH Publications - button/link

An Evidence-Based Approach

Introduction: Evaluation of IHC

Consumers & IHC Evaluation

Developers & IHC Evaluation

Policy Issues Relevant to IHC

Health Care Providers, Purchasers & IHC

SciPICH Final Report

SciPICH home icon and link

SciPICH Publications IconWired for Health and Well-Being: The Emergence of Interactive Health Communication

Editors: Thomas R. Eng, David H. Gustafson

Suggested Citation: Science Panel on Interactive Communication and Health. Wired for Health and Well-Being: the Emergence of Interactive Health Communication.  Washington, DC: US Department of Health and Human Services, US Government Printing Office, April 1999.

Download in PDF format:  [Entire Document] [References]

Chapter V.

Major Issues for Key Stakeholders

The Panel has identified several stakeholder groups that need to participate in IHC application development, evaluation, and quality assurance activities if meaningful evolution and quality improvement of IHC is to occur. These include consumers (including patients, families, caregivers, and communities), IHC developers, purchasers and investors, health professionals, and policymakers, all of whom have related roles.7 Developers of IHC applications are a vital link between the consumer and various sources of information and support. They also have ultimate control of evaluation and quality assurance aspects of these interventions, and are clearly influenced by the needs of the purchasers and users. Potential health care purchasers, including health plans and employers, determine whether IHC applications are implemented for their plan members or employees and directly influence the financial success of commercial developers. Investors determine which commercial applications receive funding to reach the marketplace. Health care and public health professionals often mediate the use of these applications with patients, and are involved in the development of IHC applications. Policymakers influence the climate in which stakeholders make decisions about the development, use, or purchase of IHC applications.

Understanding the various concerns, motivations, and potentially different perspectives of all stakeholders may be valuable in establishing collaborative efforts and in arriving at a consensus on appropriate directions and policy for IHC. In this chapter, the Panel outlines some of the major IHC-related issues among key stakeholders.


As mentioned previously, many types of developers create applications. Although their background and available resources for product development and evaluation may vary widely, they face similar obstacles in producing effective and useful programs (Jimison et al., 1999; Henderson et al., 1999). Many of these barriers have their origins in the competitive marketplace. The rapidly shifting nature of technology and the marketplace for IHC makes economic success difficult for many developers.

Balancing Science and the Realities of the Marketplace

How do developers balance their responsibility to use scientific principles in product development with the realities of the marketplace? IHC developers—commercial or nonprofit, large or small—are typically under tremendous pressure to deliver their products on time and within budget. Developers are pressed to produce high-quality applications in an extremely competitive environment with limited resources. This is the norm for the software industry. The emphasis on releasing products to the market as quickly as possible often results in applications that have fewer features than originally envisioned or ones that have not been fully tested and evaluated. The competitiveness of the market, from some developers’ perspectives, precludes them from investing appropriate time and resources to fully evaluate a system before release.

Investing in Marketing Versus Evaluation

What is the appropriate balance between investing in marketing and evaluation activities? Some developers believe that aggressive marketing efforts, rather than positive evaluation results, ultimately drive sales and commercial success. In some larger companies and organizations, decisions about evaluation activities may be made by managers and executives, and developers may be frustrated by the need to convince these decisionmakers about the importance of evaluation.

Obtaining Funding for Development and Evaluation

How do developers get adequate resources for application development? Venture capitalists and large corporations may invest in commercial developers. Government or private foundation grants are often the primary sources for smaller developers. This is particularly true for developers who are part of nonprofit organizations. Competing for capital is of utmost concern for many developers. As a result, many are reluctant to collaborate or share their ideas with potential competitors. A list of potential resources for locating funding for IHC development is presented in Appendix C.

The scenario in Box V-1 illustrates some of the difficult issues that developers face when they try to balance time-to-market considerations with their responsibilities for quality assurance and evaluation.

Box V-1
A Developer’s Dilemma

The PDM company was formed by the Jones brothers, Paul (a programmer), Bob (a business school professor), and Greg (a physician), in response to the news that their father had been diagnosed with prostate cancer. As their father informed each of them about his condition, he also expressed his confusion and doubt about what to do. Each brother commiserated with him about the difficulty of working through the available options for prostate cancer treatment. One evening not long afterwards, the brothers were having dinner together and talking about their father. They agreed that men like him needed a program to help sort through their options in a way that would help them make the best possible decisions during a difficult time. Collectively, they felt that their unique assemblage of talents gave them a special understanding of what was needed and how to do it. As a result, they created the PDM Company (for prostate decisionmaking) with a goal of creating (and selling) an interactive application that would help men decide about treatment options–thus making some money while meeting a real need.

With expert medical guidance recruited by the physician brother and decisionmaking expertise from the business school brother, the programmer began creating their flagship program. Due to major disagreements among the medical experts about which treatment options actually were most desirable, the project schedule started slipping. Meanwhile, Bob was holding focus groups to determine what men thought would be most useful and to get their reactions to mock-ups of the program. Enthused by the reactions of the men he spoke with, Bob began encouraging Paul to expedite development of the program. Greg was very concerned about the potential to lead men into the wrong decisions. He was worried about both legal liability and the moral/ethical issues of encouraging specific decisions, even inadvertently. Bob, an expert in marketing, explained how easy it is to influence decisions while providing seemingly neutral information. Paul suggested an extensive program evaluation to make sure the program actually helped men make the right decisions. They all agreed this sounded like a good idea.

Paul was coordinating the production of multimedia assets (graphics, audio files, and video) as well as programming. The video subcontractor kept delaying production and Paul could not finish the programming without the assets. Then, a major new microcomputer operating system (OS) version was announced and Paul realized his program would need to be changed if it was to run under the new OS. Each delay meant more days and weeks of effort, rent, subcontracts running over budget, and other expenses. Money was becoming a real worry. Finally, Paul announced that a pre-beta version was available for testing. Bob tried to run it on his office computer but he did not have the right multimedia software installed. He gave the program to others but all of them ran into some type of bug or installation glitch. Paul tried to create new versions for each problem that was encountered. Bob began to worry about just what types of computers could run the program. Was the installed base a big enough market? Maybe they should simplify the program so that it would run on more computers or make it a Web-based application? Paul was beginning to feel stressed out. Greg kept asking about the big delay.

Not much later, Paul produced a "final candidate" program and they began pilot testing with five men recently diagnosed with prostate cancer. Despite some glitches, the program worked pretty well and the men generally liked it. They felt it definitely was useful and had helped them make good treatment decisions. Greg was called in and the three Jones brothers began discussing an evaluation that would include enough men to have a statistically meaningful sample and would use a randomized control trial design. Greg and Paul agreed that anything less would not really tell them whether the program worked well or lead to any bad decisions. It would take about six months, but it was all going to be worth the effort. The euphoria of being so close to completion lasted three days. Then, Bob was at a conference where he overheard two men talking about the prostate decisionmaking program they had recently seen demonstrated. "Very slick," said one man. "Yes, they’ve got a winner I’d guess," added the other. "Do you really think they can ship it within the next three months like they claimed?" asked the first. "I think so," was the reply. Bob ran to a phone and called Paul. To his surprise, Greg answered and immediately interrupted his brother to announce that Paul had completed the "final" program. "Great news!" he added. "We’re ready to begin the randomized clinical trial. My friends, the ones with that huge urology practice I told you about, have agreed to start referring patients Monday." Bob brusquely replied, "Forget the trial, we’ve got to ship right away or we’ll get scooped! I just heard about another program coming out." What ensued was a bitter argument about whether or not they had sufficient data to begin shipping the program.

"We can get data from actual users," said Bob. "Without random assignment, we cannot be sure what that kind of data would mean," said Greg. Paul added, "I don’t want to do any harm but I don’t really see how we could." Greg explained how leading a man into the wrong decision could mean an unnecessary end to his sex life, on top of urinary incontinence. "Think about no more intercourse and being incontinent the rest of your life," Greg said. "But we probably have been more careful about creating a balanced program than our competitors. If we don’t go to market now we may not make anything out of this!" Bob said with emotion. "I don’t think we should market it unless we have data to back up its efficacy and safety," said Greg. "But that’s what the pilot data says, isn’t it?" asked Paul. "It’s not conclusive, there were only five men," said Greg. "Yes, but it’s the best we can do for now," added Bob. "We can run the trial while we start marketing, can’t we?" said Paul. Silence followed as each brother thought about the choices and implications. Paul spoke, "Well, what about it? We could start the trial as planned and Bob can start the marketing plan. By the time we ship a significant number, we should have preliminary data from enough patients to see if there are any major problems. We will keep analyzing the trial data as they come in and if we see any problems we yank the program and fix whatever has come up." Greg shook his head, "If we cause a problem I’d have a hard time living with it. First we test, then we ship." Bob looked glum. He said, "You do understand we could lose everything we’ve put into this?" Greg sighed and said softly, "Yes, I do."

The next three months were tense ones for PDM. After enrolling the first 20 men, they discovered that about half of the users were missing a critical section of the program due to a confusing interface element. The experienced computer users understood the interface metaphor Paul had programmed, but the others missed it. It was a serious problem, but easily fixed. Moreover, their competitor did release a comparable program. They heard about it while reading an industry newsletter. There was a review of their competitor’s program by a physician. Their dismay turned to glee as they read the review. The gist of the review was that the program had offered terrible advice, "evidently based on faulty algorithms," concluded the reviewer. The reviewer strongly suggested that the program be pulled off the market before the developers (and any clinics making it available to patients) faced lawsuits. The brothers carefully reviewed their own program to make sure it did not have similar problems. Once the randomized clinical trial had produced enough data, Greg gave a peer-refereed presentation of the scientific data at a national meeting. A medical director for a very large HMO approached Greg after the talk and asked if he was prepared to negotiate for "the purchase of a rather large number of copies to be used across all of their sites."

Key Considerations

  • The first guiding principle must be: "Do no harm."
  • Two key ingredients are content expertise and accurate knowledge of the target population (i.e., their desires, capabilities, and needs). Both are necessary for optimal program development.
  • Key design decisions (e.g., specific behavioral goals, target platform, structure) must be made carefully—and early in the development process—to avoid wasted efforts and costly delays.
  • Adequate testing of the programming and the content are critically important—both to ensure efficacy and to avoid liability.
  • Premature marketing can result in problems worse than reduced market share.
  • Good efficacy data can be an important sales factor.

Health Professionals

Health care and public health professionals have a vested interest in offering patients and consumers reliable and useful health information because providing access to quality health information has the potential to both improve the quality of health care and lower costs (Jimison et al., 1999). The challenge for health professionals and purchasers is to identify systems that are effective and warrant further investment of resources, time, and integration into clinical practice and preventive programs. In a shifting paradigm where patients may have just as much or more access to the latest health information, clinicians and other health and safety professionals will be challenged to find ways to become involved in the health information-seeking process and be able to add value to this process.

Employee health and safety directors, occupational health nurses, industrial hygienists, and others in workplace settings, are examples of health and safety professionals who could benefit from IHC applications. The recommendations for reducing the risks associated with workplace hazards often require the development and implementation of complex engineering and administrative controls—measures which require expertise and often need to be tailored to individual work settings. As a result, controlling occupational health and safety hazards can be costly and time-consuming for employers and workplace health and safety professionals. IHC has the potential to reduce costs and occupational risks by alerting health and safety professionals about newly identified hazards, and by providing accessible, comprehensive, and tailored information for developing and implementing controls.

Impact on Role of the Clinician

How will IHC impact on the traditional role of clinicians? How should IHC applications be integrated into clinical practice? It is unclear exactly how IHC will change the role of clinicians but it is likely to be a gradual rather than an abrupt process. As discussed previously, some health care professionals view IHC as a threat to their traditional relationship with patients. Others become discouraged at the thought of managing patients who are avid health information seekers given the confines of a 15-minute appointment. These concerns are somewhat magnified for some clinicians because they are occurring in the context of the growth of managed care. Regardless of whether their practice setting supports use of IHC, clinicians can make use of applications to improve patient care. For example, referring patients to quality Web sites for additional information or to well-regarded online support groups for peer support may help supplement the information and support provided during an office visit.

A related concern for clinicians is whether they will be reimbursed for their activities as information intermediaries as described in the next chapter. In many cases, administrators rather than individual clinicians ultimately make decisions about integrating IHC into clinical practice, particularly if they involve the purchase of hardware or software. Administrators, therefore, have a key role in successful implementation efforts.

Development of New Skills and Training

A 1997 survey among a nationally representative sample of 1,905 physicians showed that 42 percent use a computer for personal or professional practice purposes (Trotto, 1998). It is clear that as the roles of health professionals in the context of IHC evolve, additional training in new areas of responsibility will be required. These areas may include both technology-related skills (e.g., computer use, information searching and information evaluation skills) and communication-related skills (e.g., facilitating shared decisionmaking, interfacing with proactive information-savvy patients). Such training will need to be available during both health professional student and post-graduate training. In addition, IHC-related technology is rapidly becoming an essential tool for educating health professionals.

The scenario in Box V-2 illustrates several issues that may challenge clinicians in integrating IHC applications into their practice.

Box V-2
The Proactive Health Care Professional

Dr. Emily Chen is a senior physician with more than 20 years of experience in a large oncology group practice. During the last year or so, the clinicians in the practice have noticed a dramatic increase in the number of people who have been coming in with realms of printouts from Medline searches and other online resources. The clinical staff have been overwhelmed by these patients because much clinician time is required to address all their questions and concerns. Some of the clinicians also believe that most of the time spent with these patients is spent correcting inaccurate information or explaining why the information is not relevant to the patient’s situation. As a result, these clinicians have been discouraging their patients from using online health resources. In addition, because the practice receives a fixed pre-negotiated reimbursement rate from the insurance company for most services regardless of the actual time involved, the practice administrator has strongly encouraged time limitations on many types of visits.

Dr. Chen has been reluctant to accommodate the more information-savvy patients in the practice and will sometimes only brusquely answer these patients’ questions. Then, one day, a long-time patient of Dr. Chen’s abruptly announces that, while he respects the quality of clinical care at the practice, his information and support needs are not being met. He is switching to a new practice across town because they not only answer all his questions, but they actually encourage him to conduct online research about his illness. In fact, this practice encourages him to e-mail the clinic staff with any questions he might have at any time. Asking around, Dr. Chen finds out that several other clinicians in the practice have had similar experiences recently. Concerned, Dr. Chen asks the practice’s accountant to review the clinic logs. She finds out that there has been a significant drop in the number of new patients coming into the practice during the last year, and an alarming number of long-time patients have elected to leave the practice.

At their next meeting, the practice management team decides to conduct a survey to assess the information and support needs of patients. The team learns that an overwhelming number of patients are dissatisfied with the information and support they receive from their clinician. It becomes clear to the team that the policy of not fully supporting patients who use the Internet to access health information and support is shortsighted and potentially ill-fated. Using the survey results, the team sets out to develop a comprehensive "patient information support" policy with input from an IT consulting company.

The team decides that the new policy would revolve around a proactive and comprehensive approach to managing patient information and support needs. Their goal is that all patients will be fully supported in seeking information and support, and fully satisfied with their clinical encounters.

The management team quickly realizes that, although most of the office staff have used personal computers, only a few of them have experience in searching for health information on the Internet. They hire a local medical librarian to hold a series of in-service training sessions on locating and evaluating health information on the Internet. The librarian also identifies several well-respected online cancer support groups that the clinicians start to use to get a sense of what questions and concerns are circulating among cancer patients and caregivers. The nurses, technicians, and other office staff are included in these training sessions because they will assume much of the responsibility for supporting patient information and support needs under the new policy.

A subgroup of the team is tasked with building the Web site for the practice. They start by locating and conducting reviews of relevant, health-information Web sites. This group carefully evaluates these Web sites using standardized evaluation criteria and creates links to sites that seem to be of highest quality and relevance to the practice’s patient population. Sites that inform and guide cancer patients to appropriate clinical trials of new therapies are highlighted. They also contract with a regional cancer center to use some of their online content and adapt it to their patient population. To complement the health-information resources on their site, they purchase licenses for several risk-assessment and clinical decision-support tools from a university group after examining several peer-reviewed research articles documenting the effectiveness of these tools. In addition, during the course of the next few months, each clinician submits several "frequently asked questions" from patients and the answers are posted on the practice’s Web site. All of the clinicians are now available by e-mail to handle nonemergency questions. Patients really like this because they no longer feel that they have to think of all their questions during their visits, and many have been reluctant to call their clinician for nonurgent questions.

The team also wants to make sure that all patients have some access to their Web site. On a trial basis, several interactive kiosks are installed in the practice waiting room. An "information assistant" is available to show patients how to use the system. New patients are instructed to use the kiosks to provide a detailed medical history so that the clinicians can focus on their responses rather than administering the history during their visit. Because the kiosks are linked to the Internet, patients also can access the online resources on the practice Web site while they wait. The office finds that the kiosks are well received, and that they actually free up time that the clinicians would have otherwise spent interviewing the patients and explaining basic concepts about cancer.

Several months after implementing their new "patient information support" policy, the team implements structured and open-ended surveys of patients to get their feedback on the new approach. They find that most patients are satisfied with the information and support they receive through the practice. They use the survey data to further refine their quality improvement initiatives. Dr. Chen and the other clinicians also have the following observations: 1) most patients seem to be well informed about their illnesses and their questions during visits are better focused; 2) many patients have become active participants in their care and seem to be more satisfied with their care; and 3) the number of patients in the practice has increased.

Key Considerations

  • Clinicians have important roles in helping patients locate, assess, and interpret health information.
  • Many health professionals will need training and skills-building in health information technology to adequately address the information and support needs of their patients.
  • A proactive and comprehensive approach to addressing patient information and supporting needs can help improve patient satisfaction.
  • Encouraging patients to use online resources does not necessarily increase the amount of time that clinicians have to spend with patients during visits.


Currently, the major purchasers of IHC applications are health plans, other health care organizations, and large employers. Emerging purchasers include government agencies, schools, and voluntary health organizations. Direct-to-consumer applications are becoming increasingly common. Health plans may be primarily interested in the potential of IHC to promote healthy behaviors and manage demand for services. Many employers consider purchasing an IHC application to help decrease health care insurance premiums and to reduce losses in worker productivity due to illness. Ultimately, both types of purchasers want to improve satisfaction among members or employees and lower health care costs. After purchasers make a purchasing decision, they then have responsibility for implementation and evaluation of the selected product. A checklist of evaluation-related questions for potential purchasers is presented in Appendix D.

Selecting and Evaluating Applications

How should purchasers compare competing products and what outcomes should they consider besides price? Major outcomes of interest to most purchasers include cost (i.e., all expenditures potentially influenced by the use of the system) and quality, including clinician and patient satisfaction, increased knowledge and healthy behaviors, improved health status outcomes, process control, improved access to and more appropriate utilization of health care services, and concordance between utilization and expressed preferences (Jimison et al., 1999). One outcome of importance to health plans is improved market growth with the use of IHC applications that may increase client satisfaction, retention, and new enrollments. Cost-savings may result from applications that facilitate disease management, self-care, and self-triage. Purchasers should keep in mind, however, that many individuals are much more interested in outcomes that reflect improvements in health, service, convenience, social support, and general ease of use, rather than cost-effectiveness and improved market share. Therefore, the former types of outcomes should also be important criteria for purchasers.

Strength of evidence is also important to consider when examining evaluation results (Robinson et al., 1998). The results should show internal validity (this is higher with randomized controlled trials, lower with observational studies), statistical significance (higher with larger samples and/or larger effect size), and clinical or system significance (sufficiently large effect). In addition, purchasers may wish to consider how applicable evaluation results are to their own circumstances and population. For example, in the case of a health plan, comparison of its patient demographics and organizational characteristics in terms of culture, incentives, and willingness to change or adapt, may shed light on whether evaluation findings are "generalizable" to their circumstances. As with all interventions, it is important to document whether the reported effect of a system is actually obtained in routine use among the purchaser’s population.

Supporting Evaluation of IHC

From the purchaser perspective, the importance of supporting the evaluation of IHC applications is based on both quality-of-care and business criteria. Evaluation information can be used to identify more effective products, and the quality of health care will be improved if patients are using more effective systems for their health care decisionmaking. Without purchaser demand for evidence, there is little incentive for developers to evaluate their systems and evaluation information on competing IHC applications would be limited. Some purchasers, however, may not be concerned about demanding health status outcome evaluations because they are more interested in providing IHC applications as a marketing mechanism or as a means for improving member or employee satisfaction rather than in the actual impact of the product on health status. Appropriate evaluation results, however, would provide information about user satisfaction rates that could predict satisfaction in the purchaser’s population.

Developing and Implementing IHC

What types of applications are most appropriate for members and employees? In considering whether to implement IHC in a health care organization, administrators will be concerned about several major issues. First and foremost may be whether IHC can address the needs of their members or employees. Addressing this question requires information about the population’s health information and support needs, and an understanding of realistic outcomes of the application. Another major consideration is whether implementing a technology-mediated application adds real value compared with a more traditional method of providing information and support. Administrators also need to determine how to relate the IHC application to clinical information systems and to care delivery. Another important issue they face is whether to purchase external systems or develop applications in-house.

The scenario in Box V-3 describes several major issues related to how health care organizations may approach the implementation of IHC applications into their organization.

Box V-3
Implementing IHC in a Health Plan

Janice Rodriguez, an accomplished project manager, has just been recruited from an interactive health software company to be the regional chief information officer for a large progressive health plan with several facilities in the area. The CEO of the health plan wants the organization to be on the "cutting-edge" of using IHC applications to improve member health and to reduce costs of delivering care.

Eager to please, Janice embarks on an ambitious plan to eventually implement a comprehensive suite of Web-based applications to provide seamless health information and support for members, allow electronic member transactions, provide clinical decision support for members and clinicians, and facilitate electronic communication between clinicians and members. For the first phase of her plan, she focuses on selecting and implementing a shared decision-support application for members and clinicians.

Janice ponders whether to outsource the development of the application or to do it in-house. After an assessment of her department’s resources and an environmental scan of available applications, she decides on the former option. She surmises that, with a pilot project, it is better to minimize uncertainty. The health plan’s IT department has some expertise in general Web site development, but they are unfamiliar with decision-support applications. If she outsources to a company with an established product, she is assured of prompt delivery and can more easily predict costs. In addition, she is worried about potential liability issues. Relying on an established and rigorously evaluated application, she concludes, may help in that regard.

She sends out notice of what she is looking for to her former company and a few other contacts. Within a week, Janice is deluged with calls from more than a dozen companies that want to sell their product. Each of them claims that their product is of high quality and very effective.

Janice understands that she needs to base her decision on more than surface appeal or which salesperson can do a better presentation, because her boss is looking for bottom-line benefits to the health plan, such as increased member satisfaction and retention. She promptly assembles a team of people from the IT and clinical research departments with experience in evaluation. They review the literature on evaluation of IHC, talk to colleagues in other plans, and brainstorm, coming up with a list of questions for themselves and prospective vendors. The questions include: What are reasonable objectives for such an application within the context of the organization? Was the application evaluated with regards to these objectives, and what were the results? Has it been shown to be cost-effective? Is the product flexible and adaptable to the plans changing needs?

After reviewing additional information from the vendors, the team decides to purchase a hundred licenses of the application from a company that presented very convincing data on effectiveness which was published in a respected scientific journal. Janice and the team make plans to deploy the application in the primary care department in one of the plan’s facilities because their medical director happens to be a "techie" and is very enthusiastic about the program. Things look positive; the application runs well with their existing clinical information system, the training session is well attended, and several of the primary care staff say that they are impressed with the technology.

Janice and her team check in weekly to see how things are going. After a month, only a small proportion of the clinicians has used the application with their patients. Janice interviews the staff. "I don’t want to support using something that may eventually replace me," says one physician. "I like the program but I don’t have time to use it during my appointments…I am also afraid that it will actually increase the amount of time I have to spend with some patients," says another. Yet a third comments, "It’s too much of a hassle for me to use it because it crashes my personal scheduler program when I do. When I ask our IT administrator to help, he says he will not support it because he is too busy troubleshooting the clinical information system that the plan just purchased last year."

Janice’s team realizes that they have formidable implementation problems on their hands, so they start again from square one, and recruit more troops. They fan out, interview additional stakeholders, hold extensive "town meetings," and form a plan-wide implementation team.

The team is divided into a number of smaller groups that are dedicated to a specific aspect of the implementation process. The development team takes responsibility for assessing the value of available applications, rating them according to compatibility with the health plan, and modifying and designing interfaces to reflect the organizational culture. The implementation working group will oversee the process of defining constituencies and determining how best to represent their needs in the dissemination process. The operations working group considers how the application will be merged with existing clinical care, how quality will be assessed, and how the application will be maintained. The evaluation group defines how the impact of the program will be determined, designs usability testing protocols, analyzes potential return on the investment, and determines how study results will be disseminated. The technology working group hammers out the "nuts and bolts" of the installation and designs the architecture to support the smooth operation of the application. Finally, a planning working group is formed to examine future opportunities and directions for new applications and development.

Meanwhile, a communication plan is set into action so that support for the program can be gathered from the stakeholders and opinion leaders throughout the organization. Key decisionmakers are identified to ensure continued financial support; they are encouraged to join the working groups and inject their concerns into the development process. Clinical staff members from all the facilities are asked to visit and report on changes in medical care and health plan policies. In addition, organization leaders continually address the working groups to inform them of organizational changes so that the applications are consistent with organizational goals.

Finally, the working groups meet every few months with their customers, including members of the health plan, clinical staff, and plan administrators. They define and refine the dissemination process, provide feedback on the success of the program, and brainstorm on future options for added technology.

A year later, a survey shows that 70 percent of members who have used the decision-support application are highly satisfied with their experience and believe that it not only helped them make a better decision, but also improved their relationship with their health care provider. A similar survey among providers shows that more than half have already adapted the application into their routine practice and believe that it improves the quality-of-care they deliver.

Key Considerations

  • Health care organizations and other potential purchasers should carefully assess their internal capabilities for application development and maintenance before attempting to develop their own applications.
  • Purchasing decisions require careful consideration of the objectives for the application and evidence of application effectiveness.
  • Substantial attention to, and planning of, the implementation process are critical. The process should be consistent with the organization’s structure and culture, and address institutional barriers.
  • Continuous evaluation and quality improvement are essential.
  • Successful implementation is a team process; involvement of, and communication with, all stakeholders are necessary.


Consumers, perhaps more than any other stakeholder group, vary in terms of their ability and experience in evaluating applications, and, thus, may be the most "vulnerable" stakeholder group. This is because a "consumer" may be a scientist or health professional by training, someone trained in a different field, or someone who has no formal education. Therefore, of all the stakeholders, consumers are probably at greatest risk of potential harm and need to be cautious because of the general lack of disclosure of information about developers and sponsors of IHC applications. Many applications are currently being used with no or limited guidance from a health professional. Most consumers are concerned about being able to select and use the best applications for their needs and require guidance and tools for doing so (Gustafson, Robinson et al., 1999). A "consumer’s guide" for interpreting evaluation results reported by developers is presented in Appendix E.

Selecting Applications for Use

How do users select the most appropriate applications? Even if consumers recognize the potential for harm and wish to evaluate applications before use, will they be able to? The growing array of applications available makes the selection process extremely daunting. If independent evaluations of products are available, and they are presented in consumer-friendly terms, consumers would be better able to make informed selections of applications for their use. In many cases, however, no evaluations of specific products will be available, and the consumer will need to independently judge the quality of the application.

Consumers need to consider more than just the quality or accuracy of the information or content. Some products may make users feel anxious or overly confident, or may affect the trust they have in people important to them. What kind of emotional support does the product offer? Did it actually help them? For example, a program that offers only information and no social support may be less likely to promote behavior change or enhance coping with a disease (Brennan et al., 1995; Gustafson, Hawkins et al., 1999). Moreover, because different consumers learn in different ways, potential users need to consider which style of learning works best for them. Two applications that have identical information but different presentation styles may have very different effects on the same person. In essence, consumers may consider developing their own evaluation framework that includes consideration of content quality and outcomes related to personal coping and self-confidence in decisionmaking. Tools will be needed to assist consumers on how to be an educated consumer and implement an evaluation framework.

Building Skills to Use IHC Applications

What skills and training do consumers need to use IHC applications? Because of the newness of IHC, relatively few consumers have the skills needed to take full advantage of applications. In some cases, skills must be acquired under circumstances of high stress, such as in the case where an individual has just been diagnosed with a life-threatening illness. Unless application design takes into account the diversity of potential users (e.g., educational/computer skills level, disabilities), many resources will be unusable. Allowing users to provide feedback to developers is important to improving usability. In addition, enabling users to share their experiences with others may help them use applications more effectively.

The scenario in Box V-4 illustrates several important issues faced by consumers in selecting and evaluating appropriate IHC applications.

Box V-4
A Consumer’s Experience with IHC Applications

Mary Smith returned home from the hospital with her week-old twins, a girl and a boy. Like many first-time mothers, Mary was both enchanted and overwhelmed with the challenges of her new family. The practical advice offered during the childbirth preparation classes seemed to escape her awareness at the very moment that questions about nursing the babies entered her mind. Her questions seemed too minor and insignificant to warrant a call to the nurse midwife. However, she remained worried, and a bit unsure about many of the day-to-day rituals of mothering twins. Encouraged by her partner, Mary decided to use some of the health information resources available on the Internet. During one of her first trips away from home, Mary stopped at the public library to use their Internet computer.

First Time: Finding Resources

Mary began looking for health information at her hospital’s Web site. She brought along the address for the hospital’s Web site, which she had found in her discharge instructions. Mary typed the address into the browser, and soon pages appeared on the computer screen presenting information by her hospital. Most of the information addressed logistics such as how to get to the hospital or where to park; however, several sets of keywords suggested that specific clinical advice might be available. Selecting the phrase "Help for New Parents" led Mary to discover a series of pages with helpful hints and explanations about taking care of infants.

One page described common sleep patterns for newborns, noting that smaller babies had more sleep difficulties than larger babies. This information confirmed Mary’s experience with her five-pound twins, neither of whom slept more than three hours at a time. Mary read with eagerness the suggestions regarding how to establish successful sleep routines with small infants. Another page addressed bathing infants. Mary was able to view a short movie demonstrating how to hold small infants while bathing them. She learned that many soaps caused skin irritation in newborns, and reviewed a list of recommended bathing products.

None of the material presented on the hospital’s Web site specifically addressed the needs of twins. Mary noticed the message "Search For Other Resources" just as she needed to leave the library to return home. She made a note to come back and check the new information.

Second Time: Follow-up and Exploration

One week later Mary returned to the library. By this time, she had begun to feel a bit more confident feeding and bathing the babies, but had found herself becoming increasingly weepy and sad over very small events. The babies seemed to be nursing successfully, but she continued to worry about her ability to feed them. Mary launched another session to her hospital’s Web site.

Mary located the appropriate section within the hospital’s Web site and looked through the listing until she found the message "Search For Other Resources." At this point Mary initiated a search. In response to a question requesting her topic, Mary typed in the words "newborn twins." She received a list of 27 different Web sites, all denoting "twins" somewhere in the title of the pages. She began reviewing the pages one at a time.

A commercial baby food company presented the first page that Mary discovered. Included on this page were pointers to descriptions about various formula preparations and strategies for promoting health nutrition. Pretty colors and pleasing pictures of well-nourished babies made the page seem interesting; however, with her commitment to breast-feeding, Mary determined that the content on the page was not relevant to her current situation.

A second page led Mary to the Web site developed and supported by a national voluntary society, Parents of Twins. This page differed from the ones Mary had already viewed. In addition to the usual helpful reference materials, this page had a public bulletin board area where interested persons could post or read messages. Mary focused her attention here.

Mary discovered several hundred messages with topics ranging from "Twins—double trouble" to "Help—where can I find cheap diapers." As Mary reviewed the messages, she noticed that parents of twins posted most of the messages. Many messages offered humorous or unnerving stories about child care experiences. Some posed specific questions, such as "Has anyone tried to nurse both babies at the same time?" to which other participants offered suggestions or advice. Occasionally a user identified himself or herself as a health care professional; their comments seemed different from those posted by the parents, offering clinical descriptions or recommendations to consult with clinical care providers.

Mary observed that the quality of advice in the postings ranged from very believable to almost ludicrous. Some postings clearly were advertisements in disguise, suggesting that a specific product may help with a given problem. Others offered helpful strategies, such as how to use pillows to position the babies comfortably during feedings. Occasionally a poster would direct readers to additional Web sites, which Mary herself sometimes subsequently viewed.

Mary noticed that a small set of individuals seemed to do most of the posting. Five or six users seemed to post several times a week, sometimes addressing their messages directly to other individuals or to the group as a whole. Phrases such as "Twin Moms Unite" or "Parents in the Know" created a sense of community that Mary knew she would like to join. Since no funds were needed to use the computer or to participate in the conversations in the bulletin board area, Mary resolved to return when she could and partake in this shared experience of parenting twins.

Maintenance: Linking IHC With Professional Care Services

When the twins were 4-weeks-old, Mary returned to the midwife for her post-partum checkup. In the conversation about her emerging role as mother, Mary described for the midwife the assistance and peer support she had discovered on the Web. Mary talked about the range of helpful (and sometimes unhelpful) advice provided by fellow users, and about the different Web sites that addressed child-care and parenting. Her midwife listened attentively, expressing support for Mary’s abilities to seek out and interpret health-related information for herself. She reviewed Mary’s strategies for determining the quality and relevance of the health information provided and identified some additional Web sites that she already reviewed and believed Mary would find helpful.

Key Considerations

  • IHC applications are an important resource for consumers and public access to these resources is critical.
  • Consumers need to be critical of the quality and relevance of online health information and support.
  • The use of IHC applications can be more effective when their use is linked with regular care from a health professional.


Policymakers, both in the public and private sectors, determine the nature of the environment in which IHC development and implementation occur (Patrick et al., 1999). IHC policymakers in the private sector typically include executives of health care organizations and large corporations, and they are driven by their primary responsibility to ensure the financial viability of their organization. Their purchasing and investment decisions essentially determine the commercial viability and development of specific types of applications. Policymakers in the public sector, especially legislators and regulators, have decisionmaking power over the general field of IHC, and have to ensure that their decisions are consistent with the interests of their constituents and the general public. Both types of policymakers help determine the role of IHC in the overall mix of public and private investment in individual and community health, in disease prevention, and in health care and rehabilitation. Ultimately, policymakers are concerned with the impact that IHC applications may have on the structure, process, and outcomes of health and health care. Understanding how IHC applications relate to, enhance, and/or potentially detract from other determinants of individual and community health is important in developing policy. Major policy issues related to IHC are reviewed in the next chapter.

Promoting Quality and Access

How do policymakers promote quality and ensure safety of IHC while not hindering innovation and use? Policymakers will need to weigh the role of voluntary standards and self-regulation versus government regulation, and decide if or when it is appropriate to intervene or regulate. In some cases, such as in the case of fraudulent online marketing of health care products, Federal agencies such as the Federal Trade Commission have taken some action (FTC, 1997). Policymakers may consider implementing appropriate incentives to promote voluntary adoption of quality goals.

How do policymakers ensure that all populations have access to IHC? Some policymakers are increasingly concerned that the growing reliance on technology in health care and public health may leave the most needy populations without access to health information and support (Eng et al., 1998). The debate about who should pay to enhance access to telecommunication services is being played out in the context of the universal service provisions of the Telecommunications Act of 1996 (FCC, 1998). In addition, there are concerns about how applications that address issues of relatively small populations ("orphan" issues) can be supported if they do not have commercial value.

The scenario in Box V-5 outlines some critical issues that policymakers must deal with in developing and implementing IHC applications that contain personal health information.

Box V-5
A Policymaker’s Dilemma

A county with a population of 2.5 million and a handful of dominant health care providers is working to develop a community-wide immunization registry for all children and youth under the age of 21. The vision is to have "anytime, anywhere" instant availability of information on the immunization status of the target population. The rationale for this is that having such information available at the time of either medical office visits or school and college health immunization initiatives, will reduce the likelihood that these individuals will "fall through the cracks" and remain un- or under-immunized against important and sometimes deadly diseases such as measles, rubella, and hepatitis B.

More than two years ago, the county director of public health convened leaders from each of the major health care providers, the council of community clinics (the members of which serve most of the uninsured), three major employers who, combined, provide approximately 60 percent of the county’s jobs, and expert faculty from the local schools of public health and medicine. With support from staff of the health department, they have been meeting monthly to discuss ways in which the immunization registry can be built. Many of the technical problems that existed several years ago when the idea of a registry first surfaced are now more tractable in the face of growing acceptance of Internet-based technologies. These include problems such as interfacing disparate databases, sufficient connectivity between health care providers and the health department, acceptance of common definitions and language to use for procedures, and methods for secure and reliable transmission of information via the Internet, including safe storage behind adequate "firewalls."

However, in recent meetings, the group has become bogged down over three issues. First, while there is a desire to fully engage consumers in the process of helping to keep immunization data on their children up to date, there is worry over whether public access to even part of the registry might be put to unintended purposes (e.g., separated parents contesting custody of a child might use the registry to locate one another). Second, the general issue of making available personal medical information outside the confines of traditional health care settings is beginning to be the subject of call-in talk shows in the community. One county supervisor is particularly entrenched in his views about "big government" and how the potential for misuse exists in any governmental collection of information about individuals. Finally, there is concern about the accuracy and general quality of the information to be contained in the registry. The health care providers readily admit that when they perform audits on their own internal data sets for immunization information, they find errors and problems. This is in spite of substantial efforts on their part to maintain the currency and quality of this information.

What role can the local health department play to help this group move beyond these problems and create a functional immunization registry? What liability issues are raised by the first concern? How can these issues be mitigated? Can the trade-offs between concerns about how access will be used and the need to engage consumers fully in helping the registry stay current be made to come out in favor of consumer involvement?

As for the second concern, how can the group move to community acceptance of the functions served by the registry? Is it acceptable to characterize the registry as similar in kind to other public health data functions and thus invoke the public health department authority and responsibility? If the county does not assume this responsibility what sort of entity could be developed to operate the registry that might reassure those concerned about "big brother?" What safeguards can and should the data communication and storage technology partners provide to assure stakeholders that data will be secure?

Finally, how can the group handle the third area of concern, that of the quality of the data present in the registry and any liability that may pertain? This is considered especially important given the potential for consumer access to the registry to enable, for example, a simply query about the need for a vaccination for a child. What if the registry says there is no need when, in fact, there is? Who is responsible for this—the public health department, all participants, the "health provider of record" for the child, or is there some other chain of responsibility? If it is an independent new entity, how can these problems best be anticipated and lessened?

Key Considerations

  • Policymakers should educate and work closely with all community stakeholders.
  • The potential for abuse or harm from publicly accessible databases should be carefully evaluated.
  • Policymakers should help define and coordinate roles and responsibilities of professionals who are involved in delivery of services.
  • Public health goals need to be balanced with the rights of individuals to privacy and confidentiality.

7 The term “purchaser” is used here to refer to purchasers of health services and products, such as health plans, other health care organizations, large employers, and government agencies.

Return to Table of Contents

Comments:   Updated: 05/01/08