You Are Here: AHRQ Home > Quality & Patient Safety > Measuring Health Care Quality > Quality of Health Information on the Internet

Quality of Health Information on the Internet

Enabling Consumers to Tell Fact from Fraud

Executive Summary
Introduction
Methods
Results
Next Steps
References
Acknowledgments

Executive Summary

Health Improvement Institute presented a workshop on the Quality of Health Information on the Internet on November 17, 1997. Representatives from over 60 healthcare, Internet, consumer interest, Government, telecommunications, and public health organizations attended the workshop to explore possibilities for developing mechanisms to enable consumers to tell fact from fraud on the Internet.

The workshop consisted of three parts:

Expert presentations of the scope of the problem and potential solutions.
Interactive work group discussions on criteria for, and costs and benefits of, possible mechanisms to enable consumers to tell health fact from fraud.
Plenary session to hear work group results and discuss next steps.

Participants identified several requirements for systems to enable consumers to evaluate the quality of health information on the Internet. Such systems must:

Be feasible and practical to implement.
Benefit both the consumer and producer.
Build upon existing systems.
Be voluntary, cooperative efforts by producers, independent agencies, and other public and private organizations to provide consumers the means to judge the value of information for themselves.

They discussed the costs and benefits of the following potential enabling mechanisms:

Empowering consumers to evaluate health-related Internet sites—empowerment through such mechanisms as checklists and filtering software would promote cost-effective availability of health information, foster self-care, and potentially improve health status, but producers would have to bear the cost of updating sites and responding to consumer feedback, for example, and there is the potential for fraudulent disclosures.
Consumer feedback—excellent for communicating consumer opinions quickly, but the spectrum of consumers' ability to discern reliable information and producers' potential to manipulate consumer feedback limits their utility.
Independent ratings systems—potentially useful watchdogs, but their success will depend on increasing the public's awareness of the value of such systems.
Industry standards—a code of conduct for producers may require standards tailored to each health care segment; it may be easier to monitor a code of conduct listing what "should not" be done, rather than specify what "should" be done.
Accreditation of Internet producers—an enormous and costly task requiring a large staff with expertise in varying fields to monitor thousands of health-related sites; it is also vulnerable to fraudulent seals and claims, and may shut out small producers who are unable to afford accreditation fees.

The potential promise of the Internet as a vital tool to improve self-care and the public's health is too great to burden it with unworkable regulations or restrictions. The goal is to encourage producers, independent rating agencies, and other public and private organizations to work together voluntarily to provide consumers with the means to judge for themselves what information is valuable to them, and what is best ignored. The results of this workshop represent an important beginning toward achieving this important goal.

Return to Contents

Introduction

Consumers are connecting to the Internet in growing numbers. One in six adults is regularly online, and use of Internet applications beyond E-mail has increased over three-fold in the last 2 years.¹ Through the World Wide Web, people are gaining unprecedented access to health information from over 10,000 health-oriented Web sites.² The plethora of health information and the slickness of its presentation make it difficult for consumers to tell high-quality, authoritative information from high-technology frauds and self-promoting cure-alls. This year, the Journal of the American Medical Association drew national attention to this important and growing problem.³ The editorial suggested modest criteria to which all health-oriented Web sites should adhere. Other organizations have drawn attention to this same problem and have proposed similar solutions, but these efforts often fall short of what is needed.

To address this issue of vital importance to Americans' health, the Health Improvement Institute presented a workshop in Bethesda, Maryland, on November 17, 1997, on the quality of health information on the Internet. This workshop brought together professionals from healthcare, Internet, consumer interest, Government, telecommunications, and public health organizations to explore possibilities for developing mechanisms to enable consumers to evaluate the quality of health information on the Internet.

The workshop's goals were:

To formulate criteria/requirements for systems to enable consumers to evaluate the quality of health information on the Internet.
To assess the benefits and costs of various enabling mechanisms.
To identify useful next steps towards developing mechanisms to enable consumers to evaluate health information on the Internet.

This report summarizes workshop results and lists next steps that the Health Improvement Institute is prepared to pursue toward enabling consumers to evaluate the quality of health information on the Internet.

Return to Contents

Methods

Health Improvement Institute structured the workshop in three major sections:

Presentations by experts.
Work groups composed of participants.
Plenary session for participants to hear the results of all work group deliberations and to discuss next steps.

Opening presentations by experts were designed to enable participants to contribute to work group discussions. They provided attendees with a broad overview of current issues and concerns regarding the quality of health information on the Internet.

Work groups were designed to be interactive discussions involving moderators, experts, and participants. There were two types of work group. One work group considered criteria for mechanisms to inform consumers about the quality of health information on the Internet.

Three work groups discussed various enabling mechanisms, including:

Empowering consumers to evaluate health-related Internet sites.
Consumer feedback systems; including complaints/compliments, awards of excellence, law suits, and independent ratings of health-related Internet sites.
Industry standards and ratings, including, accreditation of Internet health information producers.

The plenary session was designed to permit work groups to present conclusions formulated by each work group. A work group spokesperson presented an overview of various enabling mechanisms' advantages and disadvantages, and the practicality of each mechanism. Finally, workshop participants discussed next steps in the context of workshop results.

The Health Improvement Institute established a Workshop Advisory Group (WAG), representatives of workshop underwriters, supporting organizations, and select consultants to assist in:

Setting the workshop agenda.
Suggesting speakers and expert resources.
Selecting workshop invitees.
Deciding how to best report and disseminate workshop results.

Workshop announcements were sent to over 4,000 healthcare, Internet, consumer interest, Government, telecommunications, and public health organizations. Announcements were distributed via the Internet, E-mail, and mail, and included information on how to register for the workshop, suggest speakers, cite relevant references, and to submit statements for inclusion in the workshop workbook.

The Health Improvement Institute provided a workbook to workshop participants to inform them about the day's objectives and activities. The workbook consisted of the following five sections:

Workshop information—Agenda, description of workshop, list of underwriters and participants, and evaluation forms.
Speakers information—Biographical sketches, copies of presentation materials.
Work group information—Work group assignments, descriptions, points for discussion.
Resources—Relevant articles, glossary of Internet terms.
Statements—Statements submitted.

Return to Contents

Results

Participants, experts, and moderators participated in one of four work groups to discuss assigned topics and to develop next steps toward enabling consumers to evaluate health information on the Internet. These work groups addressed:

Requirements of systems to tell fact from fraud.
Empowering consumers to evaluate health-related Internet sites.
Consumer feedback and independent ratings systems.
Industry standards and accreditation of Internet producers.

Requirements of systems to tell fact from fraud.

The work group on system requirements developed the following criteria for any proposed mechanism to educate consumers about the quality of health information on the Internet:

Feasible and practical to implement.
User-friendly and direct.
Relevant to the format, for example, WWW site, chat room, bulletin board.
Built upon learned experience; uses existing systems such as codes of conduct, dedicated health super-sites.
Provides material benefit for both producer and consumer.

Empowering consumers to evaluate health-related Internet sites.

The work group on ways to empower consumers to evaluate health-related Internet sites divided consumer empowerment into two levels:

Providing valid, accurate, and balanced information.
Enabling consumers to access, judge, and utilize this information.

Providing valid, accurate, and balanced information requires disclosure of such information as:

Name of company and/or author with relevant credentials.
Date the site was developed.
Frequency of updates.
Basis of any claims put forth by the site, such as references to the scientific literature and evaluation of the claims by others.

Enabling mechanisms for consumers to access, judge, and utilize the above information include:

Criteria/checklist systems.
Rating systems.
Filtering software that guides consumers towards certain grades of health information.

Mechanisms to empower consumers should:

Educate patients.
Link to appropriate sites or super sites.
Evaluate sites.
Provide quality seals.
Allow consumers to communicate through feedback mechanisms such as chat rooms and message boards.

The benefits of such mechanisms would include:

Promoting a cost-effective way for consumers to obtain health information.
Improving the quality of information on the Internet.
Increasing self health care.
Potentially improving consumer health status.

Drawbacks of such mechanisms would include:

Cost for producers to monitor and update sites.
Cost of responding to consumer feedback.
Potential inhibition of small, non-mainstream sites that would be unable to sustain the above-mentioned costs.
Potential for fraudulent disclosures, such as false quality seals, bogus credentials, and phony references.

Consumer feedback and independent ratings systems.

The work group on consumer feedback systems and independent ratings systems profiled two levels of consumers:

Health care professionals who are generally knowledgeable in the subject and would be interested in criteria used to rate the health information provided on the Internet.
Consumers who have varying knowledge of healthcare and experience in using the Internet, and who are generally interested in ratings rather than using criteria to rate information.

Consumer feedback systems are mechanisms that allow consumers to register their views about any aspect of health information on the Internet. Examples of such systems are compliment/complaint systems, consumer choice awards, and law suits. The benefits of consumer feedback systems include:

Identifying exceptionally excellent sites (via consumer choice awards of excellence) or fraudulent sites (via law suits).
Communicating consumer opinion quickly (by incorporating compliments/complaints into feedback to consumers).

Obstacles to the development of consumer feedback mechanisms include:

Potential to manipulate consumer feedback, since Webmasters have the power to broadcast favorable comments and to censor unfavorable comments.
Difficulty in implementing meaningful consumer choice awards because of the broad spectrum of consumers' ability to discern reliable health information.
Fear of liability that would result from excessive litigation or an atmosphere of over-regulation that may inhibit the presentation of valuable information.

Independent ratings systems or agencies use their own discovery mechanisms, rating criteria, and format for presenting information about the best (or worst) sites to the public. Rating systems that excel would be expected to prevail in the information market place. To succeed, a rating system must be simple and straightforward. Collectively, rating systems' success will depend on their ability to promote, and to increase, the public's awareness of the value of such systems.

Industry standards and accreditation of Internet producers.

The work group on industry standards and accreditation of Internet producers discussed the topic from the industry's perspective. Industry mechanisms are those developed by the industry to guide Internet providers (who give consumers access to the Internet) and producers (who develop its content) in providing quality health information on the Internet. Industries often rely on codes of conduct to provide uniform services to consumers. However, codes of conduct for "health information" are uniquely difficult, because:

The impact or result of misinformation is serious and can be a health risk.
Health care encompasses many segments, each of which would require different standards for the presentation of information to the public.

Participants concluded that it may be easier to outline what producers "should not" do than to create codes of conduct that specify what they "should" do. A list of "should nots" would clearly delineate inappropriate conduct which would facilitate monitoring compliance with the code.

Accreditation of health-related Internet sites by an organization that develops criteria to guide Web site producers as well as verifying that accredited sites are consistently following those guidelines would be an enormous and costly task. It would involve, for example:

Broad staff with expertise in all areas of health care.
Large staff constantly monitoring thousands of health-related Internet sites.
Financial costs that would inevitably be passed on to producers.

Transferring these costs to producers may give rise to three major problems:

Creation of fraudulent accreditation certificates or seals.
False claims of accreditation by legitimate accreditors.
Potential to shut out small producers who cannot afford accreditation fees.

Return to Contents

Next Steps

Workshop participants from over 60 organizations believed that:

There is a pressing and apparent need for enabling consumers to evaluate the quality of health information on the Internet.
Enabling mechanisms would lead to improvement in the quality of health information on the Internet.
We should proceed cautiously, and with the same caliber of quality, as that desired for health information on the Internet.

There was a remarkable consensus on ideas for next steps towards improving the quality of health information on the Internet. They included:

Identifying all parties involved and establishing a cooperative relationship.
Forming a neutral coordinating council to promote and guide development in this area
Conducting a needs assessment of consumers and producers.
Developing codes of conduct for producers that involve full disclosure so that consumers could better judge the usefulness and reliability of health information.
Educating the public about the need to become discriminating consumers of health information.
Continuing discussion of this topic.

Health Improvement Institute will explore these next steps with relevant organizations; specifically its potential to:

Develop a coordinating council.
Establish an on-line clearinghouse of independent rating agencies.
Operate a compliment/complaint Web site to which consumers could direct their comments and view feedback.
Conduct surveys and or focus groups of what consumers want in the way of disclosures, information, and enabling mechanisms and/or test prototype tools for these purposes.

Return to Contents

References

1. Corcoran E. 1 in 6 US adults regularly online, study indicates. Washington Post 1997; May 7: C10,13.

2. Virtual doc. U.S. News and World Report; 1997; July 14:62.

3. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet. JAMA 1997; 277 (15): 1244-1245.

Return to Contents

Acknowledgments

Financial support for the workshop was provided by: Agency for Health Care Policy and Research, American Association of Working People, Medical Care Management Corporation, and United States Pharmacopoeia.

Assistance in developing and presenting the workshop was provided by: Consumer Coalition for Quality Health Care, Food and Drug Administration, and World Health Organization.

Views expressed during the workshop by participants do not necessarily reflect the official views of the Health Improvement Institute, workshop supporters, participants' organizations, or any other organization associated with workshop development.

Workshop Report prepared by Health Improvement Institute for the Agency for Health Care Policy and Research, under contract No. 97R40188401D.

Return to Contents

Internet Citation:

Goldschmidt PG, Liao J. Quality of Health Information on the Internet—Enabling Consumers To Tell Fact from Fraud. Bethesda, MD: Health Improvement Institute, 1998. http://www.ahrq.gov/qual/hiirpt.htm