We undertook the following six research tasks for this project:

1. Review background materials from ODPHP's prior work on Objective 11-4 and from related online health quality initiatives
2. Finalize a methodology for determining the denominator of health websites
3. Develop technical specifications that identify discrete elements required to comply with the disclosure criteria
4. Create protocols and instruments for reviewing and scoring health websites
5. Conduct a preliminary test of the protocols and revise them, based on findings
6. Collect and analyze data for the baseline analysis

In the following sections, we describe our methodological approach to these tasks.

To assure that our working assumptions appropriately reflected the prior work of the Objective 11-4 Technical Expert Workgroup, we reviewed background materials the ODPHP Project Officer provided:

• "Measuring Healthy People 2010 Objective 11-4: Disclosure of Information to Assess the Quality of Health Web Sites; Discussion Paper," prepared by Carol Cronin, Consultant to ODPHP, September 15, 2004
• "Measurement Options and Approaches; Defining the Denominator - An Estimate of the Number of Health Web Sites," prepared by Carol Cronin, Consultant to ODPHP, September 23, 2004
• "Measuring Healthy People 2010 Objective 11-4: Disclosure of Information to Assess the Quality of Health Web Sites," September 30, 2004, Technical Expert Workgroup Meeting Summary

We also examined the standards and protocols produced by the two organizations that have developed processes to review and assess the quality of health websites, to assure that our definitions of terms, identified disclosure elements and criteria, and language were consistent with those of these established and recognized quality initiatives. These included:

• URAC Health Web Site Standards, Versions 1.0 (©2004) and 2.0 (©2006)
• Health Improvement Institute's health information rating instrument, Version 2.0 (revised April 16, 2005), prepared for Consumer Health WebWatch

This review assured that the assumptions guiding our work and our definitions of terms and concepts were consistent with evolving standards in the field.

Several assumptions guided our definition of the universe of health websites on the Internet (the "denominator") and the sampling strategy for this project. First, we assumed that many (if not most) Internet users search for health information using standard search engines, such as Yahoo or Google, and then visit several different websites identified. Second, we assumed that topics of interest, and therefore search terms, will vary, depending not only on individual interests, but also on health issues in the news. Thus, while a small number of heavily used consumer-oriented health information websites may address common areas of interest or serve as portals to specialized sources of information, any given search might in fact lead the consumer to a large number of websites designed for many different audiences and many different purposes. We therefore defined "health-related website" in inclusive terms consistent with those suggested by URAC, Consumer Health WebWatch, and the eHealth Code of Ethics, to include websites associated with a wide variety of sponsoring organizations that provide information for staying well, for preventing and managing disease, and for making decisions related to health, health care, health products, or health services.³

For the purpose of defining the denominator, we sought to enumerate an inclusive universe of health websites visited by Internet users. We therefore relied on information generated by a commercial vendor (Hitwise) that tracks Internet traffic, in lieu of smaller or more selective lists available from other sources. Hitwise uses information from Internet service providers to monitor traffic and categorizes websites with a minimum number of visits by market type, based on their subject matter and content⁴. Market share is then reported based on the number of visits to each website within the market, the number of pages viewed per visit, and the amount of time spent per visit.

We purchased data from Hitwise on websites classified in its Health and Medical market category, based on Internet traffic in the United States during October 2005⁵. Although information was also available on subgroups within this market (including the Health Information subgroup), we defined the "universe" for this project as all 3,608 websites within the overall Health and Medical category, in order not to exclude websites that might contain health information consistent with our working definition. Of these 3,608 websites, 213 accounted for 60 percent of all visits within this market. The dataset we purchased from Hitwise included the following information for all websites: domain name, ranking (from 1 to 3,608) within the Health and Medical category (based on number of visits), percentage of market share by visits, percentage of market share by number of page views, and average length of each visit (expressed in minutes: seconds). We used market share defined by number of visits to identify the heavily trafficked sites, since page views and the average length of visits could reflect navigational difficulties rather than user interest. Hitwise also provided domain names and limited information on market share of the top 100 sites within each of 11 subcategories within the Health and Medical category⁶. We used this information to assure diversity in our sample.

< PREVIOUS     |      TABLE OF CONTENTS     |      NEXT >