Building on the work of ODPHP and the Objective 11-4 Technical Expert Workgroup, and in collaboration with the ODPHP Project Officer, we drafted technical specifications that further defined each of the six disclosure criteria: (1) Identity, (2) Purpose, (3) Content and Content Development, (4) Privacy and Confidentiality, (5) User Feedback, and (6) Content Updating. Draft technical specifications included the primary question that each criterion was intended to address, the content or disclosure elements associated with each, and guidelines for determining the accessibility of each disclosure element. Consistent with the prior recommendations of the Technical Expert Workgroup, the technical specifications required most disclosure elements to be accessible to users within two clicks of the home page¹⁰. To the extent possible, we also sought to capture critical elements from the URAC and Consumer Health WebWatch standards and language.

Based on the draft technical specifications, we drafted a data collection instrument and protocols for the review of health websites. The protocols guided the reviewer through a series of questions to determine compliance with disclosure criteria. We designed the protocols to reflect, to the extent feasible, the average Internet user's navigation of the website, starting from the URL of the home page. We also aimed to minimize the need for reviewers to make independent judgments about what does and does not "count" toward compliance and to ensure consistency across reviewers by giving closed-ended response options accompanied by clear directions and definitions of terms. In addition, we included space in the instrument to record the URL where specific information was found in order to facilitate validation of data, as well as a comments field to capture issues that might require additional discussion or followup. The protocols also directed reviewers to review three separate items of health content on each website in answering specific questions about Content and Content Development as well as Content Updating.

We pretested the draft protocols on a sample of 10 websites broadly representative of the pool from which the final sample of 100 sites would be drawn. The pretest sample included 5 sites from the target stratum and 5 from the remainder, and a mix of nonprofit (.org), commercial (.com), government (.gov), and educational (.edu) domains. Two members of the MPR project team who would not be the primary reviewers of the full sample conducted the pretest (1) to determine whether the protocol appropriately elicited information about compliance with disclosure criteria from these sites, and (2) to identify any adjustments needed for the full review. Although the pretest was designed primarily to test the protocols, we also obtained preliminary feedback on the proposed mode of administration for the full review. Findings from the pretest addressed the length of time required to review each site, the mode of administration, sources of discrepancies between reviewers, and the content and wording of the protocols. Findings from the pretest are attached as Appendix A. The revised final protocols are in Appendix B.

< PREVIOUS     |      TABLE OF CONTENTS     |      NEXT >