Chapter 2. Methods for Development of the Abbreviated Evaluation Tool

Identification of Expert Reviewers

A panel of three authors of the original tool (S.C., E.H., and M.J.) and external experts in disaster drill execution and evaluation was established. Nineteen external experts were invited to participate and 12 agreed to join the group. In addition to the three authors, four of the external experts had participated in the development of the original evaluation tool. External experts included representatives from government agencies (Department of Health and Human Services/Health Resources and Services Administration [DHHS/HRSA], AHRQ, the New York City Department of Health and Human Services [YC-DHHS] and the Navy Medical Office of Homeland Security), professional organizations (Joint Commission on Accreditation of Healthcare Organizations [JCAHO] and the American Association of Colleges of Nursing [ACN]), and academic medical institutions in several states (AL, GA, MD, PA, OR, CO).

Return to Contents

Selection of Items for Inclusion

A modified Delphi technique was used to establish consensus on the items that should be included in the abbreviated tool for evaluating hospital disaster drills.² The authors of the original tool independently identified items from the complete modules that were felt to be of greatest importance and then met to obtain consensus internally. In addition, the authors identified target numbers of questions for reviewers to identify as important for each module and addendum.

The external experts were supplied with the original workbook, Evaluation of Hospital Disaster Drills: A Module-Based Approach, as well as a set of response sheets listing all item numbers for each module in the workbook. They were asked to identify independently the critical items from each module with a target number of items set by the authors. They were also asked to indicate the five most critical items for each module. Items above the targeted number and narrative comments were also accepted.

All responses from the expert panel were reviewed, and the abridged modules were drafted based on reviewer responses. Questions that were considered critical by at least two reviewers or those that were recommended for inclusion by more than half of the reviewers were retained. In response to reviewer comments, some language was modified to be more specific or inclusive and to reduce redundancies.

The revised modules were sent to the reviewers for a second review of (1) clarity of items and (2) appropriateness of content for effective drill evaluation. Based on feedback received, the modules were further revised as necessary.

Return to Contents

Summary Items

Following discussion with AHRQ and DHHS personnel, summary questions were added to the beginning of each section of the zone modules. The purpose of these questions is to identify whether key activities of the zone occurred appropriately and to highlight points to keep in mind during the evaluation process.

Return to Contents
Proceed to Next Section