Appendix 4. Developing an Interactive Response Tool: The HELP Model (continued)

6.0 Interactive Response Tool Evaluation Document

6.1 San Luis Valley Region Exercise

The San Luis Valley Region is made up of six rural counties located in the Southwest of Colorado. Most of public health agencies in the valley only have one or two public health nurses assigned to them. The total census population for the region is 48,000 but population can increase to greater than 75,000 in the summer months. We decided to test a prototype version of the QI Monitoring application in conjunction with a planned influenza vaccination exercise in this rural area.

Exercise Design

In order to test the vaccination plans that would be used during a disease outbreak, such as for pandemic influenza, public health nursing services in the six counties in the San Luis Valley conducted free flu shot clinics beginning on Saturday, October 15, 2006. The exercise began with a fictitious health emergency being declared. Area clinics were opened and flu shots were provided to the public to test the ability of the local public health agencies to respond to this staged incident. As part of the response, each county public health nursing service also practiced the ability of its staff to isolate ill persons and to quarantine people who have been exposed to ill individuals.
A series of questions was asked before each person was admitted to the vaccination clinic to segregate out those who were already ill and may need to be quarantined. A "Yes" answer to any of the following questions was sufficient for the person to be assigned to home quarantine:

• Do you have a fever now?
• Do you have body aches now?
• Do you have a new cough or new sore throat?
• Have you been around someone who has the flu or the above symptoms?

Methodology

Two citizens in each county, a total of 12, volunteered to be mock "isolation cases" and help test the QI Monitoring application. Data on individuals to be placed into isolation were collected (see sample Case/Contact Investigation Report Form—Appendix 3-A) and faxed to the Regional Epidemiologist who entered them into the State Health Department's Outbreak Management System. Data were transferred twice each day (prior to start of calling periods) as a flat file to test our ability to utilize this for directing the QI Monitoring application to call people. We were also provided the list of all mock cases and their contact information to verify that the application received the complete list and made the appropriate calls. The Quarantine/Isolation (QI) Monitoring application called the 12 volunteers identified for quarantine and isolation during the exercise to check on their status (whether they answered the provided phone number) two times (calling periods) per day for 2 to 4 days.

Results

The application was able to utilize the received flat files for contacting volunteers prior to each calling period. There was difficulty in reaching one person who had a telephone number that blocked solicitation calls (they also use IR systems). This was disabled to permit receipt of calls from the IR system. Otherwise the application successfully dialed all listed phone numbers. All 12 volunteers successfully interacted with IR application (call registered as answered by a person) at least once during their quarantine. Four of the volunteers answered the IR application calls within 1 or 2 attempts for all calling periods. The Regional Epidemiologist was notified whenever a volunteer had not answered for two consecutive calling periods, signifying a need for a home visit. This occurred for 8 of the volunteers who failed to answer the IR application calls within 2 attempts for two to four consecutive calling periods.

• Total number of calling periods for all 12 volunteers = 74.
  • # Calling periods with calls answered with 1 attempt = 38 (51 percent).
  • # Calling periods with calls answered with 2 attempts = 14 (19 percent).
  • # Calling periods without calls answered with 2 attempts = 22 (30 percent).
• Notifications to Regional Epidemiologist (2 consecutive calling periods without an answer) = 14.

The feedback from volunteers and exercise organizers suggested several potential improvements for the QI Monitoring application (followed by our assessment of each suggestion):

• Have the option to repeat messages. (Such an option was added).
• Call up to three times during an hour to attempt contact. (Application could be programmed to call as many times per day as needed.)
• IR system should recognize telephone prefixes and call the appropriate county health agency when someone does not answer. (This is not possible with the IR system.)
• Recorded voice should speak slower. (Recordings updated with best "voice" for speed, clarity, tone and volume.)
• If cell phone is called, integrate GPS information to insure the person is still at home. (This is not possible with the IR system.)
• IR should give a phone number to call back on if the person cannot wait to speak to someone. (It is important that the quarantined person complete calls but the application does provide an announcement of a direct number to use when received, if assistance is needed afterwards.)
• Cancel call forwarding feature to limit calls to a land line telephone at the person's home. (This would require cooperation of telephone service provider and be subject to applicable laws.)
• Provide an electronic link between State Health Department and the call center to transfer case or contact data. (A flat file was used to export list of contacts to make prior to each calling period—it is a manual process for loading this file into the application.)

There were also suggestions for public health agencies using this application:

• Have someone make at least one home visit during the home quarantine or isolation period.
• Develop protocols for children.
• Develop protocols for more than one case or contact at the same location.
• Provide an information sheet to home quarantine or isolation candidates explaining:
  • What to expect.
  • What to do in certain instances.
  • What not to do.
  • Who to call if you need help.

Summary

There were several key lessons learned from the San Luis Valley Region Exercise. In a pandemic influenza or other infectious disease (i.e. SARS) scenario, home quarantine or isolation may be an important, early component of the public health response. With effective risk communication messages to the public and adequate support for those in quarantine, we expect a good percentage of compliance. Small public health agencies (and perhaps larger agencies) will have difficulty keeping track individuals in home quarantine or isolation. This will require extensive personnel resources without the benefit of a system like the IR and the QI Monitoring application (or some similar monitoring strategy). In this test the application was able to monitor up to 70 percent of the quarantined persons which demonstrated compliance with little personnel resources. This may help public health agencies free up resources to focus on non-compliant individuals or manage the myriad of other response actions required.

Return to Appendix 4 Contents

6.2 North Central Region Exercise

The North Central Region (NCR) consists of 10 counties in the Denver metropolitan area with a population of 2.7 million people. The geography of the NCR ranges from mountainous rural to suburban foothills to urban plains areas. The NCR was established by Colorado as a Homeland Security region. The NCR is participating in the Cities Readiness Initiative (CRI) with the goal of "get pills into people within 48 hours using every method available to save as many lives as possible." The antibiotic medications the CRI is preparing to distribute will come from local caches as well as the Strategic National Stockpile (SNS), if necessary.

Exercise Design

The goal of the NCR Exercise was to test the ability of the four IR applications to initiate contact and effectively communicate key information to users. In addition to a more fully developed version of the QI Monitoring application, public health volunteers also tested the Drug Identification (DI), Point of Dispensing (POD) and Frequently Asked Question (FAQ) Library applications. Up to ten volunteers from each county participated in the exercise beginning on May 2, 2006. A total of 96 volunteers tested the QI Monitoring application, with approximately one third assigned to test one of the three other applications. The goals for testing each application were:

• QI Monitoring: initiate contact and assess health status of individuals in quarantine reflective of their assigned "health" scenario.
• DI: provide accurate pill identification based upon supplied drug photo.
• POD: provide accurate location for drug distribution based upon entered zip code.
• FAQ Library: to provide accurate information that was retrievable by the user about an assigned health concern.

Evaluation Methodology

Unlike the first exercise in the San Luis Valley, this exercise was strictly scripted so that we could evaluate IR application accuracy as well as how the user assessed the applications upon a set of eight criteria. Volunteers were provided a call log (go to Appendix 3-B) to record their experiences with the QI Monitoring application. An additional evaluation form was used to record the answers from their use of one of the three assigned other applications (Appendixes C—E). The volunteers evaluated the QI Monitoring and other assigned application on specific aspects using a 5-point scale (5 = Strongly Agree, 4 = Agree, 3 = Undecided, 2 = Disagree, 1 = Strongly Disagree). The eight criteria used to evaluate each IR application were:

• Directions given by the IR were easy to follow.
• Recorded voice on the IR was easy to understand.
• Recorded voice on the IR went at a proper speed.
• Recorded voice on the IR was at a proper volume.
• User satisfied with experience using the IR.
• User had a positive opinion of the IR. (This criteria was tailored to a specific question about each application.)
• User would trust receiving supportive contact or information via an automated system such as the IR during a public health event.
• User would prefer to receive supportive contact or information via an automated system such as the IR versus a person during a public health event.

The QI Monitoring application attempted to contact all volunteers during two call periods (9:00am-11:00am and 2:00pm-4:00pm) each day for two consecutive days. Each volunteer was assigned one of four "health" scenarios so we could determine, once contacted, if the QI Monitoring application correctly reported a person's health status.

The following is an example "health" scenario:

"You are a 65-year-old man who has been quarantined since May 1st and is expecting to receive automated calls verifying your health status on May 2nd and May 3rd. You begin experiencing symptoms including body aches and fever above 100oF on the afternoon of May 3rd. You will be contacted by the IR for your health status and should respond according to the following:

• May 2nd - morning: Temperature <100°F (no assistance needed)
• May 2nd - afternoon: Temperature <100°F (no assistance needed)
• May 3rd - morning: Temperature <100°F (no assistance needed).
• May 3rd - afternoon: Temperature >100°F (assistance needed)

Volunteers returned call logs after the exercise and their evaluations of the applications and comments were reviewed to determine if any modification would be needed.

Results

We received 93 evaluations (97 percent return rate); 88 evaluations were completed representing 92 percent of volunteers. The results for each application will be discussed in the following sections.

Quarantine/Isolation (QI) Monitoring

The modified version of this application was again able to utilize a flat file for contacting volunteers. A problem was encountered during the first calling period with dialing three long distance numbers. The application was quickly modified to allow for long distance dialing ("1" added to front of long distance numbers) for the three remaining calling periods. Otherwise the application successfully dialed all listed phone numbers for all calling periods.

All but two volunteers successfully interacted with IR application (call registered as answered by a person) at least once during their quarantine. Fifty-nine (61 percent) of the volunteers answered the IR application calls within 1 or 2 attempts for all calling periods: 20 answered for three periods, 14 for two periods, and 1 for just one period. Four times a person made a mistake in the language selection (and could not go on to select a health status—instructions what to do if this occurs could be provided upon enrollment).

A county public health contact was notified whenever a volunteer had not answered for two consecutive calling periods, signifying a need for a home visit. This occurred for 15 of the volunteers who failed to answer the IR application calls within 2 attempts for two to four consecutive calling periods.

• Total number of calling periods for all 96 volunteers = 382.
  • # Calling periods with calls answered with 1 attempt = 278 (72 percent).
  • # Calling periods with calls answered with 2 attempts = 45 (12 percent).
  • # Calling periods without calls answered with 2 attempts = 59 (15 percent).
• Notifications to Regional Epidemiologist (2 consecutive calling periods without an answer) = 19.

The majority of volunteers indicated favorable scores (4 or 5) for seven of the eight criteria used to assess the QI Monitoring application (n=88):

• Directions given by the IR were easy to follow—99 percent (mean score 4.6, median score 5.0).
• Recorded voice was easy to understand—99 percent (mean score 4.7, median score 5.0).
• Recorded voice went at a proper speed—98 percent (mean score 4.6, median score 5.0).
• Recorded voice was at a proper volume—97 percent (mean score 4.7, median score 5.0).
• Satisfied with experience using the IR—93 percent (mean score 4.6, median score 5.0).
• Had a positive opinion of the IR—93 percent (mean score 4.6, median score 5.0).
• Would trust receiving supportive contact via an automated system like the IR during a public health quarantine—88 percent (mean score 4.3, median score 5.0).

The only criteria not scored favorably by a majority of users concerned their preference in receiving health department contact from an automated system versus speaking to a live person: 43 percent with scores 4 or 5, mean score 3.4 and median score 3.0.

In hindsight we thought we should have asked users to assess if it would be acceptable to receive supportive contact from an automated system during public health quarantine. Indeed some of the comments received from users seemed to indicate that the IR would be acceptable for use during such an emergency. One comment raised the question whether this application could work for everyone. It was never our intention that this application could work for everyone. It would be at the discretion of public health agencies coordinating quarantine efforts to decide which individuals this application could assist in monitoring—the goal being that many individuals could be supported by this approach. That would reduce the overall number of individuals requiring personnel to monitor them, so those resources could concentrate on special needs cases.

• "Much faster automated… was on hold 11 to 12 minutes awaiting to speak to a live person—a reasonable time frame—but this will dramatically increase during a true emergency, using up cell phone batteries, increasing public anxiety, and overtaxing responders."
• "Prefer a real person, but this was easy to understand."
• "If an actual emergency occurred, I wouldn't mind being called by IR as long as I could get to a live person if my questions weren't answered."
• "But will it work for everyone? Especially the very elderly living with relatives."

Drug Identification (DI)

This application was tested to determine how effective it would be in assisting the public in identifying antibiotic drugs that may be distributed during certain public health events. The underlying challenge is that multiple medications will be distributed to the same household during an emergency and not all of them will look the same. For example there are multiple manufacturers of doxycycline, all of which are contained in the local and SNS stockpiles. Rather than calling their doctor or pharmacist to question them about these medications, this IR application offers a self-service alternative.

Thirty volunteers were assigned to evaluate this application. Rather than provide the actual medication, they were provided pictures of both sides of one of the doxycycline drugs, asked to identify it with the application and record the answer on an evaluation form (see Appendix 3-C). We received completed evaluations from 26 volunteers (83 percent return rate) assigned to test this application. A total of 24 (92 percent) correctly identified the pictured drug: 22 identified it as doxycycline, 2 identified it as an antibiotic, 1 did not provide an answer and 1 indicated that the color of the pictured drug did not match any of the options (a printer issue—see comment below).

• "The prompt asked for a brown pill not pink with this imprint."

To increase the ability of callers to correctly identify drugs, the application includes drug type (capsule, tablet), drug shape, drug color and drug imprint for identification. In a real event, this caller could have been routed to an Information Provider for assistance (an option that could be added to the application) or directed to another resource.

The majority of volunteers indicated favorable scores (4 or 5) for seven of the eight criteria used to assess the DI application (n=26):

• Directions given by the IR were easy to follow—96 percent (mean score 4.6, median score 5.0).
• Recorded voice was easy to understand—88 percent (mean score 4.5, median score 5.0).
• Recorded voice went at a proper speed—85 percent (mean score 4.3, median score 5.0).
• Recorded voice was at a proper volume—92 percent (mean score 4.5, median score 5.0).
• Trust automated system to correctly identify drug—77 percent (mean score 4.2, median score 4.5).
• Satisfied with experience using the IR—73 percent (mean score 4.2, median score 5.0).
• Would trust receiving information via an automated system like the IR during a public health emergency—77 percent (mean score 4.2, median score 4.5).

The only criteria not scored favorably by a majority of users concerned their preference in receiving health department information from an automated system versus speaking to a live person: 42 percent with scores 4 or 5, mean score 3.4 and median score 3.0. Again, we thought we should have asked users to assess if it would be acceptable to receive information about medications from an automated system during public health emergency. Indeed one of the comments received from users seemed to indicate that the IR would be acceptable for use during such an emergency.

• "Talking to a real person is always comforting but this system did work well."

Other comments identified improvements for the application including recorded messages (spelling out drug names, using better voice for recording) and an option to identify more than one drug appearance per call. We made the appropriate modifications to the application to address these suggestions.

• "I could not understand the name of the medication. Need to pronounce it better and slower. Spelling it would help."
• "It would be nice to be able to identify more than one pill per call if there is potential for family members to have different products. I wouldn't want to listen to the introduction more than once."

Point of Dispensing (POD)

This application was tested to determine if zip code specific messages could assist users with getting POD locations where drugs are being distributed. The application allows users to call and receive very specific messaging according to their entered 5 digit zip code. This application provides self-service information delivery where messages could be modified for a variety of events besides mass drug dispensing. Major application benefits include:

• Provides consistent, accurate information dependent upon entered zip code.
• Collects zip code data to characterize events (situational awareness—where are callers from and should more media messaging be used).
• Expands capacity for handling surges since calls are handled without personnel.
• Adaptable to any emergency where zip code specific messaging is needed including shelter in place strategies.
• Supports mass prophylaxis/immunizations.

Thirty-six volunteers were assigned to evaluate this application. Callers were instructed to enter a 5 digit zip code, record that zip code and the location they were given on an evaluation form (see Appendix 3-D). We received all 36 evaluations back (100 percent return rate) and all recorded the correct POD location for their entered zip code.

The majority of volunteers indicated favorable scores (4 or 5) for seven of the eight criteria used to assess the POD application (n=36):

• Directions given by the IR were easy to follow—83 percent (mean score 4.3, median score 5.0)
• Recorded voice was easy to understand—92 percent (mean score 4.4, median score 5.0).
• Recorded voice went at a proper speed—81 percent (mean score 4.1, median score 4.0).
• Recorded voice was at a proper volume—92 percent (mean score 4.4, median score 5.0).
• Information received was accurate based upon entered zip code—94 percent (mean score 4.6, median score 5.0).
• Satisfied with experience using the IR—86 percent (mean score 4.4, median score 5.0).
• Would trust receiving information via an automated system like the IR during a public health emergency—86 percent (mean score 4.4, median score 5.0).

The only criteria not scored favorably by a majority of users concerned their preference in receiving health department information from an automated system versus speaking to a live person: 47 percent with scores 4 or 5, mean score 3.5 and median score 3.0. Again, we thought we should have asked users to assess if it would be acceptable to receive information about POD locations from an automated system during public health emergency.

Comments identified improvements for the application including recorded messages (using better voice for recording, providing major intersections to locations, eliminating confusing words) and an option to repeat location messages. We made the appropriate modifications to the application to address these suggestions.

• "I had to play the message several times to hear the location - still not sure what the name of the stadium was."
• "Need 'directions to location' option. I really liked the message about 'plenty of antibiotics' and impression that people would be taken care of properly."
• "Speak at different speeds for different parts of the message. Eliminate words like prophylaxis which don't help most callers."

Frequently Asked Question (FAQ) Library

This application was tested to evaluate the ability of users to navigate a library of messages and to obtain the desired information. Our HELP program uses this library for handling calls after hours and with great success. One of the benefits of the FAQ Library application is allowing self-service information delivery that is consistent and accurate. The application collects structured data to characterize the information needs of the public (situational awareness—what are public information needs and should media messaging be used for them). The application has the ability to expand capacity for handling surges and is capable of adapting to different events.

Thirty volunteers were assigned to evaluate this application. Callers were instructed to call, obtain the answer to the question they were assigned and record the answer on an evaluation form (see Appendix 3-E). We received 28 completed evaluations (93 percent return rate): 27 (96 percent) provided correct answers to one of the following assigned questions:

• What is the incubation period for West Nile Virus in humans?
• What are the symptoms of mold exposure?
• How do I know if I have West Nile Virus?

The one incorrect entry gave mold-growing conditions instead of symptoms of mold exposure.

The majority of volunteers indicated favorable scores (4 or 5) for all eight criteria used to assess the FAQ Library application (n=28):

• Directions given by the IR were easy to follow—96 percent (mean score 4.6, median score 5.0)
• Recorded voice was easy to understand—96 percent (mean score 4.6, median score 5.0)
• Recorded voice went at a proper speed—82 percent (mean score 4.1, median score 4.0).
• Recorded voice was at a proper volume—100 percent (mean score 4.7, median score 5.0).
• Able to easily navigate through the options to find information—100 percent (mean score 4.7, median score 5.0).
• Satisfied with experience using the IR—100 percent (mean score 4.7, median score 5.0).
• Would trust receiving information via an automated system like the IR during a public health emergency—93 percent (mean score 4.5, median score 5.0).
• Would prefer receiving information via an automated system versus a live person during a public health emergency—57 percent (mean score 3.7, median score 4.0).

This application received the most favorable evaluations from users. However, the lowest rating again concerned the preference in receiving health department information from an automated system versus speaking to a live person. The responses suggest that most users would find it acceptable to retrieve information from an FAQ library during public health emergency. Comments identified improvements for the application including recorded messages (using better voice for recording, eliminating confusing words), an option to repeat messages and an option to return to the main to answer another question. We made the appropriate modifications to the application to address these suggestions.

• "I felt that the symptoms were listed very quickly. IR needs to slow down just a little bit, otherwise very good!"
• "He went too fast to write it all down However, if I was calling to see if I had the symptoms, I would know what I was listening for and would probably not write all of them down."
• "When you get to the answer you are looking for your only option was to repeat the message or start all over again. Could you also provide an option to go to the West Nile Virus main menu?"
• "Need way to slow down or pause recording when listing things, otherwise too much info too fast."

Return to Appendix 4 Contents

6.3 Evaluation Summary

Exercise objectives were met and we obtained excellent feedback to help us improve the tested applications. We also obtained important information on what the user acceptance was for these IR applications. Although there were mostly favorable evaluations for all four applications, it was apparent the FAQ Library application seemed more acceptable than the DI application (perhaps because the latter concerned medications to be taken). The comments and evaluations of these applications should help Public Information Officers in determining which ones may be acceptable for different events and in developing messaging strategies. These results also suggest areas for potential community outreach efforts for public health agencies to create a more informed public. One lesson learned is that the applications will only be as good as the information that is developed for them and how it is provided to the public.

Return to Appendix 4 Contents

6.4 Future Research

These IR applications have yet to be tested with vulnerable and at risk populations (Spanish-speakers, seniors, etc). Such testing may be necessary to determine if some groups would respond as positively to the IR applications as the test groups did. Additional research will be needed to determine other information and resource needs for the public and how to provide them health emergency events.

Return to Appendix 4 Contents
Return to Report Contents
Proceed to Next Section