Appendix B. Conducting a Web-Based Survey

As mentioned earlier in this guide, we strongly recommend that nursing homes administer the Nursing Home Survey on Patient Safety Culture as a paper-only survey. Nursing home staff's access to E-mail and the Internet, as well as staff computer skills, may be very limited. Current research and evidence show that Web-based surveys typically have lower response rates than paper-based surveys (Groves, 2002; Shih and Fan, 2008). This general finding seems to hold true in health care settings. For example, as reported in the AHRQ Hospital Survey on Patient Safety Culture 2008 Comparative Database Report (Sorra, et al., 2008), the average response rate for hospitals that administered the survey by paper was highest (60 percent), followed by mixed mode using Web and paper (52 percent) and Web only (44 percent). We believe it is likely that similar differences in response rates by survey mode will occur with the Nursing Home Survey on Patient Safety Culture. Response rates are important because low rates may limit your ability to generalize your results to your entire nursing home.

If you have never conducted a nursing home Web-based survey, you will need to allow for the extra time and resources that are usually needed when conducting a Web-based survey for the first time. However, if your organization is going to use a vendor or has a history of conducting nursing home Web-based surveys and you are familiar with their advantages and disadvantages, your nursing home may want to consider this type of approach. To help you decide which approach is best suited to your situation, or if a combination approach is warranted, we present some of the pros and cons of conducting a Web-based survey. We also outline special considerations that need to be taken into account and present guidelines that will help you make the most of a Web-based survey, if you decide to take that approach.

A major factor, of course, is cost. Although the costs of a Web-based survey may seem lower because there are no printing, postage, or data entry expenses, do not overlook the labor costs associated with Web survey programming and testing. At the same time, a Web-based approach generally tends to be more economical as the survey sample size becomes larger. Surveys of only a few hundred individuals are likely to be more cost-effective using a paper-based survey. Cost, however, is just one of the many factors that need to be considered in deciding which approach to take.

Consider the Pros and Cons of Web-Based Surveys

There are a number of pros and cons to conducting Web-based surveys. The relative weight given to each of these advantages and disadvantages, and the final decision on whether to conduct a Web-based survey, will be determined by your nursing home's specific circumstances, capabilities, resources, and goals.

The primary advantages to Web-based surveys are:

• Simpler logistics. Web-based surveys can be virtually paperless, making them easier in some ways to manage. There are no surveys to print; no handling of letters, labels, envelopes, or postage; and no completed paper surveys to manage.
• No need for data entry and minimal need for data cleaning. Web-based surveys typically are programmed to prevent invalid responses. Moreover, the responses are automatically copied to a database, so the need for separate data entry is eliminated and the need for data cleaning is greatly reduced.
• Potential for faster data collection. Although not always the case, Web-based surveys can facilitate shorter data collection periods. Web-based surveys involving E-mail notification and followup correspondence are received immediately after being sent, so the time interval between survey administration steps often is reduced.

Web-based surveys also have several disadvantages:

• Time and resources needed for development and testing. Time and resources are needed to program a Web-based survey so that it meets acceptable standards of functionality, including usability requirements. Of equal importance are security safeguards for protecting the data. In addition, the Web-based survey must be pretested thoroughly to ensure that it works properly and that the resulting data set is established correctly.
• Limited access to the Internet or E-mail. A Web-based survey should be accessible to all the individuals in your sample group. If staff Internet access is limited, your nursing home will have poor response rates. If computers with Internet access are located centrally for staff to use to complete the survey, staff may be concerned about the privacy of their responses. In addition, all staff may not have E-mail access or may not access their E-mail regularly. In such cases, E-mail notification or E-mail messages with hyperlinks to the survey Web site may not be effective ways to get respondents to complete the survey.
• Individual differences in computer and Internet use. The intensity of computer and Internet usage is the most important predictor of cooperation in a Web-based survey (Groves, 2002). In nursing homes, it is likely that many staff will not be computer or Internet savvy. Therefore, they may not respond to the survey if this is their only way to complete the survey.

Design and Pretest the Web-Based Survey

If you decide after weighing the pros and cons of conducting a Web-based survey that this is the approach your nursing home or health care system will take, there are a number of Web survey design aspects to consider. If your nursing home plans to use off-the-shelf commercial software, rather than having a vendor design and develop a custom Web-administered survey, assess the various software applications available to you. Select the product that best accommodates the features and recommendations we outline below.

Web-Based Survey Design Features

Although research on the best ways to design Internet-administered surveys continues to evolve, current knowledge suggests that a good Web-based survey has the following elements:

• Do not force respondents to answer every question. Allow staff to continue completing the survey after choosing not to answer a particular question. Forcing respondents to answer each question before being allowed to move to the next question may annoy respondents. It also is not advisable on the Nursing Home Survey on Patient Safety Culture because some respondents may have legitimate reasons for not answering an item. Forcing a response may cause them to make a wild guess, rather than provide an informed answer. In addition, you will want the Web version to be similar to the paper version, which does not require an answer to every question.
• Provide respondents with a way to assess their survey progress (optional). Because it is difficult to know the length of a Web-based survey, it is sometimes helpful for respondents to have some type of indicator showing their overall progress in the survey. However, for a relatively short instrument like the Nursing Home Survey on Patient Safety Culture, we think a progress indicator is optional. If you prefer to use one, there are several ways to indicate progress in completing the survey. For example, you could use a graphical progress bar that indicates completion percentages at various points (e.g., "Survey is 50% complete."). Other options include allowing respondents to move forward and backward through a multiple-page format at their convenience so that they can view the entire length of the survey. If a multiple-page format is used, however, avoid using an extreme one-question-per-page design.
• Include username and/or password protection (optional). Unless access is restricted in some way, Web sites are accessible to the public. Your survey Web site can be restricted through the use of a password that is common to all users or groups of users. You also can use individual usernames and/or passwords (which require the use of confidential identifiers to link individuals to usernames/passwords). The use of passwords and/or usernames is also an effective way to ensure that respondents will not be able to complete the survey more than once, even when the survey is published to part of a restricted company or organization Intranet. We recommend providing usernames and/or passwords and hyperlinks to the survey Web site in all your E-mail survey notifications. Respondents will be able to click directly on the hyperlink, then copy and paste their individual username and/or password directly from the E-mail.
• Allow respondents to print a hard-copy version of the survey and complete it on paper (optional). Some respondents will prefer to complete a paper version of the survey, and providing this option may boost your response rate. It is possible to design your Web-based survey so that it can be printed in paper form, but this functionality must be tested thoroughly to ensure that it prints properly on different printers. Attention must be given to line lengths and page lengths in the design of the survey page template. Alternatively, you can include a link to a portable document file (PDF) version of the survey on the Web site. With either alternative, respondents will need instructions to know where to return the completed paper surveys, and designated personnel then must enter the responses into your data set (paper survey data can be entered via the Web site).
• Thoroughly pretest the survey (essential and mandatory). Conduct thorough pretests of the survey using low-end computers with slower Internet connections, with various Internet browsers (e.g., with different iterations of Internet Explorer, Safari, Mozilla, Opera), and with different display settings (screen resolutions set at 800 x 600 pixels versus 1152 x 864 pixels), and so forth. Pretesting will help to ensure that the survey appears and performs as it should, despite the different settings and personal preferences selected on individual computers. For more information on Web survey design principles and pretesting, go to Dillman, 2007.

Return to Contents

Develop a Web-Based Data Collection Plan

A Web-based survey data collection plan is similar to a paper-based data collection plan in its basic steps. Rather than reiterate all the necessary data collection steps in this section, we have chosen to highlight various steps and strategies unique to Web-based surveys, while offering advice on the best approaches.

A Combination of Web- and Paper-Based Survey If you prefer to use a combination of Web-based and paper survey approaches, it is most economical to implement the Web-based survey first. Later, you can distribute paper surveys to those who did not respond to the Web-based instrument.

Survey Administration Timeline

Although you may need to add several weeks to the planning/preparation period in the overall timetable for developing and testing the Web-based instrument, actual administration time for a Web-based survey may be shorter than for a paper-based survey. Go to the sample timeline in Figure B-1. The data collection period is 3 weeks from the time the E-mail invitation to participate in the survey is sent until the closeout of data collection.

Prenotification/Publicity

For a Web-based survey, we recommend E-mailing staff a prenotification letter telling them about the upcoming nursing home survey and alerting them that they will soon receive an invitation to complete this Web-based survey. Prenotification messages are easy to prepare and inexpensive to deliver in most Web-based surveys. Also, because E-mail messages can be easily overlooked in a crowded inbox, the prenotification message will help to alert providers and staff to expect the upcoming survey invitation E-mail. Prenotification messages do require an up-to-date list of the E-mail addresses for those individuals in your survey population.

To boost response rates, we recommend personalizing any E-mail prenotification messages (i.e., addressed to each respondent, using first and last name). If E-mail notification is used, the name or E-mail address in the "From" line should be easily recognizable to staff to prevent them from mistaking your E-mail for spam and deleting it. For example, you might use the title and name of the nursing home or health care system senior executive leader, or another recognized staff executive, to ensure that the E-mail gets opened and read (FROM: Dr. Joe Smith, with "X" Nursing Home, or FROM: Jane Smith, CEO, with Health System "X").

As we recommended with paper-based surveys, you should also publicize the survey by posting paper flyers in the nursing home and posting messages on office or health care system Intranet sites. In addition, you can promote the survey during staff meetings.

Followup

Followup steps improve response rates for Web-based surveys in the same way they help with paper surveys (Groves, 2002). It is important to follow up with nonrespondents in a timely manner to ensure that the data collection period does not drag on too long.

If you have the means to conduct all contact steps via E-mail, time intervals between followup steps can be reduced. Consider sending the first E-mail reminder within a few days after the survey Web site link has been E-mailed (rather than using a 1-week reminder, as recommended with a paper survey). Include the hyperlink to the survey Web site in each E-mail reminder, along with the individual's username and/or password, if applicable. Then send a second E-mail reminder 1 week after the first reminder. A third E-mail reminder can be sent the following week. Use a larger, colored font to make the heading of the reminder E-mail more noticeable, and ensure that the subject lines and texts of the first and second reminder messages are slightly different, to capture the recipients' attention.

If you have used individual identifiers and can determine which providers and staff have completed the survey, you can send reminder notices only to nonrespondents. However, if you did not use identifiers, reminders must be sent to everyone. It is important in such cases to include a sentence thanking those who have already completed their surveys and asking them to disregard the reminder.

We recommend using a combination of printed reminders and electronic reminders—even for those with the capabilities to conduct all contact steps through E-mail—to ensure that at least one of the messages reaches each eligible provider and staff member. Remember that individuals respond differently to various forms of communication. You may decide to send the first and second reminders via E-mail, then distribute a final reminder card or letter to nonrespondents. The final reminder card could be printed on brightly colored card stock, thanking those who have responded for their help and asking those who have not responded to please complete the survey in the next 7 days.

If all followup reminders are printed on paper and distributed to staff, more distribution time will be needed between data collection steps. The followup steps for a Web-based survey are the same as those associated with a paper survey (Go to Chapter 4: Establishing Data Collection Procedures).

Return to Contents

References

Dillman DA. Mail and Internet surveys: The tailored design method, 2nd edition. New York: Wiley; 2007.

Groves RM. Survey nonresponse. New York: Wiley; 2002.

Shih, T, Fan, X. Comparing response rates from Web and mail surveys: A meta-analysis. Field Methods 2008;20(3):249-71. Available at: http://fmx.sagepub.com/cgi/content/abstract/20/3/249.

Sorra J, Famolaro T, Dyer N, et al. Hospital Survey on Patient Safety Culture 2008 Comparative Database Report. (Prepared by Westat under contract No. 233-02-0087). Rockville, MD: Agency for Healthcare Research and Quality. March 2008. AHRQ Publication No. 08-0039. Available at: http://www.ahrq.gov/qual/hospsurvey08/

Return to Contents
Proceed to Next Section