Chapter 6. Preparing and Analyzing Data and Producing Reports
At the end of data collection, you will need to prepare the collected survey data for analysis. As mentioned in Chapter 2, you may want to hire a vendor for data entry or data analysis or to produce feedback reports for your medical office. If you decide to do your own data entry, analysis, and report preparation, use this chapter to guide you through the various decisions and steps. If you decide to hire a vendor, use this chapter as a guide to establish data preparation procedures. If you plan to conduct a Web-based survey, you can minimize data cleaning by programming the Web survey to perform some of these steps automatically. In addition, if you plan to administer the survey in more than one medical office, you will need to report the results separately for each participating medical office.
You or your vendor will need to accomplish a number of tasks to prepare the survey data for analysis. During the data preparation process, several data files will be created. It is important to maintain the original data file that is created when survey responses are data entered. Any changes or corrections should be made to duplicate files, for two reasons:
- Retaining the original file allows you to correct possible future errors made during the data cleaning or recoding processes, and
- The original file is important should you ever want to go back and determine what changes were made to the data set or conduct other analyses or tests.
Each survey needs to be examined for completeness before the survey responses are entered into the data set. Exclude surveys that were returned completely blank or those with only background demographic questions answered. In addition, you may want to visually or programmatically (during data cleaning) omit surveys where the respondent gave the exact same answer to all the questions in the survey. Because the survey includes negatively worded items, respondents should use both the positive and negative ends of the response scales to provide consistent answers. If every answer is the same, the respondent did not give the survey his or her full attention and the responses are probably not valid.
Return to Contents
After you have identified which returned surveys will be included in the analysis data file, you can use the following formula to calculate the official response rate:
Number of returned surveys – incompletes
Number of surveys distributed – ineligibles
This formula differs from that used for calculating preliminary response rates (shown in Chapter 4) only in the numerator. The numerator may be smaller than in your last preliminary response rate calculation because, during your examination of all returned surveys, you may find that some of the returned surveys are incomplete and you may have to exclude them from the analysis data file.
Return to Contents
In this section we describe several data file preparation tasks.
Illegible, Mismarked, and Double-Marked Responses
Some survey responses may need to be edited or coded before the data are entered into an electronic data file or statistical analysis program. Editing and coding involve decisionmaking regarding the proper way to enter ambiguous responses. These editing and coding steps will probably not be necessary if you are using a Web-based survey or scannable forms.
One potential issue is survey responses that are difficult to determine. For example, some respondents may write in an answer such as 3.5 when they have been instructed to circle only one numeric response. Or they may circle two answers for one item. Develop and document decision rules for these situations and apply them consistently. Examples of such rules are to use the highest response when two responses are provided (e.g., a response with both 2 and 3 would convert to a 3) or to mark all of these types of inappropriate responses as missing. After surveys have been edited and coded as necessary (most surveys will not need to be coded), you can enter the data directly into an electronic file by using statistical software such as SAS®, SPSS®, or Microsoft Excel®, or you can create a text file that can be easily imported into a data analysis software program.
Individual Identifiers on Your Data File
If you used identifiers (identification numbers or codes) on your surveys, after you close out data analysis, destroy any information linking the identifiers to individual names. You no longer need this information, and you want to eliminate the possibility of linking responses on the electronic file to individuals. After destroying the linkage information, you may enter the identification number in the electronic data file.
If no identifiers were used on the surveys, you will need to include some type of individual identifier in the data file. Create an identification number for each survey and write it on the hard copy surveys in addition to entering it into the electronic data file. This identifier can be as simple as numbering the returned surveys consecutively, beginning with the number 1. This number will enable you to go back and check the electronic data file against a respondent's original answers if there are values that look like they were entered incorrectly.
Open-Ended Comments
Respondents are given the opportunity to provide written comments at the end of the survey. Comments can be used to obtain direct quotes for feedback purposes. If you wish to analyze these data further, the responses will need to be coded according to the type of comment that was made. For example, staff may respond with positive comments about patient safety efforts in their medical office. Or they may comment on some negative aspects of patient safety that they think need to be addressed. You may assign code numbers to similar types of comments and later tally the frequency of each comment type. Open-ended comments may be coded either before or after the data have been entered electronically.
Return to Contents
After the surveys have been coded and edited as necessary and entered electronically, you will need to check and clean the data file before you begin analyzing and reporting results. The data file may contain data entry errors. You can check and clean the data file electronically by producing frequencies of response to each item and looking for out-of-range values or values that are not valid responses. Most items in the survey require a response between 1 and 5, with a 9 coded as Does Not Apply/Don't Know. Check through the data file to ensure that all responses are within the valid range (e.g., that a response of 7 has not been entered). If you find out-of-range values, return to the original survey and determine the response that should have been entered.
Return to Contents
Analyze the Data and Produce Reports of the Results
Feedback reports are the final step in a survey project and are critical for synthesizing the survey responses. Ideally, feedback should be provided broadly—to medical office management, health system boards of directors, medical office committees, and medical office staff, either directly during meetings or through centralized communication tools such as E-mail or newsletters. The more broadly the results are disseminated, the more useful the information is likely to become. The feedback also will serve to legitimize the collective effort of the respondents and their participation in the survey. It is gratifying and important for respondents to know that something worthwhile came out of the information they provided. Different types of feedback reports can be prepared for each different audience, from one- or two-page executive summaries to more complete reports that use statistics to draw conclusions or make comparisons.
Frequencies of Response
One of the simplest ways to present results is to calculate the frequency of response for each survey item. A Microsoft PowerPoint® presentation is available from the AHRQ Web site that you may use to communicate results from the Medical Office Survey on Patient Safety Culture (see sample on page 51).The feedback report template groups survey items according to the safety culture dimension each item is intended to measure. You simply insert your medical office's survey findings in the charts to create a customized feedback report. The two lowest response categories are combined (e.g., Strongly Disagree/Disagree or Never/Rarely), and the two highest response categories are combined (e.g., Strongly Agree/Agree or Most of the Time/Always) to make the results easier to view in the report. The midpoints of the scales are reported as a separate category (Neither Agree nor Disagree or Sometimes). The percentage of answers corresponding with each of three response categories then are displayed graphically—see the example below.
Figure 3. Sample Graph Displaying Frequencies of Response to an Item
Most of the survey's items include a Does Not Apply/Don't Know response option. This option is included so that staff who do not have enough information or do not know about a particular issue can select this answer rather than guessing or providing an answer about something they are not familiar with. In addition, each survey item will probably have some missing data from respondents who simply did not answer the question. Does Not Apply/Don't Know and missing responses are excluded when displaying percentages of response to the survey items. When using a statistical software program, you will recode the "9" response (Does Not Apply/Don't Know) as a missing value, so it is not included when displaying frequencies of response.
An example of how to handle the Does Not Apply/Don't Know and missing responses when calculating survey results is shown in Table 1. As Table 1 shows, respondents who answered Does Not Apply/Don't Know are treated the same way as those who did not answer the item (missing). The column labeled "Correct Percentages of Response" shows the correct percentage for each response option in the example. The column labeled "Correctly Combined Percentages" shows the correct percent negative, neutral, and positive scores, which do not include the Does Not Apply/Don't Know responses or missing responses.
The two shaded columns on the right in Table 1 labeled "Incorrect Percentages of Response" and "Incorrectly Combined Percentages" show the incorrect or wrong way to compute results if you were to mistakenly include the Does Not Apply/Don't Know responses as valid responses. Again, the easiest way to ensure that percentages are computed correctly is to recode all "9" responses to missing so that they are not included in the frequency and/or percent negative, neutral, and positive score calculations.
Item and Composite Percent Positive Scores
It can be useful to calculate one overall composite score for each dimension. To calculate your medical office's composite score on a particular safety culture dimension, simply average the percent positive response on each item that is included in the composite. Here is an example of computing a composite score for Office Processes and Standardization:
- There are four items in this composite—two are positively worded (items C9 and C15) and two are negatively worded items C8 and C12). Keep in mind that DISAGREEING with a negatively worded item indicates a POSITIVE response.
- Calculate the percent positive response at the item level (go to the example in Table 2).
In this example, there were four items with percent positive response scores of 46 percent, 52 percent, 46 percent, and 56 percent. Averaging these item-level percent positive scores (46% + 52% + 46% + 56% / 4 = 50%) results in a composite score of .50 or 50 percent on Office Processes and Standardization. That is, an average of about 50 percent of the respondents responded positively to the survey items in this composite.
Do Not Report Results If There Are Not Enough Respondents
To protect the confidentiality of individual responses, do not provide any type of survey feedback report for a medical office if fewer than five respondents have answered the survey. In addition, even if a medical office has five overall respondents, if fewer than three respondents answered a particular survey item, do not report percentages of positive, neutral, or negative response for that item—simply indicate that there were not enough data to report results.
|
There are placeholder pages in the Microsoft® PowerPoint® survey feedback report template for highlighting a medical office's strengths and areas for improvement with regard to patient safety issues covered in the survey. You may decide to define patient safety strengths as those positively worded items that about 75 percent of respondents endorsed by answering Strongly Agree/Agree or Always/Most of the Time (or, for negatively worded items, where 75 percent of respondents disagreed or responded Never/ Rarely). The 75 percent cutoff is somewhat arbitrary, and your medical office may choose to report strengths using a higher or lower cutoff percentage.
Similarly, areas needing improvement may be identified as those items that 50 percent or fewer respondents answered negatively (they either answered Strongly Disagree/Disagree or Never/Rarely to positively worded items, or they answered Strongly Agree/Agree or Always/Most of the Time to negatively worded items). The cutoff percentage for areas needing improvement is lower, because if half the respondents are not expressing positive opinions about a safety issue, improvement is probably needed.
It also is important to present information about the background characteristics of all the respondents—how long they have worked in their medical office, their staff positions, and so forth. This information helps others to better understand whose opinions are represented in the data. However, be careful not to report item results if the total number of respondents is fewer than 3, where it may be possible to determine which employees fall into those categories (e.g., 2 employees working in the office for 1 to 4 hours per week). You can choose to combine respondents from one or more response categories to ensure enough respondents in the group to report the statistic (e.g., report the number of respondents working from 1 to 32 hours per week).
Return to Contents
The Agency for Healthcare Research and Quality (AHRQ) has posted initial comparative data from the pilot study of the Medical Office Survey on Patient Safety Culture on its Web site (http://www.ahrq.gov/qual/hospculture ). In the future, AHRQ will ask all medical offices that have administered the survey to voluntarily submit their data files to the Medical Office Survey on Patient Safety Culture Comparative Database. This database will be modeled on the Hospital Survey on Patient Safety Culture Comparative Database, which contains comparative data for users of AHRQ's Hospital Survey on Patient Safety Culture. You will be able to compare your medical office results with the overall medical office comparative data.
When you submit your data file, you will be asked to provide some background information about the characteristics of your medical office, such as the type of practice, majority ownership, total number of providers (MDs, DOs, PAs, NPs, etc.) by specialty, etc. (Appendix B). This information may be used to conduct analyses of aggregated data files by selected medical office characteristics. Participating medical offices will not be identified by name. Only aggregate data will be reported, and only when there are sufficient data so that such aggregation will not permit reidentification of participating medical offices. If your medical office is interested in submitting its data to the medical office database, send an E-mail to DatabasesOnSafetyCulture@ahrq.hhs.gov.
Return to Contents
For free technical assistance on the Medical Office Survey on Patient Safety Culture regarding survey administration issues, data analysis and reporting, or action planning for improvement, you can E-mail SafetyCultureSurveys@ahrq.hhs.gov. AHRQ is also sponsoring periodic in-person User Group Meetings so that users of the medical office survey, along with users of the hospital and nursing home surveys, can come together to network and learn from one another.
Return to Contents
Dillman DA. Mail and Internet surveys: The tailored design method 2nd ed. New York: Wiley; 2007.
Groves RM. Survey nonresponse. New York: Wiley; 2002.
Shih T, Fan X. Comparing response rates from Web and mail surveys: A meta-analysis. Field Methods 2008: 20(3):249-71. Available at: http://fmx.sagepub.com/cgi/content/abstract/20/3/249.
Sorra J, Famolaro T, Dyer N, et al. Hospital Survey on Patient Safety Culture 2008 Comparative Database Report. (Prepared by Westat, Rockville, MD, under contract No. 233-02-0087). Rockville, MD: Agency for Healthcare Research and Quality: March 2008. AHRQ Publication No. 08-0039. Available at:
http://www.ahrq.gov/qual/hospsurvey08/ .
Return to Contents
Proceed to the Survey
540 Gaither Road Rockville, MD 20850