Advances in Patient Safety: From Research to Implementation Advances in Patient Safety: From Research to Implementation Volume 4. Programs, Tools, and Products

An Employee Questionnaire for Assessing Patient Safety in Outpatient Surgery

Pascale Carayon, Carla J. Alvarado, Ann Schoofs Hundt, Scott Springman, Amanda Borgsdorf, Peter L.T. Hoonakker

Abstract

This paper provides information on the reliability and validity of an employee questionnaire developed in a study of patient safety in outpatient surgery. The Systems Engineering Intervention in Outpatient Surgery (SEIPS), a collaborative community perspective project currently underway at the University of Wisconsin-Madison, examines the impact of a systems engineering intervention on both employees and patients. In this paper, we describe the SEIPS employee questionnaire, which surveys various elements of the work system (e.g., communication, workplace, supplies, and patient safety climate), the care process, and employee outcomes (e.g., job satisfaction, stress, perceived quality and safety of care provided). Data from a sample of 289 staff members in 5 outpatient surgery centers (53 percent response rate) are used to examine reliability, construct validity, convergent validity, and predictive validity. The results provided evidence for the reliability and validity of the SEIPS study's employee questionnaire.top link

Introduction

The Systems Engineering Intervention in Outpatient Surgery (SEIPS), a collaborative community perspective project currently underway at the University of Wisconsin-Madison, is applying the SEIPS work system model to patient safety in outpatient surgery centers. 1 The objective of the project is to examine the impact of a system intervention on the work system, employee and organizational outcomes, and quality and safety of patient care. The five major outpatient surgery centers in Madison are cooperatively participating in the study. All five sites are members of the Madison Patient Safety Collaborative, a group of local providers committed to improving patient safety in the community who have agreed to not use patient safety to their competitive advantage. In the course of this research study, all employees working in the five outpatient surgery centers were asked to voluntarily respond to an employee survey, designed to assess patient safety in outpatient surgery from the viewpoint of the employees. In this paper, we describe the steps used to develop the content, reliability, and validity of the survey.

There is a conceptual framework underlying the structure of the employee questionnaire. In the SEIPS model of work system and patient safety, we integrated Donabedian's 2 structure-process-outcome framework and the work system model. 3, 4 The structure of an organization ---or, more generally, the work system ---affects the care process, and the means of caring for and managing the patient (the care process) affects patient safety (patient outcome) and employee and organizational outcomes. 1 The employee questionnaire asks about various elements of the work system (e.g., communication, workplace, supplies, and patient safety climate), the care process, and employee outcomes (e.g., job satisfaction, stress, perceived quality and safety of care provided).top link

Methods

Study setting

In 2002, representatives of the five outpatient surgery centers discussed a joint research effort aimed at addressing quality and safety issues at their respective and combined centers. The "pilot team" of SEIPS researchers and representatives from each of the centers agreed that a baseline had to be established to determine the focus of the intervention they would ultimately implement at each center. Through an open-ended survey, medical, nursing, technical, and clerical staffs were asked to identify quality and safety issues at their centers, as well as working conditions issues that either interfered with or facilitated their ability to perform their jobs. 5, 6

From the information collected with the initial employee questionnaire, the pilot team agreed that each center could choose its own specific intervention, but that any intervention should address "coordination and communication of care" between providers prior to outpatient surgery. 7 The interventions were evaluated using two data collection instruments: an employee questionnaire and a patient telephone survey. Data were collected prior to the implementation of the intervention and again 12 months later. In this paper, we describe the employee questionnaire tool that we developed to evaluate the systems engineering intervention and report baseline data that were collected before implementation.top link

Sample

Employees (physicians, nurses, and other technical and administrative personnel) from the five outpatient surgery centers in Madison, Wisconsin, were asked to complete the questionnaire. A total of 531 questionnaires were distributed to the surgery centers' staffs; 289 surgery center employees participated, yielding an overall response rate of 53 percent (ranging from 48 --91 percent). Thirty-six percent of the respondents were physicians (60 percent of whom were surgeons, 40 percent anesthesiologists), and 38 percent were nurses. The rest of the sample included technicians, office personnel, and schedulers. Women represented 67 percent of the sample. The age distribution was as follows: younger than 25: 2 percent; 25 --34 years: 16 percent; 45 --54 years: 31 percent; 55 --64 years: 43 percent; and older than 65: 1 percent. The majority worked either 31 --40 hours per week (41 percent) or more than 41 hours per week (40 percent). Tenure with current employer was 9.6 years on average (standard deviation [SD] = 7.5 years), and the average number of years in current job was 8 (SD = 7.3 years). Ninety-six percent of respondents were white.top link

Employee questionnaire

The employee questionnaire 11 included a total of 71 questions covering the following domains:

Table 1 provides information on the source of the questions, the number of questions for each concept, and examples of the questions. Because the system intervention implemented in the outpatient surgery centers focused on improving communication and coordination within the centers, as well as between the centers and other units, the questions on the work system included in the survey specifically focused on several dimensions of communication and coordination.

Table 1. Description of the variables

Concept # of questions Source Example of question
Communication/ openness 4 Shortell et al.a Communication with the nursing staff in this unit is very open.
Communication/ accuracy 4 Shortell et al.a I can think of a number of times when I received incorrect information regarding patient care from nurses in this center.
Communication /timeliness 3 Shortell et al.a I get information on the status of patients when I need it.
Time pressure affecting patient safety 2 Singer et al.b I have enough time to complete patient care tasks safely.
Workload 1 Caplan et al.c How often is there a great deal to be done?
Staffing 1 Gray-Toft and Andersond How often is there not enough staff to adequately cover the center?
Coordination mechanisms 5 Shortell et al.a How effective are written rules, policies and procedures for the coordination of staff activities?
Workplace design 6 Shortell et al.a Your workplace is... ... unnecessarily noisy / reasonably quiet
Equipment design 6 Shortell et al.a The equipment you work with is... ... modern / outdated
Access to supplies 2 Shortell et al. a Supplies provided for your use are... ... usually out of stock / available when needed
Patient safety climate 17 (2 questions answered only by physicians) Singer et al.;b Gaba et al.;e Nieva and Sorraf I feel that it is just pure luck that more serious mistakes don't happen around here.
Unit effectiveness 5 Shortell et al.a Given our patient population and the procedures that we perform, our center's patients experience very good outcomes.
Satisfaction with care provided 1 Bertram et al.g In general, I am satisfied with the quality of care that I provide.
Job satisfaction 1 Quinn et al.h All in all, how satisfied would you say you are with your job?
Fatigue 2 McNair et al.i How have you been feeling during the past week... ...fatigued
Tension 2 McNair et al.i How have you been feeling during the past week......nervous
a Shortell SM, Rousseau DM, Gillies RR, et al. Organizational assessment in intensive care units (ICUs): construct development, reliability, and validity of the ICU nurse-physician questionnaire. Med Care 1991 Aug;29(8):709 --27.
b Singer SJ, Gaba DM, Geppert JJ, et al. The culture of safety: results of an organization-wide survey in 15 California hospitals. Qual Saf Health Care 2003 Apr;12(2):112 --8.
c Caplan RD, Cobb S, French JR, et al. Job demands and worker health; main effects and occupational differences. Cincinnati: National Institute for Occupational Safety and Health; 1975.
d Gray-Toft P, Anderson JG. The nursing stress scale: development of an instrument. J Behav Assess 1981;3(1):11 --23.
e Gaba DM, Howard SK, Jump B. Production pressure in the work environment. California anesthesiologists' attitudes and experiences. Anesthesiology, 1994 Aug;81(2):488 --500.
f Nieva VF, Sorra J. Safety culture assessment: a tool for improving patient safety in healthcare organizations. Qual Safe Health Care 2003 Dec;12(Suppl 2):ii17 --23.
g Bertram DA, Hershey CO, Opila DA, et al., A measure of physician mental work load in internal medicine ambulatory care clinics. Med Care 1990 May;28(5):458 --67.
h Quinn R, Seashore S, Kahn R, et al. Survey of working conditions: final report on univariate and bivariate tables. Washington, DC: U.S. Government Printing Office; 1971. Document No.2916-0001.
i McNair DM, Lorr M, Droppleman LF. EITS manual for profile of mood states. San Diego, CA: Educational and Industrial Testing Service; 1971.

11 A copy of the complete employee questionnaire is available from the corresponding author.top link

Study procedures

As the study subjects were drawn from five separate outpatient surgery centers with varying case mixes, types of employees, and a range of physical settings (i.e., in-hospital setting, free-standing clinics, and clinics in close proximity to the hospital), several data collection procedures were used for each site. The survey questionnaire was originally formatted in Microsoft Word[reg]. However, a Web-based format was offered in addition to the paper-based questionnaire, in hopes of catering to the needs and schedules of busy surgical center staff. The Microsoft Word paper questionnaire survey was converted to a Web-based survey using Macromedia Dreamweaver[reg] MX Education Version. The distribution of the personalized e-mail cover letters, the electronic questionnaire survey, and anonymous survey tracking was accomplished using WSMS1.1© Web Survey Mailer System. 8 The choice of the survey format (electronic or paper) for distribution was ultimately the individual decision of each surgical center's top management. All the outpatient surgical centers, with one exception, chose to distribute the paper questionnaire survey. One center distributed the Web-based survey to its physicians, reasoning that ease of computer-access surveys might increase the number of surveys completed by this group.

A member of the research team introduced and explained the outpatient surgery research project at each outpatient surgery centers' regularly scheduled staff meetings. Upon completion of the project presentation and discussion, the investigators explained and distributed the employee questionnaire to all surgery center staff. Attendance was taken at each site's staff meeting, and questionnaires were left for those absent employees to complete. The surgery centers' supervisory staff explained and distributed questionnaires to the absent employees at a later time. An institutional review board-required cover letter and information sheet, explaining the research project, accompanied each questionnaire. The surveys were returned via a locked, secure drop box at any of the five sites, the U.S. Postal Service, university interdepartmental mail, or via the Internet, if subjects chose to complete the Web survey. All nonphysician staff were given work time to complete and return the questionnaires. Completed questionnaires were retrieved from each site's locked drop box twice weekly by a research assistant or placed in self-addressed, sealed envelopes and returned to the principal investigator.top link

Data analysis

In this paper, we report data on reliability and validity for the measures of work system, perceived performance, and quality of working life. The analysis of reliability and validity of the patient safety climate is reported elsewhere. 9 Cronbach's alpha scores were used to assess the reliability of scales. We conducted confirmatory factor analysis, using structural equation modeling with a maximum likelihood procedure, in order to assess construct validity. This analysis was performed with the AMOS© software. For convergent validity, we compared answers to various scales and questions among three job categories: nurses, physicians, and other staff. In order to evaluate predictive validity, we examined the relationship between measures of the work system and the measures of perceived performance and quality of working life. This analysis was based on a correlational analysis and a series of stepwise regression analysis.top link

Results

Table 2 displays basic statistics of the variables. The variables had adequate range, and the Cronbach's alpha scores of most scales were satisfactory (above 0.70). The scales of communication timeliness and access to supplies both had a Cronbach's alpha score of 0.67.

Table 2. Basic statistics of the variables

Scale No. of items Mean (SD) Range Cronbach's alpha score Coding (meaning of high score)
WORK SYSTEM
Communication openness 4 4.06 (.62) 1.75 --5 .80 High comm. openness
Communication accuracy 4 2.07 (.75) 1 --4.25 .88 Low comm. accuracy
Communication timeliness 3 3.32 (.42) 2.33 --5 .67 High comm. timeliness
Time pressure affecting patient safety 2 4.01 (.73) 1.40 --5 .76 Low time pressure
Workload 1 3.80 (1.07) 1 --5 N/A High workload
Staffing 1 2.62 (1.32) 1 --5 N/A High staffing problem
Coordination effectiveness 5 3.76 (.68) 1.60 --5 .76 High effectiveness
Workplace design 6 3.06 (.79) 1 --5 .78 Little workplace problem
Equipment design 6 3.77 (.74) 1.83 --5 .83 Little equipment problem
Access to supplies 2 3.81 (.90) 1 --5 .67 Little supplies problem
PERCEIVED PERFORMANCE
Unit effectiveness 5 4.23 (.54) 1.40 --5 .81 High effectiveness
Satisfaction with care provided 1 4.42 (.61) 1 --5 N/A High satisfaction
QUALITY OF WORKING LIFE
Job satisfaction 1 1.41 (.54) 1 --4 N/A Low satisfaction
Fatigue 2 2.21 (.97) 1 --5 .84 High fatigue
Tension 2 1.65 (.81) 1 --5 .79 High tension
SD = standard deviation

The initial step of the confirmatory factor analysis involved the replacement of all missing data (some questions were not answered by some people) for each variable by the mean on that variable. The highest number of missing data values was, in eight cases, for the following question: "How effective are computerized information systems to the coordination of staff activities?" The confirmatory factor analysis involved two steps: (1) confirmatory factor analysis of each of the scales in order to find out whether the a priori model (concept) fits the data, and (2) a second-order factor analysis on two groups of scales (scales of communication and scales on environment/equipment/supplies) to find out whether each group of scales could be represented by only one underlying factor. This second procedure has as an additional benefit as compared to the first step because information can be retrieved from the scales (models) for which insufficient information is available (the models are "underidentified"). Table 3 shows the results of the confirmatory factor analysis. In the analysis we allowed for covariance between items that belong to the same scale, but not for covariance between items that belong to different scales.

Table 3. Results of the confirmatory factor analysis

Model X2 DF GFI AGFI CFI PGFI RMR RSMEA
Communication openness 1.1 1 .998 .981 1.000 .100 .005 0.02
Communication accuracy 13.2 1 .978 .782 0.982 .180 .057 .205
Communication timeliness - - - - - - - -
Communication model with one factor 111.4 32 .931 .877 .954 .522 .041 .083
Communication model with three factors 132.4 36 .921 .859 .940 .504 .060 .096
Coordination mechanisms 5.8 4 .992 .970 .993 .265 .018 .040
Workplace design 8.5 6 .990 .966 .995 .283 .030 .038
Equipment design 18.4 7 .979 .936 .982 .326 .036 .075
Access to supplies - - - - - - - -
Design model with one factor 272.3 73 .886 .836 .870 .616 .090 .097
Design model with three factors 168.2 69 .926 .887 .935 .608 .070 .070
Unit effectiveness 5.8 4 .992 .970 .996 .265 .011 .039
Quality of working life 11.9 4 .984 .940 .984 .262 .019 .083
X2 (chi square) = difference between model and data
DF = degrees of freedom (indication of the complexity of the model)
GFI = goodness-of-fit index
AGFI = adjusted goodness-of-fit index (fit index that takes model complexity into account)
CFI = comparative fit index (fit index that takes sample size into account)
RSMEA = root mean square error of approximation (takes error of approximation in the population into account)
RMR = root mean square residual (represents the average value across all standardized residuals)
Nonsignificant X2 and goodness-of-fit indices in the 0.90s accompanied by parsimonious fit indices in the 0.50s are not unexpected. (Mulaik SA, James LR, Van Alstine J, et al. Evaluation of goodness-of-fit indices for structural equation models. Psychol Bull 1989;105:430 --45.)

The results show that, in general, the models fit the data well, although not always optimally. Results of the analysis of the three communication scales (i.e., communication openness, communication accuracy, and communication timeliness) show that a model with one underlying factor fits the data slightly better than a model with three factors. The results of the analysis of the scales on environment/equipment/supplies show that three underlying factors fit the data much better than a model with only one underlying factor. The results show that most of the models could not be improved by allowing for covariance between items that do not belong to the same scale. This is an indication of the construct validity of the items and scales. However, there was one exception to this finding: the fit of the model for the questions on the environment, equipment, and supplies can be improved by allowing for covariance (a crossloading) between question number 27 (good versus poor layout of the work place) and question number 31 (modern versus outdated equipment).

In order to examine convergent validity, we compared answers to the various measures across the three job categories of nurses, physicians, and other staff. A multivariate analysis of variance showed that the three groups were different on two of the three groups of variables: measures of work system (Wilks' lambda = 0.605, P < 0.001), measures of perceived performance (Wilks' lambda = 0.994, not significant), and measures of quality of working life (Wilks' lambda = 0.949, P < 0.05). For the measures of work system, all measures except the measure of coordination effectiveness displayed differences between the three job categories. In general, physicians had more positive perceptions of the work system than nurses and other staff (Figure 1). As for quality of working life, univariate tests were statistically significant for job satisfaction (P < 0.05) and approached significance for fatigue (P = 0.06). Physicians were more satisfied with their job and reported less fatigue than nurses and other staff.top link

Figure 1. Comparison of three job categories (nurses, physicians, and other staff)

top link

Predictive validity was examined by conducting a correlation analysis between measures of the work system and the measures of perceived performance and quality of working life (Table 3). Results showed that many of the measures of the work system showed a statistically significantly correlation with most measures of perceived performance and quality of working life. Communication openness, time pressure affecting patient safety, coordination effectiveness, workplace design, and equipment design showed a statistically significant correlation with both measures of perceived performance and the three measures of quality of working life. The other measures of work system (except for workload) showed correlation with the measures of perceived performance. Job satisfaction was correlated with all the measures of work system, fatigue was correlated with all the measures of work system except communication timeliness, and tension was correlated with only six measures of work system.

We also conducted a series of stepwise regression analyses with each of the measures of perceived performance and quality of working life as the dependent variables, and the 10 measures of work system as independent variables (Table 4). The measures of work system predicted a significant amount of variance of unit effectiveness (33 percent), satisfaction with care provided (39 percent), job satisfaction (25 percent), fatigue (19 percent), and tension (5 percent). The measures of work system that were the most consistent predictors of patient safety were communication openness and time pressure. When communication was reported to be open, the survey participants also reported high unit effectiveness, satisfaction with care provided, job satisfaction, and low tension. High time pressure affecting patient safety contributed to low unit effectiveness and job satisfaction, and high fatigue.

Table 4. Correlation and regression analyses between measures of work system (independent variables) and perceived performance and quality of working life (dependent variables)

Perceived performance Quality of working life
Work system Unit effectiveness Satisfaction with care provided Job satisfaction Fatigue Tension
CORRELATION ANALYSIS
Communication openness .52*** .42*** -.37*** -.21*** -.18**
Communication accuracy -.34*** -.32*** .30*** .16** .07
Communication timeliness .19*** .22*** .23*** -.04 -.06
Time pressure affecting patient safety .39*** .59*** -.36*** -.26*** -.19**
Workload -.03 -.05 .17** .25*** .13*
Staffing -.14* -.12* .27*** .13* .10
Coordination effectiveness .39*** .28*** -.31*** -.23*** -.15*
Workplace design .24*** .19*** -.35*** -.31*** -.19***
Equipment design .41*** .24*** -.41*** -.20*** -.18**
Access to supplies .32*** .24*** -.31*** -.21*** -.11
STEPWISE REGRESSION ANALYSIS
Adjusted R2 33%*** 39%*** 25%*** 19%*** 5%***
Significant predictors (beta-coefficients) Comm. open (.38) Pressure (.54) Equipment (-.23) Workplace (-.20) Workplace (-.16)
Supplies (.14) Comm. open (.27) Pressure (-.19) Workload (.21) Comm. open (-.14)
Coord. (.17) Workplace (-.13) Comm. Open (-.17) Coord. (-.15)
Pressure (.13) Staffing (.14) Pressure (-.14)
*** p<0.001
Discussion

Data reported in this paper provide evidence for the reliability and validity of the SEIPS study's employee questionnaire. The Cronbach's alpha scores for all the scales except two were above 0.70, which demonstrates acceptable reliability. 10 Two scales had Cronbach's alpha scores of 0.67, very close to the 0.70 limit.

The confirmatory factor analysis using structural equation modeling shows overall, the scales of the work system (three scales on communication, one scale on coordination, three scales on environment/equipment/supplies), the scale of unit effectiveness, and the measures of quality of working life demonstrated construct validity. A few results, however, provide some indication that construct validity was not fully achieved. The model combining the three scales of communication yielded a slightly better fit of the data as compared to a model keeping the three scales separate. Because our systems engineering intervention focuses on various aspects of communication, we decided to keep the three scales of communication separate. It is likely that the intervention affects some of the communication dimensions, but not others. Another unexpected result concerns the covariance (a cross-loading) between question number 27 (good versus poor layout of the work place) and question number 31 (modern versus outdated equipment). This result is difficult to interpret, given the very different concepts tapped by those two questions.

Convergent validity was examined by comparing responses of the three job categories (nurses, physicians, and other staff) on three groups of variables, i.e., work system, perceived performance, and quality of working life. We found evidence of convergent validity for the measures of the work system and the measures of quality of working life. In general, physicians reported more positive perceptions of the work system and higher quality of working life than nurses and other staff. There was no difference between the three job categories on the measures of perceived performance ---i.e., unit effectiveness and satisfaction with care provided. The lack of differences between nurses, physicians, and other staff on these measures actually demonstrates their similarity in reporting information on the quality and safety of care provided by their surgery centers, as well as themselves.

In order to assess predictive validity, we examined the relationships between the work system and perceived performance and quality of working life. The measures of the work system explained a significant proportion of the variance for all measures of perceived performance (33 percent and 39 percent) and all measures of quality of working life (5 --25 percent). In particular, the measures of communication openness and time pressure affecting patient safety were strong predictors of perceived performance and quality of working life. The systems engineering intervention implemented in our SEIPS project tackles the issue of communication and therefore should impact one of these two important work system characteristics.

A major weakness of the data used to examine the reliability and validity of the SEIPS study employee questionnaire is the cross-sectional nature of the study design. All data were collected at one point in time; the researchers were not able to evaluate relationships over time ---in particular, relationships between work system and perceived performance and quality of working life. However, since our SEIPS study involves the implementation of a systems engineering intervention, 7 we will be able to see whether the employee questionnaire can capture changes in perceptions of work system, perceived performance, and quality of working life. This, of course, assumes that the systems engineering intervention is successful at actually implementing changes in the work system and processes.

The employee questionnaire provides a unique opportunity to hear from health care providers. According to our conceptual framework, 1 we collected information on the work system, perceived performance (including unit effectiveness and satisfaction with care provided), and quality of working life. We have demonstrated how a structured method (i.e., the employee questionnaire) can be developed for assessing perceptions and opinions from the outpatient surgery centers.top link

Conclusion

Various methods can be used to gather input from health care providers regarding the quality and safety of care provided by their organizations. Interviews, focus groups, and questionnaires are some of these methods. In this study, we chose to use a questionnaire approach in order to collect structured, quantitative information on the work system and various outcomes (e.g., perceived unit effectiveness and job satisfaction). The data presented provide evidence for the reliability and validity of our employee questionnaire. Further steps of our research project involve the implementation of a systems engineering intervention that will be evaluated by this employee questionnaire, as well as a patient survey.top link

Acknowledgments

This research is funded by AHRQ Grant # P20 HS11561-01 (Principal Investigator Pascale Carayon). We would like to thank the five outpatient surgery centers that participated in this study.top link

Author affiliations

All authors are affiliated with the University of Wisconsin-Madison.

Address correspondence to: Pascale Carayon, Ph.D., University of Wisconsin-Madison, 610 Walnut Street, 575 WARF, Madison, WI 53726; phone: 608-265-0503 or 608-263-2520; e-mail: carayon@engr.wisc.edu.

References

1. Carayon P, Alvarado CJ, Brennan P, et al. Work system and patient safety. In: Luczak H, Zink KJ, editors. Human factors in organizational design and management VII. Santa Monica, CA: IEA Press: 2003. pp. 583 -- 9.

2. Donabedian A. The quality of care. How can it be assessed? JAMA 1988. 260(12):1743-8. (PubMed)

3. Smith MJ, Carayon-Sainfort P. A balance theory of job design for stress reduction. Int J Ind Ergon 1989. 4:67-79.

4. Carayon P, Smith MJ. Work organization and ergonomics. Appl Ergon 2000. 31:649-62. (PubMed)

5. Carayon P, Alvarado CJ, Hundt AS, et al. Patient safety in outpatient surgery: the viewpoint of the healthcare providers. Submitted for publication 2004. (Note: this paper is available from the author.)

6. Carayon P, Gurses AP, Hundt AS, et al., Performance obstacles and facilitators of healthcare providers. In: Korunka C and Hoffman P, editors, Change and quality in human service work. Vol. 4. Munchen, Germany: Hampp Publishers; 2005.

7. Carayon P, Hundt AS, Alvarado CJ, et al., Implementing a systems engineering intervention for improving safety in outpatient surgery. In: Henricksen K, Battles J, Marks E, et al., editors. Advances in patient safety: from research to implementation. Vol. 3, Implementation issues. AHRQ Publication No. 05-0021-3. Rockville, MD: Agency for Healthcare Research and Quality; Feb 2005.

8. Barrios E. Web Survey Mailer System (WSMS1.1). In: CQPI Technical Report Series. Madison, WI: University of Wisconsin-Madison Center for Quality and Productivity Improvement; 2003.

9. Alvarado CJ, Carayon P, Hundt AS. Patient safety climate (PSC) in outpatient surgery centers. Presented at the Human Factors and Ergonomics Society Conference, 2004; New Orleans.

10. Nunnally JC. Psychometric theory. 2nd ed. New York, NY: McGraw-Hill; 1978.