Guidance for Industry and FDA Premarket and Design Control Reviewers - Medical Device Use-Safety: Incorporating Human Factors Engineering into Risk ManagementDocument issued on July 18, 2000
This document replaces the draft guidance document of August 3, 1999,
entitled
Device Use Safety: Incorporating Human Factors in Risk Management.
 |
U.S. Department of Health and Human
Services |
Comments and suggestions may be submitted at any time for Agency consideration to: Ron Kaye or Jay Crowley at 1350 Piccard Dr. (HFZ 230), Rockville, MD 20850. Comments may not be acted on by the Agency until the document is revised or updated. For questions regarding the use or interpretation of this guidance contact Ron Kaye or Jay Crowley at (301) 443-2436 or by electronic mail at: HFSO@cdrh.fda.gov.
Additional copies can be obtained from the Center for Devices and Radiological Health's (CDRH) World Wide Web site at http://www.fda.gov/cdrh/HumanFactors.html or CDRH's Facts-on-Demand at 1-800-899-0381 or 301-827-0111 (specify number 1497 when prompted for the document shelf number).
Medical Device Use-Safety: Incorporating Human Factors
Engineering
into Risk Management
Identifying, Understanding, and Addressing Use-Related Hazards
| Authors: | Ron Kaye Jay Crowley |
Center for Devices and Radiological Health
Office of Communication, Education, and Radiation Programs (OCER)
Division of Device User Programs and Systems Analysis
Table of Contents
1.2 Use Scenarios Resulting in Hazards
3.1 HF in the Use of Medical Devices: Overall Considerations
3.2 Human Factors Considerations for the Device-User System
4.0 Advantages and Level of Effort
5.0 Apply Human Factors Engineering (HFE) Approaches Within the Risk Management Process
5.2 User Interface Design Information in Standards and Guidelines
5.3 Apply Analytic and Empirical Approaches to Identify and Understand Use-Related Hazards
5.5 Empirical HFE Approaches (Use Studies)
5.6 Prioritize and Assess Use-Related Hazards
5.7 Mitigate and Control Use-Related Hazards
5.8 Verify and Validate User Interface Design
6.0 Document Risk Management Activities for Device Use
6.7 Verification and Validation
7.0 References for Further Reading
This guidance1 describes how hazards related to medical device use should be addressed during device development as part of the risk management process. Potential use-related hazards are best identified and addressed using human factors engineering (HFE)2. The process of incorporating these approaches in the risk management processes is explained. Documenting these efforts can demonstrate that the device manufacturer has undertaken efforts to control use-related hazards. The goal is to minimize use-related hazards, assure that intended users are able to use medical devices safely and effectively throughout the product life cycle, and to facilitate review of new device submissions and design control documentation.
Addressing use-related hazards should be undertaken within the context of a thorough understanding of how a device will be used. Essential components of this understanding include:
Following a thorough understanding of device use, specific ways that devices could be used that are likely to result in hazards should be identified and investigated through analysis and testing. In addition to investigating known or suspected problems with device use, testing prototype devices with users can identify ways of using devices that could be hazard-related that were not anticipated. This is important because it is extremely difficult to identify all significant device use problems in advance.
_____________________
1This document is intended to provide guidance. It
represents the Agency's current thinking on this topic. It does not create or confer
any rights for or on any person and does not operate to bind FDA or the public. An
alternative approach may be used if such approach satisfies the requirements of the
applicable statute, regulations, or both.
2The term Human Factors Engineering and its acronym
(HFE) are used extensively in this document. On occasion the term Human Factors (HF) is
also used. The two terms distinguish between application (HFE), and the scientific
principles and academic research that provides the basis for it (HF).
After use-related hazards are understood, the hazards are mitigated or controlled by modifying the device user interface (e.g., control or display characteristics, logic of operation, labeling) or the abilities of users to use the device (e.g., training, limiting use to qualified users). The field of human factors provides a variety of useful approaches to help identify, understand, and address use-related problems.
This guidance does not focus on any specific kind of medical device, but applies to all medical devices and accessories that involve interaction with users (e.g., thought, perception, decision-making, and manipulation with hands). It is intended for medical device manufacturers, the Food and Drug Administration (FDA)’s Center for Devices and Radiological Health (CDRH) reviewers of pre-market submissions and design controls, and as a general reference for post-market surveillance activities associated with use-related hazards. It is assumed that readers have some understanding of design controls, risk management and HFE. Some readers might find it helpful to review references 10, 16, 19, 28 and 33 listed in Section 7.0.
A hazard is a potential source of harm. Hazards arise in the use of medical devices due to the inherent risk of medical treatment, from device failures (or malfunctions), and from device use. Hazards resulting from medical devices impact patients, family members, and professional healthcare providers. This document addresses hazards resulting from interactions between users and devices. It does not focus on hazards inherent to medical treatment or caused by device failure.
Hazards associated with device use are a common and serious problem3. Evidence from researchers (Cooper, Leape, and others) suggests that the frequency and consequence of hazards resulting from medical device use might far exceed those arising from device failures. Therefore, it is essential to ensure safe and effective device use if all hazards are to be controlled effectively. An Institute of Medicine report (reference 19, section 7.0) released in November 1999 estimates that as many as 98,000 people die in any given year from medical errors that occur in hospitals, which is more than the number who die from motor vehicle accidents, breast cancer, or AIDS. Though many of these errors are not related directly to the use of medical devices, some are, and the importance of incorporating HFE principles into device design to reduce device related medical errors was highlighted.
Medical device designers are interested in developing highly reliable devices. To do this, they consider the possibilities of hazards arising from failures of the device and its components. These kinds of failures can be identified through conventional reliability analyses. Designers need a more complete and accurate understanding of device use and approaches to include consideration of unique limitations and failure modes of device users as critical components of the device-user system. Relatively few user actions that can cause the device to fail other than the most apparent (e.g., fire or explosion), or well-known instances of use problems are considered by designers. This limitation during device design increases the likelihood of unexpected use scenarios (see Section 1.2) and use-related hazards for users and patients.
__________________________
3Incidents of death or serious injury resulting from
how a device is used, "user error" are reportable events under FDA's Medical
Device Reporting (MDR) program.
Hazards typically considered in risk analysis include:
These hazards most often result from instances of device or component failure that are not dependent on how the user interacts with the device, unless the way a device is used causes it to fail. In addition to the hazards mentioned above, there are certain kinds of hazards for medical devices that often result from device use. Hazards caused specifically by how a device is used are referred to in this document as use-related hazards (Figure 1). These include misdiagnoses (e.g., failure to identify disease or measure physiological parameters accurately), failure to recognize and act on information from monitoring devices, and improper treatment (e.g., ineffective or dangerous therapy).
Figure 1. Device Failure Hazards and Use-Related Hazards
Use-related hazards occur for one or more of the following reasons:
Use-related hazards often occur as a result of a sequence or chain of events involving device use. For instance, a user might not understand the calibration procedure required for a home-testing device. The user could calibrate it incorrectly, use the device and then act on the results it provides. Although the user’s technique following the calibration might be appropriate, inaccurate results are obtained because the device was not calibrated accurately. Decisions involving a patient’s health could be made on these inaccurate results. This use scenario involves the failure to understand the procedure, the incorrect calibration and the subsequent use of the device and the inaccurate results. In this example, a hazard occurred due to how the device was used. In this document, the concept of "use scenarios that result in hazards" refers to problematic use of the device in its entirety. An alternate terminology could be "failures" or "faults" in which scenarios resulting in hazards can be viewed essentially as failures or faults that occur during the interaction between the user and the device.
The guidance presented in this document describes how HFE approaches can be integrated into Risk Management to help identify, understand, control, and prevent failures that can result in hazards when people use medical devices. Risk management is a systematic application of policies, procedures, and practices to the analysis, evaluation, and control of risks. It is a key component of quality management systems, and is a central requirement of the implementation of Design Controls in the Quality Systems Regulation. Risk management involves the identification and description of hazards and how they could occur, their expected consequences, and estimations or assessments of their relative likelihood. The estimation of risk for a given hazard is a function of the relative likelihood of its occurrence and the severity of harm resulting from its consequences. Following the estimations of risk, risk management focuses on controlling or mitigating the risks.
Estimates of the risk of use-related hazards can be difficult to make. They can also be misleading. Problems with device use that could result in hazards are often difficult to anticipate due to the many ways and conditions under which users interact with devices. This causes some use-related hazards to not be included in risk management. Also, when they are anticipated, their true likelihood is difficult to estimate analytically. Even use-studies can be misleading measures of likelihood because the rate of use-related hazards observed is likely to be less than in actual use (Section 5.3.1). From a perspective of human factors engineering (HFE) in medical device use, the risk associated with low-likelihood use-related hazards could be misleading because users are often less able to react appropriately to situations that occur infrequently. Therefore, it is important to carefully consider the severity of harm in the management of device use risks.
Thorough consideration of use-related hazards in risk management processes should include the following tasks:
This process will be discussed in Section 5 in conjunction with HFE approaches.
To understand use-related hazards, it is necessary to have an accurate and complete understanding of how a device will be used. Understanding and optimizing how people use and interact with technology is the subject of human factors engineering (HFE). HFE considerations important to the development of medical devices include device technology, the users, environment in which the technology will be used, how dangerous device use is, and how critical the device is for patient care. An introduction to human factors (HF) considerations for medical devices can be found in the FDA document, Do It By Design.
Several general HFE concepts should be considered before proceeding with a discussion of HFE approaches in the context of risk management.
3.1.1 User Preference does not Necessarily Indicate Safety and Effectiveness A focus solely on user preference in the development of a design does not assure that safety and effectiveness have been adequately considered. Users generally prefer devices that are easy and satisfying to use and are aesthetically pleasing. Too often, device manufacturers and users emphasize these device characteristics at the expense of safety and effectiveness.
Although design features that assure safety and effectiveness could decrease user preference in some instances, they are necessary nevertheless. For instance, safety-related user interface design features such as shields over critical controls, mechanical or software-based interlocks, or verification requirements could slow down the use of a device or effect its aesthetics.
3.1.2 Use Scenarios with a Low Frequency of Occurrence that Result in
Hazards Require Careful Consideration
Rare or unusual use scenarios resulting in hazards with serious consequences often prove
to be the greatest threat to safe and effective medical device use after a device becomes
available for general use. Users are often not prepared for infrequent, unexpected use
scenarios because they are often not dealt with adequately in device design, training, or
operating instructions. Infrequent but dangerous use scenarios are often difficult to
identify, which underscores the necessity for careful application of the analytic and
empirical approaches (Section 5.5.) early in, and throughout
the design process.
3.1.3 Direct Inspection or Paper-based Analyses of a Device Might not
Identify all Hazards
Use-related hazards involve interactions among aspects of the use environment, user, and
the device (see Figure 2). Many hazards involving unsafe or
ineffective device use can be identified through careful inspection and analyses of
existing information pertaining to the use of similar devices. Some use scenarios are
rare. Some involve unusual or unexpected ways of interacting with a device, or involve use
in unusual circumstances. Use scenarios of this kind are difficult to identify by using
only analytic approaches (see Section 5.4). Therefore, it is
important to obtain information from the intended user population and test devices under
actual or simulated use conditions (see Section 5.5).
Safe, effective, or unsafe, ineffective use of medical devices is determined by the following major components of the device-user system: (1) Use Environments, (2) User Characteristics, and (3) Device User Interface Characteristics. This interaction and its possible results is depicted graphically in Figure 2.
Figure 2. Interaction of HF Considerations Results in: (1) Safe and Effective Use, or (2) Unsafe or Ineffective Use
3.2.1 Medical Device Use Environments
Use environments for medical devices can vary widely and can have major impacts on device
use and use-related hazards. The amount of thinking and concentration a person exerts
while using a device is called mental workload. The mental workload imposed on
users by the environment in which they use devices can exceed their abilities to use
devices properly. For instance, in an operating room, there could be too many alarms on
different devices for an anesthetist to be able to identify the source of any single
alarm. Mental workload is often used synonymously with mental "stress". There
can be a physical component to workload associated with medical device use (physical
workload) that also adds to the stress experienced by the user. Under high stress
levels, the user is distracted and will have less time to make decisions, consider
multiple device outputs, follow complex operating logic, or physically manipulate device
components. Devices that can be used safely under conditions of low stress (i.e., low
workload) could be difficult or dangerous to use under conditions of high stress.
Use environments can also limit the effectiveness of visual and auditory displays (lighted indicators, auditory alarms and other signals) if they are not designed appropriately. If the users cannot understand critically important information, errors are likely. For devices used in noisy environments, the user might not be able to notice alarms if they are not sufficiently loud or distinctive. When multiple alarms occur for different devices or on the same device, the user could fail to notice them or to make important distinctions among them. Similarly, motion and vibration can affect the degree to which people are able to perform fine physical manipulations such as typing on the keyboard portion of a medical device. Motion and vibration can also affect the ability of users to read displayed information.
Important considerations for displays (including visual alarm indicators) and device labeling include ambient light levels, viewing angles, and the presence of other devices in the use environment. If the device will be used in low light conditions, display scales or device status indicators might not be clear to the user. Some scales will be read inaccurately when viewed from an angle due to parallax or because part of the display is blocked. Other display information can be lost under brightly-lit conditions due to insufficient contrast. When certain types of equipment are used in close proximity with other devices, it could be difficult for users to associate visual displays and auditory signals with the corresponding equipment. With too much distraction, important information could be missed.
3.2.2 Medical Device Users
A device that is easy for one person to use safely and effectively might present problems
for another person. Similarly, a device that is easy for a certain group of users to use
safely and effectively could be difficult for another group. Users need devices that they
can use safely and effectively. To assure that these needs are met, it is necessary to
understand abilities and limitations of the intended users.
It is convenient to refer to the group of users who use a given device as its user population. It is then helpful to describe the user population with respect to the abilities and limitations of its members. For any device, the abilities and limitations of the user population might be relatively uniform. On the other hand, the user population might contain sub-components that have significantly different abilities. Examples are young and old users, or home users and professional healthcare providers. Fatigue, stress, medication, or other temporary mental or physical conditions can temporarily effect ability levels of device users.
Important characteristics of user populations include:
For example, older users might have difficulty remembering specific sequences for operation, using their hands to do tasks that require fine manipulation, or sensing device outputs such as auditory alarm sounds or information displayed visually. Highly trained and motivated users (i.e., developers, sales personnel, participants in previous use-studies, expert users) are often much more capable of operating complex devices than typical users. They are also likely to adapt better to unexpected or variable circumstances. Motivated and adaptable users are more likely to take actions to compensate for problems with the design of a device. But, if the same device is placed in the hands of more typical users, unexpected use scenarios possibly resulting in hazards could occur.
With proper application of HFE, the design of a device can often be made to compensate for limitations in user ability. For example, diabetics often suffer from some degree of retinopathy (degenerative disease of the retina) resulting in impaired eyesight. These users have difficulty reading the results of blood glucose test kits when the meter displays are very small. Blood glucose meters with small displays were not a good design for this user population. After this problem was understood, subsequent models with larger displays mitigated this hazard.
User experience and expectations are important considerations. Users will expect devices and device components to operate in ways that are consistent with their experience with other similar devices or device interface components. For example, users are likely to expect that the flow-rate of a given substance (such as a gas or liquid flow) will increase by turning a control knob counter-clockwise. Hazards result when an electronically driven device control operates in the opposite direction.
3.2.3 Medical Device User Interfaces
HFE considerations relate directly to the device user interface and responses of
the device to user actions. A well-designed user interface will facilitate correct actions
and will prevent or discourage actions that could result in hazards.
The user interface includes all components of a device with which users interact while using it, preparing it for use (e.g., calibration, set-up, unpacking), or performing maintenance (e.g., repairing, cleaning). It includes hardware features that control device operation such as switches, buttons, and knobs and device features that provide information to the user such as indicator lights, displays, auditory, and visual alarms. The user interface also includes the logic that directs how the system responds to user actions including how, when, and in what form information (feedback) is provided to the user. An important aspect of the user interface is the extent to which the logic of information display and control actions is consistent with users’ abilities, expectations, and likely behaviors.
Increasingly, user interfaces for new medical devices are computer-based. In these cases, interface characteristics include: the manner in which data is organized and presented, control and monitoring screens, screen components, prompts, navigation logic, alerting mechanisms, data entry requirements, help functions, keyboards, mouses, and pointers. The size and configuration of the device are important parts of the user interface, particularly for hand-held devices. Device labeling, packaging, training materials, operating instructions, and other reference materials are also considered part of the user interface.
An important concept pertaining to user interface use-safety is error tolerance. Error tolerance is the quality of a user interface that prevents or mitigates dangerous or disastrous consequences when an error occurs. Humans make errors. Some kinds of error can be anticipated and are essentially unavoidable – such as inadvertently pressing an adjacent key on a keypad, or bumping the keypad inadvertently while doing other tasks. The application of HFE approaches to device design will increase the likelihood that the design is tolerant of errors that are likely to be made by users. There are many ways to do this; one example is the placement of a shield over the button that initiates a beam of radiation to prevent inadvertent activation. The logic of device operation can also determine its degree of error tolerance. For example, some devices include "interlocks," or mechanisms that prevent a critical process from being initiated without users verifying their intent to initiate it or necessitating extra control steps to be performed before proceeding. In other cases, devices can be designed to do tasks that users do not do well, such as timing certain steps in home-testing procedures, remembering set-up parameters, or test dates, or performing calculations. For complex procedures, devices can prompt users to perform the appropriate action at critical points in the procedure.
The advantages of addressing use-related hazards through application of human factors engineering (HFE) in risk management extend beyond improved safety. Device manufacturers have found competitive advantages from the application of HFE in the design of their products. Also, these efforts reduce the necessity for modifications during implementation and reduce costly updates. When HFE approaches are used in the design of devices, particularly if the perspective of users is obtained, the overall ease of use and aesthetics of a device can be improved with the same effort. Users appreciate medical devices that are easy to use, if they also know the devices are safe. With increased safety, the likelihood of incurring expenses associated with recalls or liability is also reduced. For the process to be integrated well, personnel conducting HFE efforts should be integrated into the design and risk management team.
The type and extent of HFE in design and risk management efforts necessary to control risk associated with device use will vary. Effort applied to identification, description, and mitigation of use-related hazards scenarios should be determined by reasonable assessment of the potential harm of each scenario. In general, the set of scenarios to be considered should be kept manageable, although care should be taken not to dismiss scenarios involving atypical, unexpected, or unusual device use that could result in serious consequences.
The central question to be answered in use-related hazard identification and control efforts is: "Can the intended users use the device safely and effectively?" The processes necessary to answer this question are described in the following section. For some devices, relatively small efforts could be adequate to answer the central question while others will require more effort. The extent of effort required for a given device is often difficult to estimate accurately prior to beginning the process of incorporating HFE into risk management. The variability in approach and the amount of effort required results from the unique characteristics of devices, their expected use, characteristics of the population of users, and the risks of use-related hazards.
This section provides an overview of how HFE considerations and approaches can be incorporated into the design and risk management processes. These four central steps are essential:
HFE efforts are used to identify, describe, and mitigate use scenarios that result in hazards. Figure 3 shows the structure of a use scenario resulting in a hazard for a medical device. The figure shows how the use of a device is influenced by human factors characteristics that can be separated into the three broad human factors areas: 1) Use Environment, 2) User Characteristics, and 3) Device User Interface Characteristics. When identified, these influences can be described as causes or contributing factors to the use scenario. HFE approaches are used to identify these use scenarios, to understand the causes and contributing factors, and to develop mitigation strategies.
Figure 3. Use Scenario Resulting in a Hazard
Figure 4 depicts the risk management process for addressing use-related hazards. Certain HFE approaches should be applied to allow this process to work effectively. Table 1 provides a cross-reference between risk management activities and the sections in which corresponding HFE approaches are discussed.
Figure 4. Addressing Use-Related Hazards in Risk Management
Table 1
Risk Management Activities and Associated HFE Approaches
| Risk Management Activity | Applicable HFE Approaches |
|---|---|
| 1. Identify and describe use scenarios resulting in hazards. | Section 5.1:
Device use Description Section 5.2: Standards and Guidelines Section 5.3 – 5.5 Analytic and Empirical Approaches |
| 2. Prioritize and assess use-related hazards. | Section 5.3 –
5.5 Analytic and Empirical Approaches Section 5.6: Prioritization and Assessment |
| 3. Develop and implement mitigation and control strategies for use-related hazards. | Section 5.2:
Standards and Guidelines Section 5.3 – 5.5 Analytic and Empirical Approaches Section 5.7: Mitigate and Control Use-related Hazards |
| 4. Verify mitigation and control strategies. | Section 5.3
– 5.5 Analytic and Empirical Approaches Section 5.8: Verification and Validation |
| 5. Determine if the risks related to device use are acceptable. | Section 5.2:
Standards and Guidelines Section 5.3: Analytic Approaches |
| 6. Determine if new hazards have been introduced. | Section 5.2:
Standards and Guidelines Section 5.3 – 5.5 Analytic and Empirical Approaches |
| 7. Validate safe and effective device use. | Section 5.5:
Empirical Approaches Section 5.8: Verification and Validation |
Describing the intended use of a device is an essential initial step for understanding device use accurately and completely. This description should include the following information:
The device use description can be developed from documents on device operation that do not necessarily focus on user interaction as long as they describe the intended use of the device. Input from design team personnel can be very useful at this stage; however, input from intended users should also be obtained. The level of detail contained in the device use description should be sufficient to explain interactions between the user and the device user interface.
Some use-related hazards are evident from the device use description. It also provides a basis for analytic approaches and is necessary for creating valid test scenarios for usability testing (see Section 5.5.2). For example, if a device is intended to be used on emergency vehicles including helicopters, potential use-related hazards could involve failure to hear audible alarms, or inability to perform device connections or manipulations if they require significant time, attention, or manual dexterity. These potential use-related hazards will guide subsequent HFE activities. For instance, when developing scenarios for usability testing, the possible impact on device use caused by noise and motion of a helicopter environment should be simulated, or an actual helicopter should be used for the testing.
The development of the device interface should include review and incorporation of relevant standards and guidelines that are applicable to the design. To facilitate pre-market review and assist manufacturers, FDA has published device-specific and general guidances, some of which contain specific recommendations for device user interface characteristics. FDA has also officially recognized device-specific and general standards published by standards bodies such as Association for the Advancement of Medical Instrumentation (AAMI) and International Electro-technical Commission (IEC). FDA general and specific guidances as well as standards recognized by FDA are listed on FDA’s home page. www.fda.gov/cdrh
Some device-specific standards contain information for developing specific user interface features such as auditory alarms (preferred loudness and pitch), visual displays (size or brightness), printed or displayed text (size, color, and contrast), as well as the overall layout of the user interface. Some general standards also contain considerations applicable to the design of the user interfaces.
It is difficult for standards and guidelines to stay current with changes in technology that influence interface design. Designers should carefully evaluate the applicability and appropriateness of existing standards and guidance to any new device user interface design.
This document describes two broad classifications of approaches for identifying, understanding and evaluating use-related hazards for medical devices: analytic and empirical. These approaches are discussed separately; however, they are interdependent and should be used together.
Analytic approaches (see Section 5.4) involve the description and systematic decomposition and analysis of device use. They are based on the expected use of new devices and on existing information about the use of similar devices. They should be used to identify use-related hazards early in development of the user interface and operating logic of a device. The application of analytic approaches is particularly useful for identifying and resolving use-related hazards that occur infrequently, are too dangerous to "force" in an evaluation involving actual users, or might be too difficult to simulate.
The unpredictability of human behavior, the complexity of medical device user interfaces, and the variability of use environments produce use-related hazards that can be difficult or impossible to identify or understand analytically. Empirical approaches (see Section 5.5) derive information from actual or simulated use of devices. Because empirical approaches evaluate actual or simulated device use they allow for previously unanticipated use scenarios resulting in hazards to be identified and described and for identified use-related hazards to be understood.
Some use-related hazards can be understood through the application of analytical approaches only, while others might not be detected or could require empirical approaches to be sufficiently understood. Both kinds of approaches are used to identify and understand safety-critical user actions, use scenarios resulting in hazards, and the contexts in which these occur. The results of analytic approaches are useful to guide the development and application of empirical approaches.
5.3.1 Analytical and Empirical Approaches in
Clinical Evaluation Research
Medical device manufacturers conduct clinical evaluations in a variety of ways for devices
(or prototypes). These evaluations can be done prior to approval or after a device is on
the market. Regardless of when clinical evaluation is undertaken, the main goal is to
evaluate or demonstrate the safety and effectiveness of the device. For any device that
requires user-interaction, the use of the device by intended users is a vital part of its
overall safety and effectiveness. Therefore, clinical evaluations should include a means
to demonstrate that use-related hazards do not occur or that the strategies for their
control or mitigation are adequate.
To the extent that appropriate HFE approaches are used in clinical evaluations, their results can validate that device use is safe and effective in the hands of the intended users. To clearly demonstrate that intended users can use a device safely and effectively, the design of clinical evaluations should include user-based approaches. Measures of overall safety and effectiveness can be painstakingly planned, recorded and analyzed, while issues corresponding to use, which could be simultaneously collected with relative ease, are omitted. Of particular importance, occasional problems or anomalies experienced by users are usually not captured, described, or investigated.
Certain characteristics of clinical evaluation research should be carefully considered when the intent is to demonstrate safety and effectiveness of device use:
The approaches used in clinical evaluation research should include provisions for collecting information specific to device use. Users should be encouraged to identify and describe problems they experience when using the device: close calls, confusion, or repeated difficulties of any kind. Use problems identified should be investigated. If use problems can be dealt with easily, they can be resolved and reevaluated during the study. If not, strategies for mitigation and review should be developed.
There are a variety of analytic approaches that are used by human factors and systems engineers. Analytic approaches used for HFE investigations including function and task analysis, heuristic analysis, and expert reviews. These approaches can be applied within more comprehensive approaches such as Operational Analysis, Analysis of Similar Systems, Failure Modes Effects Analysis (FMEA), Fault Tree Analysis (FTA), Critical Incident Technique, Hazard and Operability Studies (HAZOP), and others. Regardless of the choice of technique, the first step should be to identify and describe use scenarios that result in hazards from the information developed in the device use description. Subsequent analyses will evaluate these scenarios further.
5.4.1 Identify and Describe Use Scenarios Resulting in Hazards
Two perspectives are necessary to identify and describe use scenarios that could result in
hazards. The "top-down" perspective identifies possible hazards first, then the
analyst determines all the possible use scenarios that could lead to that hazard. The
"bottom-up" perspective begins with known, likely, or suspected use scenarios
that involve difficulty using a device prototype, similar devices or similar components,
and then determines the hazards that can result from these problems analytically.
The best source of information on use-related hazards associated with similar devices (known hazards) is likely to be complaint and customer feedback files. Other sources of information on known hazards are discussion (focus groups) with device users, journal articles, proceedings of professional meetings, newsletters, and relevant internet sites, such as:
The device use description (see Section 5.1) and task analyses provide information to help the analyst identify and describe use-related hazards. With respect to the overall HFE process, use scenarios identified from this analysis can be thought of as anticipated use scenarios. Unanticipated use scenarios that result in hazards are identified and described through the application of empirical approaches such as usability testing (see Section 5.5).
Answering the following questions can help identify and describe potential scenarios that could result in hazards (Note: This list is not exhaustive):
5.4.2 Function and Task Analyses
Descriptions of exactly what functions and tasks are vary among function
and task analysis techniques available. These differences are not critical; the important
contribution of applying function or task analysis techniques is the systematic breakdown
of the device-use process into discrete steps or sequences for the purposes of description
and further analysis. With respect to safety, function and task analyses can contribute
by:
A simplistic example of a task analysis component for a hand-held blood glucose meter includes the following tasks:
This set of tasks includes examples that are performed by the "user" by the "device" or by a combination of the user and the device ("user+device").
After functions and tasks have been identified, they are analyzed to determine if, and how HF considerations apply. For instance, in Task 2 above, the user places a sample of blood on a test strip. There are five fundamental questions that should be investigated:
To begin to address these, the analyst should pose more specific questions, such as:
In early glucose monitors, the user had to perform Task #4 manually (the sample is allowed to react with reagents in the test strip for a specific time). Users had difficulty doing this task well, and the accuracy of the results too often suffered from the users’ failure to time the process accurately. In subsequent models, this task was done automatically by the device. Modification in device design and operation removed that use scenario and the resulting hazard.
Analyzing functions and tasks in this way will allow identification of possible hazards associated with device use. Function and task analyses can provide a foundation for subsequent HFE efforts. For instance, test scenarios (see Section 5.5) should be developed to address use scenarios that involve tasks identified as critical or error-prone.
5.4.3 Heuristic Analysis
Heuristic analysis is an analytic process in which the device’s user-interface is
formally evaluated from the perspective of users. The object is to identify possible
use-related hazards with a focus on the interaction of the user with the user interface
and operating logic of the device. Design team members often perform heuristic
evaluations, but they are more effective if they involve clinical and HFE personnel. This
technique is particularly useful for early identification of difficult or
counter-intuitive aspects of the device user interface. Another application is the
evaluation of candidate interface design alternatives. The output of heuristic analysis is
limited because evaluators typically do not represent real users, use scenarios considered
might not be comprehensive, and the evaluation environment is not representative of actual
use.
Heuristic analyses should include careful consideration of accepted concepts for design and operation of the user interface, sometimes known as "de-facto" standards or "population stereotypes" which are essentially social and cultural norms and constraints for the use of device components. A simple example is a light switch oriented in a vertical direction being "on" when it is in the "up" position and "off" when in the "down position". For medical devices, general de-facto standards are applicable at times, while others are unique for certain kinds, or types, of medical devices.
5.4.4 Expert Review
Expert reviews rely on clinical and HF experts to analyze device use, identify problems,
and make recommendations for addressing them. The process is quite similar to the
heuristic analyses. The difference is that expert review relies more heavily on the
assessment of individuals with expertise in a specific area. The success of the expert
review depends on the expert’s knowledge of the device technology, its use, clinical
perspectives, characteristics of the intended users, as well as the expert’s ability
to predict actual device use. This kind of review can provide very useful information,
particularly early in the design process, but might not be comprehensive since it does not
involve actual device use and might not include the perspective of actual users.
Use studies are applicable to several risk management activities. They can be used early in the design process to identify unanticipated use-related hazards. They can also be used to clarify suspected or known problems with device use, demonstrate that use-related hazards have been addressed, evaluate candidate design alternatives, and to validate safe and effective use by intended users. Beyond application to the safety and effectiveness of device use, use studies provide a powerful means for creating effective labeling (including directions for use), and device designs that are user friendly, satisfying to use, and desirable to users. For the consideration of device use-related risks to be complete, empirical methodologies should include efforts that focus on identification and analysis of unanticipated use-related hazards and the incorporation of the results into the overall risk management process. Use Studies can identify problems that were noticed by test participants but did not manifest themselves as errors during use.
Use studies will provide accurate results to the extent that test participants represent actual device users, the test conditions represent realistic use environments, and the test is well run. Members of the team who are developing the device should not participate as users since their knowledge of how the device operates (or should operate) will influence how they use it. If the intended users have certain limitations in their abilities, one focus of the testing should be to establish whether these limitations affect device use. If so, further effort is required to determine whether potential use problems associated with user limitations can be mitigated by modifying the design of the device interface or the operation of the device.
Although user studies are effective in identifying and understanding device use, care should be taken not to underestimate the frequency of problems based on the experiences of test participants. Participants could be (despite good efforts of test coordinators) unrealistically well trained, capable, or careful. Also, when people are observed they often try to "do their best" and often do not use the device long enough to experience infrequent problems.
When applied to medical devices, empirical approaches should support identification, understanding, and mitigation of hazards resulting from device use. Demonstrating how well users like using a device is not sufficient to do this. However, both use-safety and user preference can be addressed through proper application of empirical approaches.
5.5.1 Walk-Through
A simple kind of study involving users is the walk-through. It is less
time-consuming and less formal than Usability Testing. In a walk-through, a user or small
group of users are "walked-through" the process of using a device. During the
walk-through, participants are questioned and encouraged to provide feedback on
difficulties they notice while using the device. Evaluators can also collect subjective
information from participants about thought processes, mental models, and perceived
workload when using the device. The walk-through technique can provide valuable
information but is limited by a lack of realism. The walk-through technique is most useful
early in the development process, and for developing and evaluating usability testing
scenarios.
5.5.2 Usability Testing
Usability testing (also called user testing) is a powerful technique used to
assess user’s interaction with a product. This technique can also be used to identify
and understand previously unanticipated or poorly understood use scenarios resulting in
hazards if care is taken to focus on the safety and effectiveness perspectives. The
central advantage of usability testing is that device use is realistic and the results of
the process are more representative of actual use than results obtained through analytic
approaches. If usability testing is employed early in the development process, it can
identify potential use-related hazards so that they can be addressed early in the design
life cycle.
Usability testing involves systematic collection of data from users (participants) using a device (or device component) in realistic situations. Data are obtained in a variety of ways, including user feedback, manual and automated measures of user performance, and observation. Often, the most convenient data collection methods focus on subjective user feedback. User feedback includes descriptions by test participants of difficulties encountered, good and bad aspects of the device user interface characteristics, including the logic of device operation, and suggested changes. Careful collection of subjective assessment of device use can identify problems that were noticed by test participants ("concerns," or "close calls") but did not manifest themselves as errors during use and not identified in objective performance measures.
Objective user performance measures include the type and number of errors, time required to do tasks, requests for help, accuracy, and the success or failure on individual tasks and overall performance. The application of specific, objective user performance measures enhances and focuses subjective user feedback. Performance measures are particularly useful for complex devices, where users might not be aware of (and therefore unable to evaluate) potentially hazardous use scenarios. These measures are also important for home-use devices where users are often not aware that they are inadvertently effecting the performance or accuracy of the device in some way. Outlier data from performance measures is often informative and should be investigated to determine the nature and pattern of the use scenarios associated with them.
Usability testing can be done in a variety of ways in various degrees of complexity and formality. However it is done, it should include the following:
The validity of use testing depends on the extent to which realistic or simulated environments are used during the testing. For example, in clinical settings users must perform multiple tasks simultaneously. These tasks involve individual devices, multiple devices, and duties unrelated to device use. Users must constantly trade-off accuracy for speed. In home environments, users might be distracted or have medical conditions that affect their abilities to interact with the device. Home users can also drop devices or expose them to various temperatures and humidity levels in various parts of the home. Clinical and home users might try to cut costs. There are many aspects of the use environment that can affect device use. By the time use testing is undertaken, anticipated use environments should be understood (device use description).
Use-related hazards identified by analytic and empirical approaches should be assessed to determine their priority for subsequent risk control efforts. This process can involve obtaining and combining input from several individuals who provide perspective from a variety of areas of expertise. In addition, it should also incorporate valid and useful information about likelihood and consequences (i.e., risk) of use-related hazards for similar devices when available.
Important perspectives include those from:
These individuals should then assess the likelihood of these hazards and their consequences to estimate the risk for each. Within the general process described in this guidance, assessing preliminary results through group consensus is most useful for:
Identifying and addressing use-related hazards early in the design process reduces the time and expense necessary to correct them. The most effective strategies to address use-related hazards focus on improvements to the design of the device user interface. The user interface should convey the concept for correct operation through its appearance and operation ("look and feel") so that safe and effective use is intuitive. Addressing use-related hazards through this kind of modification is preferred because it reduces or eliminates the need for users to rely on instructions, labeling, or training "patches."
Use-related hazards often require a combination of mitigation and control strategies. The following list presents the order of overall priority for applying strategies to control or mitigate risks of use-related hazards:
Modify device design to remove hazard or reduce its consequences: Making the interface intuitive and ensuring that critical information is effectively communicated to the user can reduce the likelihood or eliminate certain use-related hazards. If hazards cannot be eliminated, the design should act to mitigate the consequences.
Make user interface, including operating logic, error tolerant (safety features): When users can make errors using the device, such as pressing an adjacent key on a keypad, the device should act to preclude a hazardous outcome from occurring. Safety mechanisms such as physical safety guards, shielded controls, or software or hardware interlocks will make the design more tolerant of errors that users occasionally make.
Alert users to the hazard: When neither design nor safety features will effectively eliminate a use-related hazard or mitigate the consequences, the device should detect the condition and provide an adequate warning signal to alert users.
Develop written procedures and training for safe operation: Where it is impossible to eliminate hazards through any of the previous strategies, or to enhance other control or mitigation strategies, written procedures, labeling enhancements, and training for safe operation should be used.
Instructions, labeling, and training can influence users to use devices safely and effectively and are critical HFE considerations for safe device use. But because they rely on the user to consistently use the device as directed, these approaches are less effective than modifications to the design of the user interface. Therefore, mitigation of use-related hazards should not focus on instruction, labeling, or training fixes in isolation, since these "patches" might not be adequate. Often, a combination of these strategies is the best solution. Regardless of the strategy used, subsequent testing should be done to ensure that the use-related hazards have been successfully controlled.
Verification confirms that the specific functional and operational requirements for the design of a device user interface have been met. The process for verifying individual user interface requirements will likely require focused effort for each. For instance, if a device will be used by a user population of elderly users with hearing ability ranging from normal to moderate impairment, a specification should be developed to assure that the device’s alarm volume can be adjustable to a sufficient level to accommodate these users. The verification process would involve testing the device alarm to ensure that the volume adjustment capability (and any other specifications developed to assist users) has been implemented successfully.
Validation establishes that the device meets the needs of the intended users. The primary need of medical device users is the ability to use the devices safely and effectively under the actual use conditions. Applying usability testing approaches can directly validate a user interface design. For the purpose of validation, it is particularly important to use a production version of the device, representative device users, actual or simulated use environments, and to address all aspects of intended use. If small-scale iterative testing of interface components is done adequately as the device was developed, it might not be necessary for validation efforts to be extensive at the end of the design process. However, some degree of testing of the entire system under realistic conditions with representative users is warranted. In the alarm volume example above, determining whether users with moderate hearing loss can hear the alarm well enough to allow them to use the device safely and effectively is the essential component of validation of this user interface requirement.
Documenting the incorporation of human factors engineering (HFE) in risk management can help demonstrate that a manufacturer has adequately addressed the needs of the intended users. Submitting this documentation can streamline and facilitate that part of the pre-market review process concerned with safe and effective device use.
When information pertaining to device use safety is extensive, it is helpful to provide it in summary form that highlights the most important issues, considerations, resolutions, and conclusions. When portions of this information are contained in various parts of a submission a comprehensive cross-reference should be provided.
The level of detail of device use documentation submitted should be consistent with the level of concern of use-related hazards for the device. The information that should be included with the device use documentation is described below.
AAMI/FDA Conference Report: Human factors in medical devices: design, regulation, and patient safety, Association for the Advancement of Medical Instrumentation, Arlington, VA, 1996.
AAMI HE48, Human factors engineering guidelines and preferred practices for the design of medical devices. Association for the Advancement of Medical Instrumentation, Arlington, VA, 1993.
Bahr, N., System safety engineering and risk assessment: A practical approach. Washington DC: Taylor and Francis; 1997
Chapanis, A., The business case for human factors in informatics. In: Shackel and Richardson (eds.). Human Factors for Informatics Usability. New York: John Wiley & Sons; 1994.
Chapanis, A., Human factors in systems engineering. New York: John Wiley & Sons; 1996.
Cooper, J., Preventable anesthesia mishaps: A study of human factors. Anesthesiology 1978, 49:399-406.
Cooper, J., An analysis of major errors and equipment failures in anesthesia management: considerations for prevention and detection. Anesthesiology 1984, 60:34-42.
Dumas, J., and Redish, J., A practical guide to usability testing. Ablex, New Jersey, 1993.
Ehrlich, K. and Rohn, J., Cost justification of usability engineering: A vendor’s perspective. In: Bias and Mayhew (eds). Cost justifying usability. New York: Academic Press; 1994, 73-110.
FDA, Design Control Guidance for Medical Device Manufacturers. March 11, 1997.
FDA, Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices. May 29, 1998.
FDA, Quality system regulation (current good manufacturing practice (CGMP) final rule). October 7, 1996.
Goodstein, L., Anderson, H. and Olson, S., Tasks, errors, and mental models. New York: Taylor and Francis; 1988.
IEC TR 513, Fundamental aspects of safety standards for medical electrical equipment. International Electrotechnical Commission; 1994. Geneva, Switzerland.
ISO Guide 51, Guidelines for the inclusion of safety aspects in standards, first edition. International Organization for Standardization, 1990. Geneva, Switzerland.
ISO 14971-1, Medical Devices – Risk Management – Part 1: Application of Risk Analysis, first edition. International Organization for Standardization, 1998. Geneva, Switzerland.
Kirwan, B., and Ainsworth, L.K., A guide to task analysis. London: Taylor & Francis Ltd; 1992.
Klemmer, E. (ed.), Ergonomics: Harness the power of human factors in your business. Norwood, NJ: Ablex, 1989.
Kohn, L., and Corrigan, J., (eds.), Building a Safer Health System. Institute of Medicine (IOM), Committee on Quality of Health Care in America. Washington, D.C.: National Academy Press; 1999.
Leape, L., Error in Medicine. Journal of American Medical Association, 1994 Dec; 21(3) 272.
Leveson, N., Safeware: system safety and computers, New York: Addison-Wesley; 1997.
Meister, D., Human factors testing and evaluation. Amsterdam: Elsevier; 1986.
MIL-STD 1472D, Human engineering guidelines for military systems, equipment and facilities. Washington, D.C.: Department of Defense, January, 1996.
MIL-STD 882C, System safety program requirements. Washington,DC: Department of Defense. January, 1993.
Modarres, M., What every engineer should know about reliability and risk analysis. New York: Marcel Dekker, Inc; 1993.
Mosenkis, R., in Grutting, C., Medical devices: international perspectives on health and safety. Amsterdam: Elsevier; 1994.
Nielsen, J. and Mack, R., Usability Inspection Methods. New York: John Wiley & Sons; 1994.
Norman, D., The Design of Everyday Things. New York: Doubleday; 1988.
O’Brien, T., and Charlton, S., Handbook of Human Factors Testing and Evaluation, New Jersey: Lawrence Erlbaum Assoc; 1996.
Reason, J., Human Error. Cambridge, Mass: Cambridge University Press; 1992
Rubin, J., Handbook of Usability Testing. New York: John Wiley and Sons, Inc; 1994.
Salvendy, G. (ed.), Handbook of Human Factors and Ergonomics. New York: John Wiley and Sons, Inc; 1997.
Sanders, M., and McCormick E., Human Factors in Engineering and Design. New York: McGraw Hill; 1993.
Sawyer, C., Do It By Design: An Introduction to Human Factors in Medical Devices. FDA; 1997.
Schneiderman, B., Designing the User Interface: Strategies for Effective Human-Computer Interaction. New York: Addison-Wesley; 1993.
Trautman, K., The FDA and Worldwide Quality Systems Requirements Guidebook for Medical Devices. ASQC Press; 1997.
Wickens, C. D., Gordon, S. E., and Liu, Y., An Introduction to Human Factors Engineering. Addison-Wesley (Longman Imprint): New York, NY; 1998.
Updated July 24, 2000
CDRH Home Page | CDRH A-Z Index | Contact CDRH | Accessibility | Disclaimer
FDA Home Page | Search FDA Site | FDA A-Z Index | Contact FDA | HHS Home Page
Center for Devices and Radiological Health / CDRH
