Involving Users in Web Accessibility Evaluation
Page Contents
Introduction
Web accessibility evaluation often focuses on evaluating conformance to accessibility standards, such as the Web Content Accessibility Guidelines (WCAG). Broadening evaluation to involve people with disabilities helps better understand accessibility issues and implement more effective accessibility solutions.
One of the benefits of including people with disabilities is that Web developers can learn how people with disabilities interact with the Web and with assistive technologies. For example, consider a developer who does not know what it is like to use a screen reader. To meet the Web accessibility guideline "Provide text alternatives for all non-text content", the developer might provide the alternative text: "This image is a line art drawing of a dark green magnifying glass. If you click on it, it will take you to the Search page." However, observing a person use the site with a screen reader will clearly show the developer that his alternative text is ineffective and "search" is all that is needed.
When Web developers, managers, and other project stakeholders see people with disabilities use their Web site, most are highly motivated by a new understanding of accessibility issues. Collaborating with people with disabilities who are target "users" of a Web site early in a project helps Web developers be more efficient in addressing accessibility, thus maximizing the results from investment in accessibility.
Involving users with disabilities in evaluation has many benefits; however, it alone cannot determine if a Web site is accessible. Combine user involvement with evaluating conformance to WCAG to ensure that accessibility is provided to users with a range of disabilities and situations.
This document introduces involving users in Web accessibility evaluation, provides guidance on some considerations for involving users, and links to additional information. It is part of a multi-page Evaluating Web Accessibility resource suite that outlines different approaches for evaluating Web accessibility.
Involving Users Effectively
A first step in evaluating Web accessibility is conducting a preliminary review of the Web site to check for any obvious accessibility problems. This allows you to fix any significant accessibility barriers before spending time evaluating with users.
Even Web developers with little accessibility knowledge can find some accessibility issues through a preliminary review. An accessibility expert with first-hand experience of how people with different disabilities interact with the Web can:
- evaluate accessibility issues for a broad range of users, which might not be found by individual users;
- help fix any known barriers before bringing in users; and
- focus the evaluation with users on potential areas of concern.
Users with disabilities can be included in a wide range of evaluation activities, from brief consultations to large-scale usability studies. There are many options in between these extremes:
- Informal evaluation of a specific accessibility issue can be as simple as asking a person in your office who uses a screen reader to find some data in an early draft of a data table that you are developing, observing her interaction, and discussing issues.
- Formal usability testing of a Web site follows established protocols to gather quantitative and qualitative data from representative users performing specific tasks. Formal usability tests can be optimized to focus on accessibility issues.
Conducting informal evaluations throughout development is more effective than formal usability testing at the end of a project. In most cases, including users in evaluation involves:
- finding a few people with disabilities,
- including them throughout the development process to complete sample tasks on prototypes, and
- discussing accessibility issues with them.
Carefully consider all feedback and avoid assuming that feedback from one person with a disability applies to all people with disabilities. A person with a disability does not necessarily know how other people with the same disability interact with the Web, nor know enough about other disabilities to provide valid guidance on other accessibility issues.
Including Diverse Users
People with disabilities are as diverse as any people. They have diverse experiences, expectations, and preferences. They use diverse interaction techniques, adaptive strategies, and assistive technology configurations. People have different disabilities: visual, auditory, physical, speech, cognitive, and neurological — and some have multiple disabilities. Even within one category, there is extreme variation; for example, "visual disability" includes people who have been totally blind since birth, people who have distortion in their central vision from age-related degeneration, and people who temporarily have blurry vision from an injury or disease.
Include users with a variety of disabilities and user characteristics. In most cases evaluators have limited time and budget and cannot include many users in evaluation. Selecting the optimum number of users with the best suited characteristics can be difficult. There are resources on the Web that provide guidance on determining participant characteristics for a particular situation and on finding participants with disabilities.
Users' Experience Interacting with the Web
A primary consideration in selecting users to help with evaluation is their experience interacting with the Web. For example, some assistive technologies (AT) are complicated and difficult to learn. A user with insufficient experience may not know how to use the AT effectively. Problems identified may be due to the user's lack of knowledge of the AT, not problems with the Web site being evaluated. On the other hand, a very advanced user might know uncommon work-arounds to overcome problems in the site that the "average" user would not be able to handle.
Just as with any evaluation with users, whether you include novice, average, or advanced users depends on your target users. For example, if you are developing a Web application to be used by accountants inside a company, you probably want advanced AT users; for a public Web site to apply for disability benefits, you want novice AT users.
Analyzing Accessibility Problems
Web accessibility depends on several components of Web development and interaction working together, including Web browsers, assistive technologies, and Web content. For any accessibility problems you identify, determine which components are responsible. For example, if a user has trouble with a data table, it could be because:
- the developer did not markup/code the data table properly, or
- the user's AT does not facilitate reading data tables effectively, or
- the user does not know how to use the ATs table reading feature.
In addition to finding accessibility problems, evaluating with users with disabilities usually reveals general usability problems that impact all users, including those without disabilities.
Drawing Conclusions and Reporting
As explained in the introduction, involving users with disabilities makes accessibility efforts more effective and more efficient. However, it alone cannot determine if a Web site is accessible; even large-scale usability studies cannot cover the diversity of disabilities, adaptive strategies, and assistive technologies. Combining user involvement with evaluating conformance to WCAG ensures that the broad range of accessibility issues are covered.
Reports should include the scope of the study and the evaluation parameters, such as the testing methods and the user characteristics. For example, if a study included only usability testing with participants who are blind, its report should clarify that it did not evaluate conformance to accessibility guidelines and that it does not apply to all people with disabilities. Thus the report can help readers draw appropriate conclusions.
For More Information
This document briefly addresses a few points of a very complex topic. Many resources on other aspects of involving users in evaluation are available on the Web.
- Accessibility in the User-Centered Design Process
provides guidance on incorporating accessibility throughout design of Web
sites, and other information technologies and communications technology
products. The chapter on Usability
Testing for Accessibility includes:
- Planning for usability testing for accessibility – determining participant characteristics, recruiting participants, providing compensation
- Preparing for usability testing for accessibility – preparing test materials, ensuring the facility is accessible, setting up and testing the assistive technology, conducting a pilot test, using screening techniques
- Conducting a usability test for accessibility – interacting with people with disabilities, setting up the room
- Reporting usability testing for accessibility – distinguishing between accessibility and usability issues, drawing conclusions, writing about people with disabilities
- Many books, articles, conference presentations, and other resources cover usability evaluation techniques, including different types of usability testing; test design; developing test protocol including questionnaires, tasks, data collection; conducting pilot tests; and how many participants to include in usability testing.
- There are organizations around the world that specialize in helping recruit people with disabilities and conduct evaluations with users with disabilities.
Terminology and Notes
- adaptive strategies
- Adaptive strategies are techniques that people with disabilities use to improve interaction with the Web, such as increasing the font size in a common browser. Adaptive strategies include techniques with mainstream browsers or with assistive technologies.
- assistive technologies
- Assistive technologies are software or equipment that people with disabilities use to improve interaction with the Web, such as screen readers that read aloud Web pages for people who cannot read text, screen magnifiers for people with some types of low vision, and voice recognition software and selection switches for people who cannot use a keyboard or mouse.
- user characteristics
- User characteristics typically include things like age, job responsibilities, software, hardware, environment (for example, home, shared office, private office, shared public terminal), computer experience, and Web experience.User characteristics can also include type of disability, adaptive strategies used, and experience with specific assistive technologies.
- Web content
- Web "content" generally refers to the information in a Web page or Web application, including text, images, forms, sounds, and such. More specific definitions are available in the WCAG documents, which are linked from the Web Content Accessibility Guidelines (WCAG) Overview.
Optimizing Usability Testing for Accessibility Issues
Note for usability professionals: When defining usability tests specifically to find accessibility issues, the protocol will be different from a typical general usability test; for example:
- you would likely use a think-out-loud technique with high facilitator interaction;
- data collection would focus on understanding errors related to accessibility issues, rather than on time-on-task or user satisfaction; and
- tasks would concentrate on specific areas of concern for potential accessibility problems, rather than general site usage.