Usability Updates Newsletters

---

July 2005

The Value of Iterative Design

by Dr. Bob Bailey

Introduction

Several years ago, Skip Bailey (G. Bailey, 1993) published the results of a study where he had tracked user performance through a series of three or four iterations. Each of his programmer participants created a software system, and then made improvements to the system based on the results of usability tests. He reported many usability improvements over the duration of the study, including an overall 16% improvement in tasks successfully completed, and a 26% improvement in performance time.

This was one of the first, carefully controlled studies to report convincing evidence that the "iterative design" method actually worked. What this means is that if we created a prototype system, then conducted a usability test, then made changes to the site based on the test results, and then tested again, the site's user interface would elicit substantially better user performance. The more "test, make changes, retest" cycles, the better will be the system's usability.

University Tests

In 2001, Wei-Siong Tan (Tan, et.al., 2001) and his colleagues at the University of Nebraska reported a study with impressive human performance improvements. They had 15 test participants complete scenarios while using a preliminary version of a commercial website. Like all good tests, their scenarios were created to represent tasks that were typical of those encountered in real life, and reflected the major functionality of the site. After doing their best to eliminate the problems identified during the baseline usability test, they then conducted a follow-up test by having 20 different users complete the same task scenarios. The changes to the website resulted in:

• A 28% faster average completion time for the scenarios, and
• An overall 37% reduction in the number of usability problems.

Government Tests

More recently, a test and retest (Bailey and Wolfson, 2005) were conducted on the FirstGov (now USA.gov) search engine. The baseline test had 12 participants complete 10 task scenarios, and was conducted in December, 2004. Based on those test findings, the design team made many changes to the search capability. The site was retested in May, 2005 and showed many measurable improvements:

• The overall percent correct increased from 65% to 73%
• Five of the 10 task scenarios showed substantial improvements
• Nine of the 10 scenarios took less time
• The satisfaction score increased from 63 to 73
• Nine out of 10 satisfaction categories were higher in the follow-up test

Company Tests

Probably the best recent study of how iterative design methodology can lead to impressive user interface improvements was reported by Lori Caldwell LeDoux, Ellen Connor and Thomas S. Tullis at Fidelity Investments (2005).

Their study was intended to show the benefits of a user-centered design process. They conducted a baseline usability test using 16 participants to identify problems with the original website, then determined the best changes to apply to the new design, and then conducted a follow-up usability test on the final site. This approach enabled them to clearly observe and explain how usability issues were identified and corrected.

The baseline usability test identified over 100 potential usability problems. In addition, they conducted an expert review that identified 89 usability issues. After making many changes to the website, they conducted a follow-up usability test with 14 participants, and found that:

• The overall percent correct increased from 67% to 87% (30% improvement)
• The average time to complete the task scenarios went from 68 seconds to 51 seconds (25% improvement)
• The overall satisfaction score improved from 49 to 82 (67% improvement)

The results of the usability testing clearly showed a significant improvement in both human performance and user satisfaction. They also reported that the client was very happy with the new design.

Conclusions

Improving the usability of a system is difficult. There continues to be considerable good evidence that the test, make changes, and retest (iterative) approach clearly works. In fact, it continues to be the only evidenced-based way to improve the quality of computer-based systems, including websites and web applications. There is no evidence that revising a system based solely on designer observations about needed "improvements," or user comments (usually complaints) actually leads to better user performance or increased user satisfaction.

________

Bailey, G.D. (1993), Iterative methodology and designer training in human-computer interface design, INTERCHI '93, 198-205.

Bailey, R.W. and Wolfson, C. (2005), Evaluation of the FirstGov search capability, A follow-up usability test report, May.

LeDoux, L.C., Connor, E. and Tullis, T.S. (2005), Extreme makeover: UI edition, Proceedings of the Usability Professionals Association Annual Conference.

Tan, W., Liu, D., Bishu, R. R., Muralidhar, A. and Meyer, J. (2001), Design improvements through user testing, Proceedings of the Human Factors and Ergonomics Society 45th Annual Meeting, 1181-1185.