Introduction

Selecting a Web content management system (WCMS), as many can attest, is a daunting process. As part of its selection process, U.S. Department of Health and Human Services (HHS) reviewed the market and selected several applications to assess. Usability testing the WCMS "out of the box" was a key component of the evaluation process.

From Many to Few

In Phase I of the evaluation process, respected industry resources were consulted and a report was written that summarized which applications met our desired technical and functional requirements. We invited several vendors to participate in lab-based testing at our facilities. Phase II involved usability testing the applications "out of the box," as well as scoring the applications based on our requirements.

Evaluation Methodology

We recruited test participants to test four systems. Participants represented expected users of the system. Prior to testing an application, participants received basic training on the application; next, they attempted to complete a general set of typical tasks tailored to the application. WCMS applications were scored based on how many of the tasks the participants successfully completed.

An overall score included the weighted technical evaluation score and the weighted usability test score. To reach a total score for each application, usability was treated as an additional evaluation component, and the usability test scores were converted by multiplying the percent correct by 100 to match the technical evaluation scoring framework.

Conclusion

When we began the process of selecting a WCMS, we knew that it would be important that it be easy to use. If the system was not flexible and user-friendly, we knew that users would not take advantage of the application to manage their content. Too often, we only consider usability in the context of Web sites and neglect to include it in evaluating products for organization use. By combining two objective approaches to evaluation in order to inform our technology acquisitions, we addressed not only the technical appropriateness of an application, but also its expected acceptance by a user base. Usability testing provided HHS with an objective measure of how well users will be able to accomplish their work with the new application.