Building Usability into Electronic
Data-Collection
Forms for
Economic Censuses and Surveys[1]
Elizabeth
D. Murphy, Elizabeth M. Nichols, Amy E. Anderson,
M.
Diane Harley, and Kimberly D. Pressley
U.S.
Census Bureau, Washington, DC� 20233
Abstract
Because the economic census[2]
is the primary source of information about the Nation�s economy, collecting
detailed economic data from business establishments is one of the Census
Bureau�s key activities.� In planning
for the 2002 Economic Census, the Census Bureau intends to offer all
respondents an electronic alternative to paper forms.� Providing an electronic alternative requires translation of over
650 paper formats into Computerized Self-Administered Questionnaires (CSAQs).� Because of the differences between paper and
the electronic medium, and because of the capabilities made available by the
electronic technology, the translation process is not simply a one-to-one
mapping.� Consideration must be given to
the cognitive and usability issues associated with electronic questionnaires to
ensure that respondents who choose to report by CSAQ find it to be an effective
and efficient means of providing their data.�
As an aid to developers, the Census Bureau developed a style guide
documenting numerous design decisions.�
This paper describes the process of developing the user-interface style
guide with input from cognitive and usability testing.� Informed by user-interface design principles,
standards, best practices, and the results of low-fidelity-prototype testing, a
style guide of this kind is a tool for building usability into electronic forms
for data collection.��
Keywords:� usability, style guide, Computerized
Self-Administered Questionnaires, cognitive testing, usability testing
Introduction
The economic census provides a detailed portrait of the Nation's economy once every five years.� Data collected in the Economic Census are essential for government, business, and the public[3].� For example, business establishments use Census data to make business decisions and to evaluate their performance.� Trade associations and news media study economic indicators and projections based on Census data.� Legislators at local, state, and national levels use Census data in the preparation and evaluation of new laws. Consultants and researchers use Census data to analyze changes in industrial structure and location.� To meet these many and diverse needs, the data collected must be as complete and accurate as possible.�
In previous economic censuses, the Census Bureau has mailed out paper questionnaires to collect economic data from business establishments.� In 1997, an electronic form was offered to a selection of large companies in the retail sector.�� According to plans for 2002, the Census Bureau intends to offer all business respondents the option of reporting their data on paper or by electronic means.� The electronic forms are called Computerized Self-Administered Questionnaires (CSAQs).��� Electronic technology offers the opportunity to streamline the process of collecting data.� However, any change in data-collecting techniques must result from a detailed design-and-development process that includes user testing.
There are more than 650 versions of the economic census form, each customized to particular industries. �To convert the paper forms into a set of consistent electronic questionnaires, developers need detailed specifications on such topics as font style and layout.� These specifications must cover the hundreds of design decisions that would otherwise need to be made �on the fly� by individual developers.� A style guide documents design rules to a level of detail that ensures consistency across developers. Since the universe of possible design rules is enormous, the design team must consider and rule out many options before reaching consensus on the content of a style guide.��
Development of the 2002 Economic Census Style Guide
The purpose of a style guide is to foster consistency across forms and across the various programmers who may be working on different pieces of a software development project, in this case, a CSAQ.� To ensure quality and consistency in the design of more than 650 electronic forms for the economic census, a cross-divisional team was assembled to develop a style guide for Windows-based, economic CSAQs.� Working in a top-down fashion, the team determined that the style guide should include design rules for layout and screen design, navigation, graphics, error handling, help, feedback from users, and other internal feedback mechanisms.��
Once the initial set of topics to be covered had been identified and grouped, members of the team signed up to develop various sections of the style guide.� The team met weekly to discuss and review the draft content for the various sections.�� Content evolved gradually as team members, individually and collectively, consulted existing usability principles, user-interface design standards and best practices in creating design rules and sample screens. The project's developer advised the team on alternative approaches that were technically feasible.� In some instances, cognitive testing and usability testing were conducted to help the team decide between alternative design rules.�
Usability Principles
By definition, principles are stated at a high level of abstraction and offer general, top-down guidance.� The style guide team consulted the following kinds of usability principles, which were collected from two major sources of expertise in usability: Nielsen (1993) and Shneiderman (1998):��
� Ease of navigation through the user interface:� Users should be able to move easily from their point of origin to their destination.� They should not have to work their way down menu hierarchies and back up again to get to where they want to be.� Users who need to complete forms for multiple units should be able to use shortcuts, such as alternative keyboard commands, to navigate quickly within the user interface.�
� Ease of training/learning: The user interface should reflect an understanding of the user's thought process and the logical flow of the user's work.� Required training should not be lengthy.
� Consistency of "look and feel" within and across applications:� Consistency is the key to building usability into an automated system.� Everything from terminology to system feedback should be phrased and located consistently from screen to screen.� System behavior should be consistent from one instance to the next.� Consistency in look and feel allows the user to build up expectations with the reasonable hope that they will be met.� Consistency within and across electronic questionnaires allows the user to build expertise quickly.
Although principles define general criteria for a design, developers need much greater specificity about the attributes of design elements. �Thus, the style-guide team consulted more detailed user-interface guidelines and "best practices," e.g., conventions used by other CSAQ-development teams.� This continued the top-down approach to identification of topics and style-guide content.
User-Interface Design Guidelines
In the absence of any standards for the design of electronic questionnaires, the team consulted available user-interface design guidelines (e.g., Compuware, 1998, Microsoft, 1995).� The team also consulted an internal Census Bureau report on research into user-interface guidelines.� The guidelines and the research report helped the team identify topics to cover, provided a general overview of approaches to the design and layout of visual elements, and gave ranges of specific guidance on some design elements.�
Best Practices in User-Interface Design
Although the literature on human-computer interaction was the primary source of best practices, the team often referred to design techniques used successfully in other Census Bureau economic CSAQs, such as the Annual Survey of Manufactures (ASM), the Company Organizational Survey (COS)[4], and the Quarterly Financial Report (QFR).� For example, the COS uses what we have termed a "stacked design" in formatting the questions on status, payroll, and employment (Figure 1):
�����������
����������� Status
of establishment...�� ������������������� ����������� ��������
�����������������������
����������� Employees
during pay period...
����������� Payroll
for first quarter...����� ���� �����������������������������������
����������� Total annual payroll...���������������������������������������
���� Figure 1.� Example of a stacked format as an
���� alternative to horizontal strings of items; recommended
���� to fit all items within the viewable screen space.
Using a stacked format deviates from the horizontal layout used on the paper form.� However, this format eliminates the need for the respondent to scroll to the right.� Because respondents are likely to miss the subtle cues for horizontal scrolling, they are likely to omit items that are not completely visible within the computer screen�s display area.� Thus, the stacked format is preferable for the electronic form.
The error-handling mechanism implemented in the QFR is another design element adopted for the 2002 Economic Census Style Guide.� In the QFR, an icon consisting of a red circle with a white X was used successfully to identify edit failures (i.e., answers that are out of predefined ranges or inconsistent with prior answers).� As in QFR, this icon will appear to the right of an edit-failing data field in the electronic forms for the 2002 Economic Census.� Clicking on the icon will open a dialog window containing the edit message, which will provide at least an explanation of the problem.� The icon will disappear once the respondent deals with the edit failure.�
In developing wording for edit messages, the team referred to wording that had been used for ASM, COS, and QFR edits as well as the 1997 Economic Census.� In those CSAQs, some messages were preceded by the word "Warning" or "Error," depending on the severity of the discrepancy.� Based on recommendations in the literature, an early team decision for the 2002 Style Guide was not to use the word "error" in any message because of its negative implications (e.g., Shneiderman, 1998).� The general feeling was that a respondent's data can be correct even if certain entries technically fail an edit check.
Giving the respondent a Remarks section to explain anything that fails an edit check is expected to help the Census Bureau understand the respondent's rationale for not changing values in response to edit messages.� In contrast to the paper form, where respondents often write explanations right beside an entry instead of going to the Remarks section, respondents in the electronic mode can enter explanations only in the Remarks section.�
Cognitive and Usability Testing in the Laboratory
To resolve some user-interface design issues, the style-guide team needed further guidance beyond the available literature and best practices.� In those cases, alternative user-interface designs were prototyped and submitted to cognitive and usability testing.� Results helped to further define the contents of the style guide.� For example, members of the team tested proposed wording of edit messages for the electronic forms.�
Method:� We conducted ten concurrent, cognitive, think-aloud laboratory interviews to test respondent comprehension, task difficulty, and sensitivity of proposed edits for the 2002 Economic Census electronic forms.� Using paper prototypes, we also performed preliminary usability testing on the presentation of the edits. Test participants were members of the Census Bureau's economic staff.
We gave participants a partially completed paper questionnaire mimicking an on-line mortgage application. We asked participants to pretend that they were reviewing their own application.� Participants read through the questionnaire aloud.�
In order to �fail the edits,� some pre-filled answers did not make sense.� For example, one question on the mortgage form asked respondents to indicate the percentage of their down payments and closing costs that would come from several types of funds (e.g., savings, investments). The pre-filled data totaled 110%, thus tripping the edit message, "Sum of down payment percentages is 110%.� It should be 100%.� If values are correct, explain discrepancy in the remarks section."� We tested several types of edit messages.�
Scattered through the questionnaire were three questions for which participants were to receive preventive edits.� That is, the edit message for each of these questions was to be presented immediately instead of later, at the end of the session.� Immediately after the participant read each of the three questions designated for preventive edits, the test administrator explained how those questions were to be answered, and the edits were presented on paper.� Once participants completed reading the entire questionnaire, we gave them two sheets of paper with twelve edit messages relating to the responses they had just read.�� The participant's task was to find the item referenced by an edit message, and then to correct the answer to that item or go to the Remarks section and explain why the answer was correct.�
In evaluating the edit messages, we needed to determine the importance of and placement of three attributes in the message:� item number, item topic, and actual response.� When the item topic was present, we varied its capitalization.� For example, for several of the variable names, we used the following capitalization styles:� TOTAL HOME COST, Homeowner's Fee, and found a new home. That is, variable names were given in all caps, initial caps, or no caps.� We found that the following confusing edit message was the most confusing to participants: "Since you reported found a new home in item 15, please complete signed a sales contract in item 16," where "found a new home" and "signed a sales contract" were actually variable names inserted into the sentence.� Deciphering this sentence was especially burdensome to respondents.
Results and Recommendations: We developed a set of general recommendations on the content and style of edit messages. These recommendations may apply in other contexts beyond the 2002 Economic Style Guide.� The following examples are selected from the full set of recommendations (Nichols, Murphy, & Anderson, 2001):
� Provide a location, a description of the problem, and an action to take. Several error messages were written with key pieces of information missing, such as the location of the problem or what action was needed to resolve that particular error.� Respondents reacted by spending extra time determining the location of the error and interpreting their course of action.� Participants commented that working with the error messages would have been easier if these key pieces of information had been included.� When the location of the problematic question was not readily available, respondents used other pieces of information within the edit to help give them "clues" as to where the problem was.� The most readily used clues were subject-matter content that was inserted into the message, or the actual amounts reported in answer to the troublesome questions.�
� Beware of mismatches between language used in the messages and the questions. The subject-matter content that was provided in the error messages was intended to match back to the content in the actual questions.� For testing purposes, the subject-matter content that was provided in the error messages was slightly altered to see how participants would react.� When respondents encountered this mismatch, they reported "thinking twice" about whether they were working with the correct question.� This mismatch was another obstacle for the respondent.�
� Avoid jargon. Avoiding the use of jargon is a standard usability recommendation when designing a computer application.� This rule also extends to the wording that is used in edit messages.� When presented with jargon, respondents noted that they were unfamiliar with its meaning.
� Be polite. Instead of phrasing the messages as commands, the word "please" was used to introduce instructions for respondents.� It was intended to convey a pleasant tone and to reduce any negative perception of the Census Bureau as an imperative authority.�
� Be sensitive to the respondent. Certain words that are synonymous with error messages can be very judgmental and imply that the user is wrong.� Words typically associated with error messages that should be avoided are "error", "discrepancy", and "mistake."
� Beware of offering only one of many possible solutions. Several of the messages used in testing included reminders such as "please check for typing mistakes."� These messages were meant to help the respondent figure out what could have gone wrong.� Participants spent more time than necessary focusing on these reminders.� The reminders seemed to be another hurdle that the respondents encountered in working with error messages. Reminders often led respondents down the garden path and away from thinking about the real problem.�
� Capitalize variable names.� Variable names referring to subject-matter content were used in many of the error messages.� To the respondent, the variable name was a very important part of the error message, especially when it was the only piece of information given in the message.� Participants noted that they preferred seeing the variable name presented in all uppercase because that style made this key piece of information stand out more.�
� Keep messages brief. Participants noted that lengthy messages were more difficult to deal with than shorter messages that seemed to "get to the point" quicker.� It is important to ensure that messages contain all of the necessary pieces of information in a succinct way.
� Use good grammar. Participants stumbled when awkward grammar was used in a few error messages.� The difficulties resulting from insertion of variable names that did not "read" in context suggest that variable names encoded in the metadata need to make sense in any context in which they are inserted for the respondent.�
� Present edit messages as soon as possible. Several respondents felt added burden when presented with their mistakes at the very end rather than at the point where the mistake had occurred.� They would rather have dealt with any errors that occurred at the time and place where they occurred.�
� Format data-entry fields to prevent errors.� Participants noted that an edit could have been very easily avoided if the format that the application required for an answer was presented as part of the actual question.� This convention is widely used in several computer applications and on web pages on the Internet.�
Based on test participants' performance and verbal feedback, we also developed recommendations for wording the various kinds of edit messages.� The following table provides examples of the wording that was tested and the generic re-wording recommended after testing (Nichols, Murphy, & Anderson, 2001):
Table 1.� Examples of Tested Wording and Recommended Wording
|
Tested Wording |
Recommended Wording |
|
Homeowner's Fee is $1,200. Typical values are between $25 and $300.� Please check for typing mistakes.� If entries are correct, explain discrepancy in the remarks section. |
In Item X, typical values are between A and B.� You reported C.� If this value is correct, please explain in the remarks section. |
|
Since you reported found a new home in item 15, please complete signed a sales contract in item 16. |
Please complete Item X, since you completed Item Y. |
|
Sum of down payment percentages is 110%.� It should be 100%.� If values are correct, explain discrepancy in the remarks section. |
In Item X, the values you reported add up to A%.� The total should be 100%. |
The recommended, generic wording for the various types of edit messages is documented in the style guide.
Review of Form Prototypes
As prototypes were developed for the paper versions of the economic forms, a programmer built corresponding prototypes of the electronic form.� These electronic prototypes were revised according to style-guide team consensus.� Decisions made during team discussions were documented for inclusion in the style guide.� In some cases, situations arising from the prototypes called for topics to be added to the style guide (e.g., retention of column headers, indentation, presentation of running totals).� This bottom-up approach complemented the top-down approach taken earlier in the evolution of the style guide.�
As necessary, the team
called in subject-matter experts from the various economic areas (e.g., Retail,
Manufacturing) to assist in making decisions about necessary departures from
the design of the paper forms.� For
example, to avoid horizontal scrolling, the grid design of the paper form for
the manufacturing survey needed to be streamlined to fit the limited width of
the computer screen.� A representative
from the manufacturing subject-matter area attended style-guide team meetings
and participated in developing several options that departed from the
corresponding paper form.� One option
was to use the stacked design that has been used successfully by the COS
(Figure 1).� Other manufacturing
subject-matter experts agreed to this design change, and their representative
brought this decision back to the style-guide team. Rather than risk loss of
data, the electronic form was re-designed to present all the items within the
horizontal display area of the screen.��
Cognitive and Usability Testing in the Field
In the second round of user testing, a partially functioning electronic prototype of the retail form was administered to participants at several companies in the Washington, DC Metropolitan Area.� Accompanied by one or two other team members, a survey researcher visited business locations and administered the prototype questionnaire to participants.� The prototype CSAQ was administered to participants using a laptop from the Census Bureau, i.e., it was not loaded onto the participants' own personal computers (PCs) at their worksites.� During each test session, the laptop's screen was videotaped to record the participant's actual navigational path through the prototype CSAQ.�
Researchers wanted to determine whether the electronic format would support the respondent in comprehending what data to provide and where to enter data.�� Of interest was the respondent's use of navigational features.� As shown in Figure 2, three navigational features were embedded in the prototype:� a set of menu options in the left navigation bar, �next� and �previous� navigational buttons, and hyperlinks for skip instructions and data verification.�
Figure 2.� Prototype of electronic form for the 2002 Economic Census showing a hyperlinked skip instruction (...go to 29), hyperlinked menu options in the left navigation bar, and navigational buttons at the lower right.
Participants in the testing used all three of these navigational features appropriately.� As expected, the hyperlinked skip instructions were the least likely to be used, and the next and previous buttons were most likely to be used.�
Another issue of interest in testing was the method provided for respondents to select answers.� Experience in QFR usability testing had indicated that combining radio-button functionality with tabbing can have serious, undesired consequences.� For example, tabbing from one question to the next can cause the respondent's selected radio button to jump from "Yes" to "No" without the respondent noticing this change.�� Thus, for "select one" choices, the style-guide team is considering checkboxes instead of radio buttons.�
The difficulty with checkboxes is that multiple responses can be selected.� In test situations, participants have quickly discovered that they can select more than one option.� As of this writing, all test participants have noticed the instruction to select only one response and have assumed initially that the computer will allow them to check only one.� To change a response, however, participants have found that they need to perform the additional step of unchecking a box to proceed.� One participant assumed that this "feature" would be corrected in the final form. The team has documented the desired behavior of checkboxes and other user-interface elements in the style guide.���
Summary
Development of the 2000 Economic Census Style Guide drew upon several different but related methods.� Consulting usability principles and user-interface design guidelines got the team started in a top-down direction, but it was necessary to test alternatives and use bottom-up approaches in formulating design rules at the required level of specificity.� Past results (what worked and what did not work for participants) from usability testing of other CSAQ's was valuable in informing team decisions.� Cognitive and usability testing were useful in providing human performance data and observations on which to base decisions.� Once the style guide is complete, for designers will use it in conjunction with forms-design software to design the CSAQs for the 2002 Economic Census.� Revision of the Style Guide for future use can occur as necessary now that the foundation has been laid.��
References
Compuware, Inc.�
(1998).� GUI Guidelines
(Help template).� Madison, WI:� Author.
Microsoft, Inc. (1995).�� The Windows Interface Guidelines for Software Design.� Redmond, WA:� Author.
Nichols, E., Murphy, B., and Anderson, A. E.� (2001).�
Results from Cognitive and Usability Testing of Edit Messages for the
2002 Economic Census (Human-Computer Interaction Memorandum Series #39).
Washington, DC:� U. S. Census Bureau.
Nielsen, J. (1993).�
Usability Engineering.� New York:�
AP Professional/Academic Press.
Shneiderman, B. (1998). Designing the User Interface (3rd ed.). Reading, MA:
Addison-Wesley.��