Interactive edit messages that display a message to a user about a
previous entry are a critical part of most Web-based data collection
systems, but essentially no experimental evidence exists concerning
their effectiveness or impact on the user. Generally, systems use one of
two basic approaches when implementing edits, called hard or soft edits.
With a hard edit, a message is displayed somewhere on the user's screen
(or in a separate window), and the user must take some action to resolve
the action before forward navigation is allowed in the instrument. This
action is often as simple as re-entering a value to ensure that a keying
error did not occur, or the action might require the user to enter some
type of explanation. On the other hand, soft edits do not require a
concrete action by the user. In this case, a user can totally ignore the
message by clicking a navigation button and proceeding through the
instrument.
The design of edit messages can vary on a wide variety of
characteristics, but as can be seen from numerous examples on the Web,
these designs can vary widely. The assumption is generally made that
edits will improve the quality of data collected by getting it right at
the "source." Although this seems perfectly reasonable when dealing with
users who are highly motivated to complete an online form, this
assumption seems less tenable in other applications such as surveys,
where the Web is offered as one of several reporting options (for
example, mail, phone, and fax). In this context, the Web often competes
with other data-collection modes, and a user's perceived burden is an
important consideration. Although edits are potentially useful for
improving data quality, if overused, poorly designed, or confusing, they
might increase respondent burden significantly and, therefore, have
negative impacts on survey response or data quality.
This study was motivated by the frequent observation in usability
tests that soft edits were often missed by users. Therefore, two
high-level research questions were posed:
- How often are edits missed?
- What can be done to ameliorate the effect?
To address these questions, this exploratory study varied the timing
and location of edit messages, as well as the type (hard/soft) of edit
message used. Key dependent variables in this study were the rate at
which edit messages were noticed on their first appearance, and the
success rate that resulted if a edit message was seen (i.e., Was the
suggested action taken?) In addition to the preceding questions, several
other research questions were pursued — specifically, which approach for
presenting edits is preferred by users? Is there a correlation between
users' subjective ratings of overall form usability and observed
performance? And, does the relative location of an edit message in a
form have an impact on its effectiveness?
The results showed that a phenomenon known as "change blindness" had
a major impact on the overall results, resulting in large numbers of
edit messages being completely missed on their first appearance. This
was especially troublesome for soft edits, since many users would
proceed past these without responding to them. Hudson warned about this
phenomenon,[1]
and this study confirmed the warning. Since this study looked at the
location of edit messages in a relatively long online form, the results
also showed that messages were missed far less frequently when they
appeared toward the end of a form, rather than at the very beginning.
This result strongly suggests that familiarity with a new interface is
an important factor in change blindness, and a user's computer expertise
may play a role as well. Other results showed that users preferred to
have edit messages displayed under the item versus at the top of the
page, and that subjective measures of usability correlated with each
other, but did not correlate with an objective measure of performance
(how often an edit message was missed).
[1] "Designing for the Grand Illusion" In SIGCHI
Bulletin, November/December 2001 http://www.syntagm.co.uk/design/articles/grandillusion.htm
Bureau of Labor Statistics
