Testing Your Documents
Testing your documents should be an integral part of your plain language writing process, not something you do afterwards to see if your document is a success. It’s especially important if you're writing to hundreds, thousands, or even millions of people. The information gained in testing can save time in answering questions about your document later.
Types of testing
You can do either qualitative (What do people think about it?) or quantitative (Do the numbers show that this is a success or failure?) testing.
- Protocol testing and focus groups produce qualitative dat.
- Control groups produce quantitative data.
Focus groups are conducted with a small group of people (usually 8-12). They are valuable for gathering information about how people feel about a document. Participants will tell you if they like or dislike something. They will tell you if they understand what you are doing. And they may even tell you a better way to do something.
Prepare a moderator's guide (a list of questions or script for the moderator) in advance. Preferably, the moderator should be a neutral person—not someone involved in developing the document. Design the questions to generate a discussion among the participants about the document they are reviewing. Be objective; don't ask leading questions that will bias the answers. In professional settings, you can use a one-way mirror for observers (usually those with a stake in the project) to watch the participants. This is valuable because it prevents observers' expressions from swaying the participants. Focus groups can be taped to allow the moderator to write an accurate final report. Of course, you need the group members’ permission to watch or tape the session.
While focus groups are valuable in some situations, they are not usually an effective way to test the usability of a document,or to learn how well an individual really understands what you have written.
Protocol Testing involves a one-on-one interview with a reader. It will tell you what a reader thinks a document means. It is valuable in determining if the reader is interpreting your message the way you intended.
You should conduct 6 to 9 interviews on each document. Ask the reader to read to a specific cue (usually a dot identifying a stopping point). Each time the reader reaches a cue, ask for an explanation of what that section means. At the end of the document, ask additional questions, such as.
- What would you do if you got this document.
- Do you think the writer was trying to help you.
- Do you think your friends would understand this document?
This last question is important because sometimes people are more comfortable telling you what they think others might find confusing, rather than admitting that they don’t understand something themselves.
You should use a different type of protocol testing when evaluating long documents, like booklets and regulations. You may know this type of testing as Usability Testing. In this testing, not only do you test for comprehension, but you are also make notes about the way the reader uses the document. For instance you would note how often a reader has to flip from page to page to find references. In other words, you test the document as a whole, not just individual paragraphs.
Protocol testing is time consuming. But the time invested is worth it. Taking the time to test your document up front may save you hundreds of hours later answering questions from your readers, or even producing a second document clarifying the first one.
In this example, the information was so general that it would have generated calls:
Veterans Benefits Administration tested a letter in which readers appeared to understand every word. However, when asked what they would do if they got this letter, most people said they would call VBA’s toll-free number.
The letter was about a replacement check sent because the original check was now out of date. The letter said, "You will receive the new check shortly." Readers indicated that they would call if they didn't receive the check at the same time as the letter. Changing the sentence to show an approximate date they would receive the check eliminated countless phone calls.
In this example, a “term of art” that VBA thought veterans understand would have caused readers to take the wrong action:
When testing a multi-use letter, some readers were confused by the term "service-connected disability." To VBA it means that a veteran has a disability that can be traced back to time in military service." Protocol tests showed that one veteran thought it meant a disability that happened at work. Another thought it meant you had to be injured while in the military, but not necessarily while on duty. Another thought you had to have gotten the disability during combat for it to be considered service-connected.
When each reader was asked a general question about understanding the letter, they all said that it was clear. Yet several would have done something different than what VBA wanted because they had a different definition of "service-connected." The solution was to explain the phrase so that everyone was working from the same definition.
In this example, adding a word to make something more legally sufficient would have caused readers to give incorrect information:
A team working on a form wanted to use the question, “When were you last (gainfully) employed?” They felt that the term “gainfully employed” would gather more legally sufficient and accurate information than just the word “employed.”
Testing showed at least three different definitions of “gainful” employment:
- Any job
- A job that provides benefits or where you can put money away
- A job that keeps you above poverty level
In fact, we learned that different government agencies may have different definitions of “gainful.” But, more importantly, because each reader had a different definition of the word, the agency would have gotten less accurate information if the word had been in the document.
Remember, the goal of protocol testing is to ensure that your readers understand your document, and therefore won't have to call you for an explanation. Although this technique is very valuable, it probably isn't worth the time to test documents that go to only one or a very few people.
Control Studies allow you to collect quantitative data on how well the general public uses the final document you've produced. Control studies can be done in several different ways, but they all have similar characteristics. Before you do a control study, you should know what results you will consider a success. For instance.
- Do you want more calls regarding a certain program.
- Do you want fewer calls asking for clarification.
- Do you want more people to return an application or a payment.
- Do you want fewer errors on forms people fill out?
Having answers to these questions will help you determine whether your document is successful.
Send a small test group of people the new version of your document. Let's say you're sending the new version to 700 people. You should also send 700 people, your control group, the old document. Track the responses to all 1400 documents and compare the results. Note that it is significantly easier to test results when people return a written response than when you try to track the number of phone calls you receive. (If you have a statistician or actuarial staff, they can tell you how many people you should use to make your study scientifically valid. If your agency doesn't have an expert on staff to help you, statistics books will give you a formula to determine a good sample size for your study.)
There are numerous other ways of collecting quantitative data. For instance, you can record what percentage of your "before" letters generates correct responses compared to your "after letters," or what percentage of each letter results in your customer calling you asking for an explanation.
You should use control studies after your qualitative testing is completed and you believe you have the best possible document. That's because control testing will tell you if the new document is a success, but it won't tell you why it is or isn't a success.
When to use what tool
Like any good tool, focus groups, protocol tests and control studies are most successful when used for their intended purpose. The chart below will show you the best times to use different methods of testing documents. Focus Groups and Control Groups are optional depending on what type of document you are rewriting. But for documents that will be read by a large number of people, protocol testing is essential to help you know where you should change your document. The chart below shows the most effective times to use each tool.
Testing Method | When to Use It | What You will Get |
---|---|---|
Protocol Test/ Usability Testing (qualitative) |
After completing a final draft of your document |
Base any revisions to the document on this information. |
Focus Groups (qualitative) |
Before rewriting an old, usually lengthy, document |
|
After rewriting to compare the format of different versions of a document |
|
|
Control Group (quantitative) |
After protocol testing and revising a document or during a pilot |
This information can help demonstrate the success of the rewrite. Or, it you are doing a pilot project and people disagree about how to write a document, a test mailing of two different versions should produce a clear decision on which is more usable for your audience. |
Make sure to pick the right tool, or your results will not give you the information you want.