US-CERT Usability Lessons Learned

Executive summary

The United States Computer Emergency Readiness Team (US-CERT) conducted two rounds of usability testing on a Web site that has two main audiences: technical and non-technical (home) users. Each audience has its own sets of needs when using the site and the site has to accommodate both levels of users. In addition to the two primary audiences, the site also has a small audience of government users.

After the first, or baseline, usability test was completed, US-CERT made a variety of changes to the site including design of the home page, terminology, and presentation of content. After the completion of these changes, US-CERT conducted a second, comparison, test to gauge the success. This comparison test yielded a number of favorable results. Overall, technical users' success improved 24%, their satisfaction improved 16%, non-technical (home) users' success improved 20%, and their satisfaction improved 93%.

The purpose of the usability evaluations was to

• measure users' abilities to perform tasks
• obtain users' impressions of the Web site
• identify difficulties involved in using the site
• develop recommendations for improvement
• compare users' performance and satisfaction to the baseline usability test

The goals of the testing were to ensure that the site was

• easy to navigate and use
• useful to the intended audiences
• satisfying to use

Introduction

US-CERT wanted to improve navigation, usefulness, and satisfaction of its Web site (http://www.us-cert.gov/). This posed a challenge because there are two main target audiences with different information-level needs. The site serves technical (e.g., computer professionals) and non-technical users (e.g., home and business users), and a smaller audience of government users.

We conducted a baseline and a follow-up usability study. Following the first usability test, we changed the design of the home page, terminology, and presentation of content. Next, we conducted a comparison test to gauge the success of the changes, which yielded a number of favorable results.

Overall, users' ability to successfully accomplish tasks and their satisfaction ratings of the site increased significantly. Most notable was the task completion. Users were far more successful in correctly completing the assigned tasks. (See Graphs 1 and 2.) This improved task completion impacted the participants' satisfaction with the site. Because they were able to figure out how to use the site, their answers to the ten satisfaction questions were far more favorable after the follow-up test.

Graph 1: Task completion for the baseline usability test	Graph 2: Task completion for the follow-up usability test

Test participants

Before we recruited test participants, we created several personas to represent our target audiences. (A sample persona describes a home user.) The test participants were chosen based on their compatibility with the personas. We used the same personas for both rounds of the usability tests. A key characteristic for test participants, even the home users, was the initiative to search for solutions and resolve computer issues on their own. This characteristic is necessary because our target audience is made up of people who seek out solutions and who would come across our Web site by searching for information to solve computer issues.

The nine participants selected for the baseline test and the eight participants from the follow-up test reflected different aspects of each of our personas:

• Five baseline and follow-up test participants were non-technical users.
• Four baseline and three follow-up test participants were technical users.

The tests

Before completing assigned tasks, all users first attempted to solve a computer issue by themselves.

During the tests, participants individually

• completed a user profile questionnaire
• answered questions about initial site impressions
• performed real-world tasks on the site while thinking aloud
• rated satisfaction using a post-test questionnaire
• answered questions about the site in a post-test interview

In addition, non-technical home users were given content to review. They read the content, answered questions designed to evaluate their comprehension, and answered subjective questions about the content.

Lessons learned

There were five lessons learned:

1. Give fewer options.
2. Choose link labels carefully.
3. Put subscription options for mailing lists in multiple places.
4. Make mailing list instructions detailed and in numbered steps.
5. Keep the color palette simple.

1. Give fewer options.

   The original site tried to allow for multiple pathways to information. The home page included Prepare and Respond links as well as Technical and Home User entry points. Instead of being empowered by the ability to choose how to use the site, users were confused by the variety of choices.

   By simplifying how to find information and giving users fewer choices, we improved the usability of the site and allowed users to find the information appropriate for them.





2. Choose link labels carefully.

   Participants did not understand many of the link titles, and they thought the link labels were ambiguous. Also, one of the main categories on the site, Home Users, was unclear.

   We rearranged titles on content pages, removed the numbers and dates from the titles of non-technical documents, and grouped content in meaningful categories. These changes greatly improved usability for our non-technical audience. They were able to quickly find the information they needed and understood which sections of the Web site were designed specifically for them.









3. Put subscription options for mailing lists in multiple places.

   We had a persistent link in the left hand navigation for users to sign up for our mailing list. This link appears on all pages of the site. Unfortunately, users did not see it. Also, once they got to the instruction page for signing up, they were confused because it was not clear that links in the product descriptions led to index pages for individual products. They expected to be able to subscribe to the mailing list from within the main content areas of the site. We kept the persistent link in the navigation and added links to the index pages for each product.

   

   This image shows the main page for subscription information. Users can click "View all" for any of the alerts and be taken to the index page. From that index page, they will link back to the anchor for "Subscribing to a mailing list." This format proved successful and allowed users to understand the products to which they were subscribing without leading them to believe they were subscribing by clicking a link in the descriptive text.

4. Make mailing list instructions detailed and in numbered steps.

   Because the process for signing up for our mailing list requires multiple steps, users often failed to properly subscribe and unsubscribe. Changing the instructions from three short sentences to a numbered step-by-step process improved the success rate for subscriptions and eliminated many frustrated email messages to our Web master from users unable to subscribe or unsubscribe to our lists.

5. Keep the color palette simple.

   Participants were startled by the overuse of the red, white, and blue palette; they felt that the site was cluttered. We simplified the color palette by using only a few different shades of the same color throughout the site and streamlining all graphics and content to ensure that the site had a clean look and feel.

   

   

   Sample Persona

   John, a home user

   John is a 68-year-old retiree. He closed down his accounting business at 65 to spend time traveling with his wife and spoiling his grandchildren. He and his wife of 40 years live in the same neighborhood as their son and daughter-in-law in suburban San Diego. They spend a lot of time with their grandchildren and even more time emailing pictures of the family to relatives who live back East.

   Most of John's computer knowledge is self-taught. He is taking his time learning how to alter photographs and gets up early in the morning to use his DSL Internet access to do research on vacation spots and to keep up with his email correspondence. He takes a methodical approach to everything that he does, and tends to find information slowly and through a trial and error process. Unfortunately, this means that he hits a good deal of "bad" Web sites in his search for good information.

   John would rather figure something out on his own than ask anyone for help, so he often uses search engines and his bookmarked sites to solve any technical problems he has with his computer or software. After a good deal of searching, he can usually fix the problem himself; if he can't, he'll ask his son or call the customer support line listed for his computer.