Return Home

Usability and Methodology
By Michael Korn

September column

This new column is written by members of the San Diego chapter of CHI (Computer Human Interactions). Click here to learn about SandCHI.

Author Michael Korn [see bio] is a senior user interface designer at AOL.

He was SandCHI's inaugural chapter chair and founder, and he currently serves as program chair.

In last month’s column, I wrote that usability could be viewed as:

• a process
  
• a methodology
  
• a bureaucratic gate-keeping mechanism
  

In practice, usability usually contains a bit of all three of these definitions.

While last month’s column focused on process, this column will focus on methodology.

Usability test (Utest) observers and visitors are known for making comments such as "I can’t believe I saw what I just saw."

This phenomena can be very powerful for those who see real people interact with technology. The long-term effects caused by observing what seems unbelievable are not as resilient and long lasting as quantitative data.

In corporate America, numbers and statistics rule the day, rising to the largest offices in the highest towers. In my opinion, behavioral methodology and quantitative data collection are the keys to bolstering the value of usability testing results and conclusions.

As is the case with most things, there is more than one way to sink a putt. Usability is no exception. While there are multiple techniques used to collect and manage data, some usability procedures and techniques are fairly standard across the technology industry. Generally, usability methodology falls into three phases and 10 to 20 procedures.

I list the phases and steps associated with sound methodology below. The purpose of the list is to emphasize the importance of ensuring sound data collection. Without that, usability differs little from basic market research. I have bolded steps that I will describe in more detail.

Concept Phase

1. Understand the business and user objectives.
2. Formulate testing questions or hypotheses.
3. Determine what success and failure looks like, and get buyin from stakeholders.
4. Identify the user characteristics that will influence the achievement of the objectives.
5. Identify the relevant product tasks.
6. Prioritize the tasks.
7. Conceptualize the test flow.
8. Choose the primary metrics.

Implementation Phase

9. Recruit participants.
10. Create pretest (data collected before the utest).
11. Create posttest and interview objectives (data collected after the utest).
12. Run the utest sessions.

Analysis Phase

13. Calculate the results.
14. Formulate conclusions.
15. Formulate design suggestions.
16. Make design changes.
17. Modify step 2.
18. Repeat steps 7 through 18.

In theory, every utest should maintain this level of methodological rigor (Rubin 1994). In practice, some of these steps can be removed, tweaked, or replaced. Why then do I waste your time outlining the theoretical procedures? The reason for knowing the formal procedures is to help demonstrate how utesting succeeds in helping designers create more usable products.

Once a professional has breadth of data and experience, some of the steps can be skipped. Problems emerge, however, when steps are changed without an understanding of the consequences of the change.

I can’t tell you how often I’ve heard folks ask me to run a "quick usability test" on the software engineers down the hall. While this is feasible if the software engineers do not work on the product and the product is being designed for software engineers. It is not feasible, however, if a utest participant works on the product or if the participant is not even remotely similar to the real participants.

Formulate testing questions or hypotheses.

As is customary in the behavioral sciences, research goes from the abstract to the concrete and back again. Research ideas, concepts, and hypotheses are derived from sources such as previous research papers, customer feedback, personal experience, and pie-in-the-sky dreaming. The resultant "great ideas" or hypotheses are the basis for designing utests.

Hypotheses formulated before a utest occurs add credibility to the test results. The increased credibility is due to the decreased chances of serendipitous occurrences being identified as repeatable, common human effects.

Determine what success and failure look like, and get buyin from stakeholders.

If I hypothesize that novice Web users will find it easy or pleasurable to buy movie tickets online, I accept the possibility from the outset that I might be wrong. If the usability engineer (uengineer) and the business stakeholders agree on what a successful movie ticket purchase is, failure can be defined as what’s left. These rigid definitions might seem a bit black and white, but usability doesn’t provide maximum value to the team by focusing primarily on behavior’s shades of gray.

Once success is defined in behavioral terms, it's time to choose your metrics.

Choose the primary metrics.

Once success and failure have been defined in conceptual terms, we look for observable, repeatable human manifestations to represent the construct outlined in the concept.

WHAT?!

When one tries to measure a concept, one accepts that one is trying to associate something nonconcrete, like motivation, and view it in concrete terms. This is not always easy, but it’s a great challenge for the uengineer.

This is one of my favorite parts of the process. We look to identify observable behaviors, effect, and types of comments that can serve as data points. This is one place where a uengineer must be wary of contaminating his/her own test. The goal is choosing the right metrics, not the metrics that are likely to make the uengineer correct.

In the movie tickets example mentioned above, we may want to measure motivation. We then decide that "successfully purchasing movie tickets online" represents high motivation.

We then need to break down this construct into measurable subtasks. These subtasks would include:

• finding the Web site
  
• registering as an online user
  
• choosing a movie
  
• entering personal information
  
• making the purchase
  

If we measure success and failure as time to completion or number of errors, for example, we set a foundation for knowing whether one version of the product design is better than another.

At this point, it should be noted that the uengineer should not be the utest moderator. It’s far too difficult for someone close to the utest concept and design to inadvertently influence participants. The utest moderator, who is experienced in the collection of utest data, is responsible for maintaining the integrity of the testing session.

Once the metrics are established and the utest moderator is in place, the credibility of the testing sessions and, hence, the results are enhanced.

Recruit participants.

The implementation phase begins with the recruitment of participants. Recruiting the most common and/or important user populations is the initial step to implementing the test plan.

Say, for example, a software product is being designed for all White House personnel. The software must serve home-maintenance personnel who may be the most common users while also serving the President who is clearly the most important. The utest must address ease of use for both populations.

Run the utest sessions.

The utest moderator executes the utest plan written by the uengineer. The participant follows the directions outlined by the utest moderator during the 1- to 2-hour sessions.

A usability test session is a laboratory simulation of the user experience associated with the tasks being tested. In other words, the laboratory test should take into account situations where users will encounter the product. Therefore, if the product is going to be used at home 90 percent of the time in a home office right before the dinner hour, efforts should be made to simulate that experience.

The session is usually broken down into small tasks that the participant is asked to complete without help. The utester generally interjects as little as possible while the participant uses the product. When more information is needed than is offered by the participant, the utest moderator will attempt to elicit information in an unbiased fashion.

Interviewing styles vary, but techniques necessary to garner valid verbal responses do not. An experienced utester with advanced interviewing techniques is a pleasure to watch because the utester asks only the right questions at the right time, thereby enlisting rich, valid qualitative data from utest participants.

When the participant completes the assigned tasks, participants usually complete a posttest questionnaire. The utest moderator will then do an end-of-session interview to collect rich, qualitative feedback that will add depth and humanity to the quantitative data.

Often this brief interview is the icing on the cake. This process answers many questions and inevitably brings up new ones.

Make design changes.

Once the uengineer formulates conclusions from the utest sessions, it is incumbent on the utest professionals to interact with the designers to come up with design suggestions. A laboratory environment is not real life. It is not the actual experience. As a result, we need to take the utest results and see how they fit in the real-world environment and design.

The utest professional’s job is not done until the designers understand how best to modify the product design to make the product more usable (Norman 2002).

Note

There is a lot of information in this column. This column is not intended to be a recipe for doing utesting but, rather, an outline and description of some of the procedures for this type of testing. I hope that usability methodology matures over the next few years to the point that it can become a pillar that business owners can rely on to shorten their development time and decrease their costs.

References
Rubin, J. 1994. Handbook of Usability Testing. New York: Wiley Technical Communication Library.

Norman, D.A. 2002. Interactions, Volume IX.4.