Quick Turnaround Usability Testing, Part II

Written by: Paul Nuschke

In Part I, I discussed how to make the first three steps of Quick Turnaround Usability Testing (QTUT)—Sales & Kickoff, Recruitment, and Preparation—as short and efficient as possible. In Part II, I discuss the final two steps: Testing and Analysis & Reporting.

Steps in the QTUT Process

  • Step 1: Sales & Kickoff
  • Step 2: Recruitment
  • Step 3: Preparation
  • Step 4: Testing
  • Step 5: Analysis & Reporting

Testing

It’s testing day. You have successfully recruited enough participants for the first day, but you feel a bit of panic as you make the finishing touches before the first participant arrives. You have a rough but solid test script. You have five attentive stakeholders in the observation room ready to begin taking notes. Now you need to execute the test and you need to compile results as you go along.

Up to this point, the lack of time you had to plan and to refine your method creates a bit of a panic as you begin the testing phase. Often, we are working on the script until the very last second, incorporating changes from the stakeholders that they hand off when they arrive at our testing facility.

Early on the test day, I print out a screenshot of every important page and component (e.g., the primary navigation). I number these screenshots and then tape them above a large whiteboard that we keep in an “idea” room that is adjacent to our usability lab’s observation room. We use the whiteboard to keep track of issues and metrics across participants.

After you finish each participant session, immediately note changes that you need to make to your test script or the application. Then go talk with your stakeholders about the results. Here, the time that you have budgeted between sessions for discussion really pays off. If your stakeholders have been watching and taking notes, they are likely already talking about the results. They may also be already talking about potential fixes.

The whiteboard can be useful for focusing the discussion on issues. It’s often useful to set ground rules for the discussion. First, the discussion should focus on results and not solutions. It is important to manage your stakeholders by telling them to be patient and to let the results play out over several people before drawing conclusions. Second, as the person facilitating the study, you should lead the discussion on each topic by first summarizing your notes on the whiteboard. After we summarize a page, we ask for any additional feedback from the stakeholders and then quickly move on to the next page. Occasionally, we need to remind the stakeholders that since we have limited time between participants, we cannot dwell on any one finding.

You may think that stakeholders will not have much valuable feedback to add, but we have found that they often see things that we don’t because of their knowledge of the history of the application. For example, one client had just made a political decision to change a button label. Since the participants understood the new button name, we didn’t think to list it as a finding. But for the stakeholders, it was helpful to track it and mention it in the report.

After the first participant’s session, you will likely realize that some of your tasks and questions were not worded correctly. Through the whiteboard session, you may think of additional questions to ask participants, or you might even want to add, delete, or change tasks. Do not rely on your memory to do any of this. Instead, make the changes to your script and print out a new copy for the next participant.

At the end of your first day of testing, you should have a board full of findings, including some task completion data. If you test five participants per day, by the fifth participant trends are becoming clear. You may decide to stop a task because you already know the issues, which allows you to test other tasks that you couldn’t fit in. Discuss this at the end of the day so that you can make the necessary edits to your test script before you leave for the day.

In addition, if your stakeholders are already discussing potential fixes while you facilitate the test, it is important for you to be a part of that discussion. In their eagerness to fix things, our stakeholders occasionally solve the problem in such as way that a larger problem is created.

If you are in a fast-paced development environment, where developers are staying until 10 p.m. at night to make changes, your stakeholders may want to change things that night. Because of the potential to affect your testing, you should be very careful to attempt only easy fixes. You do not want to rework the navigation or radically alter a task flow overnight. However, you may want to change a button’s behavior or the text in a label.

As sessions progress, your whiteboard may start to fill up. Usually it is easy to condense the notes. We use a Post-It note for each issue and then start writing participant numbers on the note for each person that experiences the same issue (see Figure 1). We take pictures of the whiteboard throughout the process, prune non-issues, and type up results for tasks and questions that we have stopped using.

whiteboard

Figure 1: Whiteboard full of findings after two days of testing. Post-It notes contain reoccurring issues.

Finally, as you reach the last few sessions, increase the level of detail about potential recommendations. If possible, test these ideas with your participants. Get a sense from your stakeholders about what recommendations are feasible and cost effective on a short timeline, and which require long-term attention.

Testing Tips

  • Keep the testing as simple as possible—for QTUT avoid technology such as eyetracking and anything else that makes your job more difficult or confusing.
  • For changes to your tasks and questions, modify the test script and do not rely on your ability to remember the changes.
  • Be willing to change questions and tasks. It is better to find and to fix the issues with your testing script early on.
  • Remind your stakeholders that because of the aggressive timeline your test script is not perfect, so you may have to change tasks and questions on the fly for the first couple of participants.
  • Keep a whiteboard in the observation room and use it to keep track of and to discuss problems with your stakeholders after each session.
  • At the end of the day, summarize the results and discuss potential recommendations. Consider making a list of the most important findings from the day.
  • Expect long work hours.
  • Don’t duplicate effort by typing up the results before you have tested all of the participants; write up the results only when you have stopped testing a task or asking a question.

Analysis & Reporting

You have made it through two days of testing. The stakeholders have left and now you have to create a report that clearly communicates the issues and your recommended fixes.

The beauty of the whiteboard method is that your report becomes simply a summary of what you have already written on the whiteboard, including completion metrics, findings, and recommendations that have been vetted by key stakeholders.

Although the people who attended the sessions may understand the issues, you still need to translate them from shorthand whiteboard notations to a format that a wider audience can read. We use a presentation-style report because it is much faster to create than a report with lots of text.

Because time is important to the stakeholders, we classify recommendations as short-term and long-term fixes. Short-term fixes can be finished in a few days, or before the application is released. Long-term fixes require more time than your clients have, but they are severe enough that they need to be addressed eventually.

Our QTUT ends when we present the results to the stakeholders, which often includes several interested people who were not able to attend the sessions. We present our findings in person so that the stakeholders feel more comfortable asking questions about the test and so they can get immediate clarification on findings that have been inadequately described.

Analysis and Reporting Tips

  • Use a presentation-style report format that does not require long text passages.
  • If you will be presenting results to stakeholders who did not attend the sessions, give the necessary context to your findings.
  • If you change recommendations that you discussed with your stakeholders, let them know in advance so that they do not feel undermined.
  • Classify your recommendations in terms of the expected timeline for the fix.
  • Take pictures of your whiteboard in order to preserve your notes quickly and safely, enabling you to write the report from another location.

Quick Turnaround Usability Testing

Written by: Paul Nuschke

It starts with any number of scenarios: Design and development have taken too long to produce a prototype, you need to release in three weeks, and you suspect there may be design flaws. You are trying to incorporate usability testing into an Agile development process. Or maybe you simply want to pare down your process to make it shorter and less expensive.

Completing usability testing quickly is a challenge anywhere but especially in consultancies, which have to overcome additional challenges, such as learning a new application. To assure success on these projects, I’ve developed a quick turnaround usability testing methodology (QTUT) that minimizes the time needed to complete testing. In Part I of this article, I discuss how to make the first three steps of QTUT—Sales & Kickoff, Recruitment, and Preparation—as short and efficient as possible. In Part II, I will discuss the final two steps: Testing and Analysis & Reporting.

Steps in the QTUT Process

Step 1: Sales & Kickoff
Step 2: Recruitment
Step 3: Preparation
Step 4: Testing
Step 5: Analysis & Reporting

Sales & Kickoff

A new client or a group within your company has approached you about doing usability testing. They need the results next week, which works out to six business days from today. What should you do?

Before a project kicks off, we typically have a number of discussions with the client to understand their goals and deadlines for our engagement. However, in QTUT, each day spent in the sales and kickoff process takes a day away from testing and analysis. To help our sales team conduct initial discussions efficiently, we’ve prepared a one-page guide outlining the do’s and don’ts of QTUT.

One critical “do” for our sales team is that they should discuss the method with the client and immediately set deadlines for our testing results. Starting with the final due date, an experienced tester can work backwards to determine the dates for testing, beginning recruitment, and finishing the test plan.

It’s also important to review expectations with the client. For example, current clients typically expect a certain level of quality in our deliverables. When we compress a week of work into one day, delivering a perfect document or presentation is impossible, so simply reviewing the timeline and discussing how we plan to shorten the process is extremely helpful.

Since the project must start very quickly, our sales staff and project team use part of the kickoff as a working session. During this working session, we develop specific goals, learn what types of results will be helpful, develop an initial list of testing tasks, and learn about the users that we need to test the application. We also compile a list of what we need from the client. Depending on the project, we may need:

  • Information about users for both recruiting and for writing the test script. For example, do people regularly use the application or do they only use it occasionally?
  • Training materials for the application and a subject matter expert who can answer questions.
  • Background information, feedback, or previous testing materials that give context to the current design or may help us to write screeners and test scripts.
  • Access to a stable application, especially when it is under development.

Sales & Kickoff Tips

  • Create a “Do’s & Don’ts” guide to help team members through the process.
  • Avoid clients who are rigid, who prefer not to participate in the process, or who want long reports.
  • Use the kickoff as a working session to learn about the participants and potential tasks.
  • Have a facilitator available with the domain knowledge needed to quickly learn the application.
  • Refine your methodology before you have a project.

Recruitment

Before starting the project, you called several of your favorite recruiters to see if they could meet the deadline. You’ve gotten information about participants during the kickoff meeting, and now you simply need to write a screener in less than two hours!

Recruiting quality participants is a big challenge. There are two key components to recruiting: writing the screener and scheduling the participants. The screener is a list of questions used to filter out participants who are not appropriate for your study. Scheduling involves calling potential recruits, asking the questions on the screener, and scheduling participants who qualify and are available.

Because scheduling is time-consuming, it’s critical to write the screener immediately; in fact, we often start and finish the screener immediately after the kickoff meeting. That’s why it’s important to get information about participants during this meeting.

The level of effort required to recruit depends upon whether you’ve recruited similar participants before and whether you handle the scheduling internally or externally. If you’ve recruited similar participants before, you can reuse the same screeners almost verbatim. To facilitate recruitment for new participant types, we’ve built a list of standard questions so that, at worst, we need to write only a few new questions. In addition, we use a screener template that has our facility procedures, location, and contact numbers; the facilitator need only fill in the specific test questions.

External schedulers. We typically use external schedulers, which is particularly helpful for QTUT, because it takes burden off the facilitator. I highly recommend contacting your external recruiters before agreeing to do a study to see if they can meet the timeline that you need. If at least one of your external recruiters cannot meet your timeline, do not hesitate to ask your client for assistance before agreeing to the study.

Internal schedulers. If you do your own scheduling for QTUT, you may have a list of participant leads in house, so writing a screener isn’t always necessary. However, depending on your timeline and your success rate when calling, it’s often necessary to ask your client or a co-worker to make the calls for you.

You might be tempted to schedule testing sessions very closely together in order to complete more sessions per day. However, we find it more efficient to leave time between sessions in order to quickly debrief and begin analyzing what we observed during the previous session (I’ll discuss this analysis method in more detail in Part II). You may also need the extra time to modify the testing script as needed before the next session.

Recruitment Tips

  • Start recruiting as soon as possible.
  • Create a screener template so that you only need to write the questions.
  • Reuse questions from past screeners.
  • If you do your own recruiting, ask a coworker to make the calls for you.
  • Budget more money than usual when using third party recruiting firms.
  • Screen carefully but avoid particularly restrictive screeners.
  • Schedule backup participants (“floaters”) to cover multiple time-slots in case a participant fails to arrive.
  • Leave time between participants to summarize results and to change the tasks as needed.

Preparation

You’ve started recruiting. You are familiar with the application type but you need to learn this particular application. You need to write the tasks and questions for your test script and you need to clear them with your client.

With recruiting underway, you can turn your attention to learning the application and writing the test script.

One critical note in our sales guide is to avoid aggressive timelines for applications in unfamiliar domains or those which require extensive training. Even if you’ve followed this advice, you will still likely need to learn certain aspects of the application.

It may seem obvious, but you cannot learn an application if you do not have access to it. Often the clients who come to us are still developing the application, so be sure to schedule adequate access to a stable version.

After you’ve had a chance to learn the application, writing tasks and questions for QTUT is similar to a normal study. We rely on client stakeholders more than usual because they help to prioritize testing needs and to identify key tasks. In QTUT, most clients are releasing soon so they’re more interested in finding easy-fix problems than in statistical data or qualitative feedback. Consequently, we tend to focus our effort on creating tasks vs. writing a lot of questions. You also need to consider what you can realistically summarize quickly. For example, if it takes two hours to compile error rates, then eliminate error tracking from your test plan.

We run pilot tests as early as possible because they help us more rapidly iterate and improve the script. Keep in mind that you probably will not have perfected your script when your first participant arrives, so you may need to modify slightly the tasks and questions between sessions.

Preparation Tips

  • Avoid gathering data that takes a lot of time to compile and analyze.
  • Use open-ended questions such as “Tell me about what you did yesterday at work” and allow time for discussion with the participant in order to learn about them during their session.
  • For tasks that require more information that you currently know about participants, use a series of questions to build more realistic tasks.
  • Use Likert-style scales to get data if you need it, and rely less on task completion data (as your tasks may often change).

Next time: Part II

In Part II, I will discuss how to facilitate the sessions, and I’ll describe a novel approach to analyzing and reporting the results.