It starts with any number of scenarios: Design and development have taken too long to produce a prototype, you need to release in three weeks, and you suspect there may be design flaws. You are trying to incorporate usability testing into an Agile development process. Or maybe you simply want to pare down your process to make it shorter and less expensive.
Completing usability testing quickly is a challenge anywhere but especially in consultancies, which have to overcome additional challenges, such as learning a new application. To assure success on these projects, I’ve developed a quick turnaround usability testing methodology (QTUT) that minimizes the time needed to complete testing.
In Part I of this article, I discuss how to make the first three steps of QTUT—Sales & Kickoff, Recruitment, and Preparation—as short and efficient as possible. In Part II, I will discuss the final two steps: Testing and Analysis & Reporting.
Steps in the QTUT process
Step 1: Sales and kickoff
Step 2: Recruitment
Step 3: Preparation
Step 4: Testing
Step 5: Analysis and reporting
Sales and kickoff
A new client or a group within your company has approached you about doing usability testing. They need the results next week, which works out to six business days from today. What should you do?
Before a project kicks off, we typically have a number of discussions with the client to understand their goals and deadlines for our engagement. However, in QTUT, each day spent in the sales and kickoff process takes a day away from testing and analysis. To help our sales team conduct initial discussions efficiently, we’ve prepared a one-page guide outlining the do’s and don’ts of QTUT.
One critical “do” for our sales team is that they should discuss the method with the client and immediately set deadlines for our testing results. Starting with the final due date, an experienced tester can work backwards to determine the dates for testing, beginning recruitment, and finishing the test plan.
It’s also important to review expectations with the client. For example, current clients typically expect a certain level of quality in our deliverables. When we compress a week of work into one day, delivering a perfect document or presentation is impossible, so simply reviewing the timeline and discussing how we plan to shorten the process is extremely helpful.
Since the project must start very quickly, our sales staff and project team use part of the kickoff as a working session. During this working session, we develop specific goals, learn what types of results will be helpful, develop an initial list of testing tasks, and learn about the users that we need to test the application. We also compile a list of what we need from the client. Depending on the project, we may need:
- Information about users for both recruiting and for writing the test script. For example, do people regularly use the application or do they only use it occasionally?
- Training materials for the application and a subject matter expert who can answer questions.
- Background information, feedback, or previous testing materials that give context to the current design or may help us to write screeners and test scripts.
- Access to a stable application, especially when it is under development.
Sales and kickoff tips
- Create a “Do’s & Don’ts” guide to help team members through the process.
- Avoid clients who are rigid, who prefer not to participate in the process, or who want long reports.
- Use the kickoff as a working session to learn about the participants and potential tasks.
- Have a facilitator available with the domain knowledge needed to quickly learn the application.
- Refine your methodology before you have a project.
Recruitment
Before starting the project, you called several of your favorite recruiters to see if they could meet the deadline. You’ve gotten information about participants during the kickoff meeting, and now you simply need to write a screener in less than two hours!
Recruiting quality participants is a big challenge. There are two key components to recruiting: writing the screener and scheduling the participants. The screener is a list of questions used to filter out participants who are not appropriate for your study. Scheduling involves calling potential recruits, asking the questions on the screener, and scheduling participants who qualify and are available.
Because scheduling is time-consuming, it’s critical to write the screener immediately; in fact, we often start and finish the screener immediately after the kickoff meeting. That’s why it’s important to get information about participants during this meeting.
The level of effort required to recruit depends upon whether you’ve recruited similar participants before and whether you handle the scheduling internally or externally. If you’ve recruited similar participants before, you can reuse the same screeners almost verbatim. To facilitate recruitment for new participant types, we’ve built a list of standard questions so that, at worst, we need to write only a few new questions. In addition, we use a screener template that has our facility procedures, location, and contact numbers; the facilitator need only fill in the specific test questions.
External schedulers. We typically use external schedulers, which is particularly helpful for QTUT, because it takes burden off the facilitator. I highly recommend contacting your external recruiters before agreeing to do a study to see if they can meet the timeline that you need. If at least one of your external recruiters cannot meet your timeline, do not hesitate to ask your client for assistance before agreeing to the study.
Internal schedulers. If you do your own scheduling for QTUT, you may have a list of participant leads in house, so writing a screener isn’t always necessary. However, depending on your timeline and your success rate when calling, it’s often necessary to ask your client or a co-worker to make the calls for you.
You might be tempted to schedule testing sessions very closely together in order to complete more sessions per day. However, we find it more efficient to leave time between sessions in order to quickly debrief and begin analyzing what we observed during the previous session (I’ll discuss this analysis method in more detail in Part II). You may also need the extra time to modify the testing script as needed before the next session.
Recruitment tips
- Start recruiting as soon as possible.
- Create a screener template so that you only need to write the questions.
- Reuse questions from past screeners.
- If you do your own recruiting, ask a coworker to make the calls for you.
- Budget more money than usual when using third party recruiting firms.
- Screen carefully but avoid particularly restrictive screeners.
- Schedule backup participants (“floaters”) to cover multiple time-slots in case a participant fails to arrive.
- Leave time between participants to summarize results and to change the tasks as needed.
Preparation
You’ve started recruiting. You are familiar with the application type but you need to learn this particular application. You need to write the tasks and questions for your test script and you need to clear them with your client.
With recruiting underway, you can turn your attention to learning the application and writing the test script.
One critical note in our sales guide is to avoid aggressive timelines for applications in unfamiliar domains or those which require extensive training. Even if you’ve followed this advice, you will still likely need to learn certain aspects of the application.
It may seem obvious, but you cannot learn an application if you do not have access to it. Often the clients who come to us are still developing the application, so be sure to schedule adequate access to a stable version.
After you’ve had a chance to learn the application, writing tasks and questions for QTUT is similar to a normal study. We rely on client stakeholders more than usual because they help to prioritize testing needs and to identify key tasks. In QTUT, most clients are releasing soon so they’re more interested in finding easy-fix problems than in statistical data or qualitative feedback. Consequently, we tend to focus our effort on creating tasks vs. writing a lot of questions. You also need to consider what you can realistically summarize quickly. For example, if it takes two hours to compile error rates, then eliminate error tracking from your test plan.
We run pilot tests as early as possible because they help us more rapidly iterate and improve the script. Keep in mind that you probably will not have perfected your script when your first participant arrives, so you may need to modify slightly the tasks and questions between sessions.
Preparation tips
- Avoid gathering data that takes a lot of time to compile and analyze.
- Use open-ended questions such as “Tell me about what you did yesterday at work” and allow time for discussion with the participant in order to learn about them during their session.
- For tasks that require more information that you currently know about participants, use a series of questions to build more realistic tasks.
- Use Likert-style scales to get data if you need it, and rely less on task completion data (as your tasks may often change).
Next time: Part II
In Part II, I will discuss how to facilitate the sessions, and I’ll describe a novel approach to analyzing and reporting the results.
I was involved in a QTUT last month. I didn’t think of the structure as you have outlined above, but basically set the same expectations. My project was smaller: 8 hours to test a website with three users. 8 is better than nothing. To get the most out of the testing, I suggested that the client sit in on the testing. Although I didn’t have access to recording software, I didn’t see much point in using it, either. It takes a long time to set up, test, review, and write up, and the site wasn’t that complicated. Plus they had done their user research before designing the site, so they just needed confirmation that they had reflected what the research told them.
I spent 1.5 hours setting expectations (this is going to be quick, you won’t hit on more subtle problems, the report will be quick and dirty, then worked on scenarios and the test script. Two people from the client sat in while I tested the user. It was definitely informal. I wrote up a bulleted list in Word.
QTUT is a great concept and its outlined in a lucid and clear way, also emphasized the need for initiating from the frontline team itself. on the other hand, this also highly depends on the project thats been focussed and relied on. Mostly projects that has high end creativity or which actually focussed more on the creative segment to create the difference cannot have any strict do’s and dont’s as we would be expecting it to be, which actually creates the bottleneck of whats the usability technique.
will sites like disney or dreamworks animation be concentrating on the creative difference that the site will have with the world when they create a movie site or will they be involved in the structured way of what we as usability engineers would love to prefer. The case was also highly reluctant when we actually start the QTUT and inturn assign the participants for the same. I ve been involved in such projects where the dilemma arrived when the perspective varied when the client saw the output.
This article is a perfect start for such discussion which actually discusses more on to whats to be done for QTUT right fromt he recruitment of screeners. for site which has larger audience the challenge lies even in recruiting as outlined with more dimensional vision. So the user research part of the section of each of the site is the key start before the site is started. Its a good read for all the Account Managers who are involved in the signoff process. i am waiting for your part II which is going to talk about the analysis. i hope you would take my point of sites having different scope and expectation also in consideration when you publish the analysis so that it would be of great use for professionals like me.
This is a very timely post, especially since so many projects are increasingly becoming “agile-ish”. One additional point on setting expectations during the Sales & Kickoff phase. I’ve found it’s really important beforehand to get at least semi-commitments from all the stakeholders that something will be done with the results once the usability testing is completed. After all, why bother setting up costly time-consuming tests if a manager is not going to budge from a code freeze date within the next week? Or if there are already plans to re-architect a particular feature and the current UI is just a temporary placeholder. Without setting these kinds of expectations the usability tests become an exercise in frustration for everyone involved.
Good point, Anthony. One nice thing about consulting work is that clients have typically already gotten the buy-in for our services before they contact us. But for companies doing internal testing, getting buy-in should be a key part of the process.
I, too, agree with Anthony. Getting a commitment from all shareholders to do *something* with the results before testing begins is important. Thanks for the great article, Paul, I’m looking forward to pt. 2. At my office, where everything is “quick turnaround”, this is, with a couple minor modifications, definitely useful.
Nice Article.
I conducted a QTUT last month for a client with a small site site redesign. The site had limited functionality and I had only a week to conduct the usability tests and compile a list of recommended design changes.
We actually recruited the participants and conducted the tests via email. For the participants, we chose a set of web savvy and creative people via our collective social networks. We offered a downloadable gift and found a number of well-screened and enthusiastic participants!
We then conducted all the tests via email by sending a link to the beta site and a link to a web form with the test questions. Our method was much like the one you listed in the article above and we got great and useful feedback. Of course, we were only able to identify the low hanging fruit and easy-to-fix items…but, we caught many copy, layout, and functional snafus that our design and developement team missed because we were too close to the project!
I’m looking foward to the second part of your article!
I’m glad that you shared this Melissa. I had a limited amount of space to describe QTUT, so I chose to leave out testing online and recruiting through friends. When you are working with a consumer-type application, it can be pretty easy to find qualified applicants through social networks. They also have a bit more motivation to pay attention than normal, since they have a connection to you. Internet testing gains you is a bit more flexibility, although I haven’t found that it makes much of difference in terms of recruiting faster. But our (Electronic Ink) testing facility is pretty conveniently located in Philly, so that may be part of it. What do you think?
Hi Paul, I like that you have documented QTUT and how it should be ran. Following Anthony’s comment that almost most of the projects are becoming “agil-ish”, I have found myself running many QTUT already. So, in my experience:
* Sales & Kickoff. I agree with you, we have to help our sales team and set up proper expectations of the exercise to the client. But I don’t agree with the part that you say to “avoid” clients that are rigid. We cannot always get the perfect client, so we have to accommodate and be flexible with out methodologies and deliverables. In this case, setting up proper expectation with the clients is the key part of the testing, so they know the benefits of the test and what they will get out of it.
* Recruitment: We usually use an external company to do our recruitment and scheduling. We found that this part is very time consuming if done internally.
* Preparation: This is something that I work together with the client. Using the Kick-off session to gather data for this stage is a great recommendation. We usually create the initial script and give it to the client to review. On the script we highlight any conditions/assumptions/dependencies of the testing, for example, if it is a prototype, only few pages and features are working, if it is a major website or application, we highlight that we will only be testing an specific section, etc. This goes back to the “Sales and Kick-off” step of setting up correct expectations.
Good article, looking forward for part 2.
Nice article Paul.
Specially step 1: Sales & Kick off together with last recruitment tip “Leave time between participants to summarize results and to change the tasks as needed “ should be considered throughout the usability testing process. I found myself several times creating a “Do’s & Don’ts” guide to help team members through the process but after testers feedback I had to reconsider the list and updating it. Time limitation is a factor that leaves you no space to look back on your results in detail but is good to save some time between sessions to examine your results so far. This might help you avoid future mistakes in the same study and optimise the user testing throughout its own process.
Wait to read Part II.
Good article paul. Thanks for the write up of tips. I particularly liked the fact that you put this statement in “Use Likert-style scales to get data if you need it, and rely less on task completion data “. As a practitioner of QTUT one should realise that there are things more important than completion times or stats like that. QTUT is very useful in situations where one needs to summarize (as results) the usability problems that are out of reach of heuristic analysis and yet with in reach of a time restricted quick testing. So time data or error data might not be as useful as where exactly are the people getting lost or where exactly is the system failing to give status notification etc… In other words “wheres” and “whys” become more important than stats due to time contraints.
thanks again and waitin for part II
Being particularly clear about the test goals and what specifically you’re testing for is key to drafting the correct set of questions and getting the right test participants. Great article!