Woman pointing to phone explaining something.

Usability Testing Guide

The process of conducting a usability test

Posted by

Usability testing is a core component of User Centered Design and can be used at any stage in the process. It provides valuable insight into the mind of the user, giving us a better understanding of users’ mental models, and it helps to highlight issues that might negatively impact the experience, while also pointing to solutions. If you are new to Usability Testing and want to learn more or just interested in how someone else approaches it, this article gives an overview of how to set-up and run a usability test, and provides a checklist of things to do to complete a usability testing project.

Preparation 

Here is a brief outline of the different stages involved in setting up a Usability Testing session. I will go into each in greater detail and explain what it is and what you need to do.

  • Briefing Meeting
  • Participant Specification
  • Recruitment
  • Discussion Guide
  • Consent Form
  • Setting Up the Session

Briefing Meeting

This is a meeting with the client or the product owner to see the assets that will be used in the usability testing sessions. It is your opportunity to learn the objectives of the research, who the participants are, the journeys that will be evaluated, and discuss what the final deliverable of the research should be. It also gives you an opportunity to ask any questions about the assets or the research. Coming out of this, you should have a clear understanding of what the research is about and the specific questions that the research should answer. You should also have an idea of how you are going to approach the research in terms of methodology.

Participant Specification

Based on the briefing meeting and, if available, personas or mindsets supplied by the client, you can start to think about the different users that you want to recruit for the research. Knowing what the different user types are, and the number of user types, will help you gauge how many participants you will need for the user test. If the user base is one homogenous type of user, you can pick up most of the usability issues with 5 to 6 users. However, if there is more than one distinct user type, you may want to increase this number to accommodate the different user types. If there are many distinct user types and the budgets do not allow 5 to 6 users for each, you should aim for at least 3 per distinct user type. This will give you a good understanding of the issues.

There are two options available to specify the participants that you want to recruit. The first is a recruitment screener, which is the more detailed of the two, and a participant spec.

Recruitment screeners are particularly useful if you are using external recruiters that are not familiar with the target audience. They have the following characteristics:

  • One to two pages length
  • Can vary in complexity depending on participants being recruited
  • Generally consists of multiple choice questions
  • Moves from high level to more specific questions about the profile’s characteristics
  • Screeners can be implemented online or administrated over the phone to potential participants

Participant specs are less detailed than recruitment screeners and are useful when the profile being recruited is more generic and does not require a complex screening process. They may have the following characteristics:

  • Number of participants to be recruited
  • Demographic profile – age, gender, income
  • Technology proficiency – high, medium, low
  • Persona / segment type
  • Specific user behaviors of the product being evaluated

Recruitment

Depending on the particular user type of your research, you may be able to use 3rd party recruiters. The cost of recruitment and incentive that you need to pay the participant will vary based on how difficult the participant is to find and how time poor the participant is. You should allow at least 2 weeks for 3rd party recruiters to recruit participants, but for more complicated participant specifications you may need a longer lead time.

In financial services and highly skilled industries, you may need to get clients to help with recruitment; this is because 3rd party recruiters do not have access to this type of user or standard incentives are not sufficient to get these users to give up their time.

Discussion Guide

The discussion guide is the document that brings together all the elements of the research and shows your proposed approach. It offers the client the opportunity to feedback on the tasks and questions for the usability tests.

The document should start by outlining the objectives of the research. You can also include the participant types in this document, but it is not mandatory as you will have agreed and kicked off recruitment at this stage. Next, there should be a brief description of how the sessions will be run, for example – ‘The research will be task-based usability testing sessions. Participants will be asked to think-aloud as they complete the tasks. The sessions will be conducted in-person/remotely…’

The final section is for the tasks. Write tasks in clear and simple language so participants whose first language is not English can understand them. Each task should have a list of (mostly open) questions that probe the areas that you are interested in. It should also include observations and things you need to look out for during the sessions. 

Some other things to consider:

  • Tasks should be realistic and similar to real-life situations 
  • Participants respond better to scenarios than instructions, so tasks should be scenario based
  • Tasks should direct users to the parts of the products that are being evaluated 
  • The goal or endpoint of the task should be the page or the end of the journey that is being evaluated
  • A task can be created for each area if there is more than one

You should avoid the use of leading questions, i.e. questions that suggest the answer you are looking for. Some examples of questions are:

  • Would you rather use the old version or this improved version of the website?
  • Do you find this feature frustrating to use?

You should also avoid asking two questions at the same time, for example:

  • Is this service relevant and useful to you?

This is essentially two questions, the service may be relevant, but not useful to the participant, thus making it harder to answer. 

Depending on the length of the session and the number of elements you want to get feedback on and the number of tasks will vary. 

Consent Form

This is an important yet often overlooked part of the process. A consent form is a document signed by the participant, which shows that they agree to take part in the research and outlines the following: 

  • What the research is about
  • What the participant is agreeing to do as part of the research
  • That participation is voluntary and they do not have to complete the session or any questions or tasks they are uncomfortable with
  • What will be done with the data and how it will be used
  • Who will have access to the data and the results of the research

In addition to protecting you as a researcher against liability, consent forms are great for setting the context of the research for participants before they begin the session.

Setting Up The Session

Before the first day of testing, it is important to set everything up and run a pilot study with a participant. The participant does not have to be a user type that you’ve recruited, they can be anyone that is not familiar with the research. The purpose of the pilot is to test the set-up and the flow of the tasks and questions. It is recommended that you run the pilot a day or two before the first session so that you have time to make changes if you need to. You should run through the usability session as if it were a real session, getting the participant to complete all the tasks and answer all the questions. This will also allow you to check all the software and equipment is working before the day of testing.

Usability Testing 

Conducting The Session

Before the session starts, you should get the participant’s consent to conduct the research. This involves getting them to sign a one-page consent form, as discussed earlier.

At the beginning of the session, it is important that you set the scene and ensure the participant is relaxed. To do this, you should:

  • Ask the participant some generic questions about themselves
  • Clearly explain the purpose of the research
  • Explain that they are not being tested and there are no right or wrong answers
  • Tell them you did not design the assets (I would sometimes do this even if I did) – participants may not answer honestly if they think you designed it
  • Tell them who is viewing the session and whether it is being recorded
  • Ask them if they have any questions

Before starting the tasks, it is useful to ask the participant to tell you about themselves and how their experience relates to the session. This should help them relax and allow you to build rapport with them by showing interest in them and what they do.

You may want the participant to ‘think-aloud’ as they complete the tasks. This will be an unfamiliar concept to most participants, so demonstrating how a ‘think-aloud’ works will help them understand what you are asking them to do. During the session, some participants may feel uncomfortable doing this and may need to be prompted to keep talking.

Note: if you are using eye tracking you should not get participants to ‘think-aloud’ as it will interfere with the eye tracking data. Similarly, if you are taking note of ‘time on task’, ‘think-aloud’ should not be used, as users may take longer to complete tasks as they are thinking and talking about what they are doing as they do it.

Over the course of the session, users are likely to have questions. At the start of the session, explain that you are interested in hearing the types of questions they have, as it will help you better understand their thought process and the issues they encounter. However, also explain that you will not answer these questions as you want to make the session as realistic as possible (you will not be there in real life to help them) and you want to see how they overcome issues when they encounter them. If they encounter an issue that they are unable to overcome, make a note of it and move onto the next task. It is OK to show the participant how to complete the task if it doesn’t impact any of the subsequent tasks. 

Your discussion guide will contain mostly open questions, when you ask these questions, give the participant time to answer. If the participant doesn’t answer straight away is it easy to assume they are not going to answer and quickly move onto the next question. It is important to give them time to think and then answer the question you asked or ask a question. So be patient and allow long pauses – be comfortable in the silence. Long pauses give participants a cue that you are expecting them to say something. It is important to probe their answers as well, so that you can fully understand what they mean. For example, if a participant says that they ‘like’ or ‘prefer’ something (to something else), always ask why. Knowing that someone likes or prefers something does not give us a lot of insight on its own. 

Also, participants will often feel that it is their fault that they cannot do or find something, so it is important to stress that if they cannot find something it is a reflection of the design rather than themselves, and that these are the types of things that we want to catch before the design goes live.

At the end of the session, is it useful to get the participant to reflect on the session and get the overall impression of what they have seen. You can also ask them about the things that they liked and disliked the most from the session, and their thoughts on the assets on the whole. You can also use this time to give the participant an opportunity to talk about anything they would like to mention – by asking: “Is there anything that we haven’t talked about that you think would be useful for us to know?”

Reporting And Analysis

Depending on the level of detail required by the report, there are two options available to you – high level findings report and a full usability report, which will be discussed a bit later. First, analyzing the feedback.

Analysis

A lot of data is generated in a user testing session and it can be daunting trying to make sense of it all. There is no standard approach to analyzing the data, but the following methods are useful to draw out the findings.

Walkthrough the tasks/prototype

Step through the tasks that the users completed and make a note of all the findings that you remember for each page. Check back through your notes afterward to see if you have missed anything.

Analyze by page

During the session make a note of the participant’s issue and the page it is on, then go back through your notes afterward and check findings per page per participant. This allows trends in the data to emerge and the extent of an issue with participants to be seen.

Write a list of the high-level findings

This can help you decide how to structure the report and is a good reference to have when writing the findings up.

It is always recommended to have a second person taking notes, this will help with compiling the findings and allow you to focus on the session and the user. But this is not always possible due to budget or staffing shortages. If this is the case, there are plenty of transcription options available, such as Transcribe in Word, or apps like Recorder on Android, Descript, or Otter. These will allow you to get a full transcription of the session which you can look through in full or search for the parts that you are interested in.

Reporting

High level findings report

This is a document that contains a list of all the major findings from the usability testing sessions. It is a slimmed down version of a full usability report. Like a full usability report, it can contain screenshots of issues and recommendations. Typically, this type of report will be written in 2-3 days.

Full usability report:

This is a detailed document containing all of the significant findings from the research. It is a larger and more detailed version of the high-level findings report. It will contain information on the methodology, who the participants were, a summary of findings, and a list of the findings (usually one page per finding), with recommendations. This type of document will be written in 5 to 7 days, depending on the number of participants.

Wrap Up

I hope that gives you a good overview of the process of conducting a Usability Test. There is no one way to set-up and run a Usability Test, but preparation is key to success. Here are some key things to remember:

  • Check your participants – ensure that the participants are correct and meet the requirements before the session. If you end up with the wrong participants, you are not getting the feedback you need and your results will be off
  • Discussion guide – allow time for clients to read and feedback on the discussion guide. This ensures that everyone is on the same page and there are no surprises where the client expected a different outcome from the sessions
  • Run a pilot – this is important! The last thing you want is the camera, mic, or software to stop working at 9am the day of the session. I learned this the hard way. Run a pilot!
  • Relax – don’t be too stressed by the sessions. Relax and enjoy them. If you are relaxed, the participants will be more relaxed and you will get better feedback. Things will go wrong, and that’s OK. It is rare that something goes so wrong that the session cannot be used.

Please feel free to leave any questions or comments below. I would love to hear your thoughts.


Photo by Headway on Unsplash