To date, usability testing has been largely confined to usability labs. This ensures a controlled environment where users can interact with products or designs and researchers can field questions. The downside of this is not being able to get the context of use of what you are testing. But a recent project for a life science organization cemented the idea that taking user research out of the usability lab yields the best results.
Traditional Usability Testing
When I began my career as a user researcher, I worked at a company that had two usability labs, each had a viewing room with one-way glass. The labs were onsite, and clients would come to our office to watch users work through a set of tasks designed to evaluate a new design or concept, a set of features and functionalities, or an existing website. These sessions were valuable, they allowed us to gain insight into the world of the user, their thought process, where they struggled, what they liked and what they didn’t like.
But this was not without issues. The environment in which we conducted the sessions was sterile – there was a large mirror in the room, behind which there might be people watching. Users knew this because I told them.
Although users do get used to the idea that someone is watching the session, it will not reduce the Hawthorne effect – a psychological theory that people change their behavior when they know they are being watched. This unfortunately is unavoidable in user research, but we can try to mitigate its effects, which we will come back to later.
Furthermore, we were asking users to complete tasks in a space that they were unfamiliar with – one which didn’t have the distractions or the comfort of their workspace or home. So, considerations or things that normally add to or interrupt the flow of their journey do not crop up. At the time, usability labs like this were the gold standard. But this changed as technology moved on and other, more viable, options emerged.
Technology’s Impact on Usability Testing
The next place I worked also had a usability lab onsite, but there was no longer mirrored glass, instead it had been replaced with a video set-up, that streamed the sessions to a separate viewing room for clients. Whether this had any impact on the Hawthorne effect I don’t know, but it did create a more relaxed environment for the usability sessions to take place. As a moderator, I felt that participants relaxed more easily, and, although they were aware that they were being filmed, it was easier to build rapport and have more open conversations. In addition to recording the on screen interaction and the conversation, this set-up had the added benefit of recording the room and the session. This means that clients who are unable to make it to sessions can watch them later and to get additional insight into participants’ body language and expressions as they react to the designs.
Today this set-up is the defacto standard and one that we use in our office in New York. This option is going to work best for most companies. And companies can explore what kind of environment they want to set-up for their users – is it a relaxed one with couches and beanbags or is it something more akin to an office space? This all depends on the users, the business, and what mindset you want the users to be in.
Usability with a Life Science Organization
More recently, I was doing a project for a life science organization. The scientists and technicians do the bulk of their work in laboratories. These, unlike the labs mentioned above, are in fact sterile environments. And the idea of bringing camera equipment and researchers into that environment, while not impossible, is difficult and time consuming to set-up.
At first, we conducted user interview sessions in a usability lab, with a streamed video to a viewing room for the team to see the sessions. This worked well for exploring ideas and testing concepts, but as the project progressed it became clear that in order to really understand if the product was going to work for users, we would need to see how they used it in their natural environment. This meant going into a laboratory.
A New Way of Doing Things
As you can imagine, setting something like this up has many checks and balances. Once it has been approved, which is no small feat, all the equipment that you need for the session needs to be sterilized. This can take a couple of days to a week or two – depending on capacity. And before going into the lab, everyone needs to get suited up, so that there is no contamination by germs or pathogens from the outside world.
Once you get into the lab, the first thing that hits you is the smell – it’s overpowering. A lot of the research involves evaluating the efficacy of antibodies with mice. Once you get used to the stale smell of sawdust and mice, the next thing you notice is the noise – there are large fans and ventilation systems at work, so there is a constant hum in the background. The movement of people is also constrained, in addition to the suits, masks, and gloves that they change regularly, they record data and move between workbenches performing different tasks, all while trying not to cross-contaminate the different work areas.
Seeing this in-person provides a layer of insight that would not have been possible to glean in a usability lab setting. Sure, users could have told us about it, but seeing it first-hand allows us to experience it and in turn have a better understanding of users and the problems that we are trying to solve.
While this session was really useful and we learned a lot about the challenges user face and the environment that they work in, it was not sustainable – long lead time to set-up, equipment has to be sterilized each time, rules about how often we could go into the labs, and disruption to the work of other scientists and technicians while we were in there. This was not going to work within our two-week sprint cadence. It was clear that we needed to find a solution somewhere in the middle.
Evolving the Usability Lab
The solution we settled on was that we would build our own science lab, but in a non-sterile environment. By replicating their lab environment, we could run user research sessions whenever we wanted and still get the benefit of seeing users interact with our designs in a quasi ‘natural’ environment. In order to do this, we gathered the same equipment that is used in the lab – an isolator, a workbench, and a rack where the mice are housed. This was possible with the cooperation of the life science organization, who unsurprisingly understood the value of doing research properly and liked the idea. The new lab was kitted out with the same cameras and microphones that we used in the previous usability sessions.
Once we had set everything up, we were ready to start doing the research. One of the first sessions that we ran in our new user research ‘lab’ involved an iPad app. We were exploring the idea of replacing the cumbersome computer workstation used in the lab with an iPad application, so we created an iPad concept of the software they used to record data in the labs. In addition to evaluating the application concept, we wanted to see how the scientists and technicians would react to using an iPad instead of a computer – does this impact how they do their job? Or do they need to change their workflow because they are no longer using the computer? If yes, is this a good or a bad thing?
Feedback from Users
The feedback we got from users was very positive, they liked the iPad concept and having something that was more mobile really resonated with them. However, one of the most interesting findings was about the context of use and what they do with the iPad when they were not using it to record data. Their workbench was crowded with materials they needed to do their job, but they wanted to be able to quickly see and interact with the data on the iPad at the same time. This prompted us to look at ways in which the iPad could be in easy reach of their workbench and optimize space. This was a very important finding, and one which would have a large impact on how the scientists and technicians did their job, that would not have been uncovered by traditional user research in a usability lab.
Several Important Takeaways
First, if possible, go and see the environment in which the users will use the product or design. This was key for this piece of research. It provided us with great insight into the environment that the scientists and technicians work in and highlighted the constraints and workarounds involved in their day-to-day tasks.
Second, creating a replica of the environment will not be needed for all types of research, but if the context of use of a product can have a big impact on the experience (and the resources are available), then it is something worth considering. We found that putting users into a space that resembled their workspace helped them forget about the cameras and put them in a different mindset, where it felt more like a demo of how they would do their work, rather a ‘test’ or ‘evaluation’, which may in turn have reduced the Hawthorne Effect. Also, because it is more immersive, it makes participants more likely to consider environmental constraints that otherwise might not have come up.
Third, if the resources are not available to recreate a workspace or area, then consider setting-up your usability lab space so that it feels more like the environment in which the product or design will be used. For example, is the environment corporate, where sitting at a desk with a computer is relevant? Or is it more relaxed where there are couches and comfortable chairs? Whatever the environment, small changes in the lab set-up can have an impact on the experience and potentially yield deeper insights.
As new technology and trends emerge the way in which we conduct research will undoubtedly change. New ways of working will lead to deeper insights and help us create innovative solutions. The most important takeaway is, as always, to do research with users early and often. This is the only way to ensure that your product or service will actually work for users and is something that they would want to use.
One comment
Comments are closed.