Your Guide to Online Research and Testing Tools

The success of every business depends on how the business will meet their customers’ needs. To do that, it is important to optimize your offer, the website, and your selling methods so your customer is satisfied. The fields of online marketing, conversion rate optimization, and user experience design have a wide range of online tools that can guide you through this process smoothly. Many companies use only one or two tools that they are familiar with, but that might not be enough to gather important data necessary for improvement. To help you better understand when and which tool is valuable to use, I created a framework that can help in your assessment. Once you broaden your horizons, it will be easier to choose the set of tools aligned to your business’s needs. The tools can be roughly divided into three basic categories:

  • User testing: Evaluate a product by testing it with users who take the study simultaneously, in their natural context, and on their own device.
  • Customer feedback: Capture feedback of customer’s expectations, preferences, and aversions directly from a website.
  • Web analytics: Provide detailed statistics about a website’s traffic, traffic sources, and measurement of conversions and sales.

To better understand when to use which tool, it is helpful to use the following criteria:

  • What people say versus what people do… and what they say they do
  • Why versus how much
  • Existing classifications of online tools

The possible services are included at the latter part of the article to help you start.

What people say versus what people do… and what they say they do

What people say, what people do, and what they say they do are three entirely different things. People often lack awareness or necessary knowledge which would enable them to provide correct information. Anyone who has any experience with user research or conversion rate optimization and has spent time trying to understand users has seen firsthand that more often than not user statements do not match the acquired data. People are not always able to fully articulate why they did that thing they just did. That’s the reason it’s sometimes good to compare information about opinions to information on behavior, as this mix can provide better insights. You can learn what people do by studying your website from your users’ perspective and drawing conclusions based on observations of their behavior, such as click tracking or user session recording. However, that is based on the idea that you test certain theories about people’s behavior. There is a degree of uncertainty, and to validate the data you’ve gathered, you will sometimes have to go one step further and simply ask your users, which will allow you to see the whole picture. Therefore, you can learn what people say by reaching out to your target group directly and asking them questions about your business.

Why versus how much

Some tools are better suited for answering questions about why or how to fix a problem, whereas tools like web analytics do a much better job at answering how many and how much types of questions. Google Analytics tells you the percentage of people who clicked what thing to through to what page, but it doesn’t tell you why they did or did not do that. Having knowledge of these differences helps you prioritize certain sets of tools and use them while fixing issues having the biggest impact on your business.

The following chart illustrates how different dimensions affect the types of questions that can be asked:

Choosing the right tool—infographics

There are a lot of tools out there these days that do everything from testing information architecture and remote observation. With more coming out every day, it can be really hard to pick the one that will give you the best results for your specific purpose. To alleviate some of the confusion, many experts tried to classify them according to different criteria. I decided to include some of examples for your convenience below.

Which remote tool should I use? By Stuff On My Wall

Choosing a remote user experience research tool by Nate Bolt

The five categories of remote UX tools by Nate Bolt

Four quadrants of the usability testing tools matrix by Craig Tomlin

Tool examples

The examples of tools which I list below are best suited for web services. The world of mobile applications and services is too vast to be skimmed over and has enough material to be a different article completely. The selection is narrowed down in order to not overwhelm you with choice, so worry not.

User testing

User testing and research is vital to creating a successful website, products and services. Nowadays using one of the many existing tools and services for user testing is a lot easier than it used to be. The important thing is to find a tool or service that works for your website and then use it to gather real-world data on what works and what does not.

Survey: The most basic form of what people say. Online surveys are often used by companies to gain a better understanding of their customers’ motives and opinions. You can ask them to respond in any way they choose or ask them to select an answer from a limited number of predetermined responses. Getting feedback straight from your customers may be best used in determining their painpoints or figuring out their possible needs (or future trends). However, what you need to remember about is that people do not always communicate best what is exactly the issue they are facing. Be like Henry Ford: Do not give people faster horses when they want quicker transportation—invent a car.

Examples:
Typeform
Survey Gizmo

Card sorting: It focuses on asking your users to categorize and sort provided items in the most logical way for them or create their own possible categories for items. These two methods are called respectively closed and open card sorting. This will help you to rework the information architecture of your site thanks to the knowledge about the users’ mental models. If you aim to obtain information that balances between “what they do” and “what they say”, sorting is your best bet. Be sure to conduct this study in a larger group – some mental models might make sense, but aren’t the most intuitive for others. Focus on the responses that are aligned with each other, as it is possibly the most representative version of categories.

Examples:
ConceptCodify
usabiliTEST

Click testing/automated static/design surveys: This lets you test screenshots of your design, so you can obtain detailed information about your users’ expectations and reactions to a website in various stages of development. This enters the territory of simply gathering data about the actions of your users, so you obtain information about what they do. The study is conducted usually by asking a direct question: “Click on the button which will lead to sign-ups”. However, remember, click testing alone is not sufficient enough, you need other tools that cover the part of “why” in order to fully understand.

Examples:
Usaura
Verify App

5-Second testing/first impression test: Because your testers have only five seconds to view a presented image, they are put under time pressure and must answer questions relying only on almost subconscious information they obtained. This enables you to improve your landing pages and calls to action, as users mostly focus only on the most eye-catching elements.

Examples:
UsabilityHub
Optimal Workshop Chalkmark

Diary studies: An extensive database of all thoughts, feelings and actions of your user, who belongs to a studied target market. All events are being recorded by the participants at their moment of occurrence. This provides insights into firsthand needs of your customers, asking them directly about their experiences. Yet, it operates in a similar fashion to surveys, therefore remember that your participants do not always clearly convey what they mean.

Examples:
FocusVision Revelation
Blogger

Moderated user studies/remote usability testing: The participants of this test are located in their natural environment, so their experiences are more genuine. Thanks to the tools and software there is no necessity for participants and facilitators to be in the same physical location. Putting the study into context of a natural/neutral environment (of whatever group you are studying) gives you insight into unmodified behaviours. Also, the study is a lot cheaper than other versions.

Examples:
GoToMeeting
Skype

Self-moderated testing: The participants of the test are expected to complete the tasks independently. After that you will obtain videos of their test sessions, along with a report containing information what problems your users were facing and what to do in order to fix them. The services offering this type of testing usually offer the responses quickly, so if you are in a dire need of feedback, this is one of possibilities.

Examples:
Uxeria
UserTesting

Automated live testing/remote scenario testing: Very similar to remote testing, yet the amount of information provided is much more extensive and organized. You get effectiveness ratios (success, error, abandonment and timeout), efficiency ratios (time on task and number of clicks), and survey comments as the results.

Examples:
UX Suite
Loop11

Tree testing/card-based classification: It is a technique which completely removes every distracting element of the website (ads, themes etc.) and focuses only on the simplified text version. Through this you can evaluate the clarity of your scheme and pinpoint the chokepoints that present problems to users. It is a good method to test your prototypes or if you want to detect a problem with your website and suspect the basic framework is at fault.

Examples:
UserZoom
Optimal Workshop Treejack

Remote eye tracking/online eye tracking/visual attention tracking: Shows you where people focus their attention on your landing pages, layouts, and branding materials. This can tell you whether the users are focused on the page, whether they are reading it or just scanning, how intense they are, and what is the pattern of their movement. However, it cannot tell you exactly whether your users actually do see something or do not, or why exactly do they look at a given part. This can be remedied for example with voiceovers, where the participants tell you right away what they feel.

a) Simulated: creates measurement reports that predict what a real person would most likely look at.

Examples:
VAS
Eyequant

b) Behavioral: finds out whether people actually notice conversion-oriented elements of the page and how much attention they pay to them.

Examples:
Attensee Eyetrack Shop

These are the singular features which are prominent elements of the listed services. However, nowadays there is a trend to combine various tools together, so they can be offered by a single website. If you happen to find more than one tool valuable for your business, you can use services such as UsabilityTools or UserZoom.

Customer feedback

Successful business owners know that it’s crucial to take some time to obtain customer feedback. Understanding what your customers think about your products and services will not only help you improve quality, but will also give you insights into what new products and services your customers want. Knowing what your customers think you’re doing right or wrong also lets you make smart decisions about where to focus your energies.

Live chats: an easy to understand way of communicating through the website interface in real time. Live chat enables you to provide all the answers your customers could want. By analyzing their questions and often inquired issues you can decide what needs improvement. Live chats usually focus on solving an immediate problem, so it is usually used for smaller issues. The plus is the fact that your client will feel acknowledged right away.

Examples:
LiveChat
Live Person

Insight surveys: They help you understand your customers thanks to targeted website surveys. You can create targeted surveys and prompts by focusing them on the variables such as the time on page, the number of visits, the referring search term or your own internal data. You can even target custom variables such as the number of items in a shopping cart. However, they are very specific and operate on the same principle as general surveys, so you must remember about the risk that the survey participants won’t always be able to provide you with satisfactory answers.

Examples:
Survicate
Qualaroo

Feedback forms: They are a simple website application to receive feedback from your website visitors. You can create a customized form, copy and paste code into your site’s HTML, and start getting feedback. This is a basic tool for getting feedback forms from your customer, and receiving and organizing results. If you want to know the general opinion about your website and the experiences of your visitors (and you want it to be completely voluntary), then forms are a great option.

Examples:
Feedbackify
Kampyle

Feedback forums: Users enter a forum where they can propose and vote on items which need change or need to be discussed. That information allows you to prioritize issues and decide what needs to be fixed as fast as possible. The forums can be also used for communicating with users, for example you can inform them that you introduced some improvements to the product. Remember, however, that even the most popular issues might be actually least important for imrpoving your serive and/or website, it is up to you to judge.

Examples:
UserVoice
Get Satisfaction

Online customer communities: You refer to your customer directly, peer-to-peer, and offer problem solving and feedback. Those web-based gathering places for customers, experts, and partners enable you to discuss problems, post reviews, brainstorm new product ideas, and engage with one another.

Examples:
Socious
Lithium

There are also platforms that merge some of the functionalities such as UserEcho or Freshdesk which are an extremely popular solution to the growing demands of clients who prefer to focus on single service with many features.

Website analytics

Just because analytics provide you with some additional data about your site doesn’t mean it’s actually valuable to your business. You want to find the errors and holes within your website and fill them with additional functionality for your users and customers. Using the information gathered you can influence your future decisions in order to improve your service.

Web analytics: all movement of the users is recorded and stored. However, their privacy is safe, as the data gathered is used only for optimization, and the data is impossible to be personally identified. The data can be later used for evaluating and improving your service and website in order to achieve your goals such as increasing the amount of visitors or sales.

Examples:
Mint
Mixpanel
KISSmetrics
Woopra
Google Analytics

In-page web analytics: They differ from traditional web analytics as they focus on the users’ movement within the page and not between them. These tools are generally used to understand behavior for the purposes of optimizing a website’s usability and conversion.

a) Click tracking: This technique used to determine and record what the users are clicking with their mouse while browsing the website–it draws you a map of their movements, which allows you to see step by step the journey of your user. If there is a problem with the website, this is one of the methods to check out where that problem could’ve occured.

Examples:
Gemius Heatmap
CrazyEgg

b) Visitor recording/user session replays: Every action and event is recorded as a video.

Examples:
Inspectlet
Fullstory

c) Form testing: This allows you to evaluate the web form and identify areas that need improvement, for example which fields make your visitors leave the website before completing the form.

Examples:
Formisimo
UsabilityTools Conversion Suite

In a similar fashion to the previous groups, there is also a considerable amount of analytic Swiss army knives offering various tools in one place. The examples of such are ClickTale, UsabilityTools, or MouseStats.

Conclusion

This is it—the finishing line of this guide to online research tools. It is an extremely valuable asset which can provide important and surprising data. The amount of tools available at hand is indeed overwhelming, that is why you need to consider the listed factors of what, why and such. This way you will reach a conclusion about what exactly you need to test in order to improve your service or obtain required information. Knowing what you want to do will help you narrow your choices and in result choose the right tool. Hopefully, what you’ve read will help you choose the best usability tools for testing, and you will end up an expert in your research sessions.

Posted in Discovery, Research, and Testing, Process and Methods, Software and Tools | 20 Comments »

20 Comments

  • John Weidner

    May 12, 2015 at 2:28 pm

    Nice collection of tools. Here’s another one to consider adding – https://userbob.com It’s a un-moderated remote user testing tool.

  • Owain

    May 13, 2015 at 2:27 pm

    and another, maybe less known in the market; but offering a wider, varying degree of behavioural insights – Decibel Insight (https://www.decibelinsight.com/). Disclaimer – I work for them, so please get in contact with me if you would like a free trial. This would at least provide you the opportunity to weigh up the features against more established market players such as CrazyEgg.

  • Sofie Khan

    May 14, 2015 at 7:04 pm

    This is really awesome Bartosz! Very well-organized and helpful guide! Thanks for creating!

  • Stephen Cribbett

    May 15, 2015 at 9:34 am

    Great article Bartosz, and in particular the attention you pay to ‘reported’ behaviours versus ‘observed’. We also provide a tool that’s favoured by User Researchers including the likes of WhatIf, Uservision and Nile. It’s great for diaries, mobile ethnographies and short/long-term communities where shifting consumer behaviours are the desired outcome. It’s been designed by researchers using agile methods, so it’s always being improved. Check it out, http://www.DubIsHere.com You can also find us on twitter @dub_research

  • Justin Ewert

    May 18, 2015 at 1:42 am

    Nice job Bartosz and thanks for your hard work in putting all of this information together. Much appreciated 🙂

  • www.kangmasarif.com

    May 18, 2015 at 5:12 am

    and another, maybe less known in the market; but offering a wider, varying degree of behavioural insights – Decibel Insight Share Tips SEO

  • Mary

    May 19, 2015 at 9:12 am

    Another great tool for self-moderated testing, similar to usertesting.com, is Whatusersdo http://whatusersdo.com/

  • Patricia

    May 19, 2015 at 11:44 am

    Great article, keep on writing. I think everyone with a blog or site should use a web analytics tool. I’ve used google analytics and kissmetrics before but gave up on them. I started using gostats a couple years ago and it has been a good experience, it’s easy to use and I have all the information I need about my visitors.

  • Hugh Carling

    May 20, 2015 at 12:23 pm

    This article has needed to be written for a long time! Thanks again, this was a great read for all our team.
    For user research, you could look at Liveminds too – we provide a simple, elegant platform to many leading UX agencies, brand UX teams. The platform has integrated mobile apps to enable participants to easily capture pictures and videos and a Facebook app helping you find the best people and keep them together over time. You can find out more at http://Liveminds.co.uk.

  • kaos muslim

    May 25, 2015 at 6:59 am

    Great article, keep on writing.. thanks for sharing

  • idgamer

    May 27, 2015 at 7:14 am

    great article Bartosz! thanks for sharing

  • Your Guide to Online Research and Testing Tools | UX Central | Rob Cowie

    June 5, 2015 at 5:01 am

    […] Add to collection | Read article […]

  • Moditha

    June 7, 2015 at 4:17 pm

    I agree with this quote “business depends on how the business will meet their customers.”

  • Дайджест продуктового дизайна, май 2015 | Юрий Ветров об интерфейсах

    June 9, 2015 at 6:44 am

    […] Bartosz Mozyrko сделал обзор типов пользовательских исследов…. К каждому типу идет всего по два сервиса, но сам по себе обзор получился подробным. […]

  • Malaga

    June 14, 2015 at 2:15 pm

    Very nice collection dear Bartosz Mozyrko, I will use all for my business. Best regards and thanks for all.

  • MAS OTONG

    June 15, 2015 at 8:41 am

    Online Research and Testing Tools. Before i don’t understand about your article. But, after reading this article. I can understand. Your article is simple and powerfull. Good job 🙂

  • James Franck

    July 13, 2015 at 5:54 pm

    Excellent resource! Took a lot of time/effort to put together, I suspect. Thanks loads for doing it! Valuable!

  • Acceler8 Media

    July 22, 2015 at 9:49 pm

    Great article. There are so many useful tips in here. I am going to use this as the basis of a presentation to my team. Thanks

  • Top 100 Blogs About Web Design and Development

    October 19, 2015 at 12:04 pm

    […] Your Guide to Online Research and Testing Tools […]

Sorry, comments are closed.