A Beginner’s Guide to Web Site Optimization—Part 3

Written by: Charles Shimooka

Web site optimization has become an essential capability in today’s conversion-driven web teams. In Part 1 of this series, we introduced the topic as well as discussed key goals and philosophies. In Part 2, I presented a detailed and customizable process. In this final article, we’ll cover communication planning and how to select the appropriate team and tools to do the job.

Communication

For many organizations, communicating the status of your optimization tests is an essential practice. Imagine if your team has just launched an A/B test on your company’s homepage, only to learn that another team had just released new code the previous day that had changed the homepage design entirely. Or imagine if a customer support agent was trying to help users through the website’s forgot password flow, unaware that the customer was seeing a different version due to an A/B test that your team was running.

To avoid these types of problems, I recommend a three-step communication program:

  1. Pre-test notification

This is an email announcing that your team has selected a certain page/section of the site to target for its next optimization test and that if anyone has any concerns, they had better voice them immediately, before your team starts working on it. Give folks a day or two to respond. The email should include:

  • Name/brief description of the test
  • Goals
  • Affected pages
  • Expected launch date
  • Link to the task or project plan where others can track the status of the test.

Here’s a sample pre-test notification.

  1. Pre-launch notification

This email is sent out a day or two before a new experiment launches. It includes all of the information from the Pre-Test Notification email, plus:

  • Expected test duration
  • Some optimization tools create a unique dashboard page in which interested parties can monitor the results of the test in real-time. If your tool does this, you can include the link here.
  • Any other details that you care to mention, such as variations, traffic allocation, etc…

Here’s a sample pre-launch email.

  1. Test results

After the test has run its course and you’ve compiled the results into the Optimization Test Results document, send out a final email to communicate this. If you have a new winner, be sure to brag about it a little in the email. Other details may include:

  • Brief discussion
  • A few specifics, such as conversion rates, traffic and confidence intervals
  • Next steps

Here’s a sample test results email.

Team size and selection

As is true with many things, good people are the most important aspect of a successful optimization program. Find competent people with curious minds who take pride in their work – this will be far more valuable than investment in any optimization tool or adherence to specific processes.

The following are recommendations for organizations of varying team sizes.

One person

It is difficult for one person to perform optimization well unless they are dedicated full-time to the job. If your organization can only cough-up one resource, I would select either a web analytics resource with an eye for design, or a data-centric UX designer. For the latter profile, I don’t mean the type of designer who studied fine art and is only comfortable using Photoshop, but rather the type who likes wireframes, has poked around an analytics tool on their own, and is good with numbers. This person will also have to be resourceful and persuasive, since they will almost certainly have to borrow time and collaborate with others to complete the necessary work.

Two to three people

With a team size of three people, you are starting to get into the comfort zone. To the UX designer and web/data analytics roles, I would add either a visual designer or a front-end developer. Ideally, some of the team members would have multiple or overlapping competencies. The team will probably still have to borrow time from other resources, such as back-end developers and QA.

Five people

A team that is lucky enough to have five dedicated optimization resources has the potential to be completely autonomous. If your organization places such a high value on optimization, they may have also invested accordingly in sophisticated products or strategies for the job, such as complex testing software, data warehouses, etc… If so, then you’ll need folks who are specifically adept at these tools, broadening your potential team to roles such as data engineers, back-end developers, content managers, project managers, or dedicated QA resources. A team of five would ideally have some overlap with some of the skill-sets.

Tool selection

The optimization market is hot and tool selection may seem complicated at first. The good news is that broader interest and increased competition is fueling an all-out arms race towards simpler, more user-friendly interfaces designed for non-technical folks. Data analysis and segmentation features also seem to be evolving rapidly.

My main advice if you’re new to optimization is to start small. Spend a year honing your optimization program and after you’ve proven your value, you can easily graduate to the more sophisticated (and expensive) tools. Possibly by the time you’re ready, your existing tool will have advanced to keep up with your needs. Also realize that many of the cheaper tools can do the job perfectly well for most organizations, and that some organizations with the high-powered tools are not using them to their fullest capabilities.

A somewhat dated Forrester Research report from February 2013 assesses some of the big hitters, but notably absent are Visual Website Optimizer (VWO) and, for very low end, Google’s free Content Experiments tool. Conversion Rate Experts keeps an up-to-date comparison table listing virtually all of today’s popular testing tools, but it only rates them along a few specific attributes.

I performed my own assessment earlier this year and here is a short list of my favorites:

Entry-level
Visual Website Optimizer (VWO)
Optimizely
Google Content Experiments
Advanced
Maxymiser
Monetate
Adobe Test & Target

Here are a few factors to consider when deciding on products:

Basic features

Intuitive user interface

Luckily, most tools now have simple, WYSIWYG type of interfaces that allow you to directly manipulate your site content when creating test variations. You can edit text, change styles, move elements around, and save these changes into a new test variation. Some products have better implementations than others, so be sure to try out a few to find the best match for your team.

Targeting

Targeting allows you to specify which site visitors are allowed to see your tests. Almost all tools allow you to target site visitors based on basic attributes that can be inferred from their browser, IP address, or session. These attributes may include operating system, browser type/version, geographical location, day of week, time of day, traffic source (direct vs. organic vs. referral), and first time vs. returning visitor. More advanced tools also allow you to target individuals based on attributes (variables) that you define and programmatically place in your users’ browser sessions, cookies, or URLs. This allows you to start targeting traffic based on your organization’s own customer data. The most advanced tools allow you to import custom data directly into the tool’s database, giving you direct access to these attributes through their user interface, not only for targeting, but also for segmented analysis.

Analysis and reporting

Tools vary widely in their analysis and reporting capabilities, with the more powerful tools generally increasing in segmentation functionality. The simplest tools only allow you to view test results compared against a single dimension, for example, you can see how your test performed on visitors with mobile vs. desktop systems. The majority of tools now allow you to perform more complicated analyses along multiple dimensions and customized user segments. For example, you might be interested in seeing how your test performed with visitors on mobile platforms, segmented by organic vs. paid vs. direct traffic.

Keep in mind that as your user segments become more specific, your optimization tool must rely on fewer and fewer data points to generate the results for each segment, thereby decreasing your confidence levels.

Server response time

Optimization tools work by adding a small snippet of code to your pages. When a user visits that page, the code snippet calls a server somewhere that returns instructions on which test variation to display to the user. Long server response times can delay page loading and the display of your variations, thereby affecting your conversions and reporting.

When shopping around, be sure to inquire about how the tool will affect your site’s performance. The more advanced tools are deployed on multiple, load-balanced CDNs and may include contractual service level agreements that guarantee specific server response times.

Customer support

Most optimization vendors provide a combination of online and telephone support, with some of the expensive solutions offering in-person set-up, onboarding and training. Be sure inquire about customer support when determining costs. A trick I’ve used in the past to test a vendor’s level of service is to call the customer support lines at different times of the day and see how fast they pick up the phone.

Price and cost structure

Your budget may largely determine your optimization tool options as prices vary tremendously, from free (for some entry tools with limited features) to six-figure annual contracts that are negotiated based on website traffic and customer support levels (Maxymiser, Monetate and Test & Target fall into this latter category).

Tools also vary in their pricing model, with some basing costs on the amount of website traffic and others charging more for increased features. My preference is towards the latter model, since the former is sometimes difficult to predict and provides a disincentive to perform more testing.

Advanced features

Integration with CMS/analytics/marketing platforms

If you are married to a single Content Management System, analytics tool, or marketing platform, be sure to inquire from your vendor about how their tool will integrate. Some vendors advertise multi-channel solutions (the ability to leverage your customer profile data to optimize across websites, email, and possibly other channels, such as social media or SMS). Enterprise-level tools seem to be trending towards all-in-one solutions that include components such as CMS, marketing, ecommerce, analytics, and optimization (ie. Adobe’s Marketing Cloud or Oracle’s Commerce Experience Manager). But for smaller organizations, integration may simply mean the ability to manage the optimization tool’s javascript tags (used for implementation) across your site’s different pages. In these situations, basic tools such as Google Tag Manager or WordPress plugins may suffice.

Automated segmentation and targeting

Some of the advanced tools offer automated functionality that tries to analyze your site’s conversions and notify you of high-performing segments. These segments may be defined by any combination of recognizable attributes and thus, far more complicated than your team may be able to define on their own. For example, the tool might define one segment as female users on Windows platform, living in California, and who visited your site within the past 30 days. It might define a dozen or more of these complex micro-segments and even more impressively, allow you to automatically redirect all future traffic to the winning variations specific to each of these segments. If implemented well, this intelligent segmentation has tremendous potential for your overall site conversions. The largest downside is that it usually requires a lot of traffic to make accurate predictions.

Automated segmentation is often an added cost to the base price of the optimization tool. If so, consider asking for a free trial period to evaluate the utility/practicality of this functionality before making the additional investment.

Synchronous vs. asynchronous page loading

Most tools recommend that you implement their services in an asynchronous fashion. In other words, that you allow the rest of your page’s HTML to load first before pinging their services and potentially loading one of the test variations that you created. The benefit of this approach is that your users won’t have to wait additional time before your control page starts to render in the browser. The drawback is that once the call to the optimization’s services is returned, then your users may see a page flicker as the control page is replaced by one of your test variations. This flickering effect, along with the additional time it takes to display the test variations, could potentially skew test results or cause surprise/confusion with your users.

In contrast, synchronous page loading, which is recommended by some of the more advanced tools, makes the call to the optimization tool before the rest of the page loads. This ensures that your control group and variations are all displayed in the same relative amount of time, which should allow for more accurate test results. It also eliminates the page flicker effect inherent in asynchronous deployments.

Conclusion

By far, the most difficult step in any web site optimization program is the first one – the simple act of starting. With this in mind, I’ve tried to present a complete and practical guide on how to get you from this first step through to a mature program. Please feel free to send me your comments as well as your own experiences. Happy optimizing.

A Beginner’s Guide to Web Site Optimization—Part 2

Written by: Charles Shimooka

In the previous article we talked about why site optimization is important and presented a few important goals and philosophies to impart on your team. I’d like to switch gears now and talk about more tactical stuff, namely, process.

Optimization process

Establishing a well-formed, formal optimization process is beneficial for the following reasons.

  1. It organizes the workflow and sets clear expectations for completion.
  2. Establishes quality control standards to reduce bugs/errors.
  3. Adds legitimacy to the whole operation so that if questioned by stakeholders, you can explain the logic behind the process.

At a high level, I suggest a weekly or bi-weekly optimization planning session to perform the following activities:

  1. Review ongoing tests to determine if they can be stopped or considered “complete” (see the boxed section below). For tests that have reached completion, the possibilities are:
    1. There is a decisive new winner. In this case, plan how to communicate and launch the change permanently to production.
    2. There is no decisive winner or the current version (control group) wins. In this case, determine if more study is required or if you should simply move on and drop the experiment.
  2. Review data sources and brainstorm new test ideas.
  3. Discuss and prioritize any externally submitted ideas.
How do I know when a test has reached completion?
Completion criteria are a somewhat tricky topic and seemingly guarded industry secrets. These define the minimum requirements that must be true in order for a test to be declared “completed.” My personal sense from reading/conferences is that there are no widely-accepted standards and that completion criteria really depend on how comfortable your team feels with the uncertainty that is inherent in experimentation. We created the following minimum completion criteria for my past team at DIRECTV Latin America. Keep in mind that these were bare-bones minimums, and that most of our tests actually ran much longer.

  1. Temporal: Tests must run for a minimum of two weeks to account for variation between days of the week.
  2. Statistical confidence: We used a 90-95% confidence interval for most tests.
  3. Stability over time: Variations must maintain their positions relative to each other for at least one week.
  4. Total conversions: Minimum of 200 total conversions.

For further discussion of the rationale behind these completion criteria, please see Best Practices When Designing and Running Experiments later in this article.

The creation of a new optimization test may follow a process that is similar to your overall product development lifecycle. I suggest the following basic structure:

Process-diagram-abbreviated

The following diagram shows a detailed process that I’ve used in the past.

A detailed process that the author has used in the past.

Step 1: Data analysis and deciding what to test

Step one in the optimization process is figuring out where to first focus your efforts. We used the following list as a loose prioritization guideline:

  1. Recent product releases, or pages that have not yet undergone optimization.
  2. High “value” pages
    • 1. High revenue (ie. shopping cart checkout pages, detail pages of your most expensive products, etc…).
    • 2. High traffic (ie. homepage, login/logout).
    • 3. Highly “strategic” (this might include pages that are highly visible internally or that management considers important).
  3. Poorly performing pages

Step 2: Brainstorm ideas for improvement

Ideas for how to improve page performance is a topic that is as large as the field of user experience itself, and definitely greater than the scope of this article. One might consider improvements in copywriting, form design, media display, page rendering, visual design, accessibility, browser targeting… the list goes on.

My only suggestion for this process is to make it collaborative – harness the power of your team to come up with new ideas for improvement, not only including designers in the brainstorming sessions, but also developers, copywriters, business analysts, marketers, QA, etc… Good ideas can (and often do) come from anywhere.

Adaptive Path has a great technique of collaborative ideation that they call sketchboarding, which uses iterative rounds of group sketching.

Step 3: Write the testing plan

An Optimization Testing Plan acts as the backbone of every test. At a high level, it is used to plan, communicate, and document the history of the experiment, but more importantly, it fosters learning by forcing the team to clearly formulate goals and analyze results.

A good testing plan should include:

  1. Test name
  2. Description
  3. Goals
  4. Opportunities (what gains will come about if the test goes well)
  5. Methodology
    • 1. Expected dates that the test will be running in production.
    • 2. Resources (who will be working on the test).
    • 3. Key metrics to be tracked through the duration of the experiment.
    • 4. Completion criteria.
    • 5. Variations (screenshots of the different designs that you will be showing your site visitors).

Here’s a sample optimization testing plan to get you started.

Step 4: Design and develop the test

Design and development will generally follow an abbreviated version of your organization’s product development lifecycle. Since test variations are generally simpler than full-blown product development projects, I try to use a lighter, more agile process.

Be sure that if you do cut corners, only skimp on things like process artifacts or documentation, and not on design quality. For example, be sure to perform some basic usability testing and user research on your variations. This small investment will create better candidates that will be more likely to boost conversions.

Step 5: Quality assurance

When performing QA on your variations, be as thorough as you would with any other code release to production. I recommend at least functional, visual, and analytics QA. Even though many tools allow you to manipulate your website’s UI on the fly using interfaces that immediately display the results of your changes, the tools are not perfect and any changes that you make might not render perfectly across all browsers.

Keep in mind that optimization tools provide you one additional luxury that is not usually possible with general website releases – that of targeting. You can decide to show your variations to only the target browsers, platforms, audiences, etc… for which you have performed QA. For example, let’s imagine that your team has only been able to QA a certain A/B test on desktop (but not mobile) browsers. When you actually configure this test in your optimization tool, you can decide to only display the test to visitors with those specific desktop browsers. If one of your variations has a visual bug when viewed on mobile phones, for example, that problem should not affect the accuracy of your test results.

Step 6: Run the Test

After QA has completed and you’ve decided how to allocate traffic to the different designs, it’s time to actually run your test. The following are a few best practices to keep in mind before pressing the “Go” button.

1.  Variations must be run concurrently

This first principle is almost so obvious that it goes without saying, but I’ve often heard the following story from teams that do not perform optimization: “After we launched our new design, we saw our [sales, conversions, etc…] increase by X%. So the new design must be better.”

The problem with this logic is that you don’t know what other factors might have been at play before and after the new change launched. Perhaps traffic to that page increased in either quantity or quality after the new design released. Perhaps the conversion rate was on the increase anyway, due to better brand recognition, seasonal variation, or just random chance. Due to these and many other reasons, variations must be run concurrently and not sequentially. This is the only way to hold all other factors consistent and level the playing field between your different designs.

2.  Always track multiple conversion metrics

One A/B test that we ran on the movie detail pages of the DIRECTV Latin American sites was the following: we increased the size and prominence of the “Ver adelanto” (View trailer) call to action, guessing that if people watched the movie trailer, it might excite them to buy more pay-per-view movies from the web site.

We increased the size and prominence of the “Ver adelanto” (View trailer) call to action, guessing that if people watched the movie trailer, it might excite them to buy more pay-per-view movies from the web site.

Our initial hunch was right, and after a few weeks we saw that pay-per-views purchases were 4.8% higher with this variation over the control. This increase would have resulted in a revenue boost of about $18,000/year in pay-per-view purchases. Not bad for one simple test. Fortunately though, since we were also tracking other site goals, we noticed that this variation also decreased purchases of our premium channel packages (ie. HBO and Showtime packages) by a whopping 25%! This would have decreased total revenue by a much greater amount than the uptick in pay-per-views, and because of this, we did not launch this variation to production.

It’s important to keep in mind that changes may affect your site in ways that you never would have expected. Always track multiple conversion metrics with every test.

3.  Tests should reach a comfortable level of statistical significance

I recently saw a presentation in which a consultant suggested that preliminary tests on email segmentation had yielded some very promising results.

Chart showing conversion rates per 1000 emails sent.

In the chart above, the last segment of users (those who had logged in more than four times in the past year) had a conversion rate of .00139% (.139 upgrades per 1000 emails sent). Even though a conversion rate of .00139% is dismally low by any standards, according to the consultant it represented an increase of 142% compared to the base segment of users, and thus, a very promising result.

Aside from the obvious lack of actionable utility (does this study suggest that emails only be sent to users who have logged in more than four times?) the test contained another glaring problem. If you look at the “Upgrades” column at the top of the spreadsheet, you will see that the results were based on only five individuals purchasing an upgrade. Five total individuals out of almost eighty four thousand emails sent! So if, by pure chance, only one other person had purchased an upgrade in any of the segments, it could have completely changed the study’s implications.

While this example is not actually an optimization test but rather just an email segmentation study, it does convey an important lesson: don’t declare a winner for your tests until it has reached a “comfortable” level of significance.

So what does “comfortable” mean? The field of science requires strict definitions to use the terms “significant” (95% confidence level) and “highly significant” (99% confidence level) when publishing results. Even with these definitions, it still means that there is a 5% and 1% chance, respectively, of your conclusions being wrong. Also keep in mind that higher confidence intervals require more data (ie. more website traffic) which translates into longer test durations. Because of these factors, I would recommend less stringent standards for most optimization tests – somewhere around 90-95% confidence depending on the gravity of the situation (higher confidence intervals for tests with more serious consequences or implications).

Ultimately, your team must decide on confidence intervals that reflect a compromise between test duration and results certainty, but I would propose that if you perform a lot of testing, the larger number of true winners will make up for the fewer (but inevitable) false positives.

4.  The duration of your tests should account for any natural variations (such as between weekdays/weekends) and be stable over time

In a 2012 article on AnalyticsInspector.com, Jan Petrovic brings to light an important pitfall of ending your tests too early. He discusses an A/B test that he ran for a high-traffic site in which, after only a day, the testing tool reported that a winning variation had increased the primary conversion rate by an impressive 87%, with a 100% confidence interval.

The duration of your tests should account for any natural variations (such as between weekdays/weekends) and be stable over time.

Jan writes, “If we stopped the test then and pat each other on the shoulder about how great we were, then we would probably make a very big mistake. The reason for that is simple: we didn’t test our variation on Friday or Monday traffic, or on weekend traffic. But, because we didn’t stop the test (because we knew it was too early), our actual result looked very different.”

Chart showing new design results over time.

After continuing the test for four weeks, Jan saw that the new design, although still better than the control, had leveled out to a more reasonable 10.49% improvement since it had now taken into account natural daily variation. He writes, “Let’s say you were running this test in checkout, and on the following day you say to your boss something like ‘hey boss, we just increased our site revenue by 87.25%’. If I was your boss, you would make me extremely happy and probably would increase your salary too. So we start celebrating…”

Jan’s fable continues with the boss checking the bank account at the end of the month, and upon seeing that sales had actually not increased by the 87% that you had initially reported, reconsiders your salary increase.

The moral of the story: Consider temporal variations in the behavior of your site visitors, including differences between weekday and weekend or even seasonal traffic.

Step 7: Analyze and Report on the Results

After your test has run its course and your team has decided to press the “stop” button, it’s time to compile the results into an Optimization Test Report. The Optimization Test Report can be a continuation of the Test Plan from Step 2, but with the following additional sections:

  1. Results
  2. Discussion
  3. Next steps

It is helpful to include graphs and details in the Results section so that readers can visually see trends and analyze data themselves. This will add credibility to your studies and hopefully get people invested in the optimization program.

The discussion section is useful for explaining details and postulating on the reasons for the observed results. This will force the team to think more deeply about user behavior and is an invaluable step towards designing future improvements.

Conclusion

In this article, I’ve presented a detailed and practical process that your team can customize to its own use. In the next and final article of this series, I’ll wrap things up with suggestions for communication planning, team composition, and tool selection.

A Beginner’s Guide to Web Site Optimization—Part 1

Written by: Charles Shimooka

Web site optimization, commonly known as A/B testing, has become an expected competency among many web teams, yet there are few comprehensive and unbiased books, articles, or training opportunities aimed at individuals trying to create this capability within their organization.

In this series, I’ll present a detailed, practical guide on how to build, fine-tune, and evolve an optimization program. Part 1 will cover some basics: definitions, goals and philosophies. In Part 2, I’ll dive into a detailed process discussion covering topics such as deciding what to test, writing optimization plans, and best practices when running tests. Part 3 will finish up with communication planning, team composition, and tool selection. Let’s get started!

The basics: What is web site optimization?

Web site optimization is an experimental method for testing which designs work best for your site. The basic process is simple:

  1. Create a few different design options, or variations, of a page/section of your website.
  2. Split up your web site traffic so that each visitor to the page sees either your current version (the control group) or one of these new variations.
  3. Keep track of which version performs better based on specific performance metrics.

The performance metrics are chosen to directly reflect your site’s business goals and might include things like how many product purchases were made on your site (a sales goal), how many people signed up for the company newsletter (an engagement goal), or how many people watched a self-help video in your FAQ section (a customer service goal). Performance metrics are often referred to as conversion rates, which equals the percentage of visitors who performed the action being tested compared to the total number of visitors to that page.

Optimization can be thought of as one component in the web site development ecosystem. Within optimization, the basic process is to analyze data, create and run tests, then implement the winners of those tests.

Visual of where optimzation fits in site development
Optimization can be thought of as one component in the website development ecosystem.

 

A/B vs. multivariate

There are two basic types of optimization tests: A/B tests (also known as an A/B/N tests) and multivariate tests.

A/B tests

In an A/B test, you run two or more fixed design variations against each other. The variations might differ in only one individual element (such as the color of a button or swapping out an image for a video) or in many elements all at once (such as changing the entire page layout and design, changing a long form into a step-by-step wizard, etc…).

Three buttons for testing, each with different copy.
Example 1: A simple A/B/N test trying to determine which of three different button texts drives more clicks.

 

 

 

Visuals showing page content in different layouts.
Example 2: An A/B test showing large variations in both page layout and content.

 

In general, A/B tests are simpler to design and analyze and also return faster results since they usually contain fewer variations than multivariate tests. They seem to constitute the vast majority of manual testing that occurs these days.

Multivariate tests

Multivariate tests vary two or more attributes on a page and test which combination works best. The key difference between A/B and multivariate tests is that the latter are designed to tease apart how two or more dimensions of a design interact with each other and lead to that design’s success. In the example below, the team is trying to figure out what combination of button text and color will get the most clicks.

Buttons with both different copy and different colors
Example 1: A simple multivariate test with 2 dimensions (button color and button text) and 3 variations on each dimension.

The simplest form of multivariate testing is called the full-factorial method, which involves testing every combination of factors against each other, as in the example above. The biggest drawback of these tests is that they generally take longer to get statistically significant results since you are splitting the same amount of site traffic between more variations than A/B tests.

Other fractional factorial methods use statistics to try and interpolate the results of certain combinations, thereby reducing the traffic needed to test every single variation. Many of today’s optimization tools allow you to play around with these different multivariate methods; just keep in mind that fractional factorial methods are often complex, named after deceased Japanese mathematicians, and require a degree in statistics to fully comprehend. Use at your own risk.

Why do we test? Goals, benefits, and rationale

There are many benefits of moving your organization to a more data-driven culture. Optimization establishes a metrics-based system for determining design success vs. failure, thereby allowing your team to learn with each test. No longer will people argue ad nauseum over design details. Cast away the chains of the HiPPO effect—in which the Highest Paid Person in the Office determines what goes on your site. Once you have established a clear set of goals and the appropriate metrics for measuring those goals, the data should speak as the deciding voice.

Optimization can also drastically improve your organization’s product innovation process by allowing you to test new product ideas at scale and quickly figure out which are good and which should be scrapped. In his article “How We Determine Product Success” John Ciancutti of Netflix describes it this way:

“Innovation involves a lot of failure. If we’re never failing, we aren’t trying for something out on the edge from where we are today. In this regard, failure is perfectly acceptable at Netflix. This wouldn’t be the case if we were operating a nuclear power plant or manufacturing cars. The only real failure that’s unacceptable at Netflix is the failure to innovate.

So if you’re going to fail, fail cheaply. And know when you’ve failed vs. when you’ve gotten it right.”

Top three testing philosophies

1. Rigorously focus on metrics

I personally don’t subscribe to the philosophy that you should test every single change on your site. However, I do believe that every organization’s web strategies should be grounded in measurable goals that are mapped directly to your business goals.

For example, if management tells you that the web site should “offer the best customer service,” your job is to then determine which metrics adequately represent that conceptual goal. Maybe it can be represented by the total number of help tickets or emails answered from your site combined with a web customer satisfaction rating or the average user rating of individual question/answer pairs in your FAQ section. As Galileo supposedly said, “Measure what is measurable, and make measurable what is not so.”

Additionally, your site’s foundational architecture should allow, to the fullest extent possible, the measurement of true conversions and not simply indicators (often referred to as macro vs micro conversions). For example, if your ecommerce site is only capable of measuring order submissions (or worse yet, leads), make it your first order of business to be able to track that order submission through to a true paid sale. Then ensure that your team always has an eye on these true conversions in addition to any intermediate steps and secondary website goals.  There are many benefits of measuring micro conversion rates, but the work must be done to map them to a tangible macro conversion or you run the risk of optimizing for a false conversion goal.

2. Nobody really knows what will win

I firmly believe that even the experts can’t consistently predict the outcome of optimization tests with even close to 100% accuracy. This is, after all, the whole point of testing. Someone with good intuition and experience will probably have a higher win rate than others, but for any individual test, anyone can be right. With this in mind, don’t let certain members of the team bully others into design submission. When it doubt, test it out.

3. Favor a “small-but-frequent” release strategy

In other words, err on the side of only changing one thing at a time, but perform the changes frequently. This strategy will allow you to pinpoint exactly which changes are affecting your site’s conversion rates. Let’s look at the earlier A/B test example to illustrate this point.

Visuals showing page content in different layouts.
An A/B test showing large variations in both page layout and content.

Let’s imagine that your new marketing director decides that your company should completely overhaul the homepage. After a few months of work, the team launches the new “3-column” design (above-right). Listening to the optimization voice inside your head, you decide to run an A/B test, continuing to show the old design to just 10% of the site visitors and the new design to the remaining 90%.

To your team’s dismay, the old design actually outperforms the new one. What should you do? It would be difficult to simply scrap the new design in its entirety, since it was a project that came directly from your boss and the entire team worked so hard on it. There are most likely a number of elements of the new design that actually perform better than the original, but because you launched so many changes all at once, it is difficult to separate the good from the bad.

A better strategy would have been to have constantly optimized different aspects of the page in small but frequent tests to gradually evolve towards a new version. This process, in combination with other research methods, would provide your team with a better foundation for performing site changes. As Jared Spool argued in his article The Quiet Death of the Major Relaunch, “the best sites have replaced this process of revolution with a new process of subtle evolution. Entire redesigns have quietly faded away with continuous improvements taking their place.”

Conclusion

By now you should have a strong understanding of optimization basics and may have started your own healthy internal dialogue related to philosophies and rationale. In the next article, we’ll talk about more tactical concerns, specifically, the optimization process.

UX Researcher: A User’s Manual

Written by: Victor Yocco

This article is a guide on what to expect, and how to get the most from your UX researcher–a user manual, if you will.

You will invest a lot in your researcher and you deserve the greatest return. You should have high expectations for this critical component of your UX team, and following the recommendations presented in this article will help maximize your return.

A long and prosperous future

Congratulations on hiring a user experience design researcher!  When maintained correctly, a full time researcher will give you many years of strategic insight and validation, eliciting oohs and ahs from jealous shops that have chosen to forgo a researcher and cheers from your many satisfied clients. There are many benefits of having a researcher on staff, which include:

  • Making insights through on-site observation
  • Validating business hypotheses through customer research
  • Discovering usability issues through user testing
  • Initiating new projects in an effort to constantly expand their interests and skills

First, let’s spend a minute discussing the return component of return on investment. Incorporating user research into your product ensures its usability. According to Forrester (2009, pg. 2), product experience is what creates value and establishes power in the marketplace. Specifically, they found companies providing a superior user experience led to:

  • 14.4% more customers willing to purchase their product
  • 15.8% fewer customers willing to consider doing business with a competitor
  • 16.6% more customers likely to recommend their product or services

Investing in a UX researcher is a critical part of ensuring you provide your users with the superior experience Forrester notes as being such a critical differentiator. Everything covered in the following article applies to teams of researchers as well as those in a department of one.

Expectations

You should have high expectations for the quality and quantity of your researcher’s work. She should be a main contributor to your organization, a team player, and someone you look to for new ideas and fresh perspectives on long-standing issues. Your researcher’s unique background in asking questions and finding solutions, as well as the fact that she is likely spending ample time listening to your clients, provides her with insight she can provide your team on how to move forward with addressing various issues.

You might be saying anyone can accomplish the tasks in the paragraph above. You’re correct. I’m pointing out you should expect this from your researcher fresh out of the box, no questions asked.

You might have hired your researcher with specific duties in mind; however, you should expect her to want to know what others are working on, to be a part of the bigger picture, and to ask for feedback allowing her to become more proficient at what she does.

The following are some of the key expectations you should have for your researcher.

Asking questions

Asking the right questions is a basic expectation. Don’t laugh. This is harder than it looks. Asking questions involves the preliminary step of listening to understand what the issue actually is. Not everyone can do this.

Solving a problem isn’t as simple as asking the question you want answered.

For example, your overarching question might be “Does this website work well?” You could ask 1,000 people this question, and you wouldn’t know much after counting the “yes” and “no” responses.

What you need to know is “what about this site works, what doesn’t, and why?” Responses to these questions can be obtained in a variety of ways, allowing solutions to be identified. You can rely on your researcher to determine the most appropriate questions to ask in situations like this.

Researchers spend years listening to professors, clients, peers, and stakeholders to identify core issues to solve as well as what questions will provide data to find a solution. When meeting with project staff from a recent client, don’t assume your researcher isn’t engaged if she is quiet. It is likely she is observing verbal and physical interactions in the room as she designs a plan of attack.

Navigating relevant literature

Most likely, other researchers have published findings from studies related to what your researcher will examine. Your researcher should easily navigate and compile reports and studies from the body of knowledge in UX, HCI, and other relevant fields. The fact that someone else has explored questions similar to those of a project you’re asking your researcher to tackle helps shape their thinking on how to move forward, using existing resources to their fullest potential.

Literature can serve to inspire your researcher. For example, studies of ecommerce sites suggest trust is a key factor in determining users’ purchasing behavior. If you have a client developing a site meant to provide information, not selling a product, how might trust be developed? Your researcher can use findings from ecommerce studies to shape her questions and study design and then potentially publish a report contributing to the field, beyond the needs of your client.

Using the right method

Asking the right questions and reading up on relevant literature leads to the next critical expectation for your researcher: Using the right method.

UX research is more than usability testing. Your researcher knows methods shouldn’t dictate the questions asked, but the opposite: Your methods should be tailored to get relevant data for the questions to be asked.

Picking a method is hard work, this is why you need a researcher in the first place, they have the training and experience needed to select the right method for the question being asked. Use your researcher to do this. Your researcher carries a toolbox of methods. They might have preferences, or be more comfortable with certain methods, but they should not be a one-method pony. Some researchers are on a constant quest to define or refine new methods to answer questions. These can be exciting models to work with–the sports cars of UX researchers–willing to push the pedal to the metal to see where things go.

Regardless of the amount of planning, you often find yourself in a situation less than the ideal one written up in a methods textbook. Adapting to on-the-ground scenarios is something to expect from your researcher. Whether it’s using her smartphone to record an interview when her digital voice recorder dies, or adjusting on the fly when a busy client decides they only have 45 minutes to complete a 90-minute interview, your researcher should walk away from each scenario maximizing her ability to be flexible and still collect relevant data.

Translating findings

You’ve asked the right questions and selected the right method to collect data; now your researcher should serve as a translator for the application of research findings. Study results can be confusing if not interpreted appropriately. This includes verbal and written reports tailored to the experience and expectations of your audience. Your researcher should embrace the opportunity and challenge presented by making the results of her labor relevant to her peers.

Silo-busting

Researchers should come with the ability to break down silos, serving as ambassadors internally and externally, across teams and projects. Researchers are often deployed with surgical precision at specific intervals in a project timeline. This means your researcher might be actively involved in five or six projects simultaneously, giving her a breadth of insights. Few others within your organization are as able to communicate on the goals and achievements of multiple projects as she is. If findings from one study being conducted for client A would impact a recommendation for client G, your researcher should ensure everyone working with client G is aware of this.

Academia: A land far, far away

To make the best use of your researcher, it’s important to know where they come from. Especially if she is one of the PhD models, she was likely assembled in a far away land called “Academia.”

In Academia, your researcher gained or honed some of her most useful attributes: critical thinking; exposure to broad topics; research methods, both quantitative and qualitative; analyzing, interpreting, and presenting results; and connections with fellow researchers and academics.

Academia is the land of publish or perish. There are plenty of opportunities to give presentations to groups, write papers, teach courses, and create visual displays of data for various projects. This experience should leave your researcher well polished at speaking and presenting research in various formats well before they land at your front door. Although not all researchers are the best orators in the room, they should all be highly proficient at tailoring the message to their audience.

Additionally, your researcher has navigated an unbelievable amount of bureaucracy to escape Academia with a degree. She comes with the skills of diplomacy, patience, interpreting technical documents, and correctly filling out these documents under duress. This contributes to refining her ability to successfully reach the finish line and receive the prize. Your researcher is a doer and a finisher!

There are some things done in Academia, however, that don’t translate as well in the “real world.”

Academics have a unique language beyond the jargon typically found in professional fields. An example of research-ese is the statement, “I don’t think the items in this scale are valid at measuring the factor they purport to” translates to, “We might not be asking the right questions on this survey.”

Using obscure words–sometimes in different languages–becomes second nature to those moving through Academia. It is perfectly acceptable to tell your researcher she isn’t speaking your language. She should be able to translate for you; you just need to be clear when this is necessary.

Academia instills an unrealistic sense of time, as well. Your researcher may have spent one, two, or more years working on a single research project while earning her degree. Anyone that’s spent time in the real world knows you are lucky to have a timeline of one or two months to complete a study and, more realistically, about three weeks.

Adjusting the timeline for conducting a study is something you can expect your researcher to come to grips with rather quickly. You might see smoke coming out of her ears as gears that have been set to snail’s pace spin at hyper speed, but trust me, the adjustment will happen.

Be clear about your expectations for timelines at the beginning of a project, particularly if your researcher is fresh out of Academia.

The attributes instilled by Academia have become ingrained in your researcher. Enjoy them while you provide coaching to help her adapt to your business’s requirements. Experiences in Academia are part of what makes your researcher quirky, unique, and invaluable to your organization.

As time passes, she will become more polished, especially if you provide her with explicit feedback on what she is doing well and what she can do to improve. Patience is key when helping your researcher transition from Academia; if you exercise it, you will find the results quite rewarding.

Care and maintenance

Addressing the following will ensure your researcher stays running at optimal conditions.

Continuous learning opportunities

Researchers have an inherent love of learning. Why else would someone voluntarily go to 20th grade? Your researcher probably believes “everyone is a lifelong learner.”

It’s critical to offer educational opportunities and training. You must allot time and money for her to attend classes and seminars on topics ranging from research methods, to statistical analysis, to how to visualize data.

You should offer these opportunities to all of your staff; learning opportunities are key for ensuring a high level of morale throughout your organization. These opportunities aren’t always costly. Many organizations offer free or low cost webinars lasting the time of a reasonable lunch break.

Membership in professional organizations

Professional organizations allow your researcher opportunities to keep a pulse on the current state of their field. Professional organizations often host events and distribute publications promoting professional development and networking among professionals.

You should provide your researcher funds to join a professional organization; however, there are organizations that do not charge a fee to join. For example, I am a member and current Vice Chair for PhillyCHI the ACM chartered professional organization serving Philadelphia and the Delaware Valley region. There’s no charge to join, and monthly events are free for anyone to attend.

I suggest encouraging your researcher to attend meetings and allowing her time to serve as a volunteer or board member of professional organizations. There are numerous legitimate professional organizations at local, national, and international levels affiliated with ACM, IxDA, UXPA, and more.

Attending conferences and workshops

There’s a subconscious desire for researchers to congregate to drink beer and exchange ideas. Attending conferences allows researchers to meet peers from around the world and across topics, to learn the state of the art in their field.

Your researcher is most likely aware of the various local UX organizations such as ACM SIGCHI and UXPA sponsored groups, UX book clubs, and other UX meetups. Many of these groups offer workshops and one day events that are low or no cost (Thanks sponsors!). So, if you need convincing on the value of attending conferences, you can dip your toe in the water without blowing the budget. There’s also no shortage of national and international UX conferences that would satisfy your researcher’s needs. You can start with this list compiled by usertesting.com.

Besides getting a chance to feed off the ideas of others, interacting with professionals in her field, and allowing her to show off her work, there is another way of getting value from having your researcher attend conferences:

At Intuitive Company, staff give presentations on any conference they attend using company funds. This promotes the value of attending conferences to your staff, with the added benefit of allowing your researcher to present information to their peers, something most researchers already enjoy doing.

Reading

This was mentioned in expectations, but allowing your researcher time to read is your responsibility. She is one of those rare birds that actually recharge their batteries when reading, particularly when it relates to her research and practice interests.

Here’s a secret: You benefit from your researcher’s desire and ability to read! By allowing your researcher to read, you are actually allowing her to work, so long as you structure it correctly. For example, tell her you want her to conduct a literature review; therefore you are giving permission to read while at the same time setting up the expectation that there will be a usable product as the outcome of her reading. A literature review on a relevant topic can inform future research you engage in as well as design recommendations you make.

Win-win.

If you still can’t fathom giving your researcher time to read on the job, you should at least provide her with a book budget to purchase some of the must reads in UX.

Publishing and presenting

What good would research, professional development, conference attending, and reading do if your researcher couldn’t share her newfound knowledge with others?

Academia has hammered the need for dissemination into the fiber of your researcher’s being. Allowing time for writing and presenting is another area of maintenance that is your responsibility. You should encourage her to present at conferences and publish articles, blog posts, and white papers on relevant topics.

This is a way for her and your organization to build a strong brand in the communities you work in. For example, having your researcher cited as an expert on responsive design because she’s published on the topic is something you can include in future proposals and presentations you make to potential clients.

Conclusion

The success of your researcher is a two-way street. If you’ve already begun the journey with your researcher, this article might have highlighted expectations or maintenance that you’ve overlooked. If so, it isn’t too late to implement change; she can handle that as easily as a dead recorder, and you can enhance the relationship you have with her. If you haven’t started the journey, the advice provided can help ensure you get the most from your well maintained researcher for years to come.

What would you add or change to this manual based on your experience?

Additional resources

Forrester Report on best practices in UX (2009): https://www.adobe.com/enterprise/pdfs/Forrester_Best_Prac_In_User_Exp.pdf

Sandy Greene of Intuitive Company on evolving a creative workplace: http://boxesandarrows.wpengine.com/author/sgreene/

Five Things They Didn’t Teach Me in School About Being a User Researcher

Written by: Chelsey Glasson

Graduate school taught me the basics of conducting user research, but it taught me little about what it’s like working as a user researcher in the wild. I don’t blame my school for this. There’s little publicly-available career information for user researchers, in large part because companies are still experimenting with how to best make use of our talents.

That said, in the midst of companies experimenting with how to maximize user researchers, there are a few things I’ve learned specific to the role of user researcher that have held true across the diverse companies I’ve worked for. Some of these learnings were a bit of a surprise early on my my career, and I hope in sharing them I’ll save a few from making career mistakes I made in the past for lack of knowing better.

There’s a ton of variation in what user researchers do.

In my career, I’ve encountered user researchers with drastically varying roles and skillsets: many who focus solely on usability, a few who act as hybrid designers and researchers, some that are specialists in ethnography, and yet others who are experts in quantitative research. I’ve also spoken with a few who are hybrid market/user researchers, and I know of one tech company that is training user researchers to own certain product management responsibilities.

If you take a moment to write down all of the titles you’ve encountered for people who do user research work, my guess is that it will be a long one. My list includes user experience researcher, product researcher, design researcher, consumer insights analyst, qualitative researcher, quantitative researcher, usability analyst, ethnographer, data scientist, and customer experience researcher. Sometimes companies choose one title over another for specific reasons, but most of the time they’ll use a title simply because of tradition, politics, or lack of knowing the difference.

At one company I once worked for, my title was user researcher, but I was really a usability analyst, spending 80% of my time conducting rapid iterative testing and evaluation (RITE) studies. When I accepted the job at that company, I assumed–based on my title–that I’d be involved in iterative research and more strategic, exploratory work. I quickly learned that the title was misleading and should have been usability analyst.

What does this all mean for your career?

For starters, it means you should do a ton of experimentation while in school or early on in your career to understand what type of user research you enjoy and excel at most. It also means that it’s incredibly important to ask questions about the job description during an interview to make sure you’re not making faulty assumptions, based on a title, about the work you’d be doing.

Decisions influence data as much as data influences decisions.

I used to think the more data the better applied to most situations, something I’ve recently heard referred to as “metrics fetishism.” I’ve now observed many situations in which people use data as a crutch, end up making mistakes by interpreting “objective” data incorrectly, or become paralyzed by too much data.

The truth is that there are limitations to every type of data, qualitative and quantitative. Even data lauded by some as completely objective–for example, data from website logs or surveys–oftentimes includes a layer of subjectiveness.

At the beginning and end of any research project there are decisions to be made. What method should I use? What questions should I ask and how exactly should they be asked? Which metrics do we want to focus on? What data should we exclude? Is it OK to aggregate some data? What baselines should we compare to? These decisions should themselves be grounded in data and experience as much as possible, but they will almost always involve some subjectivity and intuition.

I’ll never forget one situation in which a team I worked with refused to address obvious issues and explore solutions without first surveying users for feedback (in large part because of politics). In this situation, the issues were so obvious that we should have felt comfortable using our expertise to address them. Because we didn’t trust making decisions without data in this case, we delayed fixing the issues, and our competitors gained a huge advantage. There’s obviously a lot more detail to this story, but you get the point: In this circumstance, I learned that relying on data as a crutch can be harmful.

What does this mean for your career?

Our job as user researchers is not only to deliver insights via data, but also to make sure people understand the limitations of data and when it should and shouldn’t be used. For this reason, a successful user researcher is one who’s comfortable saying “no” when research requests aren’t appropriate, in addition to explaining the limitations of research conducted. This is easier said than done, especially as a new user researcher, but I promise it becomes easier with practice.

You’re not a DVR.

Coming out of school, I thought my job as a user researcher was solely to report the facts: 5 out of 8 users failed this task, 50% gave the experience a score of satisfactory, and the like. I was to remain completely objective at all times and to deliver massive reports with as much supporting evidence as I could find.

I now think it’s old-school for user researchers to not have an opinion informed by research findings. Little is accomplished when a user researcher simply summarizes data; that’s what video recordings and log data are for. Instead, what’s impactful is when researchers help their teams prioritize findings and translate them into actionable terms. This process requires having an opinion, oftentimes filling in holes where data isn’t available or is ambiguous.

One project I supported early in my career involved a large ethnography. Six user researchers conducted over 60 hours of interviews with target users throughout the United States. Once all of the interviews were completed, we composed a report with over 100 PowerPoint slides and hours of video footage, summarizing all that was learned without making any concrete recommendations or prioritizing findings. Ultimately we received feedback that our report was mostly ignored because no one had time to read through it and it wasn’t clear how to respond to it. Not feedback you want to receive as a user researcher!

What does this mean for your career?

The most impactful user researchers I’ve encountered in my career take research insights one step further by connecting the dots between learnings and design and product requirements. You might never be at the same depth of product understanding as your fellow product managers and designers, but it’s important to know enough about their domains to translate your work into actionable terms.

Having an opinion is a scary thought for a lot of user researchers because it’s not always possible to remain 100% objective in bridging the gap between research insights and design and product decisions. But remember that there’s often always limitations and a subjective layer to data, so always remaining 100% objective just isn’t realistic to begin with.

Little is accomplished when data is simply regurgitated; our biggest impact is contributing to the conversation by providing actionable insights and recommendations that helps decision makers question their assumptions and biases.

Relationships aren’t optional, they’re essential.

As a student, my success was often measured by how hard I worked relative to others, resulting in a competitive environment. I continued the competitive behavior I learned in school when I first started working as a user researcher; I put my nose to the grindstone and gave little thought to relationships with my colleagues. What I quickly learned, however, is that taking time to establish coworker relationships is just as important as conducting sound research.

Work shouldn’t be a popularity contest, right? Right–but solid coworker relationships make it easier to include colleagues in the research process, transforming user research into the shared process it should be. And trust me, work is way more fun and meaningful if you enjoy your coworkers!

What does this mean for your career?

Take the time to get to know your coworkers on a personal level, offer unsolicited help, share a laugh, and take interest in the work that your colleagues do. I could share a personal example here, but instead let me refer you to Dale Carnegie’s book How to Win Friends and Influence People. Also check out Tomer Sharon’s book It’s Our Research.

Expect change–and make your own happiness within it.

Change is a constant for UX’ers. I’m on my eighth manager as a user researcher, and in my career I’ve been managed by user researchers, designers, product managers, and even someone with the title of VP of Strategic Planning. I’ve also been through four reorganizations and a layoff.

What does this mean for your career?

Change can be stressful, but when embraced and expected, you’ll find that there are benefits to change. For example, change can provide needed refreshment and new challenges after a period of stagnation. Change can also save you from a difficult project or a bad manager.

I remember a conversation with a UX leader in which he shared he once quit a job because he couldn’t get along with a peer who just didn’t get the user experience process. A few months after he quit, the peer was fired. If only he had stuck around for a while.

The U.S. Navy SEALs have a saying: “Get comfortable being uncomfortable,” which refers to the importance of remaining focused on the objective at hand in the middle of ongoing change. Our objective as user researchers is to conduct research for the purpose of improving products and experiences for people. Everything else is secondary–don’t get distracted.

For more detailed recommendations on how to deal with change as a user research, I highly recommend watching Andrea Lindman’s talk “Adapting to Change: UX Research in an Ever-Changing Business Environment.”

Concluding thoughts

I’ve been happy to see in the past two years that the user experience community has stepped up in making career advice more readily available (we could do even better, though). For user researchers wanting advice beyond what I’ve shared in this article, here are four of my favorite resources:

  • Judd Antin’s talk in which he covers many opportunities and challenges of doing user research: http://vimeo.com/77110204.
  • You in UX, an online career conference for user experience professionals.
  • Tomer Sharon’s book It’s Our Research.
  • A special issue of UXPA’s UX Magazine, with the theme of UX careers.