Teaching/Learning UX: Considerations for Academic-Industry Partnerships

Written by: Guiseppe Getto

Higher education is poised to help produce the next generation of user experience designers, but we can’t do it alone. In the wake of Fred Beecher’s recent “Ending the UX Designer Drought” and studies by Onward Search, UserTesting, and the Nielsen Norman Group, it is clear that the UX market is booming and that UX designers enjoy a high level of job satisfaction. It is also clear that too few UX professionals exist to meet current demand.

And while apprenticeship programs like Fred’s can help meet much of this demand, those of us in higher ed who have hitched our research, teaching, and service agendas–our entire professional identities–to UX are uniquely positioned to help students navigate to those apprenticeship programs, or even to help take the brunt of some of this training.

If we are to do so, however, we need members of industry to partner with us.

Obstacles to establishing UX workflows in higher ed

Before discussing ways that higher ed can help produce UX designers, let me discuss obstacles that we academics face, obstacles that make it difficult to nimbly respond to industry realities. I’m sure it won’t be news to anyone, for example, that the academy is very siloed.

At the same time that knowledge in the academy is often chopped up into discrete units called disciplines, however, there are other obstacles to establishing UX as a real concern that are specific to state- and federal-funded institutions of higher learning. The first of these that must not be underestimated is a status quo mentality that is endemic to academia but that doesn’t come from academics.

That may sound confusing, but remember that our schools are tied to state and federal budgets that are ultimately decided by politicians, not educators. Every program, major, minor, course, or even revision to an existing course, has to be vetted by administrators under increasing pressure by legislators to make every dollar count. This is often called a “strategic plan” in higher ed, and is our version of a business process model.

In this context, administrative response historically has been to veto any new programs that don’t clearly extend from an existing and well-established discipline. This may sound like a death knell for UX, an interdisciplinary practice that by its nature draws on knowledge from a wide variety of contexts, some academic and some not. And though it is a significant challenge, it is not an insurmountable one. It simply means that being a UX professional in higher ed means working within, and sometimes against, disciplinary boundaries.

What I mean by “disciplinary boundaries” is that every academic must do work that is understandable as a form of research, teaching, and/or service to a particular academic discipline and to a particular academic institution. Let me give a personal example to demonstrate what this workflow looks like. From my own discipline of technical communication and my own program seated at East Carolina University, I’m responsible for doing research, teaching, and service work that is recognizable as a contribution to the field of technical communication as well as to mid-size, doctoral-granting, state university.

This means that whatever I research has to be intelligible to other members of my field or I won’t get published and I won’t get tenure. This also means that I’m responsible for teaching a dizzying array of skill sets from technical writing to web design to UX design. Service is notoriously ill-defined in higher ed but typically requires serving on committees, serving on boards of professional organizations, and some form of community service (e.g. service-learning or volunteering).

I also should mention that there are exceptions to this workflow. Academic programs capable of reliably training students in the discipline of UX all on their own have  relatively recently sprung up at places like Kent State, Michigan State, and University of Washington. Largely interdisciplinary, these programs require a kind of perfect storm to form, such as a massive restructuring of a university system, a cluster of faculty hires in a particular specialization, or some very adventurous college administrators.

For the most part, what I’ve described is the uber-structure many academic professionals must work within. But there are ways to inject UX practices into this structure. Say I’m asked to teach a technical writing class, for instance, which is a central course in our program. A big focus of this class is the creation of documentation, because that’s part of the course description that was approved. But how is “documentation” increasingly being produced? In agile environments, where UXers or tech writers often build it directly into the information architecture of the product they’re designing. Following this trend, my recent technical writing course featured a series of learning modules that focused as much on traditional modes of documentation as it did on content strategy and IA.

For UX partnerships to work in higher ed, they must be enacted as a form of research, teaching, and/or service.

Why is this important? Because if many academic professionals must engage in work that fits this basic workflow, it means that we must always produce scholarship, teach courses, and engage in service that furthers the missions of our institutions as well as the knowledge-making practices of our disciplines. For UX partnerships to work in higher ed, they must be enacted as a form of research, teaching, and/or service.

Why academic-industry partnerships MUST exist

At the same time, it is also arguable that industry organizations cannot meet the need for new UX professionals alone, and more importantly, they shouldn’t have to. Currently, there are over 20 million students enrolled in institutions of higher education. And though of course only a fraction of those students might have the inclination to become a UX designer, that is simply too big a pool of potential new designers to pass up.

Academics can’t train UX professionals by ourselves, either. The discipline moves too fast and we are tied to a workflow that is very different from that of industry. This is thus a partnership that must happen to sustainably produce sufficient UX professionals to meet current demand. Millions of people from all walks of life pass through our classrooms, and if we can reach even a tiny percentage of those people and can set them on a path towards a career in UX, then UX professionals in both the academy and industry have an obligation to do so.

So, how can academic-industry partnerships be built that allow academics to keep their jobs? That’s what I turn to next.

Research partnerships

Academic research can be produced in many forms: empirical, theoretical, practice-based, quantitative, qualitative, and the like. All we really need from industry is access. We need to be able to see inside your organizations, to pick your brains through research interviews, and to survey you about new trends. We need to do focus groups about theories we have held for years to see if they pass muster with industry realities.

The best thing you can do for us in this context, in other words, is collaborate with us on our research projects.

Teaching partnerships

The largest venue within which UX could take off in higher ed is through teaching. In my own tenure as a college-level instructor for over ten years now I have taught 4-8 courses per year of 20-30 students per course. That means that conservatively I have personally taught over 1,500 students ranging from college freshman all the way up to graduate students, and in topics ranging from basic writing to UX design.

The only reason I know what UX is, and how to give students some practice in it, however, is because I have reached out to folks in industry and they have responded. I have gone to conferences like the IA Summit. I have completed dozens of webinars and workshops. I am an active member of my local chapter of the UXPA. And I have collaborated with industry partners on projects ranging from individual articles on how to teach UX to entire courses in UX.

If it weren’t for these partnerships, when I was recently asked to teach a graduate level course in UX for our program, I wouldn’t have been able to do so.

We know how to teach, but we will never be as cutting-edge at UX as those of you who work as UX designers every single day.

The best thing you can do for those of us who have opportunities to teach UX, then, is to help us with the subject matter we teach in our courses. We’re educators. We know how to teach, but we will never be as cutting-edge at UX as those of you who work as UX designers every single day. We need help with what to teach.

Service partnerships

There are also endless opportunities to get involved in academic service that fuels the UX discipline. This can range from the creation of book clubs and informal meet-ups to service-learning partnerships and other forms of contextualized learning. In any given community, there are thousands of local non-profits and schools in need of UX help, from content strategy to CMS development to usability testing existing websites. Hardly any of these organizations can afford to hire a UX professional, but students could work in partnership with a UX professional to serve them, and could develop expertise as a result.

On any given college campus, there are also dozens of student organizations on everything from business management to community service to web design, many of whom look for guest speakers to talk to them about career opportunities, how to land a job, how to gain traction in a new field, etc. It simply cannot be overstated that there is simply no shortage of mentoring opportunities for industry professionals to entertain.

Mentoring is thus the simplest and yet most valuable form of service. And it is incredibly simple: think about the first opportunity you got in UX and try to steer someone else toward that kind of opportunity, or even create one for them in your own organization.

This is what these partnerships could look like

We academics are the people who write those textbooks you were forced to buy in every introductory class in college. Imagine if they were relevant. Imagine if you helped us write them. Or imagine if you helped us write how-to articles for our academic journals so that other academic professionals could learn about UX in a venue that is intelligible to them. Or imagine if you helped perform research projects that were actually sound instances of UX design so that we could begin to solve our own UX problems and maybe even contribute some new solutions for you as well.

Imagine if you volunteered to come speak at our conferences about any of these topics, or even at informal meet-ups and colloquia at our institutions. We can rarely afford to pay you what you’re worth, or what you get for industry-sponsored conferences, but we will always appreciate the knowledge you bring.

Those of us in higher ed are in a position to introduce UX on a scale upon which it has never before been introduced.

Think of it this way: how many eighteen-year-olds graduate from high school and when asked by their guidance counselor what they want to be, say: “I want to be a UX designer!” Few, if any. Many of those bright young people enroll in college courses, however, part-time or full-time. Perhaps they decide to major in computer science, where they first learn about usability and its role in application development. Or perhaps they take a class in technical communication where they first encounter the central concepts of information architecture. Those of us in higher ed are in a position to introduce UX on a scale upon which it has never before been introduced.

But we need your help. If you don’t think we’re preparing students for opportunities as UX designers fresh out of our programs, then tell us that. Better yet: agree to be a guest speaker in our classes. Better yet: help us create learning modules on UX that we can use in multiple classes. Better yet: help us create and advocate for an entire class in UX. Better yet: volunteer to co-teach the class with us. Better yet: help us form an industry advisory committee for our entire program. The list goes on and on.

If you want that 18-year-old who is fresh out of high school to consider going into a field they may have never heard of, a field that probably first makes sense to them as a subset of an existing major, partner with your local university or community college to make sure that 18-year-old does hear about UX, to make sure they get some training in it, and to make sure they’re on a path toward being useful to you someday.

This is how all of this makes for a stronger UX community

If nothing else, I hope to start a conversation around expanding UX partnerships between interested professionals within both higher ed and industry. These partnerships exist, but they need to be the norm, not the exception. If every industry professional who reads this article volunteers as a research partner, teacher consultant, mentor, or service-learning partner at their local university, UX could become one of the most sought-after careers in higher ed. Students could be beating down academic doors asking for more experience in UX.

That kind of demand could create the kind of pressure we academics need to start new courses and majors in UX. If our students are demanding UX, then we can give it to them. If our students have no idea what UX is because they don’t hear about it until their senior year, then it becomes very difficult to create high-quality UX learning opportunities.

In the past three years, since my real interest in UX began, I have been working diligently to introduce students to UX in every course I teach. That’s a tall order considering that I can’t simply teach five sections (my current teaching load) of “Introductory UX Design” every year. I have had the opportunity to teach some standalone UX courses, but, as I mentioned above, mostly I introduce UX through individual learning modules and homework assignments in classes in technical communication, business writing, and web design.

But I have also met resistance, and not only from the academic side. A lot of industry-based people have helped me and inspired me in immeasurable ways, but I have also been met with downright suspicion. I have been asked if I feel like I am a form of competition for industry-based apprenticeship programs and for the newly formed Unicorn Institute, the UX field’s first standalone design school. Mostly, I have been asked repeatedly how I know enough about UX to teach it.

And the simple fact of the matter is: I don’t. I have been researching and practicing UX for three years now, research that has included in-depth practice with nearly every UX method on the market. I have published both research and practice-based articles on it (some of which were co-authored by UX designers), and have taught numerous courses in it, but I am not a full-time UX designer. Professionals like me are also all that stand between industry organizations looking to hire new UX talent and those 20 million people I mentioned earlier, few of whom start college even knowing what UX is.

My invitation is to see us as partners in helping to improve the quality of UX education regardless of where students and mentees are getting that education.

So, rather than seeing professionals like me as a form of competition–we simply don’t have the resources to compete with industry-based organizations, even if we wanted to–my invitation is to see us as partners in helping to improve the quality of UX education regardless of where students and mentees are getting that education.

Let’s face it: UX is difficult for many of us who study it, teach it, and do it for a living to define. We owe it to the next generation of UX professionals to introduce them to UX as soon as possible in their professional development, to be the frontline of UX education, so-to-speak. That’s the only way to make certain that students who are dedicated to becoming UX professionals have the opportunities they need to make that possibility a reality.

Five Things They Didn’t Teach Me in School About Being a User Researcher

Written by: Chelsey Glasson

Graduate school taught me the basics of conducting user research, but it taught me little about what it’s like working as a user researcher in the wild. I don’t blame my school for this. There’s little publicly-available career information for user researchers, in large part because companies are still experimenting with how to best make use of our talents.

That said, in the midst of companies experimenting with how to maximize user researchers, there are a few things I’ve learned specific to the role of user researcher that have held true across the diverse companies I’ve worked for. Some of these learnings were a bit of a surprise early on my my career, and I hope in sharing them I’ll save a few from making career mistakes I made in the past for lack of knowing better.

There’s a ton of variation in what user researchers do.

In my career, I’ve encountered user researchers with drastically varying roles and skillsets: many who focus solely on usability, a few who act as hybrid designers and researchers, some that are specialists in ethnography, and yet others who are experts in quantitative research. I’ve also spoken with a few who are hybrid market/user researchers, and I know of one tech company that is training user researchers to own certain product management responsibilities.

If you take a moment to write down all of the titles you’ve encountered for people who do user research work, my guess is that it will be a long one. My list includes user experience researcher, product researcher, design researcher, consumer insights analyst, qualitative researcher, quantitative researcher, usability analyst, ethnographer, data scientist, and customer experience researcher. Sometimes companies choose one title over another for specific reasons, but most of the time they’ll use a title simply because of tradition, politics, or lack of knowing the difference.

At one company I once worked for, my title was user researcher, but I was really a usability analyst, spending 80% of my time conducting rapid iterative testing and evaluation (RITE) studies. When I accepted the job at that company, I assumed–based on my title–that I’d be involved in iterative research and more strategic, exploratory work. I quickly learned that the title was misleading and should have been usability analyst.

What does this all mean for your career?

For starters, it means you should do a ton of experimentation while in school or early on in your career to understand what type of user research you enjoy and excel at most. It also means that it’s incredibly important to ask questions about the job description during an interview to make sure you’re not making faulty assumptions, based on a title, about the work you’d be doing.

Decisions influence data as much as data influences decisions.

I used to think the more data the better applied to most situations, something I’ve recently heard referred to as “metrics fetishism.” I’ve now observed many situations in which people use data as a crutch, end up making mistakes by interpreting “objective” data incorrectly, or become paralyzed by too much data.

The truth is that there are limitations to every type of data, qualitative and quantitative. Even data lauded by some as completely objective–for example, data from website logs or surveys–oftentimes includes a layer of subjectiveness.

At the beginning and end of any research project there are decisions to be made. What method should I use? What questions should I ask and how exactly should they be asked? Which metrics do we want to focus on? What data should we exclude? Is it OK to aggregate some data? What baselines should we compare to? These decisions should themselves be grounded in data and experience as much as possible, but they will almost always involve some subjectivity and intuition.

I’ll never forget one situation in which a team I worked with refused to address obvious issues and explore solutions without first surveying users for feedback (in large part because of politics). In this situation, the issues were so obvious that we should have felt comfortable using our expertise to address them. Because we didn’t trust making decisions without data in this case, we delayed fixing the issues, and our competitors gained a huge advantage. There’s obviously a lot more detail to this story, but you get the point: In this circumstance, I learned that relying on data as a crutch can be harmful.

What does this mean for your career?

Our job as user researchers is not only to deliver insights via data, but also to make sure people understand the limitations of data and when it should and shouldn’t be used. For this reason, a successful user researcher is one who’s comfortable saying “no” when research requests aren’t appropriate, in addition to explaining the limitations of research conducted. This is easier said than done, especially as a new user researcher, but I promise it becomes easier with practice.

You’re not a DVR.

Coming out of school, I thought my job as a user researcher was solely to report the facts: 5 out of 8 users failed this task, 50% gave the experience a score of satisfactory, and the like. I was to remain completely objective at all times and to deliver massive reports with as much supporting evidence as I could find.

I now think it’s old-school for user researchers to not have an opinion informed by research findings. Little is accomplished when a user researcher simply summarizes data; that’s what video recordings and log data are for. Instead, what’s impactful is when researchers help their teams prioritize findings and translate them into actionable terms. This process requires having an opinion, oftentimes filling in holes where data isn’t available or is ambiguous.

One project I supported early in my career involved a large ethnography. Six user researchers conducted over 60 hours of interviews with target users throughout the United States. Once all of the interviews were completed, we composed a report with over 100 PowerPoint slides and hours of video footage, summarizing all that was learned without making any concrete recommendations or prioritizing findings. Ultimately we received feedback that our report was mostly ignored because no one had time to read through it and it wasn’t clear how to respond to it. Not feedback you want to receive as a user researcher!

What does this mean for your career?

The most impactful user researchers I’ve encountered in my career take research insights one step further by connecting the dots between learnings and design and product requirements. You might never be at the same depth of product understanding as your fellow product managers and designers, but it’s important to know enough about their domains to translate your work into actionable terms.

Having an opinion is a scary thought for a lot of user researchers because it’s not always possible to remain 100% objective in bridging the gap between research insights and design and product decisions. But remember that there’s often always limitations and a subjective layer to data, so always remaining 100% objective just isn’t realistic to begin with.

Little is accomplished when data is simply regurgitated; our biggest impact is contributing to the conversation by providing actionable insights and recommendations that helps decision makers question their assumptions and biases.

Relationships aren’t optional, they’re essential.

As a student, my success was often measured by how hard I worked relative to others, resulting in a competitive environment. I continued the competitive behavior I learned in school when I first started working as a user researcher; I put my nose to the grindstone and gave little thought to relationships with my colleagues. What I quickly learned, however, is that taking time to establish coworker relationships is just as important as conducting sound research.

Work shouldn’t be a popularity contest, right? Right–but solid coworker relationships make it easier to include colleagues in the research process, transforming user research into the shared process it should be. And trust me, work is way more fun and meaningful if you enjoy your coworkers!

What does this mean for your career?

Take the time to get to know your coworkers on a personal level, offer unsolicited help, share a laugh, and take interest in the work that your colleagues do. I could share a personal example here, but instead let me refer you to Dale Carnegie’s book How to Win Friends and Influence People. Also check out Tomer Sharon’s book It’s Our Research.

Expect change–and make your own happiness within it.

Change is a constant for UX’ers. I’m on my eighth manager as a user researcher, and in my career I’ve been managed by user researchers, designers, product managers, and even someone with the title of VP of Strategic Planning. I’ve also been through four reorganizations and a layoff.

What does this mean for your career?

Change can be stressful, but when embraced and expected, you’ll find that there are benefits to change. For example, change can provide needed refreshment and new challenges after a period of stagnation. Change can also save you from a difficult project or a bad manager.

I remember a conversation with a UX leader in which he shared he once quit a job because he couldn’t get along with a peer who just didn’t get the user experience process. A few months after he quit, the peer was fired. If only he had stuck around for a while.

The U.S. Navy SEALs have a saying: “Get comfortable being uncomfortable,” which refers to the importance of remaining focused on the objective at hand in the middle of ongoing change. Our objective as user researchers is to conduct research for the purpose of improving products and experiences for people. Everything else is secondary–don’t get distracted.

For more detailed recommendations on how to deal with change as a user research, I highly recommend watching Andrea Lindman’s talk “Adapting to Change: UX Research in an Ever-Changing Business Environment.”

Concluding thoughts

I’ve been happy to see in the past two years that the user experience community has stepped up in making career advice more readily available (we could do even better, though). For user researchers wanting advice beyond what I’ve shared in this article, here are four of my favorite resources:

  • Judd Antin’s talk in which he covers many opportunities and challenges of doing user research: http://vimeo.com/77110204.
  • You in UX, an online career conference for user experience professionals.
  • Tomer Sharon’s book It’s Our Research.
  • A special issue of UXPA’s UX Magazine, with the theme of UX careers.

User Experience Research at Scale

Written by: Nick Cawthon

An important part of any user experience department should be a consistent outreach effort to users both familiar and unfamiliar. Yet, it is hard to both establish and sustain a continued voice amongst the business of our schedules.

Recruiting, screening, and scheduling daily or weekly one-on-one walkthroughs can be daunting for someone in a small department having more than just user research responsibilities, and the investment of time eventually outweighs the returns as both the number of participants and size of the company grow.

This article is targeted at user experience practitioners at small- to mid-size companies who want to incorporate a component of user research into their workflow.

It first outlines a point of advocacy around why it is important to build user research into a company’s ethos from the very start and states why relying upon standard analytics packages are not enough. The article then addresses some of the challenges around being able to automate, scale, document, and share these efforts as your user base (hopefully) increases.

Finally, the article goes on to propose a methodology that allows for an adjustable balance between a department’s user research and product design and highlights the evolution of trends, best practices, and common avoidances found within the user research industry, especially as they relate to SaaS-based products.

Why conduct usability sessions?

User research is imperative to the success and prioritization of any software application–or any product, for that matter. Research should be established as an ongoing cycle, one that is woven into the fabric of the company, and should never drop-off nor be simply ‘tacked on’ as acceptance testing after launch. By establishing a constant stream of non-biased opinions and open lines of communication which are immune to politics and ever-shifting strategies, research keeps design and development efforts grounded in what should already be the application’s first priority–the user.

A primary benefit in working with SasS products is that you’re able to gain feedback in real-time when any feature is changed. You don’t have to worry about obsolete versions, or download packages–web-based software enables you to change directions quickly. Combining an ongoing research effort with popular software development methods such as agile or waterfall allows for immediate response when issues with an application’s usability are found.

Different from analytics

SaaS are unique in that there is not the same type of tracking needed in-product. Metrics such as page views or bounce-rates are largely irrelevant, because the user could be spending their entire session on configuring functions of a single feature on a single page.

For example, for our application here at Loggly, the user views an average of ~2 pages (predominantly login and then search) and spends on average 8x as long on search then any other page. Progression is made within the page-level functions, not among multiple pages within the application’s structure.

Javascript-heavy applications don’t have the same URL and tree structure content-heavy sites are built around but instead make calls to different states of the application from within the same page.

Say your analytics package gives an indication that something is wrong with the setup flow or configuration screen, but you don’t yet have a good concept of at what point in the process the users are getting stuck.

Perhaps a button might be getting click after click because it is confusing and unresponsive, not because its useful. Trying to solve this exclusively with an analytics package will pale in comparison to the feedback you’ll get from a single, candid user who hits the wall. As discussed later in this article, with screensharing, you’re able see the context in which the user is trying to achieve a specific task, defining the ‘why’ in their confusing becomes more apparent than just the ‘what’ are they clicking on.

Determining a testing audience

The first component of defining any research effort should be to define who you want to talk to. Ideally, you’ll be able to have a mix of both new users and veterans that are able to provide a well-rounded feedback loop on both initial impressions of your application as well as historical perspective on evolution and found shortcomings after repeated use, but not all companies have this luxury.

Once in the door

Focus first on the initial steps the user has to take when interacting with your application. It seems obvious, but if these are not fulfilled with maximum efficiency, the user will never progress into more advanced features.

Increasing the effectiveness of the flow through set-up, configuration, and properly defining a measure of activation will pay dividends to all areas of the application. This should be a metric that is tested, measured, and monitored closely, as it functions as a type of internal bounce rate. Ensuring that the top of the stream for the majority of application users is sound will guarantee improved usage further down the road to the deeper, buried interactions.

These advanced features should be also be tracked and measured with the correlation that starts to paint a profile of conversion. Some companies define conversion as free-to-paid; others do so in a more viral sense–conversion being defined as someone who has shared on social media or similar.

As you start itemize these important features, you’ll get a better sense of the usage profile for where you’re trying to point the user to. For example, adding a listing record, or perhaps customizing a page–these might match a profile for someone who is primed for repeat visitation, someone who has created utility and a lasting connection, and ultimately ready to convert.

Avoiding overlap

If there is a focus on recruiting participants who are newly signed-up users, then you’ll likely overlap with outbound sales efforts. Because your company’s sales and marketing funnel tries as hard as possible to convert trial users to paid, or paid to upgrade, the company’s priority will likely be on conversion, not on research.

Further, if a researcher tries to outreach for usability surveys at this point, from the user’s perspective (especially those deemed potential high-value customers) it would mean different prompts for different conversations with different people from various groups within your company, all competing for spots on their calendar. This gives a very hectic and frenetic impression of your company and should be avoided.

In the case of a SaaS product, sometimes the sales team has already made contact with potential customers, and many of these sales discussions involve demonstrations around populated, best-case scenarios (which showcase the full features) of your product.

As a result, you may find the participant has been able to ‘peek behind the curtain’ through watching the sales team provide these demonstrations, giving them an unfair advantage as to how much he / she knows before trying to finally use the product themselves. For the inexperienced user, your goal is to capture the genuine instinct of the uninitiated, not those who have seen the ‘happy path’ and are trying to trace back the steps to get to that fully-populated view.

To make sure you’re not bumping heads with the sales and conversion team, ask if you can take their castoffs–the customers they don’t think will convert. You can pull these from their CRM application and automate personalized emails asking for their time. I’ll outline this method in further detail in the section following, because it pertains to the veteran users as well.

Photo of people in a conference exhibit hall.
Conferences are a great way to survey new and existing users.

As described in a previous post, guerrilla testing at conferences is a great way of fulfilling what gets seen and what parts of the interface or concept get ignored. These participants are great providers of honest, unbiased feedback and haven’t been exposed to the product other than some initial impressions of the concept.

Desiring the messy room

But what about the users that have been using your product for months now, those who have skin in the game, have already put their sweat and dollars behind customization of their experience? Surveying these participants allows us to see where they’ve found both utility and what areas need be expanded upon. Surveying only the uninitiated won’t provide feedback on any nagging functional roadblocks, those which are found only after repeated use. These are the participants that will provide the most useful feedback, sessions where you can observe the environment that they’ve created for themselves, the ‘messy room.’

Making an observational research analogy, a messy room is more telling of the occupants’ personality than an empty one. Given your limitations, how has the participant been forced to find workarounds? Despite these workarounds, they’ve continued to use the product, in despite of how we’ve expected them to use it–and these two can be contrastingly very different.

Online feedback form for Loggly UK.
Example of a feedback form, initiated via email.
User is able to schedule a 1:1 screensharing session on the confirmation page.

Automated recruitment

Find your friendly marketing representative/sales engineer at your company (or just roll your own) and discuss with them the best way to integrate a user experience outreach email into the company’s post-funnel strategy. For example, post-funnel would be after their trial periods have long since expired and the user is either comfortable in their freemium state or fully paid up.

As mentioned earlier, you can also harvest leads from the top of the funnel in the discarded CRM leads. However, you’ll likely have a greater percentage of sessions with users that are misfires–those indifferent or only just poking around the app, with not yet a full understanding of what it might do. Thankfully, the opt-in approach for participation filters this out for the most part.

Focusing again on the recruitment of the veteran, experienced users, another, more complex scenario would be to trigger this UX outreach email once a specific set of features have been initiated–giving off the desired signature of an advanced, informed user.

Going from purely legacy-based perspective, six months of paid, active use should be enough time to establish a relationship with a piece of software, whether they love or hate it. If there exists enough insight into the analytics side of the sales process, it would behoove you to also make sure that the user has had a minimum number of logins across these six months (or however long you’ll allow the users to mature).

Outreach emails triggered through the CRM should empower the recipient to make the experience of the product better, both for themselves and their fellow customers. Netflix does a great job of this by continually asking about the streaming quality or any delays around arrival times of their product.

I also recommend asking the users a couple of quantitative and qualitative questions, as this metric something you should be doing for your greater UX efforts already. These questions follow the guidelines of general SUS (System Usability Survey) practices that have been around for decades. Make the questions general enough so that they can be re-used and compared going forward, without fear of needing the change the goalposts when features or company priorities change.

Screen grab of the user's desktop.
A peek into an active user’s work environment.

When engineering this survey, be sure to track which tier of customer is filling out these surveys, because both their experience and expectations could be wildly different. Remember also to capture the user’s email address as a hidden field so you can cross reference against any CRM or analytics packages that are already identifying existing customers.

Setting boundaries

It depends on the complexities of your product, but typically 20-30 minutes is enough time to cover at least the main areas of function. Any longer, and you might encounter people not wanting to fit in an entire hour block into their schedule. If these recorded sessions are kept to just a half-hour, I find that a $25 is sufficient compensation for this duration of time, but your results may certainly vary.

In any type session, do iterate that this is neither a sales, nor a support call. You’re researching on how to make the product better. However, you should be comfortable on how to avoid (or sometimes suggest) workarounds to optimize the participant’s experience, giving them greater value of use.

Tools of the trade

For implementation of the questionnaire, I hacked the HTML / CSS from a Google Form to exist as self-hosted page but still pushing results through the matching form and input IDs to the extensible Google Spreadsheet.

There are a few tutorials that explain how to retain your branding while using Google’s services. I went through the trouble so I can share the URL of either the form or the raw results with anyone, without the need to create an account or login. As we discuss the sharing component of these user research efforts, this will become more important. Although closed systems like SurveyMonkey or Wufoo are easy to get up and running, the extensibility or a raw, hosted result set does not compare.

Insert a prompt at the end of the questionnaire for the user to participate in a compensated user research survey, linking to a scheduling applications such as Calend.ly. This application has been indispensable for opt-in mass scheduling like this. The features of gCal syncing, timezone conversion, daily session capping, email reminders, and custom messaging all are imperative to a public-facing scheduling board. Anyone can grab a 30-minute time slot from your calendar with just your custom URL, embeddable at the end of your questionnaire.

To really scale this user research effort to the point where it can be automated, you cannot spend the time trying to negotiating mutually-available times, converting time zones and following up with confirmations. Calend.ly allows for you to cap the number of participants who can grab blocks of your time, so you can set a maximum number of sessions per day, preventing a complete overload of bookings in your schedule.

As a part of the scheduling flow within Calend.ly, a customizable input field asks the participant for their Skype handle in order to screen share together, and I’d advise for the practitioner to create a separate Skype account for this usability effort. With every session participant, you’ll begin to add and add more seemingly random contacts, any semblance of organization and purity for your personal contact list will be gone.

Screen grab of Calend.ly booking utility.
Calend.ly booking utility – a publicly-accessible reservation system.

Calend.ly booking utility – a publicly-accessible reservation system.

Once the user is on the Skype call, ask for permission to record the call and make sure that you give a disclaimer that their information will be kept private and shared with no one outside the company. You might also add ahead of time that any support questions that come up, you’ll be happy to direct to the proper technicians.

Permissions granted, be sure to re-iterate to the participant the purpose and goal of the call, and provide them with a license to say whatever they want, good or bad–you want to hear it. Your feelings won’t be hurt if they have frustrations or complaints about certain approaches or features of your product.

For recording the call, there are plenty of options out there, but I find that SnagIt is a good tool to capture video, especially given the resolution and dimension of the screen share tends to change based upon the participant’s monitor size. When compressing the output, a slow frame rate of 5/10 fps should suffice, saving you considerable file size when having to manage these large recordings.

Tagging annotations

When you’re walking the participant through the paces of the survey, be sure to annotate the time started and any high/lowlights you see along the way. While in front of your desktop, a basic note-taking utility application (or even pad and paper) should suffice. This will allow you to go back after the survey is finished and pull quotes for use elsewhere, such as powerpoint presentations or similar.

I always try to write a running diary of the transcript and a summary at the end just to cover what areas of the application we explored, as well as a quick summary of what feedback we gathered. Summarizing the typed transcript and posting the relative recorded video files should take no more than 10 minutes, which will still keep your total per-participant (including processing) time to under an hour each, certainly manageable as a part of your greater schedule.

Share the love (or hate)

I want to make sure that these sessions are able to be referred to by the executive and product management team for use in their prioritization strategy. Setting up an instance of MAMP / WordPress on a local box (we’re using one of the Mac Minis that power a dashboard display) which allows me to pass around the link internally and not have to deal with some of the issues around large video file sizes being uploaded, as well as alleviate any permissions concerns with these sessions being out in the wild.

Screen grab of the session archive interface.
Our UX session archive, with hundreds of recorded and tagged sessions.

Also important is to tag these posts attached to these files when you upload them. This allows faster indexing when trying to find evidence around a certain feature or function. Insert your written summary into the post content, and you’ll be able to better search on memorable quotes that might have been written down.

These resources can be very good for motivation internally, especially among the engineers who don’t often get to see people using the product they continually pour themselves into. They’ll also resonate with the product team, who will see first-hand what’s needed to re-prioritize for the next sprint.

After awhile, you’ll start to get a great library of clips that you can draw knowledge from. There’s also a certain satisfaction to seeing the evolution of the product in the interface through these screengrabs. That which was shown as confusing at one time may now be fixed!

Follow-up

Fulfillment of a participant compensation can be done through Amazon or other online retailers; you can wire a gift card through an email address, which you’ll be able to scrape as a hidden field from the spreadsheet of user inputs. Keep a running list of those that you’ve reached out to and contacted for responses.

You might also incorporate contacts met during sessions described in the Guerrilla Usability Testing at Conferences article, so you’ll be able to follow up when attending the next year’s conference to recruit again. After enough participants and feedback, think about establishing a customer experience council that you can follow up on with specific requests and outreach, even for quick vetting of opinions.

Conclusion

This article first outlined the strategies and motivation behind the research, advocating creating an automated workflow of continually-scheduled screenshares with customers, rather than trying to recruit participants individually. This methodology was then broken down to distinct steps of recruitment via email, gathering quantitative and qualitative feedback, and automating an opt-in booking of the sessions themselves. Finally, this article went on to discuss how to best leverage and organize this content internally, so that all might benefit from your process.

User research is imperative to the success and prioritization of any software application (or any product, for that matter). Yet, too often we forget to consume or own product. Whether it be server log management as I’ve chosen, or apartment listing or ecommerce purchases, shake off complacency and try to spend 30-mins a week trying to accomplish typical user tasks from start-to-finish.

Also make it a point to conduct some of these sessions among those you work alongside; you’ll be surprised what you can find just by the simple repetition with a fresh set of eyes and ears. The research process and its dependencies does not have to be as intricate as the one listed above.

 

When your company starts to incorporate user opinion into a design and development workflow, it will begin to pay out dividends, both in the perceived usability of your application as well as the gathered metrics of user satisfaction.

 

Honing Your Research Skills Through Ad-hoc Contextual Inquiry

Written by: Will Hacker

It’s common in our field to hear that we don’t get enough time to regularly practice all the types of research available to us, and that’s often true, given tight project deadlines and limited resources. But one form of user research–contextual inquiry–can be practiced regularly just by watching people use the things around them and asking a few questions.

I started thinking about this after a recent experience returning a rental car to a national brand at the Phoenix, Arizona, airport.

My experience was something like this: I pulled into the appropriate lane and an attendant came up to get the rental papers and send me on my way. But, as soon as he started, someone farther up the lane called loudly to him saying he’d been waiting longer. The attendant looked at me, said “sorry,” and ran ahead to attend to the other customer.

A few seconds later a second attendant came up, took my papers, and jumped into the car to check it in. She was using an app on an tablet that was attached to a large case with a battery pack, which she carried over her shoulder. She started quickly tapping buttons, but I noticed she kept navigating back to the previous screen to tap another button.

Curious being that I am, I asked her if she had to go back and forth like that a lot. She said “yes, I keep hitting the wrong thing and have to go back.”

Seeing an opportunity to explore her use of the device, I asked if this happened a lot. She said it did because the buttons were too small for her fingers. She told me every time a new feature was added to the reservation system, more buttons appeared on the screen and they all kept getting smaller. The system, I have to assume, was designed to use as few screens as possible.

Within two minutes she completed the check in and printed my receipt. She did it using a small battery-powered printer she wore over her other shoulder that printed roughly three-inch wide receipts. I asked how often she had to recharge her devices: once a day, usually at break times.

Her activity and my interaction with her took no more than a few minutes, but this is what I was able to glean from it:

  • Airport rental car attendants work in a fast-paced environment, often dealing with agitated customers who are running late for their flights.
  • These attendants, at least for this company, carry two devices over their shoulders as they work the busy return lanes.
  • The app in use at this company wasn’t designed to be the most finger-friendly. A greater emphasis seemed to be placed on having more features on fewer screens.
  • In addition to monitoring incoming vehicle traffic, they have make sure their devices have enough battery power to make it to the next break.
  • They are doing this work, carrying these devices, and being polite to sometimes impatient customers in a climate where the summer temperatures easily exceed 100ºF. And they are on their feet in a concrete parking deck for much of their work day.

That’s a lot of information to gather in a few minutes by asking three questions. What I found most interesting in the experience, as someone who designs digital user experiences, was the reminder that people around us every day are having problems using the hardware and software we create. It reinforced for me that we can learn a lot about how to approach hardware and software design just by watching people use these products, even products we had no part in creating.

Now, I certainly can’t claim to have found ways to improve the reservation system in question. That would require more field study with additional users as well as talking to the product designers and other stakeholders to understand the decisions that led to the current user interface. But, the experience left me wondering if the company ever bothered to watch its employees use the system in a real-world setting.

My point here is that we can still practice these observational research techniques even if we don’t always get the chance to do so on the products we get paid to design. And we can document these moments to make a case at our companies for why this kind of research needs to be part of our product development cycles. There’s value in these exercises, even if the results don’t immediately show up in our companies’ products. It’s an activity I’d encourage everyone to try.

There are plenty of opportunities for this sort of activity, including:

  • Retail self-checkout lanes,
  • Any self-service airport application,
  • Self-serve vending machines like Redbox or mass-transit ticketing machines,
  • Hotel check-in/check-out kiosks,
  • And many others I’m sure I missed.

There are a few things you need to be aware of and take into consideration before attempting to interact with strangers in the middle of a frustrating technology moment:

  • The people in these situations may be angry and flustered, so look for body language cues that may signal you are better off leaving them alone. If they are hitting the machine and cursing, you may just want to hang back and observe.
  • Politely approach people and ask if it’s OK to talk to them about their experience with the system in question. In most cases it’s better to wait until they are done because right at the moment they are using it they are probably more focused on completing their task.
  • Make sure you don’t imply they are doing something wrong. Let them know you are interested in making products better for people and would like to ask them a few quick questions about the experience they just had. If they say “no,” respect that and move along.
  • Explain that you are not with the company in question and can have no impact on any future version of the product. The last thing you want them to think is that you are a conduit for complaints to the company or that you can actually have any impact on future versions. It’s important to be as clear as possible to avoid setting false expectations.
  • Be sensitive to your surroundings and other customers who may be waiting to use a system. The last thing you ever want to do is slow down a busy checkout lane or self-serve kiosk at an airport.
  • Be sensitive to the application at hand and the location and time of day. It’s safe to say you should never approach someone using an ATM or other financial services kiosk. You also shouldn’t approach someone using a self-serve vending machine that is outdoors late at night. That could make them feel threatened or even cause them to react physically.
  • Honor any request by the business in question to not approach their customers.

Although exercises like this won’t tell us the things we’d like to know about the products we work on, they do let us practice the techniques of contextual inquiry and observation and make us more sensitive to various design issues. These experiences may also help us build the case in more companies for scheduling time and resources for in-field research with our actual customers.

Creating Your Personal Mission Statement

Written by: Louis Rosenfeld

You’re weird. In a good way, but weird nonetheless.

Weird in the sense that people outside of work likely have absolutely no clue what it is you do. Maybe many at work as well.

For me, this weirdness manifests itself at parties. Inevitably, a new acquaintance asks me what I do. Beads of sweat form on my forehead. My eyes dart around, desperately seeking my far more articulate wife, Mary Jean. I find her, ask her to explain me, and flee.

If you’re in UX or a related field, congrats: You probably have more work than you can manage in a time when many people are underemployed. But that doesn’t diminish the discomfort those weird moments cause.

How might we explain ourselves better?

I created a simple exercise to help create personal mission statements—something short and meaningful to say about yourself—with some help and encouragement from Christina Wodtke and Anders Ramsay. It’s fun, simple, and quick; in fact, I recently tried it out with a group at the Re:Design conference and it seemed to work well. Here’s what to do.

Start with a context.

It could be public, like something to include on your business card, your Twitter bio, or blurt out at those nerve-wracking parties. Or something private, like a statement you write down and keep in your wallet for you and you only.

Tell your story.

Find a friend or colleague—someone you don’t mind being a bit vulnerable with—who will take notes while encouraging you to talk about yourself and what’s important to you. Ten minutes is plenty.

Basic questions like these can get you going:

  • What kind of change would you like to be part of?
  • What’s your superpower?
  • What’s the difference between what you’re expected to do and what you want to do with your life?
  • What has always pissed you off?

Craft a short statement.

Together, take those key terms and phrases from the notes and work them into something that fits the context you chose. Easier said than done, so plan to iterate.

At Re:Design, I was the guinea pig. My context was finding a new Twitter bio to replace this one:

Founder of Rosenfeld Media and veritable UX action hero.

“UX action hero” is an inside joke: pointless for 99.99% of the people who encounter my bio. Time to axe it.

I started with the “What has always pissed you off” question, and told my session’s attendees a story of 20+ years of frustration with the traditional business models (and their defenders) that I’ve encountered in higher ed, consulting, publishing, and professional associations. Telling one’s “true story” is never easy—especially in front of 70 people—but two things really helped:

  1. The floor is yours. Whether you’re telling your story to one person or 70, it’s your time. Take as much of the ten minutes as you wish, and make it clear that there will be opportunities to discuss your story and brainstorm after you’re done.
  2. Vulnerability is engaging. The goal isn’t to impress anyone with a slick presentation of your many fine attributes; rather, use this opportunity to begin figuring out what you’re about. Your stumbles and your quirky, imperfect, unfinished story will actually draw in your partner(s). If they have an ounce of empathy and interest in you, they’ll be able and eager to help brainstorm ideas about who you really are.

My group was anxious to brainstorm before I was even done telling my story. They came up with these options:

  • Happy but not satisfied
  • Re-architect/Assassin of business models since 1965
  • Shapeshifter
  • Internet sherpa
  • Learning by teaching
  • Step, pivot, repeat

My faves are the second and the last one especially, as verbs (like “step, pivot, repeat”) are fairly constant when it comes to describing how we live and what we do. And if I wasn’t happy with any of these, repeating the exercise with someone else would have made sense—after all, it takes less than a half hour. In fact, I’m considering repeating it as often as I get the urge—maybe once per year.

In any case, here’s my new Twitter bio:

Founder of Rosenfeld Media. Slayer of traditional business models. Step, pivot, repeat.

And here are shiny new Twitter bios from some of the session’s attendees:

helping people be less afraid of making things better

…and…

Persuade. Connect. Convince. Judging user experiences everywhere.

Though I’m not sure which questions got each of them started, I am sure they were excited. In fact, they were anxious to share their new personal mission statements with the group.

They found it fun, simple, and quick. I hope you will too.