Most software user experience and product management teams have similar questions: How do users feel about our products? How is our product experience changing over time? How do users feel about a recent change we’ve made in the product?
Our user experience (UX) and product teams are no different, so we set up a system that provides an ongoing stream of data that answers these questions and does much more.
In this case study, we describe our user experience monitoring system at Qualtrics.[1] With data collected through this system, we are able to monitor the overall UX of our software-as-a-service (SaaS) products, share dashboards and reports with stakeholders, and send automated messages to individuals and groups based on key performance indicators. This system complements our existing traditional UX research initiatives (such as interviewing, surveying, usability evaluations, and reviewing telemetry data) with an actionable stream of high quality user experience data collected with very little ongoing effort from our team.
One of our finest tasks as designers is to filter the abundance of choice into easily digestible bits. Creating great interfaces is as much about motivating, teasing, leading, and guiding users along—so that they experience value, faster—as it is to improve usability by removing friction. This requires an endeavor into product psychology and the art of designing with purpose and intent.
It’s common in our field to hear that we don’t get enough time to regularly practice all the types of research available to us, and that’s often true, given tight project deadlines and limited resources. But one form of user research–contextual inquiry–can be practiced regularly just by watching people use the things around them and asking a few questions.
I started thinking about this after a recent experience returning a rental car to a national brand at the Phoenix, Arizona, airport.
My experience was something like this: I pulled into the appropriate lane and an attendant came up to get the rental papers and send me on my way. But, as soon as he started, someone farther up the lane called loudly to him saying he’d been waiting longer. The attendant looked at me, said “sorry,” and ran ahead to attend to the other customer.
A few seconds later a second attendant came up, took my papers, and jumped into the car to check it in. She was using an app on an tablet that was attached to a large case with a battery pack, which she carried over her shoulder. She started quickly tapping buttons, but I noticed she kept navigating back to the previous screen to tap another button.
Curious being that I am, I asked her if she had to go back and forth like that a lot. She said “yes, I keep hitting the wrong thing and have to go back.”
Seeing an opportunity to explore her use of the device, I asked if this happened a lot. She said it did because the buttons were too small for her fingers. She told me every time a new feature was added to the reservation system, more buttons appeared on the screen and they all kept getting smaller. The system, I have to assume, was designed to use as few screens as possible.
Within two minutes she completed the check in and printed my receipt. She did it using a small battery-powered printer she wore over her other shoulder that printed roughly three-inch wide receipts. I asked how often she had to recharge her devices: once a day, usually at break times.
Her activity and my interaction with her took no more than a few minutes, but this is what I was able to glean from it:
Airport rental car attendants work in a fast-paced environment, often dealing with agitated customers who are running late for their flights.
These attendants, at least for this company, carry two devices over their shoulders as they work the busy return lanes.
The app in use at this company wasn’t designed to be the most finger-friendly. A greater emphasis seemed to be placed on having more features on fewer screens.
In addition to monitoring incoming vehicle traffic, they have make sure their devices have enough battery power to make it to the next break.
They are doing this work, carrying these devices, and being polite to sometimes impatient customers in a climate where the summer temperatures easily exceed 100ºF. And they are on their feet in a concrete parking deck for much of their work day.
That’s a lot of information to gather in a few minutes by asking three questions. What I found most interesting in the experience, as someone who designs digital user experiences, was the reminder that people around us every day are having problems using the hardware and software we create. It reinforced for me that we can learn a lot about how to approach hardware and software design just by watching people use these products, even products we had no part in creating.
Now, I certainly can’t claim to have found ways to improve the reservation system in question. That would require more field study with additional users as well as talking to the product designers and other stakeholders to understand the decisions that led to the current user interface. But, the experience left me wondering if the company ever bothered to watch its employees use the system in a real-world setting.
My point here is that we can still practice these observational research techniques even if we don’t always get the chance to do so on the products we get paid to design. And we can document these moments to make a case at our companies for why this kind of research needs to be part of our product development cycles. There’s value in these exercises, even if the results don’t immediately show up in our companies’ products. It’s an activity I’d encourage everyone to try.
There are plenty of opportunities for this sort of activity, including:
Retail self-checkout lanes,
Any self-service airport application,
Self-serve vending machines like Redbox or mass-transit ticketing machines,
Hotel check-in/check-out kiosks,
And many others I’m sure I missed.
There are a few things you need to be aware of and take into consideration before attempting to interact with strangers in the middle of a frustrating technology moment:
The people in these situations may be angry and flustered, so look for body language cues that may signal you are better off leaving them alone. If they are hitting the machine and cursing, you may just want to hang back and observe.
Politely approach people and ask if it’s OK to talk to them about their experience with the system in question. In most cases it’s better to wait until they are done because right at the moment they are using it they are probably more focused on completing their task.
Make sure you don’t imply they are doing something wrong. Let them know you are interested in making products better for people and would like to ask them a few quick questions about the experience they just had. If they say “no,” respect that and move along.
Explain that you are not with the company in question and can have no impact on any future version of the product. The last thing you want them to think is that you are a conduit for complaints to the company or that you can actually have any impact on future versions. It’s important to be as clear as possible to avoid setting false expectations.
Be sensitive to your surroundings and other customers who may be waiting to use a system. The last thing you ever want to do is slow down a busy checkout lane or self-serve kiosk at an airport.
Be sensitive to the application at hand and the location and time of day. It’s safe to say you should never approach someone using an ATM or other financial services kiosk. You also shouldn’t approach someone using a self-serve vending machine that is outdoors late at night. That could make them feel threatened or even cause them to react physically.
Honor any request by the business in question to not approach their customers.
Although exercises like this won’t tell us the things we’d like to know about the products we work on, they do let us practice the techniques of contextual inquiry and observation and make us more sensitive to various design issues. These experiences may also help us build the case in more companies for scheduling time and resources for in-field research with our actual customers.
A quote that I stumbled on during grad school stuck with me. From the story of the elder’s box as told by Eber Hampton, it sums up my philosophy of working and teaching:
“How many sides do you see?”
“One,” I said.
He pulled the box towards his chest and turned it so one corner faced me.
“Now how many do you see?”
“Now I see three sides.”
He stepped back and extended the box, one corner towards him and one towards me.
“You and I together can see six sides of this box,” he told me.
—Eber Hampton (2002) The Circle Unfolds, p. 41–42
Creating a Learning Resource with Aboriginal Students
My graduate school thesis project was to create a learning resource for an Aboriginal literature course for Aboriginal students at the University of Alberta. This effort was an interesting challenge since it involved me—a non-Aboriginal designer—trying to design for Aboriginal students from multiple cultural backgrounds.
While navigating the expected cross-cultural design issues, I met some wonderful people and learned a great deal about the importance of letting those with whom you design guide the research and design process.
This daunting task left me more than a bit intimidated at the end of the day. However, I felt from the outset that if I took the time to get to know my design partners and if I took the time to examine how they could guide me, somehow we could be successful.
The resource was to be a website representing Aboriginal literatures from multiple cultures to enrich the learning experience for first-year Aboriginal students. My final project was a prototype design that I then tested in paper form.
The learning resource is based on materials used in English 114: Aboriginal Literature and Culture, a compulsory, full-year Aboriginal Canadian literature course in the Transition Year Program (TYP) at the University of Alberta. The TYP program is for Aboriginal undergraduate students and is designed to prepare them for admission into one of eight faculties at the University. Some students have extensive contact with their Aboriginal communities and heritage; some have little or no contact. Most students are from Canada, but there are students in the program from all over North America.
Extra tutorial support is included in every class, to aid in the transition to post-secondary study. The material used in English 114 comes from Aboriginal authors from diverse communities in North America. Many different forms of literature are studied, including poetry, prose, and oral literatures.
Resource objectives
There were no specific learning outcomes created for the resource. It was simply meant to help students explore and appreciate the literature—particularly oral literatures—through video, audio, images, and text. Both teachers and students felt that commenting on and discussing works on the blog portion of the site was important. Teachers felt that the materials and the blog would help students become more engaged with the material.
“I found I wasn’t comfortable, until the end, talking to the professor because for me, the work I was doing was so personal I was afraid of criticism—to think I didn’t know what I was doing. I was kind of shy.”—Jesse, student
I spent many months completing a literature review. I didn’t find much research on visual design practice for multiple cultures and genres. I interviewed students, all the instructors, the teaching assistant, and many other academics and professionals working in Aboriginal studies. I also audited courses in Aboriginal literature and attended as many classes as possible for the course for which I would design the website.
There were some pretty clear needs identified for the course that the instructors, students, and I felt might be helped by the use of digital media. One of the biggest benefits centered on the importance of oral literatures. Audio and video files would clearly benefit the study of oral literatures when storytellers could not be brought into the classroom. Other benefits were to:
Help students refine independent research skills
Encourage communication between students and between instructors and students by creating a forum
Show media in class using the website
Give students the opportunity to access materials outside of class
Provide background materials to students who are unfamiliar with the material, benefiting students who hesitate to ask questions
Provide a more “low stakes” context for students to explore concepts raised in class
Allow students to seek out information
on their own and explore issues they might be too embarrassed to raise in class
Increase continuity between course sections and allow instructors to share resources
Allow instructors to check in more often on students’ progress
Give reticent students the chance to express opinions in the possibly less-intimidating forum
Use multimedia to demonstrate the “webbed” nature of the texts by allowing students to see how issues intersect
Some pseudonymous student comments about the course:
“I think there wasn’t enough visual stuff, personally. I would like to see the places described in the story.”—Sarah, student
“We did have some overheads, but we couldn’t make out the overheads.” —Trish, student
“ …when the author reads it you are like, wow, is that ever powerful, I’ve got tingles. You can hear their emotion and what they are trying to portray with the literature.”—Christine, student
“Video would be awesome—tone and facial expression are really important.” —Jason, student
Importance of Visual Design
I obviously can’t summarize here all the project complexities or discuss all the wonderful things I learned throughout this process. One aspect of the research, however, reinforced for me the power of collaboration in the design process.
Refining needs and features for the project was challenging, but my most daunting task from the outset was how to create a visual design to meet the needs of the university course, the multiple cultures of the students, and also reflect the literature studied.
Students and instructors also felt the resource had to visually represent traditional and contemporary Aboriginal culture without using imagery that was specific to one culture or geographic region, but they also believed it needed to reference Western literary traditions because many of the authors on the syllabus referenced Western literary traditions in their literature. A daunting task to say the least.
Visual design is not of minor importance to Aboriginal cultures and the visual expression of those cultures.
While progress has been made, by various institutions, to foster understanding of Aboriginal cultures in Canada and the United States, images created by non-Aboriginal and Aboriginal designers found in mainstream media are not nearly as enlightened. Visual communication design creates effective communications that can “affect the knowledge, attitudes and behavior of people” (Frascara, 1997, p. 5). If designers can affect people with the communications they create, then they might be able in some way to prevent visual stereotyping of Aboriginal peoples and cultures.
It is easy to find examples of appropriation and stereotyping of imagery in mainstream media. Logos for the Atlanta Braves, Cleveland Indians, and Chicago Blackhawks are some highly visible examples of blatant stereotyping, but more subtle examples also exist. The popularity of the “New Age” spiritual movement is one such example of cultural appropriation that furthers the misrepresentation of Aboriginal cultures (Hulan & Warley, 1999–2000).
“ …you just can’t have a dreamcatcher a wagon and a chief on every page. There is a lot more to Aboriginal culture.” —Jason, Student
“ …the fact that there’s a million different medicine wheels if you look on the internet, it’s almost like a cliché or something, even though it has real cultural worth.” —Rob, Instructor
The “Aha” Moment
My plan for the visual design process had been to conduct semi-structured interviews with students and instructors and begin collecting ideas for visuals. I had hoped that from there we’d have some ideas to start working together on creating some participatory design sessions.
However, each time I came to discuss the visual tone of the resource and what should be included, it seemed we ended up talking about placement of menus and what types of information were desired for the website. Looking at websites or thinking about them made it very hard to get away from talking about features, functionality, and content. Not that those factors were not critical to shaping the visual design, but I wanted to discuss the visual tone and symbolic references that users felt might be important to include.
My thesis supervisor suggested I try to move away from the digital medium into the print medium to avoid the focus on features. Why not look at some other medium? You’re thinking this is the big “Aha” moment right? Not yet…
My plan was to try a session with five students reviewing book covers from Aboriginal authors and discussing what visual design appealed to them. From my own training and experience as a designer, I selected what I felt was a representative sample of different visual styles: Illustration, photography, typography, and the like.
Throughout my research, a critical theme was the importance of the land and landscape to Aboriginal cultures. I had this information, but I wasn’t really sure what to do with it. Any specific references to a geographic region might exclude some students or authors referenced in the curriculum. Students were from all over North America and had connections to a huge variety of landscape—prairies, mountain ranges, coastal regions, arctic regions, rain forest… This project had to be respectful to the students, teachers, and Aboriginal literatures studied. Imagery was already a very sensitive topic, so omitting any particular region or any one group was far from ideal.
During our sessions, students gravitated to abstract images of landscape that employed natural color palettes. Interestingly, some students selected these images because they felt the images represented both contemporary and traditional cultures. Few if any students selected imagery that had specific symbolism. There was great sensitivity to the use of symbols, partially because they may exclude certain groups but also because there has been a long history of offensive appropriation of Aboriginal symbols. Most students seemed to gravitate to those images that suggested the land in an abstract way.
For example, the book Transitions was selected as both a contemporary and traditional portrayal of Aboriginal culture. One student thought that because the stones were constantly evolving and were part of the land that they were representative of the contemporary and the traditional. Another student mentioned that the industrial feel of the rock landscape suggested the contemporary but that the rocks also suggested traditional elements of the “Grandfather, the stones of the earth.”
Monkey Beach and Transitions: Photography and landscape seemed most appealing to students.
I had already been thinking about using natural imagery, but the problem seemed to me to be the suggestion of one geographic region. Abstract representations of landscape seemed to solve my dilemma. It was suggestive of some type of landscape, but it wasn’t specific. When students suggested a color palette of earth tones, images of landscape, and photography instead of illustration, I had my a-ha moment.
I now had a direction for the visual design that was so simple yet would address a very complex issue. I could create an image that was respectful and addressed what was important to those involved but was universal enough not to exclude anyone.
Putting it all together
In the discipline of design—in sharp contrast to the modernist movement—there is a concern with how to tailor communications specifically to suit audience needs. McCoy (1995) calls this “narrowcasting” instead of “broadcasting.” With this concern comes the understanding that designers cannot make assumptions about cultural groups, even their own (Steiner & Haas, 1996).
I felt so incredibly challenged by this project not only because it meant I had to try to learn how to respectfully design with students and teachers from cultures that I was not familiar with but also because I could not understand how I could possibly design something that could be tailored to so many varied groups. My session with the students helped me realize how my own background led me to subconsciously dismiss the idea of finding some kind of universal solution since it would go against my own postmodern sensibilities.
Those sensibilities closed my mind to the possibility of universals that might be meaningful to the students and also be representative of the wide variety of literatures and cultures represented in the curriculum.
In short, without the different perspectives brought to the project by the students and teachers I was working with, I would probably never have come to the solution that I did. As I have moved forward with my career and my teaching and I see so many changes in the world of design, I think that more and more we may find ourselves in a position where we may have to challenge ourselves to find those universals. My takeaway from all this: try to practice a collaborative approach whenever possible to inform your work and allow yourself to see six sides of the box.
Works Cited
Frascara, J., (1997). User-centred Graphic Design: Mass communications and social change. London: Taylor & Francis.
Hulan, R., & Warley, L. (1999–2000). Cultural literacy, First Nations and the future of Canadian literary studies. Journal of Canadian Studies, 34.3–4, 59–73.
McCoy, K. (1995). Graphic design in a multicultural world. How 10(2), 146–151.
Steiner, H., & Haas, K. (1996). Design for the global village. Applied Arts, 11(3), 46–48 and 50–52.
For centuries, the well-heeled Christian faithful in Europe made pilgrimages to Jerusalem, but most couldn’t afford these expensive and dangerous trips. In the fifteenth century, monks met the demand by setting up shrines along the roads. Together, these shrines told the Passion story, so that the faithful could take the same trip in miniature, at home. In the seventeenth century, these shrines moved inside the churches, becoming the modern Stations of the Cross.
The church had met a design challenge by constructing a narrative in an environment that advanced its messaging goals and met the goals of its audience. As the visitor moves from illustration to illustration depicting the famous trials and tribulations, they are moved both physically and emotionally.
The means of telling stories in spaces have advanced since then, of course. Display, audio, and sensor technology is getting cheaper and more ubiquitous, and many businesses and institutions are investing in integrated media environments to communicate with audiences.
Integrated media environments merge video and motion graphics, environmental and exhibit design, and interactive installations into a physical environment. The visitor moves through the space, taking in the story and the message.
Creating a successful integrated media environment involves design of every stripe (interior, 3D, graphic, UX, UI), as well as software development, hardware engineering, content creation, and curation.
The traditional approach toward applying media of any kind to a physical space has been to start with the space. You mark up a floor plan with a bubble diagram showing the storytelling opportunities, and then figure out what might best fill that space. That alcove could be perfect for an interactive table! That wall is crying out for a projection! Put a reader rail over there!
Instead, consider employing the same methodology that content strategy for the web has been using for years:
Diagram by Swanny Mouton
*Research and discovery*:
* First, understand both the messaging goals of the client and the experience goals of the audience. Understand this audience and their preconceived notions about the messaging.
* Determine the most intriguing content the client has (and what format it’s in), and what’s possible to acquire or produce within the general confines of the budget and timeframe.
*Concept model/planning*:
* Then build a solid conceptual base, with a core metaphor that can inspire both visual and content decisions later.
* Develop the narrative. What do you want the experience to be like? What do we want people to feel at each stage of the experience? What’s the transformation of understanding and mind we want the audience to feel?
*Materials evaluation*:
* At the same time, think about the space, technology opportunities, and other material constraints.
Only then can you pull them all together into a holistic environment that has the power to inspire, inform, and move people.
Of course, rolling out narratives in space continued into the twentieth century, especially in museums. Charles and Ray Eames (shown right) were innovators in this realm, designing and constructing carefully considered exhibits such as “Mathematica”:http://en.wikipedia.org/wiki/Mathematica:_A_World_of_Numbers…_and_Beyond, originally installed in 1961 at the California Museum of Science and Industry. The Eameses were essentially changing the information architecture of museum exhibits, telling a story about the world of numbers and math as a pathway of discovery. So they installed a long timeline wall on one side and a long image wall on the other. These linear, passive experiences flanked a gallery that included many participatory exhibits, including a light-bulb-cube multiplication machine, peep shows, and, famously, a large mobius strip with an arrow that moved on a track. Visitors bounced around in the center space that was anchored by the long walls. In the final exhibit, the thematic message and the visitor experience meshed perfectly, and it remains a seminal example of integrated interactive media.
Most storytelling environments such as these have involved primarily static graphics, passive media, or very simple interactions.
At “Second Story”:http://www.secondstory.com, the interactive media studio where I am a content strategist, designed and built a project for the University of Oregon that serves as a great example of how this kind of strategic thinking and planning can be applied to an integrated media environment with deep interactive content and space design.
The client came to us wanting to create a media gallery in the lobby of the new Ford Alumni Center. This building was to become the front door of the university, welcoming prospective students, alumni, and potential donors.
The final implementation, we knew, would need content, design, and user interfaces that spoke to everyone at once. We discussed the various audiences in depth, to understand their needs:
* College-bound seventeen-year-olds are intoxicated by possibility, without wanting to be overwhelmed. Some know (or think they know) exactly what they want out of their experience at UO, while others expect to explore.
* Returning alumni and potential donors want to be reminded of what makes the UO special, to reminisce about their college years, and to ensure that youth today get similar or better experiences.
Based on this and other research, we devised a visual metaphor that could resonate with the client and their audiences and act as a steady beacon for everyone involved. Taking inspiration from Oregon’s natural environment and the branding of the university, we envisioned the university as the bedrock beneath a river and the students as the water flowing past, subtly making their mark on the institution’s culture. Each student’s path through the river was fluid and determined entirely by the student. This strategic vision helped guide our design choices throughout the project.
The concept of the university as bedrock, with the students and faculty flowing through, was a source of inspiration in developing the concepts.
Our initial content survey showed that we were looking at materials covering the entire university, including:
* The identities of every alumnus in the school’s 140 years.
* Images and archival video going back almost that far.
* Imagery, video, and text from the website of every academic department, every sport in the athletic department, and every student club.
The content requirements for the Oregon Cascades prompted the development of a powerful CMS for the client.
The content volume was overwhelming, and the assets were stored in hundreds of locations online and off. In conversations with the client, we decided that they should manage the content post-installation, allowing it to grow over time and not stagnate. We proposed the development of a powerful content management system with a template-based front end that would allow the content to reflect the university as things changed – the experience would remain consistent while still being fresh.
The final front end for the project’s content management system. The CMS allows the client to keep the experience perpetually fresh.
With the visual metaphor and a vision of the content in hand, we started figuring out the story of the space. This being an alumni center, one of the anchoring elements was the long history of alumni that had come through the university. Another central point in our story was the multitude of available options at the university, which required an experience that allowed visitors to reach out and grab what interested them, at a glance. And, inevitably, we would need to serve announcements and list names of donors.
In looking at the physical space, we realized that there was no possible straight narrative, as people would wander in from multiple directions. Just as we might consider hierarchies on a web page, we thought about how visitors might engage with these stories from far away, closer, and within their interactions. The principle is the same, just extended into three dimensions.
Armed with a powerful story, we sketched the experience into the physical environment.
The placement of the story elements in the space evolved over time, but the core of the experience was consistent.
At this point, we had a holistic understanding of content, messaging, and strategy. We also understood the physical environment, as well as a powerful metaphor that could keep us on track. We designed concepts for three kinds of physical installations, each of which had its own set of communication goals, and took into account visitor pathways through the space:
* The Oregon Entry Wall standing near one of the entrances, presenting donor names flowing across the wall and displaying events and announcements.
* The Oregon Alumni Table showing information about every alumnus from the university’s history, and resembling a river that visitors can dip into.
* Nine Oregon Cascades that stand 14 feet tall, and evoke waterfalls. Six Cascades are digital interactive experiences that allow visitors to swipe through content. These are presented in unique form factors, with four large monitors stacked to the ceiling. Three Cascades are cases, displaying artifacts from the university’s history. The Cascades would be installed on tracks that allow university staff to alter their position.
This was, of course, just the beginning—we spent another year or so on design and development. The final product has a cohesive feel that joins the physical and the digital. We feel strongly that the strategic foundations we laid down early in the process were a crucial factor in the project’s success. You can read more about the project at http://secondstory.com/project/browse/featured-work/university-of-oregon.
In the final media environment, the Oregon Alumni Table is in the foreground, and the floor-to-ceiling Oregon Cascades are in the background.
Content strategists have been leading the charge on how to deliver a consistent message across channels. The advent of smaller, cheaper technology is making it possible for every surface to be a new channel. Sensors such as the Microsoft Kinect depth-sensing camera are removing the need for complex input devices or even touchscreens. The more that physical environments gain digital and interactive dimensions, the more important it is to provide clear, focused storytelling that has been considered carefully. The strategies that have become codified over time for the web and are helping us with the transition to the mobile web also provide us with a powerful framework to design great experiences in physical environments.
_Thanks to Michael Neault and Swanny Mouton for their assistance with this article._
Of course, rolling out narratives in space continued into the twentieth century, especially in museums. Charles and Ray Eames (shown right) were innovators in this realm, designing and constructing carefully considered exhibits such as “Mathematica”:http://en.wikipedia.org/wiki/Mathematica:_A_World_of_Numbers…_and_Beyond, originally installed in 1961 at the California Museum of Science and Industry. The Eameses were essentially changing the information architecture of museum exhibits, telling a story about the world of numbers and math as a pathway of discovery. So they installed a long timeline wall on one side and a long image wall on the other. These linear, passive experiences flanked a gallery that included many participatory exhibits, including a light-bulb-cube multiplication machine, peep shows, and, famously, a large mobius strip with an arrow that moved on a track. Visitors bounced around in the center space that was anchored by the long walls. In the final exhibit, the thematic message and the visitor experience meshed perfectly, and it remains a seminal example of integrated interactive media.
