Observing the User Experience: A Practitioner’s Guide to User Research

Written by: Andrew Hinton

“How do we go about learning who our users are and what they really need? And how do we do this in a way that helps us make a strong case for our design decisions to the people in charge?”

Design is disorienting. Especially when you are designing something in a collaborative environment, with multiple stakeholders, pressured deadlines, business objectives and budgetary constraints. We all go into design with the firm belief that the user is our pole star, but so often we lose that focus because of tossing waves, buffeting winds, and the crew screaming in our ears–never mind the dense cloud cover that always seems to obscure that trusty star just when a committee forms to gather requirements.

With all the attention to usability over the last five years or so and the wonderful swelling of information-architecture-related books just since 2001, you would think we would have enough methods and advice to keep our projects in perfect tack. But so many of these resources, excellent though they are, tend to be more about how to pilot the ship than how to find that all-important star and keep it in sight.

I promise not to drive this metaphor hard into the rocky shore, but think of the projects that could have been saved from being lost at sea if every team had a better grasp of user requirements through direct experience of users and their needs. Think also of how many projects could have stayed the course if only there had been an expert way to sell the findings from that experience to the stakeholders, who so easily forget the users for whom their project was intended.

For precisely these reasons Mike Kuniavsky’s Observing the User Experience: A Practitioner’s Guide to User Research is a welcome addition to the half dozen essential books on my cubicle shelf. This book provides lucid, personable, experienced advice that could only come from a seasoned consultant who has seen the good, bad, and ugly of web and application design. Its purpose is to give a solid foundation to any design team in the crucial beginning stages of a project by answering the questions: How do we go about learning who our users are and what they really need? And how do we do this in a way that helps us make a strong case for our design decisions to the people in charge?

Kuniavsky begins Observing with a cautionary tale about a failed corporate web project, a situation he experienced firsthand (changing identifying information to protect the innocent, of course). This situation involved misguided good intentions from corporate management and developers, where something they thought would be just what their users wanted turned out to be a huge waste of time and money. This is just one of many real-life lessons used as background throughout the book.

Kuniavsky introduces us to web user research methodologies by showing us how they fit into an overall process and by defining various roles within a design team. These descriptions are clear and sensible, and they are more descriptive than prescriptive: he is not trying to tell us these are the roles you have to use and this is what you have to call them in guru-speak, but describing with conventional labels what tends to happen in a successful project.

The chapters are well-organized and consistent, and content is cross-referenced from chapter to chapter where appropriate. Unlike many design-related books, this one is actually fairly heavy on text and light on visuals. Where visuals are used, they are very helpful and serve to explicate the content. A good example is his spiral model of Iterative User Research,which cycles from Examination to Definition to Creation and, as it deepens, gets more granular through Contextual Inquiry, Focus Groups, Usability Tests, and so on. Kuniavsky wisely points out that many companies already have marketing research results that may be expected to yield the results necessary for web design, but explains how the tools used by conventional marketing approaches are only part of the solution for user-centered design. Focus groups and surveys can supply valuable information, but focusing on direct experience of user behavior using a combination of appropriate methods offers a stronger core for design.

Kuniavsky goes on to provide an excellent mixture of step-by-step direction and experienced advice on the practicalities of user research. Beginning with how to put together a research plan (invaluable instruction, since planning seems to be the Achilles heel of so many projects), he explains how to make sure business goals are being considered along with user goals. He admits these instructions present a somewhat idealized situation that starts as a blank slate as far as user experience product goals are concerned. However, Kuniavsky manages to keep his advice from being so lofty that no real-world team could actually follow it.

The chapter on recruiting and interviewing is especially thorough. It provides a sample phone screening script and boilerplate recruiting communication, as well as advice on how to handle no-shows, heavily biased users, and people who do not end up fitting your model. In fact, it may be the coverage of so many aberrations and anomalies that make this book so unusually valuable. This is advice one would normally only gain on the job or working side by side with a highly experienced researcher.

Kuniavsky devotes the bulk of the book to describing a series of proven techniques for researching user needs and behaviors, including user profiles, contextual inquiry (plus task analysis and card sorting), focus groups, usability tests, and surveys, as well as more secondary-research approaches such as diaries, log files, customer support, and competitive research. He presents each method in a separate chapter, describing when each one is most appropriate and various methods of execution. Throughout, Kuniavsky glosses his text with marginal notes, giving a reality check or bit of wisdom in each one, such as the reminder that Focus groups uncover people’s /perceptions/ about their needs and their values. This does not mean that they uncover what people /actually/ need or what really /is/ valuable to them-however-knowing perceptions of needs is as important as knowing the needs themselves.

In his descriptions of various methods, there is surprisingly little dogma. In an industry that has spawned a thousand do’s and don’ts lists for design, it is refreshing to find so many techniques described with equal value and rationale. I personally have long held a bias against focus groups, surveys, and marketing research as being especially valuable for fully understanding users, but this book has helped me see these resources in a more positive light.

It is also a relief to read this book’s conversational and low-jargon voice. There are a number of books I find essential in my work that I still have trouble actually comprehending during a busy workday. Somehow this one cuts through the fog of design-speak to present some very sophisticated concepts and methods in a way so that a relative novice could read it and hit the ground running. Take, for example, his lucid description of the role of information architect: It’s the information architect’s job to make the implicit architecture explicit so that it matches what the users need, expect, and understand. The architect makes it possible for the users to navigate through the information and comprehend what they see. I have never seen my job explained with such clarity anywhere else.

Another strength is Kuniavsky’s business-savvy approach to design. In the very first chapters he does an excellent job explaining the various tensions between different groups with their own agendas encountered in any collaborative design effort. He shows how having solid and documented user research can help to defuse these tensions and keep the user as the central focus of the work. In fact, Kuniavsky even has a chapter on Creating a User-Centered Corporate Culture, an ambitious but necessary topic for any corporation finding its business model being warped into a whole new shape by the powerful gravitational pull of the web.

So much of design involves a kind of tea-leaf reading voodoo that is hard to justify or describe to managers and stakeholders. When we do the typical routine–look at some users, have some conversations, and then come back with all these ideas on how to design an expensive project–aren’t the people paying for it fully justified in asking Why do you think you really know how we should build this thing? And how can one blame them for thinking their own ideas are just as valid as ours? Observing the User Experience provides solid techniques for knowing our users from a 360-degree perspective in a way that we can document, communicate, and even sell to other team members and project owners. Think of it as a combination navigational chart, captain’s log, and sextant for web endeavors–a one-stop shop for tools that help your team stay the user-centered course.

About the book:

  • Observing the User Experience: A Practitioner’s Guide to User Research

  • Mike Kuniavsky
  • Morgan Kaufmann, 2003
  • ISBN 1-55860-923-7
  • List Price: $44.95
  • Chapters:
    • Part I: Why Research is Good and How It Fits Into Product Development
      1. Typhoon: A Fable
      2. Do A Usability Test Now!
      3. Balancing Needs Through Iterative Development
      4. The User Experience
    • Part II: User Experience Research Techniques
      1. The Research Plan
      2. Universal Tools: Recruiting and Interviewing
      3. User Profiles
      4. Contextual Inquiry, Task Analysis, Card Sorting
      5. Focus Groups
      6. Usability Tests
      7. Surveys
      8. Ongoing Relationship
      9. Log Files and Customer Support
      10. Competitive Research
      11. Others’ Hard Work: Published Information and Consultants
      12. Emerging Techniques
    • Part III: Communicating Results
      1. Reports and Presentations
      2. Creating a User-Centered Corporate Culture


Andrew Hinton is a Senior Information Architect at The Vanguard Group in Valley Forge, PA. His personal website is www.memekitchen.com.

The Book of Probes

Written by: Steve MacLaughlin

“When he began to experiment with the probe technique to sharpen awareness, McLuhan stumbled on how one cliché works to probe other clichés, especially media as clichés. This led him to realize that all forms and formal structures are clichés.”

—EricMcLuhan and William Kuhns, “Poetics on the Warpath” (pg. 412)

Combine the probing thoughts of media culture sage Marshall McLuhan with the visual insights of design guru David Carson and the result is the quintessential coffee table book for anyone that works with technology and design. The Book of Probes is an intentional chocolate-in-my-peanut-butter experiment to combine the ideas of McLuhan with the images of Carson in thought provoking ways. For the uninitiated, some introductions may be in order.

Marshall McLuhan, once referred to as the “Oracle of the Electronic Age,” is perhaps best known for his phrase turned into book title, The Medium is the Massage. His massage/message word play and other bold perceptions, which he called probes, still continue to spark debate. McLuhan’s ideas about the “global village” and his division of media into hot and cool categories are just some of the reasons why Wired magazine named him its patron saint.

David Carson, a former professional surfer and sociology professor, is considered one of the most influential graphic designers working today. Newsweek said Carson “changed the public face of graphic design” and one of his books, The End of Print, is the best-selling graphic design book of all time. Carson continues to break the rules and redefine his field through work with Levi’s, Nike, Pepsi, Sony, Coca-Cola, American Express, Lucent, MTV, Ray-Ban and many other clients.

The Book of Probes puts graduate school reading assignments into a blender with graphic design edginess. The first 402 pages of the compact book are an intricate mental and visual playground. The remainder of the book contains some commentary from the book’s editors and a little more insight into McLuhan’s thinking. The editors help to explain how they selected various probes from Marshall McLuhan’s more than 200 speeches, classes, and nearly 700 shorter writings that he published between 1945 and 1980. With such a wide range of selections, some readers might not be able to decide which part is the main course and which section is dessert.

The Book of Probes coverThe electric light is pure information.  It is a medium without a message. In the electric age, we wear all mankind as our skin.Man works when he is partially involved.  He is totally involved when at play or at leisure.When new technologies impose themselves on societies long habituated to older technologies, anxieties of all kinds result.

The art direction and design in The Book of Probes is by David Carson and is a visually entrancing encounter. Admittedly, there are segments of the book that might give some readers headaches or trigger vasovagal syncope in small children. But McLuhan’s ideas and Carson’s images have always been known for their ability to shake things up. You don’t really read The Book of Probes. You experience it. It’s more like drinking a fine scotch than downing some bottled water.

Perhaps the most striking component of the book is the variety of photographic, typographical, and visual manipulations and inspirations Carson has employed to bring McLuhan’s probes to life. The probes themselves have been divided into sections of the book like “Media Ecology” and “New Science of Communication” to bottle the lightening of an immense intellectual thunderstorm. The medium enhances even McLuhan’s messages like, “Color is not so much a visual as a tactile medium,” and “The most human thing about us is our technology.”

The key thing to be aware of about The Book of Probes is that you don’t have to know a thing about Marshall McLuhan or David Carson to enjoy the book. The thoughts and imagery are a good introduction to some big concepts and probing theories. You might also find yourself scratching your head in confusion or just plain shaking it in disbelief. McLuhan’s probes are still just as controversial and puzzling today as they were when he first voiced them. Carson’s design treatments only intensify those sentiments throughout The Book of Probes.

Marshall McLuhan believed that the print revolution begun by Gutenberg was the forerunner of the industrial revolution. David Carson has done a laudable job illustrating some of the biggest ideas about the digital revolution in print. Imagine if Michelangelo could have illustrated Gailileo’s radical ideas or if Vannevar Bush and Pablo Picasso could have worked together. The Book of Probes allows Carson to work posthumously with McLuhan and will hopefully encourage similar collaborations in the future.

The Book of Probes is a book that stands uniquely apart from what takes up space on your local bookstore’s shelves. But perhaps that is a reflection of the incomparable individuals that helped create it. The marriage of these two minds shows promise for future combinations of scientists and designers. Contrary to popular belief, these two disciplines can work well together. The future of the digital revolution could be helped by more visual representations of sometimes complex ideas and subjects. The Book of Probes helps to gets us all thinking and seeing things in a new way, and to keep moving in the right direction.

  • The Book of Probes
  • Marshall McLuhan, David Carson, Edited by Eric McLuhan, William Kuhns, and Mo Cohen
  • Gingko Press, November 2003
  • ISBN: 1584230568 § 576 pages, 410 color illustrations
  • Retail price, $39.95
  • Target Audience: Designers, usability specialists, marketers.
  • Sections:
    1. Overture/Titles
    2. From Cosmic to Existential Man
    3. The Extensions of Man
    4. Advertising
    5. Media Ecology
    6. The Laws of Media
    7. Sense
    8. New Science of Communication
    9. Poetics on the Warpath
    10. The Book of Probes
    11. McLuhan and Saussure
    12. Album: Credits

Steve MacLaughlin is an experienced Interaction Architect who has helped develop award winning sites for a variety of Fortune 500 firms, governmental agencies, and educational institutions. Steve has taught the fundamentals of interactive design at Indiana University’s School of Informatics and MIME Program and the University of North Carolina at Charlotte. MacLaughlin holds a M.S. Degree in Interactive Media from Indiana University. His new weblog, Strathlachlan.com, covers a range of issues and topics.

The Substance of Style: How the Rise of Aesthetic Value Is Remaking Commerce, Culture, and Consciousness

Written by: Steve MacLaughlin
“Not long ago if someone told you that the hardware store would be a trendy place to hang out, you might have looked at them with a skeptical eye. If someone had told you that the top programs on television wouldn’t be sitcoms or dramas, but instead those that feature drab dwellings being refurbished and stylistically challenged people getting makeovers, you probably would have laughed.
Whether you have been paying attention or not we are living in an age of aesthetics. So says Virginia Postrel in her latest book, The Substance of Style: How the Rise of Aesthetic Value Is Remaking Commerce, Culture, and Consciousness. Postrel examines how the role of aesthetics and style are transforming our culture and economy in a variety of ways. In the process we get a better understanding of what this new age means for designers, decision makers, and the customers they hope to reach.

Not long ago if someone told you that the hardware store would be a trendy place to hang out, you might have looked at them with a skeptical eye. If someone had told you that the top programs on television wouldn’t be sitcoms or dramas, but instead those that feature drab dwellings being refurbished and stylistically challenged people getting makeovers, you probably would have laughed.

But that probably meant that you missed the dawn of the age of aesthetics. Today if you walk into a Home Depot on a Saturday morning you will see a lot more than stacks of lumber and rows of lighting fixtures. If you are watching carefully you will see throngs of people participating in workshops on painting, tiling, and building backyard ponds. Walk through a new shopping area and the architecture looks more like an Italian villa than a strip mall in Davenport, Iowa. These are just the kind of strange happenings The Substance of Style explains in vivid detail.

Postrel begins The Substance of Style with a few examples to illustrate how “aesthetics is the way we communicate through the senses.” After all, human beings are visual, tactile, and emotional creatures and we are drawn towards people, places, and things that give us sensory pleasure. Postrel points out that “’form follows emotion’ has supplanted ‘form follows function’.” How else do you explain the success of the iMac, Volkswagen Beetle, and the Michael Graves Toaster at Target?

The Substance of Style goes on to explain how the age of mass production gave way to the age of mass customization. The futurists who predicted we would all be walking around in the same monotone tunics were dead wrong. For most of the 20th Century “the broad public enjoyed the expanding benefits of standardization, convenience, and mass distribution” and “the big story was not the rise of aesthetics but the spread of predictable standards of minimum quality.” This was the “age of Wonder Bread and Holiday Inn” where quality became improved and more widely available, but sadly there was little or no variety for customers. Henry Ford typified the business sentiment of this age when he said, “The customer can have any color he wants so long as it’s black.”

By the late 1970s and early 1980s the gains made in mass production, distribution, and quality reached a critical mass. Virginia Postrel explains how the ability to produce variety and utility was the tipping point for “the beginning of a new economic and cultural movement, in which look and feel matter more than ever.𔄙 The cycle of individually produced items to mass-produced monotony and finally to mass-produced distinctive items was complete. The age of aesthetics had begun and suddenly style began to appear everywhere.

The Substance of Style notes that Starbucks “is to the age of aesthetics what McDonald’s was to the age of convenience or Ford was to the age of mass production.” What prompted millions of people to spend $3.3 billion on a cup of Starbucks’ coffee last year? Postrel’s answer is that companies like Starbucks have used aesthetics to give their customers a unique sensory experience, and their customers can’t get enough of it. That same focus on aesthetics by product designers is now being echoed by everyone from retailers to homebuilders, restaurants, hotels, and nearly every facet of our daily lives.

Aesthetics is no longer the luxury that it once was, and that has allowed people to pick and choose styles that appeal to them as individuals. Advances in technology and product design combined with the mixing of cultures have all allowed for a greater range of aesthetic choices. This has also meant a huge growth in industries that focus on personal aesthetics. The rise in the number of day spas, nail salons, piercing shops, tooth whitening products, and other appearance enhancing services are another indication that we are living in the age of aesthetics.

It is this facet of the age of aesthetics that seems to be drawing the most ire from critics. Postrel points out how “the very power of aesthetics makes its power suspect.” Outlet stores mimicking a Tuscan village are one thing, but many contend “surface and substance cannot coexist, that artifice inevitably detracts from truth.” Pundits allege that people are only left with a shallow, deceptive, and decadent “world of falsehoods.” Postrel confronts this widely held belief to show that style really can have substance.

To begin with, the author believes you have to throw out the absurd notion that aesthetics are meaningless and valueless. For some reason we have come to believe that “appearance must be worth either everything or nothing” instead of accepting that “aesthetic pleasure is an autonomous good, not the highest or the best but one of the many plural, sometimes conflicting, and frequently unconnected to sources of value.” The Substance of Style illustrates this point by noting that “colas are neither good nor evil, and neither is their packaging. The packaging design adds pleasure and meaning, and thus value, to morally neutral products.”

To keep things in perspective Postrel is quick to point out that “form has its own power and worth, but it does not inevitably trump content.” Aesthetics has influence over our decisions but it does not blind “us to all other values.” Instead the value of aesthetics in many cases is its ability to give individuals personalized identity. That sense of aesthetic identity prevails when “I like that merges into I’m like that.” The substance of style consists of its ability to signal identity and that reminds “ourselves and the world of what we think is important.”

The Substance of Style also delves into the broader implications of living in an aesthetic age. For all the choices and options available to customers there are a lot of roadblocks being put up. Postrel asserts, “when ‘design is everywhere, and everywhere is now designed’ whoever determines look and feel controls a great deal of economic and personal value.” Customers begin to “demand better design, and that demand inevitably generates conflict.” The results are limits on what you can and cannot do in a new housing development, what types of architecture are permitted in public spaces, and a reminder that “your ugly house bothers your neighbors; your ugly sofa does not.” The best response to the style police “is what we might call the Italian solution – to look the other way from the stuff we don’t like.”

So what does all of this mean for designers and business decision makers? At the outset of The Substance of Style, Virginia Postrel emphatically states, “Aesthetics has become too important to be left to the aesthetics.” What she means is that people in a variety of professions need to understand the importance of aesthetics to their customers and to do something about it. Customers in today’s style-focused world have issued a challenge to potential corporate suitors: “Give us a way to be smart and pretty, and we’ll take it.”

Postrel quotes Don Norman, the well-known usability expert, and his view that “attractive things work better.“ Believing that smart and pretty can coexist is the first step to focusing on the aesthetic demands of your customers. “By bringing design to new areas or coming up with newly appealing styles, aesthetic innovators can reap rewards,” writes Postrel. The other reality is that paying attention to individual aesthetics is “a requirement to stay in the game.”

The Substance of Style is more than a surface-level synopsis of the importance of style in today’s culture. This is a serious and much needed book about the forces that are shaping today’s culture and economy. Virginia Postrel masterfully explains how the evolution of mass markets helped produce personalized aesthetics for the masses. Instead of ignoring the critics of such a trend, the book faces them head-on to point out just how much substance there is to style. Postrel’s examples are illuminating, her sources are well respected, and The Substance of Style offers a lot more than just a catchy title. Anyone who is serious about surviving in the age of aesthetics needs to read this book. The Substance of Style will not only show you what you might have been missing, but it also gives you some direction on what to do about it.


About the book:

Steve MacLaughlin is an experienced Interaction Architect who has helped develop award winning sites for a variety of Fortune 500 firms, governmental agencies, and educational institutions. Steve has taught the fundamentals of interactive design at Indiana University’s School of Informatics and MIME Program and the University of North Carolina at Charlotte. MacLaughlin holds a M.S. Degree in Interactive Media from Indiana University. His new weblog, Strathlachlan.com, covers a range of issues and topics.

Talking with Virginia Postrel

Written by: Steve MacLaughlin
“Functionality, properly understood, means doing what the end customer needs and wants the product to do. Adding pleasure may be more important than adding performance attributes. ”Virginia Postrel, former editor of Reason magazine, now writes the “Economic Scene” column for The New York Times, and her words have graced the pages of Forbes, Forbes ASAP, The Wall Street Journal, The Boston Globe, the Los Angeles Times, and The Washington Post. Her first book, The Future and Its Enemies, was met with critical acclaim.

Postrel’s new book, The Substance of Style: How the Growth of Aesthetic Value Is Remaking Commerce, Culture, and Consciousness, explores the economic, cultural, social, personal, and political implications of the growing importance of aesthetics in business and society. Boxes and Arrows contributing writer Steve MacLaughlin caught up with Virginia Postrel to get her thoughts on the age of aesthetics and what it means for design professionals.

B&A: The issue of emotion and persuasion has been the source of a lot of discussion in the technology community lately. What would you say to those technology professionals who believe that adding aesthetics to what they do is just selling out or taking a short cut?

Postrel: Aesthetics isn’t a substitute for functionality, but functionality isn’t a substitute for aesthetics either. Not adding aesthetics is taking the short cut, substituting the designer’s idea of what’s important for the customer’s. Functionality, properly understood, means doing what the end customer needs and wants the product to do. Adding pleasure may be more important than adding performance attributes. To a programmer, additional computer speed may be a legitimate improvement while a pretty case isn’t. But to many customers, the case adds value while the speed doesn’t. That’s not because speed is unimportant. It’s because personal computers are already so fast that they can do what most people want them to do.

I bought a Visor Edge a couple of years ago mostly because I thought it was beautiful (though part of that beauty came from the flip up cover, which is also quite practical). Only after I had bought it did I discover just how useful a PDA is. I don’t need much functionality for my purposes—just a contacts database and a calendar–but I want a PDA that makes me happy when I look at it. If I were buying one today, I’d get a Palm Zire, which is simple and beautiful.

B&A: In the book you quote Don Norman, the well-known usability expert, and his view that “attractive things work better.” Norman admits that some think this is a “heretical” view, but he has some research to back up his argument. Are there other voices in the design community that believe that being “smart and pretty” is not only possible, but essential?

Postrel: The success of design firms like IDEO and frog demonstrates the value of “smart and pretty” as a strategy. They pay careful attention to how customers are going to use a product, but they don’t treat that functional study as a substitute for aesthetics. Their products work well, but they also look and feel good.

B&A: What would you say to today’s design students who are still being taught the same old rules? How can they prepare themselves for a design career in the age of aesthetics?

Postrel: It’s as much a matter of mental attitude as it is a question of skills. The skills taught in school provide a basis to build on. But the common idea, for instance, that “designers are problem solvers” doesn’t answer the question of what kinds of problems you’re supposed to solve. “How can I provide pleasure and meaning?” is as legitimate and important a question as “How can I make this work?” or “How can I create a product that is easy to manufacture?” Karim Rashid’s Garbino trash can is wildly successful because it’s an almost perfect design. It’s cheap and easy to manufacture in a variety of colors. It’s highly functional–it has handles built in and is shaped so that trash goes in and out easily. But the first thing most people notice about the Garbino is that it looks and feels great.

Thinking about look and feel can actually improve functionality, because it means you get in the habit of empathizing with the end user. It’s really a matter of thinking, “What will make the customer happy?” rather than “What will make me, the designer, happy?” (In the best case, of course, you’re both happy.) The nightmarish interface design that Alan Cooper excoriates (and that I wrote a popular Forbes ASAP column about) isn’t functional for consumers. But it’s easy for programmers.

B&A: You talk about the importance of building a structure for design and aesthetics to work within a company. How can designers help build a foundation for placing greater value on aesthetics in industries that are not known for their stylishness?

Postrel: This is a tough issue of corporate culture and organization, and it’s a long-term process. The first key to success is to avoid a couple of common mistakes that can make communicating with non-designers difficult.

The first mistake is to justify design’s importance by ignoring its unique contribution. Designers say “We solve problems” and “We can do strategy,” and they forget that everyone else is also solving problems and contributing to strategy. The question is what problems can you uniquely solve? Where’s your value-added? If you try to sell yourselves as another sort of engineer, the engineers will just scoff at you—and rightly so.

Yes, you contribute important functional qualities. They’re absolutely critical. But you also contribute style, and style is valuable in the marketplace, because real people—your customers—care about it. Again, the trick is to identify with the customer and to get your colleagues to see that doing that will make the organization more successful.

The second mistake is to swing in the opposite direction and push the style equivalent of basic research when the marketplace wants style’s equivalent of applied engineering. Among themselves, designers are always looking to push the aesthetic envelope. Professional prestige comes in large part from novelty and innovation, from exploring the frontiers of design. (This is even more true with graphic designers, who face fewer material constraints, than it is with product designers. Interface designers are somewhere in between.)

But the customer isn’t at the frontier. The consumer usually wants to build on what’s familiar—to have something different, but not too different. That doesn’t mean the customer is a rube or a philistine, any more than it means that avant garde designers are useless. Theoretical physics and engine mechanics are different, and both are valuable. So are cutting-edge design and less prestigious, more mundane design. It’s important to remember that “good design” depends on context—good design for whom, for what purpose?

Some of the most difficult websites to navigate, and the most likely to crash, are those created by designers for designers. The Art Center site used to crash my browser every time I visited it.

B&A: You mentioned that interface designers are somewhere between graphic designers and product designers on the aesthetic freedom spectrum because of the nature of their medium. This is a young field that is currently going through some growing pains and a bit of an identity crisis. Any thoughts on what interface designers might become when they grow up?

Postrel: The biggest difference between graphic design and interface design today is not so much the nature of the medium—which is, of course, different—but the relative immaturity of interface design. We haven’t yet seen the coalescing of a “dominant design”—the equivalent to the basic layout of a book page, the arrangement of the steering wheel and controls on a car, the shapes of knives, forks, and spoons, the idea of a suit with jacket and pants (or skirt)—within which designers experiment. The advantage for designers is that there’s a lot of room for creativity and innovation, because user expectations are still developing. The downside is that complete failure is more likely. So is “design” that reflects what’s easy for programmers rather than what really works for the user.

I don’t know what dominant designs will emerge, but interface designers will know the field has grown up when a new design proceeds from basic structures that have proven themselves effective. Designers won’t have to invent the steering wheel.

B&A: In a business environment where everyone is looking for ROI to justify expenses it is usually the marketing or design departments that get cut first. Isn’t this just a short-term band-aid that could be causing more long-term harm to these companies?

Postrel: It may be, though it depends on the particular company’s strategy—Wal-Mart is not Target-—and on what the alternative is. Outsourcing design isn’t the same as not doing design, for instance. And companies that cut marketing and design first may be companies that aren’t using those departments particularly effectively in the first place. They don’t understand the importance of these functions, so they don’t get enough value from them, so cutbacks become justified.

B&A: The book discusses the variety of different groups and stylistic preferences, and how catering to these micro-markets will only become more important in the age of aesthetics. Do you have to be a member of one of these distinct groups to design for them?

Postrel: If you’re designing for a group you belong to—essentially designing for yourself–you have an advantage, because you’ll have tacit knowledge that’s hard to get from the outside. That said, you don’t have to be in a group to design for it. You have to be able to empathize with its members, to understand what they need and where they find pleasure and meaning. That may involve close study and, say, the use of rapid prototyping to get lots of feedback. Or it may be mostly a matter of sympathetic imagination. If you’re successful, of course, the end user may apply your design in ways you absolutely didn’t expect–because you’ve encouraged their imagination.

B&A: A popular phrase in the design world these days is: “Show me the value.” A lot of specialists in the fields of interaction design and user experience design struggle with this issue. What are some ways for them to better express the value they bring to employers or customers?

Postrel: The value of design comes from its ability to provide three things to customers: function, pleasure, and meaning. For interface designers, function is still a frontier. You aren’t designing toilet brush holders, where function is already well understood. But my message to designers of all kinds is not to sell themselves short by only emphasizing function. Designers know ways to create aesthetic pleasure that other specialists, who may understand things about function, simply don’t know. Customers value that pleasure, so they’ll pay more for a pleasurable experience. The trick, of course, is to figure out what *customers* will enjoy and, hence, value, which may or may not be the same things that designers themselves enjoy and value. Designers are often bored with things that still excite customers and want something more cutting-edge. That can make it hard to communicate the value of design to employers. It’s important not to confuse the design equivalent of basic research with the design equivalent of applied engineering. In the short run, value comes from the “applied engineering,” while in the long run new ideas emerge from the “basic research.”

B&A: Throughout the book you talk about the constant backlash from critics who are appalled by the notion of personalized aesthetics that break all of their rules. Are these “experts” just afraid of the inmates running the asylum or do they have some valid concerns?

Postrel: Specialists do know things that the rest of us don’t. They’ve learned patterns that work. This is as true of aesthetics as it is of writing or plumbing. Nine times out of ten, a brochure designed by someone with training in graphic design is going to look better—to the general public, not just to other designers—than a brochure designed by someone with no training. The tenth brochure will either be by an amateur with a great eye or a professional who’s pushing the envelope.

But that’s not the end of the story. Amateurs can learn by doing, and they have access to lots of new sources of aesthetic information, including software that has some of that professional knowledge embedded in it. Over time, I would expect a certain amount of aesthetic training to become part of basic education, just as people learn to write. (Most people can’t write any better than they can design, which is why I have a job, but that doesn’t mean nobody but professionals and gifted amateurs writes.)

Also, there’s a difference between expertise and gatekeeping. Expertise tells you how to achieve what you find aesthetically pleasing. Gatekeeping tells you what you should find aesthetically pleasing. It’s the gatekeepers who are upset—people who want to dictate the one true style, whether they’re arbiters of fashions in clothing or in architecture.

B&A: You tell the tale of a Chicago restaurant that ignored the importance of aesthetics while their competitors really focused on creating a unique dining environment for customers. The restaurant eventually goes under and the stylish restaurants thrive. Does this create an aesthetics arms race where if you are not stylish you risk losing everything, but if you are stylish the best you can hope for is to simply keep up with the competition?

Postrel: In some highly competitive businesses, such as restaurants, that is a likely outcome. The gains will go to the consumers, not to the producers’ bottom lines. But that’s not the whole story, because aesthetic leaders often command a premium, because they make smart, though sometimes costly, investments and manage to stay ahead of the competition. There’s an arms race in microprocessor manufacturing, but Intel is still profitable. There’s an arms race in retailing logistics, but Wal-Mart is still ahead of its competition. An “arms race” in aesthetics simply means that style has become another important part of operations, subject to the same competitive pressures as the rest of the business.

B&A: Have the ebbs and flows of the economy in recent years done anything to stem the tide of the aesthetic age? Is the demand for more aesthetics waterproofed from a struggling economy?

Postrel: The best answer to this question is the first paragraph of the last chapter:

“The aesthetic imperative is here to stay. The indicators may fluctuate with the economy—fewer new companies that need logos, advertising, and websites mean less work for graphic designers; fewer new products mean less work for industrial designers; fewer new hotels or restaurants mean less work for interior designers—but the underlying phenomenon remains strong. Every startup, product, or public space calls for an aesthetic touch. Personal appearance demands new forms of attention and offers new sources of pleasure and meaning. Aesthetic proliferation gives us more choices, opportunities, and responsibility than ever before. We expect look and feel to express, and to help establish, the identity of people, places, and things. What once was good enough isn’t any longer. Function alone does not suffice.”

Another way of putting it is that the total demand for aesthetics varies with the business cycle, as does the demand for other goods. But the relative importance of aesthetics—its proportion compared to other goods—has increased.

B&A: How long might the age of aesthetics last? Do we one day risk becoming desensitized to all this sensory overload? What happens after the age of aesthetics?

Postrel: It’s a long-term trend but how long, I don’t know. That depends to some extent on what other sources of new value-added emerge. Here’s what I wrote in the book, and it’s as far as I’m willing to push the prediction:

“The innovations that today seem exciting, disturbing, or both will eventually become the background of our lives. We won’t notice them unless they’re missing. Like convenience or hygiene, instant communication or rapid transportation, look and feel will simply be part of modern, civilized life. We’ll assume they were always there, like indoor plumbing or recorded music—that we couldn’t possibly have lived without their pervasive presence.

New styles and new aesthetic technologies will continue to develop, of course, and old ones to evolve or improve, but at some point aesthetics will no longer be the frontier. When we decide how next to spend our time, money, or creative effort, something else will top our priorities. Something else will disturb the familiar ways of business and culture. Something else will challenge our conventional notions of ‘real’ value. That something else may be radically new, the product of currently far-fetched technologies. It may be a major improvement in an existing good—a faster, cleaner form of transportation, an instantaneous mode of manufacturing. It may be as ancient as storytelling or exploration.”

We’ve already become desensitized in one sense: We take a lot of aesthetic quality for granted, even though it’s only a few years old. But in another sense, we’re becoming more keenly sensitized, able to notice more subtle changes or differences while also playing with new combinations.


Steve MacLaughlin is an experienced Interaction Architect who has helped develop award winning sites for a variety of Fortune 500 firms, governmental agencies, and educational institutions. Steve has taught the fundamentals of interactive design at Indiana University’s School of Informatics and MIME Program and the University of North Carolina at Charlotte. MacLaughlin holds a M.S. Degree in Interactive Media from Indiana University. His new weblog, Strathlachlan.com, covers a range of issues and topics.

Report Review: Nielsen/Norman Group’s Usability Return on Investment

Written by: Peter Merholz
“The key strategy is to get businesses to recognize that user experience is not simply a cost of doing business, but an investment–that with appropriate expenditure, you can expect a financial return.”In the business world, user experience endeavors are typically seen as a cost—a line item expense to be minimized to the greatest extent possible while still remaining competitive. User experience practitioners are, in part, to blame for this. We’ve been so focused on developing methods, processes, and solutions that we haven’t bothered to help businesses measure, and thereby understand, our financial worth.

We thought our worth was self-evident. Of course you’ll sell more products if they’re more usable! Or you’ll decrease costs because of heightened productivity! Exactly how much will you profit? I don’t know, but don’t you want to build the best product you can anyway?

As our field has matured, and as the economy’s continued belt-tightening means, striking out line items associated with costs, we’re realizing we need to prove our economic value. For consultants and agencies, this proof is necessary to sell services. Inside the corporation, employees have to show their contribution for fear of being let go. And all around there exists a strong push to increase our stake in the game, utilize our experience and methods not simply to make a product better, but to determine what to make in the first place.

The key strategy is to get businesses to recognize that user experience is not simply a cost of doing business, but an investment–that with appropriate expenditure, you can expect a financial return. Proving a return can be remarkably hard–tying user experience metrics (e.g., reduced error rates, increased success rates) to key financial metrics (e.g., increased sales, improved retention) requires access to data most of us simply don’t have. So, we look to others to help make our case, if not specifically for us, for our industry.

Nielsen Norman Group’s Usability Return on Investment
This has led to a number of essays, articles, and books on proving the value of user experience. Into this fray steps the Nielsen Norman Group’s (NN/g) famously quoted in New Scientist for saying:

“Why do we have so many unusable things when we know how to make them usable? I think it has to do with the fact that the usability advocates don’t understand business. Until they understand it and how products get made, we will have little progress.”

Unfortunately, the NN/g report does not seem to follow this advice. Although it does make a reasonable anecdotal case for investing in usability, the report methodology is so fundamentally flawed that any financial analyst worth her salt would immediately question its findings. Very simply, the authors do not make a strong business case for usability—a requirement for passing the muster with the accountants and senior managers who have ultimate accountability for profit and loss in a business.

The report is split into three sections: Cost of Usability; Benefits of Usability; and Case Studies. In “Cost of Usability,” the authors report on a survey conducted with attendees of the Nielsen Norman Group User Experience World Tour where they found that the “best practice” for the usability portion of a web design budget is 10 percent. In the heart of the report, “Benefits of Usability,” usability metrics are divided into four classes (Sales/conversion rate, Traffic/visitor count, User Performance, and Feature Use), and analysis of the case studies reveals an average improvement of 135 percent in those metric classes. The remaining 80 or so pages are devoted to the 35 Case Studies, showing before –and after states of key metrics and how usability methods helped achieve improvements.

Questionable sampling
Alert readers of this review were no doubt scratching their heads at the phrase in the last paragraph: “… survey conducted with attendees.” … Perhaps the gravest sin of this report is the extremely questionable sampling that went into both the measuring of costs and the findings of benefits. The authors acknowledge this when measuring costs:

Thus, there is an inherent selection bias that has excluded companies that do not care much for usability, because such companies would not be likely to invest the conference fee and the time for their staff to attend. This selection bias is acceptable for the purposes of the present analysis, which aims at estimating usability budgets for companies that do have a commitment to usability.

By couching this in “best practices,” all that matters is that the reader understand trends within companies whose efforts could qualify as “best practices.” And a key identifier of such a company is attending the Nielsen Norman Group User Experience World Tour. Um, okay.

For the case studies demonstrating benefits, it’s worth quoting the methodology for their collection:

Some of the case studies were collected from the literature or our personal contacts, but most came from two calls for case studies that were posted on Jakob Nielsen’s website, useit.com, in 2001 and 2002. Considering how widely the call was read, it is remarkable how relatively few metrics we were able to collect. Apparently, the vast majority of projects either don’t collect usability metrics at all or are unwilling to share them with the public, even when promised anonymity.

Simply posting a call suggests a remarkable laziness, considering what this report is trying to accomplish. No one can be expected to voluntarily submit a failing case study, so of course the findings show nothing but positive improvements from usability. In order to truly understand the benefits of usability, it’s necessary for the researchers to actually perform a little legwork in finding a range of activity. Fact is, the usability design community can learn as much from its mistakes as from its success—an analysis of cases where usability improvements did not necessarily contribute to financial success, or better, an acknowledgement of cases where the financial success was difficult to attribute, would have provided an equally valuable (and perhaps more credible) report for real practitioners in the field. While the individual case studies are basically valid, this sampling approach renders any aggregate findings and observed trends meaningless.

Case studies
About 75 percent of this report (83 of the 111 pages) addresses the 35 self-selected case studies individually. For each case you are told what NN/g identified as the important metrics to be measured before and after the usability project, and are given some background, the problem that was faced, the solutions arrived at (illustrated with before –and after screenshots), and the ROI impact.

One revealing detail is how the report refers to the improvement of a metric as an “ROI measurement,” yet never discusses what the new solution costs to develop. Yes, sales might have improved by 100 percent, but without understanding the costs needed to realize that improvement, you cannot actually state a “return.”

A number of cases are quite solid—readers will most likely find it quite clear that usable design methods had a direct impact on the key financial metrics for Performance Bikes, Broadmoor, eBags, macys.com, Junior’s Restaurant, and Deerfield.com, and perhaps a few others. However, for the bulk of cases, the link the authors make between usability and financial returns was questionable or even non-existent. Here are a few examples:

No accounting for cannibalization.
In ADC’s case study, sales increased dramatically, but the case infers that these purchases would have otherwise been made on the phone. To understand actual impact, you would need to tease out the percentage of sales actually created online from the percentage that was captured from other more expensive channels. It would also be nice to have some estimate of cost savings that placing sales online makes possible.

Not enough detail was provided in the case study.
For the Anonymous Electric Company, it is not clear why the improved customer survey is important to the company or to the customer. It appears to have something to do with energy conservation, but the case study does not provide enough detail, and as a result, no returns data can be attributed to energy conservation (which is arguably a financial return for the company, the customer, and society as a whole).

“The fundamental question when considering this report, and the driving reason for this review, is “What, exactly, am I getting for my $122?”No accounting for other mitigating factors.
In the case of opentable.com, the company was in the process of going national at the time of their re-launch. The “number of reservations made” metric does not screen for natural expansion, which is why financial analysts evaluate retail chains, such as Gap, based on “same store sales” rather than “total sales.” By the same logic, a better metric for opentable.com would be “reservations per restaurant.”

Dynamic Graphics totally changed their brand and product offering at the same time as the UX re-launch. Similarly, Omni Hotels vastly changed their visual design. The NN/g report awards “usability” as the sole contribution to these improved metrics, though other factors undoubtedly had an impact.

Vesey’s Seeds’ previous site was plagued by technical problems like slow or unsuccessful page downloads. How much of their metrics improvement was simply from technical improvements?

No clear link to financial returns.
Despite being a government agency, the Ministry of Finance, Israel, must have some idea of the monetary benefits of having a usable website (reduced phone calls, etc.). The case study makes no attempt to link changes in user behavior to return on investment and simply reports a traffic analysis.

Poor baseline data.
Any case study showing infinite improvement is an example of poor baseline data. You cannot ascribe infinite improvement just because the feature did not exist or the data was not collected before the design change.

What you get for $122
The fundamental question when considering this report, and the driving reason for this review, is “What, exactly, am I getting for my $122? (Or $248 for the site license) What can I do with this report?”

This report seems to be directed at usability practitioners, to support their efforts in increasing their budgets. Presumably, usability practitioners will, in turn, show this to management. They will tell management that current “best practice” is to devote 10 percent of a project’s budget to usability efforts. They will also tell management that, “on average,” usability provides measurable improvements of around 135 percent.

Unfortunately, unless management simply focuses on the executive summary and doesn’t actually read the report, this approach may backfire for practitioners. It is likely that a manager with any real or intuitive sense of hypothesis testing, financial benchmarking, and calculating ROI will be skeptical of the report’s validity because of the weak methodology, specious accounting, and sampling bias issues already discussed.

To its credit, some truly valuable takeaways from the report are the usability metrics–both the four classes (Sales, Traffic, User Performance, and Feature Use), and the specific metrics utilized in the individual cases. These metrics are a great starting point for practitioners to begin capturing baseline data and developing hypotheses for how these metrics are linked to financial performance. After doing this leg work, practitioners can begin the task of demonstrating the economic value of usability investments. However, these metrics are only a starting point—the report hints at linkages between usability metrics and financial returns without providing any real detailed analysis of how this was done in the individual cases or offering any guidelines for addressing this challenge at your business.

I hear some folks wonder, “But what about the 83 pages of case studies? There must be good stuff in there!” Sadly, this is not the case. The bulk of this report is simply not useful, because the cases are too wedded to particular contexts. The focus of each case study is the improvement made, which is utterly meaningless to the reader. So what if, as in the case of Deerfield, the team “[r]emoved the breadcrumb from the first page in the site, where it served no practical function,” or “[a]dded support information to the homepage.” Yes, it’s interesting that through usability methodology, they increased product downloads by 134 percent. But it’s not really interesting how they did it, unless the report authors think that you, too, can improve your metrics by doing what they did. Nor is it interesting to see screenshots demonstrating this.

The case studies’ primary function seems to pad the report to 111 pages, which is much more likely to warrant a $122 payment than, say, 40 pages.

You can get more with less
The intended audience for this report will be better served by Aaron Marcus’ “Return on Investment for Usable User-Centered Design: Examples and Statistics” [PDF], an essay that combines both literature review and some cogent, simple analysis. And, as that direct link suggests, it’s free.

The essay directly refers to 42 articles addressing different aspects of the financial impact of usability. If nothing else, it would serve as a valuable bibliography on this topic. To his credit, Aaron goes further, breaking down the metrics into three classes, each with subclasses:

Development:
Reduce Costs
Sales:
Increase Revenue
Use:
Improve Effectiveness
Save development costs Increase transactions/purchases Increase success rate
Save development time Increase product sales Reduce user error
Reduce maintenance costs Increase traffic Increase productivity
Save redesign costs Retain customers Increase user satisfaction
Attract more customers Increase job satisfaction
Increase market share Increase ease of use
Increase ease of learning
Increase trust in systems
Decrease support costs
Reduce training costs

This framework helps make sense of the metrics miasma, and readers can begin to understand which metrics they can effect and how to interpret that value.

Where to go from here?
While there have been efforts to underscore the value of usability, the state of doing so is immature. Forthwith are suggestions that combine learnings from the Nielsen Norman Group report and Marcus’ essay, along with some observations made in working with Adaptive Path clients to better ascribe financial results to user experience design.

Create a cross-functional team. Academic and professional literature on product development and design has shown again and again that cross-functional teams improve the design process. Even if it’s an informal ad hoc committee, the insights of marketers, accountants, and senior managers can really help designers attach user needs and usability interventions to business goals and financial metrics.

Example: Usability Design Managers alone may not have access to (or even be aware of) important marketing and financial data that can help them to better measure the impacts of their work.

Collect good baseline data. Meaningful evaluations of design improvements are best shown by before/after snapshots of site performance. However, not all performance metrics are strong. By decomposing aggregate data (e.g., sales) into meaningful components closely linked to usability (e.g., conversion, sales per page view), designers can gain clearer understanding of the before and after snapshots.

Example: Total sales is most likely not a meaningful measure of usability improvements to an online shopping cart because many other factors influence total sales—a better metric might be reduction in abandoned carts or a reduction in errors. Contribution to sales (rather than total sales) can then be more realistically calculated from improvements in these usability metrics.

Isolate the expected impacts of usability improvements. Often usability improvements are accompanied by larger strategic changes in the brand position, marketing, and site technology. It is best to attempt to isolate usability improvements from these other changes.

Example: A design improvement that occurs during a time of natural expansion for the business will require more finesse to accurately measure the contribution that user experience design made to that growth. For instance, increased total transactions loses its meaning if the number of products or vendors has also greatly increased—a measure such as transactions per product or per vendor will help tease out design improvements that are independent of natural growth.

Use hypothesis testing. Similarly, the linkage between design performance improvements and financial returns may not occur as expected. It can be helpful to brainstorm a list of possible metrics and returns, and analyze each of these individually to determine which best captures the improvements made through user experience design. Of particular interest are what we call “indicator” metrics, whose movement is correlated to more direct financial metrics.

Example: In the Nielsen Norman Group report, the Deerfield.com team figured out that they could directly impact the number of product downloads. They also knew that, separately, product downloads tracked to product sales. So by increasing downloads, they could increase sales.

Make user experience people specifically accountable. Too often, the people performing web design are not held accountable. Their endeavors are seen simply as a cost of doing business. We’ve seen talented user experience people used as a kind of free internal consulting, spinning wheels on half-baked projects because their efforts are not believed to have truly remunerative value. User experience workers must seek accountability for the metrics we’ve been discussing.

Example: Don’t tie an entire team to the responsibility of a single aggregate metric (such as sales). This will only engender frustration because employees will feel as if their individual contribution is futile toward this grand larger goal. Make specific groups or individuals responsible for metrics over which they have direct influence, perhaps beginning with some of the metrics from Aaron Marcus’ paper. This will be cumbersome at first, but will prove immensely valuable once underway.

Celebrate success and revisit the process. To institutionalize lessons learned in any design process, it helps to share successes with members of the cross-functional team and within the business as a whole.

Example: Many firms post internal white papers to the corporate intranet to share success and recognize valuable contributions to the business. This is also a great way to maintain an “institutional memory” of projects that have succeeded as the champions of the project move on to different endeavors.

Where this can lead
User experience practitioners have long known in their guts that their efforts truly add value when developing products or systems. And we’ve been long frustrated to see our abilities relegated to the tactical end of the product development process, where we’re given poorly considered ideas and told to make them into useful, usable, and desirable products. By concretely demonstrating our impact on the success of our works, we will find ourselves involved earlier in the process, helping determine what will be made, not just how to make it.


Peter Merholz is a founding partner of Adaptive Path, which provides user experience leadership for all manner of organizations. He is an experienced information architect, writer, speaker, and leader in the field of user-experience design. Clients include PeopleSoft, Cathay Pacific, and Intuit, and he’s spoken at the ASIS IA Summits, SXSW, and DUX2003 conferences. When he ought to be working, he’s writing on his personal site, http://peterme.com/.

Scott Hirsch
A recent graduate of Berkeley’s Haas School of Business, Scott Hirsch is passionate about web design, product development processes, and creative uses for technology. Using his MBA powers for good instead of evil, his goal is to connect user experience design efforts with financial returns through analysis of business strategy and managerial accounting techniques. He is currently working on projects in San Francisco with Adaptive Path, WellsFargo.com, and the Haas School of Business. He has also presented on business analysis at the DUX 2003 conference.