This is an excerpt from “UX Storytellers”:http://uxstorytellers.blogspot.com. If you enjoy it, consider getting the kindle edition of UX Storytellers – Connecting the Dots with all the stories!

Here’s something I believe in: stories are what make us human. Opposable thumbs? Other animals have those. Ability to use tools? Ditto. Even language is not exclusive to human beings.

From my amateur reading of science, the story behind our stories goes something like this: the human brain evolved with an uncanny knack to recognize and create patterns; and through some strange twist of natural selection, gradually over millions of years, our brains started turning the incredible power of that pattern-making machinery on ourselves, until we became self-aware.

Aware of ourselves—our own faces, bodies, journeys, homes, children, tools, and everything else around us. Over eons, we went from being creatures that lived in each moment as it came and went, to protagonists in our own myths. Everything in our midst became the material for making stories, strands of moments woven into tapestries that we call things like “nation”, “family,” “love” or “discovery.”

And “design.” Because design is, ultimately, a story we make. And designing is an act of weaving a new story into an existing fabric in such a way that it makes it stronger, better, or at least more interesting, and hopefully more delightful.

An Origin Story

My identity as an information architect happened accidentally, and gradually. I just kept doing things I liked, that people were willing to pay me for, until I woke up one day and realized I had a career. And the things I liked doing were invariably surrounded by people’s stories.

One of the earliest jobs I had out of college (after trying my hand at carpet cleaning, waiting tables and telemarketing) was as an office manager in a medical office. It was 1990, and this office of five or six providers was running entirely on a phone, a copier and an electric typewriter. No computer in sight. Every bill, insurance claim, or patient report had to be typed anew … as if the 80s had never happened. I talked the owner into getting a computer and a database management package—a sort of Erector set for database application design that I’d seen at a Mac user group a year before—so I could make the office more efficient.

It would’ve been pretty easy to create a quick application with a minimal user interface, if I were the only one using it. But the owner also had a couple of people helping in the office part-time who needed to use the system too—people who had never even used a computer before. Did I mention this was 1990?

So I had a challenge: how to make it work so that total computer newbies could use it? It was frustrating, fascinating, and probably the single most important experience of my career, because it was a crucible for acknowledging the importance of understanding the user.

To understand the people who were to use the application, I had to talk to them, get a sense of what they’d done before and what sort of forms they had used in the past. What sorts of technology? What terminology was going to make sense for them? How do they tend to learn—by written instruction or hands-on activity, by rote or through improvisation? I learned these things by watching and conversing. Eventually I had enough of a sense of those “users” that I had a full story in my head about how they came to the experience of this particular application, in this particular place.

I wasn’t conscious of this at the time; and I was working completely by intuition. I would’ve done a better job if I’d had the experience, methods and tools I’ve picked up since. But looking back, the experience itself has become a story I tell myself when I need a rudder to remind me of my direction as a designer so that, even when I have nothing else to go on, if I just watch, listen and absorb the stories of the people for whom I’m designing, my design will generally head in the right direction.

An Architecture Story

Much later, about ten years ago, I was working at a web design agency, and our client was an organization that acted as a sort of confederation of research scientists, universities and technology corporations. The organization funneled research money from “investor” companies to the scientists and their students in the universities, and in return the companies received “pre-competitive” research and dibs on some of the brightest graduates.

Their website had evolved like so many in those days—having started from a few linked documents, it had grown by the addition of ad-hoc sections and content created in response to member requests and complaints, until it had become a horribly unwieldy mass of links and text. We had been called in to clean it up and organize it. That sounded straightforward enough. But when we started interviewing its users, we found people who were unhappy with the organization and its community in general—scientists who had become more entrenched in their own sub-disciplines, and divisions between those managing the community and those merely dwelling there. Not to mention the natural enmity between academics and business leaders.

We realized that the web site had become a visible instantiation of that discord: a messy tangle of priorities in tension. A new information architecture would mean more than just making things more “findable.” It meant trying to make a digital place that structurally encouraged mutual understanding. In essence, a more hospitable online home for people with different backgrounds, priorities and personalities. It was a chance to create a system of linked contexts—an information architecture—that could help to heal a professional community, and in turn strengthen the organization founded to support it.

That project provided an insight that has forever shaped how I understand the practice of information architecture: the web isn’t just a collection of linked content, it’s a habitat. And the structures of habitable digital places have to be informed by the stories of their inhabitants.

A Survival Story

Much more recently, I had the opportunity to work with a non-profit organization whose mission was to educate people about breast cancer, as well as provide an online forum for them to share and learn from one another. When interviewing the site’s users, it soon became clear how important these people’s stories were to them. They would tell the tale of their cancer, or the cancer of a loved one, and in each case the story was one of interrupted expectation—a major change of direction in what they assumed to be the storyline of their lives.
I learned that this website was merely one thread in a great swath of fabric that the site would never, ultimately, touch. But the site was most valuable to these people when it supported the other threads, buttressed them, added texture where it was needed, especially when it helped fill in the gaps of their stories: How did I get cancer? What do my test results mean? What treatment should I choose? What can I eat when getting chemo? How do I tell my children?

They wanted information, certainly. Articles full of facts and helpful explanations. And the site did that very well by translating medical research and jargon into information people could use. But even more than the packaged articles of information, so many people wanted—needed—to share their stories with others, and find other stories that mirrored their own. The most valuable learning these people discovered tended to be what they found in the forum conversations, because it wasn’t merely clinical, sterile fact, but knowledge emerging organically from the personal stories, rich in context, written by other people like them.

One woman in particular lived on an island in the Caribbean, and had to fly to the mainland for treatment. There were no support groups around her home, and few friends or family. But she found a community on this website; one that would cheer her on when she was going to be away for tests, console her or help her research alternatives if the news was bad, and celebrate when news was good. She made a couple of very close friends through the site, and carried on relationships with them even after her cancer had been beaten into submission.

Here were stories that had taken hard detours, but had found each other in the wilderness and had become intertwined, strengthening one another on the new, unexpected journey.

This work, more than any other I’d done before, taught me that stories aren’t merely an extra layer we add to binary logic and raw data. In fact, it’s reversed—the stories are the foundations of our lives, and the data, the information, is the artificial abstraction. Information is just the dusty mirror we use to reflect upon ourselves, merely a tool for self-awareness.

It was through listening to the whole stories as they were told by these digital inhabitants that I learned about their needs, behaviors and goals. A survey might have given me hard data I could’ve turned into pie charts and histograms, but it would’ve been out of context, no matter how authoritative in a board room.

And it was in hearing their stories that I recognized, no matter how great my work or the work of our design team might be, we would only be bit players in these people’s lives. Each of them happens to be the protagonist in their own drama, with its own soundtrack, scenery, rising and falling action, rhyme and rhythm. What we made had to fit the contours of their lives, their emotional states, and their conversations with doctors and loved ones.

The Moral of the Story

Design has to be humble and respectful to the presence of the user’s story, because it’s the only one that person has. Stories can’t be broken down into logical parts and reconstituted without losing precious context and background. Even though breaking the story down into parts is often necessary for technological design, the story lives only if we serve as witness to the whole person, with a memory of his or her story as it came from that person’s mouth, in that person’s actions.

Keeping the story alive keeps the whole idea of the person alive. Whether we use “personas” or “scenarios” or task analysis or systems thinking, the ultimate aim should be to listen to, understand and remember the stories, precisely because the stories are the beating heart of why we’re designing anything at all.

So, now, when I’m working on more mundane projects that don’t touch people in quite the same way as some of the others I’ve done, I still try to remember that even for the most everyday task, something I design still has to take into account the experience of the whole person using the product of my work. That, after all, is what we should mean when we say “user experience”—that we seek first to listen to, observe and understand the experience of the people for whom we design. We honor them in what we make, when we honor their stories.

Show Time: 33 minutes 42 seconds

Download mp3 (audio only)
Download m4a (with visuals, requires iTunes, Quicktime, or similar)

iTunes     Del.icio.us     Boxes and Arrows Podcast theme music generously provided by Bumper Tunes

Ireland’s Gerry McGovern shares a few of the key ideas in his recent publication The Stranger’s Long Neck – How to Deliver What Your Customers Really Want. Mr. McGovern, who will be teaching a Masterclass series in Canada on the importance of task management this November, discusses several of the key findings in his new book, including:

Trading with strangers

- The customer is a stranger. On the Web, the customer isn’t king—they’re dictator. When they come to your website, they have a small set of tasks (long neck) that really matter to them. If they can’t complete these top tasks quickly, they leave.
- There is an existential challenge going on right now between organization-centric and customer-centric thinking. Customer-centric thinking is winning.

From Long Tail to Dead Zone

- The Long Tail theory says that the Web allows you to sell more of less, that we are seeing the decline of the blockbuster and the rise of the niche.
- The Long Tail is often a Dead Zone of extremely low demand and hard to find, poor quality products.

The rise of the Long Neck

- The Web is exploding with quantity but quality is still relatively finite. Quality is the ‘long neck’; the small set of stuff that really matters to the customer.
- Understanding and managing the long neck has never been more important.
- Remember that the customer’s long neck—what really matters to the customer—is rarely the organization’s long neck —what really matters to the organization.

A secret method for understanding your customers

- A unique voting method that identifies your customers’ long neck.
- Developed over 10 years, with over 50,000 customers voting in multiple languages and countries.
- Used by the BBC, Tetra Pak, IKEA, Schlumberger, Wells Fargo, Microsoft, Cisco, OECD, Vanguard, Rolls-Royce, US Internal Revenue Service, etc.

Organization thinking versus customer thinking

- Case study that shows how car company managers think differently about how customers buy cars to how customers themselves think.
- Explanation of how to frame the task identification question.

Deliver what customers want—not what you want

- Case study of Microsoft Pinpoint, a website to help businesses find approved Microsoft IT vendors and consultants.
- What’s the top task of US small and medium businesses when it comes to IT? Security.

Measuring success: Back to basics

- Why traditional web metrics such as page views, number of visitors, etc., are often misleading
- Observation-based technique to measure online behaviour.
- The key metrics of task measurement: completion rate, disaster rate, completion time

Carrying out a task measurement

- The benefits of remote measurement
- How to run an actual measurement session

This podcast has been sponsored by:

Publishers of world class content for students, researchers, and practitioners in the UX and HCI fields. To learn more visit http://www.mkp.com/hci

From concepts to rich prototypes and detailed specifications, all in one tool. Get your free 30-day trial at www.axure.com


Boxes & Arrows: Since 2001, Boxes & Arrows has been a peer-written journal promoting contributors who want to provoke thinking, push limits, and teach a few things along the way.

Contribute as an editor or author, and get your ideas out there. boxesandarrows.com/about/participate

This article will also outline what to expect from researchers and some ways to recognize when you’re working with a good one. These are indicators, not standards, based on what we’ve found to be effective. There are many ways to do research and every research study is different so it doesn’t mean that a researcher is incompetent if he or she doesn’t conform to these indicators. One sign of a strong researcher is that he or she will educate you throughout the process so that you know what to expect. With that in mind this article is ultimately intended to provide a useful starting point.

Recruiting

One of the most critical and time-consuming elements of test preparation is defining the right target audience and recruiting participants. Participant recruiting is usually conducted by professional recruiters who typically consult databases of potential participants. Sometimes researchers will do the recruiting themselves, but it’s usually more cost effective to use a specialist.

Recruiting will almost always take two weeks or more depending on the number of participants and the type of research, so make sure that you get started early enough for the recruiter to have enough time to find the appropriate participants for the study. Recruiting for phone interviews may take slightly less time and any kind of home visit will likely take longer (ethnography or contextual interview). Your researcher should be able to provide you with an estimate at the time of initial engagement.

A week for recruiting tends to be difficult and any less than that is pretty much unthinkable. Short-changing the recruiting could result in participants that don’t properly fit the target market segment, don’t provide quality feedback, or just don’t show up at all. All of these can have a negative impact on the data. Even if it is possible to get participants faster, it’s usually better to take the time to ensure that you are getting the right people. Your researcher should know all of this and recruiting participants is where he or she will start after getting a basic understanding of your product and schedule.

A recruiter will need a screener to get started. A screener is a description of the target user with open and close-ended questions about the participant that will help the recruiter to select the right people. What you can do to smooth the process along is to have a prepared concept of your target user. This does not need to be a full market research report—just an outline of the types of users that will use your product.

Your researcher should dig deep with questions that include more than demographic information by asking behavioral questions. Behavioral questions can include such topics as TV watching behavior, purchasing behavior, internet use, etc. Typically behavioral questions will give you a stronger understanding of those who are being recruited than demographics alone. These are important elements of market segmentation that are sometimes organized into profiles called personas.

Personas are useful because they create a consistent concept of the intended market segment that can guide the design process through multiple iterations. Personas can also be adjusted following deeper discovery research, such as in-depth interviews, as more information about the intended user comes to light. Within a few days, the researcher should present a screener that includes behavioral questions as well as demographics.

Scheduling

When creating a schedule for data collection, the researcher should know that you cannot run participants back to back. It’s generally not feasible to squeeze in 8 one-hour sessions in a single day, because of all of the activity that must occur between sessions. In an eight hour day, a researcher can perform four (maybe five) one-hour sessions but any more than that will take more time. Here are the reasons why:

One-hour sessions rarely go exactly one hour, some are shorter and quite a few will run longer. This can be due to a variety of reasons such as the product malfunctioning, the participant arriving late, or the participant providing lots of feedback. My rule of thumb is to allocate 50% of the session length as a buffer between sessions to allow for overrun, not including time needed to set up for the next session.

For sessions at an office or lab, some participants will arrive 10-20 minutes early, at which time they will need to use the restroom, sign NDAs and consent forms, and generally get comfortable. Comfortable participants give useful feedback, while uncomfortable participants tend to clam up and provide short, unemotional responses.
The researcher needs to set up and get ready. For usability or experience testing, the test will need to be reset, notes and documents need to be filed and new ones prepared. For any kind of home or location visit, the researcher will need to pack up all equipment and travel to the new location and set up equipment again.

Thus for every one-hour usability or experience testing session, there’s forty-five minutes to an hour of buffer and setup time. Home visits can take much longer.

Test Plan

A test plan should take no more than a week to develop and the researcher should give it to you for review and approval before being finalized. The test plan should specify the research and business goals associated with the project. During this period the researcher will need a significant amount of time with the product, either with a prototype or available concepts, while writing and checking the test plan. The better the researcher understands the intended final product, the more valuable the information he or she can get from the participant.

For usability or experience testing, the researcher will test the tasks with the product prior to a pilot test. He or she will need to make sure that there are no glitches, no unexpected areas under construction, and nothing giving away future tasks when performing each of the tasks with the product. With that in mind, it’s important to give the researcher a stable product or prototype and avoid drastic changes to the product prior to the test.

You should receive a well-written and organized test plan that details each research question and how it will be addressed. For usability testing this will include a list of tasks, what each task is intended to examine, approximate wording for the task (avoiding leading language), and detail on how each task will be scored or evaluated. For discovery research, it will include a list of topics to be addressed such as processes, environment and context, and expected pain points and needs.

Data Collection

When the data collection starts, it’s important to let the moderator work. During this time, the participant should feel comfortable enough to open up and provide honest feedback. In order to do this, it’s important to try to minimize observer impact during the testing session.

If you don’t have a separate place to watch the session (e.g. behind a two-way mirror or through a video feed), don’t make it obvious that you are paying close attention. Think about bringing in a laptop during the session to make it look like you’re doing other work. One way of doing this is telling the participant that you are also a researcher but you’re just going to be taking notes.

When you’re observing, remain objective and don’t make judgments based on one or two participants. It’s not uncommon to see a couple participants have a completely opposite reaction to a product compared to ten other participants. The researcher’s job is to sort through all the noise and report the real trends in the research. Take what you see with a grain of salt and listen to your researcher.

At the same time, it’s important to try to observe as many sessions as possible and give your researcher feedback between sessions if there are certain aspects of the user experience you want to know more about. The researcher should put the participant at ease and extract a great deal of information, including details that might have been overlooked or emotions that the person experiences. Different researchers will tend to achieve this in different ways as everyone has their own style, but you’ll notice by paying attention to the participant and seeing if they feel relaxed or nervous throughout testing.

Findings

Frequently, stakeholders will want to make immediate changes to a design, product, or prototype and won’t have the time to wait for the researcher’s final report. People have schedules that need to be met so it’s understandable that a project can’t always wait for the final report but the researcher should be able to provide you with quick findings within 24 hours of the last session.

For usability research, these quick findings should consist of a couple of short paragraphs including problems in the interface, possible solutions to these problems, and participants’ general reactions to the product, its look and feel, and expected usage. For ethnography or other forms of discovery research quick findings will tend to consist of expected usage of the product, expected value, high and low value features, and general trends about the intended user. Quick findings aren’t comprehensive and come before the researcher can get a complete look at the data, but it will provide you with the overall themes from the study.

When you do get the final report, make sure you take a look at it. It will tell you two things:
* Detailed findings regarding the interface, product, features, and intended user
* The quality and clarity of the report will tell you quite a bit about the quality of your researcher.

There’s one other thing to keep in mind when you are processing the findings from a usability test. The participants will tend to focus on the more obvious problems with a product or interface. There could be other, smaller or more abstract problems that are not identified in the first pass of usability testing. It’s usually a good idea to perform another test on the product after making changes to ensure that the changes you made were effective and identify any additional issues.

Summary

In summary, here are the most important points for non-researchers to know about the research process:
* Recruiting will almost always take two weeks or more.
* For every one-hour usability or experience testing session, there’s forty-five minutes to an hour of buffer and setup time, home visits can take much longer.
* The researcher will need a significant amount of time with the product (prototypes or concepts) while writing and checking the test plan.
* Try to minimize your impact during the testing session.
* Remain objective and don’t make judgments based on one or two participants.
* Ask your researcher to provide you with quick findings within 24 hours of the last session.

Any comments, feedback, or suggestions are very much appreciated.

But to date, the IA community hasn’t found an effective, simple technique (or tool) to test site structures. The most common method used — closed card sorting — is neither widespread nor particularly suited to this task.

Some years ago, Donna Spencer pioneered a simple paper-based technique to test trees of topics. Recent refinements to that method, some made possible by online experimentation, have now made “tree testing” more effective and agile.

How it all began

Some time ago, we were working on an information-architecture project for a large government client here in New Zealand. It was a classic IA situation – their current site’s structure (the hierarchical “tree” of topics) was a mess, they knew they had outgrown it, and they wanted to start fresh.

We jumped in and did some research, including card-sorting exercises with various user groups. We’ve always found card sorts (in person or online) to be a great way to generate ideas for a new IA.

Brainstorming sessions followed, and we worked with the client to come up with several possible new site trees. But were they better than the old one? And which new one was best? After a certain amount of debate, it became clear that debate wasn’t the way to decide. We needed some real data – data from users. And, like all projects, we needed it quickly.

What kind of data? At this early stage, we weren’t concerned with visual design or navigation methods; we just wanted to test organisation – specifically, findability and labeling. We wanted to know:
* Could users successfully find particular items in the tree?
* Could they find those items directly, without having to backtrack?
* Could they choose between topics quickly, without having to think too much (the Krug Test)¹?
* Overall, which parts of the tree worked well, and which fell down?

Not only did we want to test each proposed tree, we wanted to test them against each other, so we could pick the best ideas from each.

And finally, we needed to test the proposed trees against the existing tree. After all, we hadn’t just contracted to deliver a different IA – we had promised a better IA, and we needed a quantifiable way to prove it.

The problem

This, then, was our IA challenge:
* getting objective data on the relative effectiveness of several tree structures
* getting it done quickly, without having to build the actual site first.

As mentioned earlier, we had already used open card sorting to generate ideas for the new site structure. We had done in-person sorts (to get some of the “why” behind our users’ mental models) as well as online sorts (to get a larger sample from a wider range of users).

But while open card sorting is a good “detective” technique, it doesn’t yield the final site structure – it just provides clues and ideas. And it certainly doesn’t help in evaluating structures.

For that, information architects have traditionally turned to closed card sorting, where the user is provided with predefined category “buckets” and ask to sort a pile of content cards into those buckets. The thinking goes that if there is general agreement about which cards go in which buckets, then the buckets (the categories) should perform well in the delivered IA.

The problem here is that, while closed card sorting mimics how users may file a particular item of content (e.g. where they might store a new document in a document-management system), it doesn’t necessarily model how users find information in a site. They don’t start with a document — they start with a task, just as they do in a usability test.

What we wanted was a technique that more closely simulates how users browse sites when looking for something specific. Yes, closed card sorting was better than nothing, but it just didn’t feel like the right approach.

Other information architects have grappled with this same problem. We know some who wait until they are far enough along in the wireframing process that they can include some IA testing in the first rounds of usability testing. That piggybacking saves effort, but it also means that we don’t get to evaluate the IA until later in the design process, which means more risk.

We know others who have thrown together quick-and-dirty HTML with a proposed site structure and placeholder content. This lets them run early usability tests that focus on how easily participants can find various sublevels of the site. While that gets results sooner, it also means creating a throw-away set of pages and running an extra round of user testing.

With these needs in mind, we looked for a new technique – one that could:
* Test topic trees for effective organisation
* Provide a way to compare alternative trees
* Be set up and run with minimal time and effort
* Give clear results that could be acted on quickly

The technique — tree testing

Luckily, the technique we were looking for already existed. Even luckier was that we got to hear about it firsthand from its inventor, Donna Spencer, the well-regarded information architect out of Australia, and author of the recently released book “Card Sorting”:http://rosenfeldmedia.com/books/cardsorting/.

During an IA course that Donna was teaching, she was asked how she tested the site structures she created for clients. She mentioned closed card sorting, but like us, she wasn’t satisfied with it.

She then went on to describe a technique she called “card-based classification”:http://www.boxesandarrows.com/view/card_based_classification_evaluation, which she had used on some of her IA projects. Basically, it involved modeling the site structure on index cards, then giving participants a “find-it” task and asking them to navigate through the index cards until they found what they were looking for.

To test a shopping site, for example, she might give them a task like “Your 9-year-old son asks for a new belt with a cowboy buckle”. She would then show them an index card with the top-level categories of the site:

The participant would choose a topic from that card, leading to another index card with the subtopics under that topic.

The participant would continue choosing topics, moving down the tree, until they found their answer. If they didn’t find a topic that satisfied them, they could backtrack (go back up one or more levels). If they still couldn’t find what they were looking for, they could give up and move on to the next task.

During the task, the moderator would record:
* the path taken through the tree (using the reference numbers on the cards)
* whether the participant found the correct topic
* where the participant hesitated or backtracked

By choosing a small number of representative tasks to try on participants, Donna found that she could quickly determine which parts of the tree performed well and which were letting the side down. And she could do this without building the site itself – all that was needed was a textual structure, some tasks, and a bunch of index cards.

Donna was careful to point out that this technique only tests the top-down organisation of a site and the labeling of its topics. It does not try to include other factors that affect findability, such as:
* the visual design and layout of the site
* other navigation routes (e.g. cross links)
* search

While it’s true that this technique does not measure everything that determines a site’s ease of browsing, that can also be a strength. By isolating the site structure – by removing other variables at this early stage of design – we can more clearly see how the tree itself performs, and revise until we have a solid structure. We can then move on in the design process with confidence. It’s like unit-testing a site’s organisation and labeling. Or as my colleague Sam Ng says, “Think of it as analytics for a website you haven’t built yet.”

So we built Treejack

As we started experimenting with “card-based classification” on paper, it became clear that, while the technique was simple, it was tedious to create the cards on paper, recruit participants, record the results manually, and enter the data into a spreadsheet for analysis. The steps were easy enough, but they were time eaters.

It didn’t take too much to imagine all this turned into a web app – both for the information architect running the study and the participant browsing the tree. Card sorting had gone online with good results, so why not card-based classification?

Ah yes, that was the other thing that needed work – the name. During the paper exercises, it got called “tree testing”, and because that seemed to stick with participants and clients, it stuck with us. And it sure is a lot easier to type.

To create a good web app, we knew we had to be absolutely clear about what it was supposed to do. For online tree testing, we aimed for something that was:
* Quick for an information architect to learn and get going on
* Simple for participants to do the test
* Able to handle a large sample of users
* Able to present clear results

We created a rudimentary application as a proof of concept, running a few client pilots to see how well tree testing worked online. After working with the results in Excel, it became very clear which parts of the trees were failing users, and how they were failing. The technique worked.

However, it also became obvious that a wall of spreadsheet data did not qualify as “clear results”. So when we sat down to design the next version of the tool – the version that information architects could use to run their own tree tests – reworking the results was our number-one priority.

Participating in an online tree test

So, what does online tree testing look like? Let’s look at what a participant sees.

Suppose we’ve emailed an invitation to a list of possible participants. (We recommend at least 30 to get reasonable results – more is good, especially if you have different types of users.) Clicking a link in that email takes them to the Treejack site, where they’re welcomed and instructed in what to do.

Once they start the test, they’ll see a task to perform. The tree is presented as a simple list of top-level topics:

They click down the tree one topic at a time. Each click shows them the next level of the tree:

Once they click to the end of a branch, they have 3 choices:
* Choose the current topic as their answer (“I’d find it here”).
* Go back up the tree and try a different path (by clicking a higher-level topic).
* Give up on this task and move to the next one (“Skip this task”).

Once they’ve finished all the tasks, they’re done – that’s it. For a typical test of 10 tasks on a medium-sized tree, most participants take 5-10 minutes. As a bonus, we’ve found that participants usually find tree tests less taxing than card sorts, so we get lower drop-out rates.

Creating a tree test

The heart of a tree test is…um…the tree, modeled as a list of text topics.

One lesson that we learned early was to build the tree based on the content of the site, not simply its page structure. Any implicit in-page content should be turned into explicit topics in the tree, so that participants can “see” and select those topics.

Also, because we want to measure the effectiveness of the site’s topic structure, we typically omit “helper” topics such as Search, Site Map, Help, and Contact Us. If we leave them in, it makes it too easy for users to choose them as alternatives to browsing the tree.

Devising tasks

We test the tree by getting participants to look for specific things – to perform “find it” tasks. Just as in a usability test, a good task is clear, specific, and representative of the tasks that actual users will do on the real site.

How many tasks? You might think that more is better, but we’ve found a sizable learning effect in tree tests. After a participant has browsed through the tree several times looking for various items, they start to remember where things are, and that can skew later tasks. For that reason, we recommend about 10 tasks per test, presented in a random sequence.

Finally, for each task, we select the correct answers – 1 or more tree topics that satisfy that task.

The results

So we’ve run a tree test. How did the tree fare?

At a high level, we look at:
* Success – % of participants who found the correct answer. This is the single most important metric, and is weighted highest in the overall score.
* Speed – how fast participants clicked through the tree. In general, confident choices are made quickly (i.e. a high Speed score), while hesitation suggests that the topics are either not clear enough or not distinguishable enough.
* Directness – how directly participants made it to the answer. Ideally, they reach their destination without wandering or backtracking.

For each task, we see a percentage score on each of these measures, along with an aggregate score (out of 10):

If we see an overall score of 8/10 for the entire test, we’ve earned ourselves a beer. Often, though, we’ll find ourselves looking at a 5 or 6, and realise that there’s more work to be done.

The good news is that our miserable overall score of 5/10 is often some 8’s and 9’s brought down by a few 2’s and 3’s. This is where tree testing really shines — separating the good parts of the tree from the bad, so we can spend our time and effort fixing the latter.

To do more detailed analysis on the low scores, we can download the data as a spreadsheet, showing destinations for each task, first clicks, full click paths, and so on.

In general, we’ve found that tree-testing results are much easier to analyse than card-sorting results. The high-level results pinpoint where the problems are, and the detailed results usually make the reason plain. In cases where a result has us scratching our heads, we do a few in-person tree tests, prompting the participant to think aloud and asking them about the reasons behind their choices.

Lessons learned

We’ve run several tree tests now for large clients, and we’re very pleased with the technique. Along the way, we’ve learned a few things too:
* Test a few different alternatives. Because tree tests are quick to do, we can take several proposed structures and test them against each other. This is a quick way of resolving opinion-based debates over which is better. For the government web project we discussed earlier, one proposed structure had much lower success rates than the others, so we were able to discard it without regrets or doubts.

* Test new against old. Remember how we promised that government agency that we would deliver a better IA, not just a different one? Tree testing proved to be a great way to demonstrate this. In our baseline test, the original structure notched a 31% success rate. Using the same tasks, the new structure scored 67% – a solid quantitative improvement.

* Do iterations. Everyone talks about developing designs iteratively, but schedules and budgets often quash that ideal. Tree testing, on the other hand, has proved quick enough that we’ve been able to do two or three revision cycles for a given tree, using each set of results to progressively tweak and improve it.

* Identify critical areas to test, and tailor your tasks to exercise them. Normally we try to cover all parts of the tree with our tasks. If, however, there are certain sections that are especially critical, it’s a good idea to run more tasks that involve those sections. That can reveal subtleties that you may have missed with a “vanilla” test. For example, in another study we did, the client was considering renaming an important top-level section, but was worried that the new term (while more accurate) was less clear. Tree testing showed both terms to be equally effective, so the client was free to choose based on other criteria.

* Crack the toughest nuts with “live” testing. Online tree tests suffer from the same basic limitation as most other online studies – they give us loads of useful data, but not always the “why” behind it. Moderated testing (either in person or by remote session) can fill in this gap when it occurs.

Conclusion

Tree testing has given us the IA method we were after – a quick, clear, quantitative way to test site structures. Like user testing, it shows us (and our clients) where we need to focus our efforts, and injects some user-based data into our IA design process. The simplicity of the technique lets us do variations and iterations until we get a really good result.

Tree testing also makes our clients happy. They quickly “get” the concept, the high-level results are easy for them to understand, and they love having data to show their management and to measure their progress against.

You can sign up for a free Treejack account at “Optimal Workshop”:http://www.optimalworkshop.com/treejack.htm.²

References

1. “Don’t Make Me Think”:http://www.amazon.com/Dont-Make-Me-Think-Usability/dp/0321344758, Steve Krug
2. Full disclosure: As noted in his “bio”:http://boxesandarrows.com/person/35384-daveobrien, O’Brien works with Optimal Workshop.

It’s 2 a.m., and a call comes across the radio that a young man with a gun has barricaded himself and his mother in his home. No shots have been fired, and little communication has been established between the man and police officers outside. The officers on the scene report that the young man has been struggling with the loss of his job and feels like there’s no reason to live. The crisis response team has been called, and hostage negotiators are en route. It’s the negotiator’s job to ensure that the young man does not harm himself or others during this crisis.

What would you do? How would you handle this situation?

Throughout the past six years, the founders of ActiveComm Labs have not only been performing design research but also assisting the law enforcement community by conducting research on the communication patterns of hostage negotiators. Specifically, we have been analyzing the communication between the hostage negotiator and hostage taker to locate patterns that could introduce new strategies to help resolve crisis situations peacefully.

We’ve come to realize that the techniques used by hostage negotiators to resolve crises are also extremely valuable to user experience researchers. In essence, both parties are attempting to establish a relationship, both are trying to keep the communication flowing, and most importantly, both are trying to extract valuable data.

There are certain myths about hostage takers. Most of them are not bank robbers or terrorists demanding millions of dollars and a plane to Cuba. The vast majority of hostage situations are a result of domestic violence, psychological disorder, or barricade situations in which a person is threatening to commit suicide, possibly with a child in the next room. Hostage takers are usually confused, upset, and very scared. It’s also pretty rare for them to be outright hostile toward the negotiator. Hostage negotiators are trained to gather important data about the situation. Who’s in there? Is anyone hurt? What kind of weapons does the hostage taker have? How much ammo does he have? To do this, negotiators have to master a variety of communication techniques.

Have you ever worked with a research participant who will only give you “yes and no” answers? How about a participant who tells you exactly what you want to hear? These situations can be frustrating, especially when you invest so much time and energy in recruiting candidates. But these experiences don’t necessarily mean that these people can’t provide valuable data, it means that you need a different approach to extract that information.

A research session isn’t usually an emotionally charged situation and research participants aren’t typically in crisis, but the fundamentals of communication tend to hold true across different types of people and contexts. Our negotiation instructor told us that we should approach a hostage negotiation in much the same way as going on a first date; it’s important to bring a certain level of calm into the situation and put the hostage taker at ease. It is also extremely important to connect with the hostage taker on a personal level. Negotiation provides a great example of how to perform this kind of communication because it demonstrates these fundamental communication elements under the most difficult of conditions. Ultimately, negotiation is about two strangers coming together to work toward a common goal built on an understanding of each other, much like design research.

Application to Design Research

There are two types of behavior that we try to extract when conducting research:

1. What the participant does (physical interaction with product)
2. What he or she says (communication about the product)

Our goal is then to study the interaction between these behaviors in order to tell a story about the user’s experience of the product.

One of the most difficult parts of research is getting the participants to tell us their story about the product. Some researchers only focus on physical interaction data, but we think too much valuable content is lost. We’ve found that the communication piece of the equation provides the emotional and logical connection that participants make with products and how it relates to their lives.

With that said, one of the most common issues with communication-related data is how to gather accurate information. What a participant says is not always what he or she believes, and what a participant does is not always what the participant reports.

Much like a hostage negotiator, who must build trust in order to successfully resolve the crisis, a user experience researcher must establish a relationship with the participant in order to extract useful and accurate information. So, the fundamental element of becoming a better communicator, and also researcher, is to establish a relationship. Hostage negotiators focus on establishing relationships in order to save lives, there’s much we can learn from the methods that they have established.

Learning from Negotiators

Dominick J. Misino is a retired NYPD crisis negotiator who has been involved in more than 200 hostage and barricade incidents. He is recognized for his successful resolution of the Lufthansa hijacking in 1993 and numerous other successful negotiations. When it comes to communicating, Dominick knows what he’s doing. Since retiring, Dominick began training other hostage negotiators. To date, he’s trained thousands of negotiators across the country and around the world. A few years ago, we had the privilege to attend all three phases of Dominick’s negotiator training and certification program, which included hands-on practice as a negotiator and hostage taker. We learned communication techniques that we currently employ when interacting with research participants. These techniques include building rapport, building alliances, and using a team approach.

Building Rapport

Rapport is established through trust, open communication and empathy. Negotiators know that rapport is essential in their job. They use rapport to influence the hostage taker and gather information. If you can effectively build rapport with the participant, there is a higher likelihood he or she will trust you and disclose more information.

The following techniques used by hostage negotiators can help you build rapport with research participants:

1. Go slow – Engage in small talk at first. If you dive right into business, the situation can become uncomfortable.
2. Communicate openly – While you can’t disclose everything, it’s important to encourage an atmosphere of open communication. Tell the participant that there are certain aspects of the study that you can’t reveal, but he or she shouldn’t feel that you’re hiding something.
3. Actively listen – When you are listening to a participant’s story, listen for the emotions behind the words. Ask open-ended questions that dig for the source of those emotions.
4. Discuss personal topics – In a hostage situation, some of the most valuable topics that lead to a peaceful resolution are personal ones. The more a person feels that you accept them, the more comfortable they will feel with you.
5. Share your experiences – Building rapport is as much about sharing your experiences as it is about listening to the other person’s. Negotiators know that the more you reveal about yourself, the more the participant feels like he or she knows you and therefore trusts you.
6. Show you care – Hostage negotiators build rapport through empathy. Empathy is extremely important because it shows that you care about the other person and that you have their best interests in mind. As a researcher, you should do this also. If you show that you care, the participant will appreciate it and respond with more openness.

Alliances

In a hostage situation, the negotiator works for the police department but he has to show the hostage taker that he’s on his side. In order to do this, the negotiator can never be the one in charge; it cripples his or her ability to negotiate. Anytime the negotiator has to tell the hostage taker “no” it’s because his boss is being a jerk. Anytime the negotiator says “yes,” whether it’s a pack of cigarettes or just some extra time, it’s because the negotiator fought hard to get it for him. The negotiator intentionally shifts the blame for anything negative and takes credit for anything positive. It convinces the hostage taker that the negotiator is on his side.

In user experience research, the researcher is on the side of the user. In our work, we establish this by telling the participant that we are not the people who designed the product and that their comments, whether good or bad, will not offend us. This establishes objectivity and allows a certain freedom in the research session. In most cases, participants open up when they hear that you have nothing at stake. Also, if the participant can see that you share his or her common goal of improving the product, the participant is more likely to be truthful in his or her evaluation.

Team Approach

Hostage negotiators always work in teams, and so should you. In the event of a hostage situation, a negotiation team is called to manage the situation. Each person in that team holds a different but critical role in the event. One of the most important positions on that team is the coach. As the negotiator acts as the primary point of contact with the hostage taker, the coach sits with the negotiator and functions as another pair of ears. The communication can move very quickly during a negotiation, and the negotiator can have a hard time catching all of it. The coach specializes in listening, controlling access to the negotiator, generating questions and helping guide the communication process by passing notes to the hostage negotiator. Through crisis negotiation training, my partner and I have learned that the ability to gather useful and accurate information dramatically increases when you work in teams. For example, when conducting an expert interview we have one person ask the questions and another person as a secondary moderator. Like the coach in negotiation, the secondary moderator listens closely, takes detailed notes and chimes in when he feels that something is being missed. This type of setup will reap maximum data in the shortest amount of time. We can’t always work in teams for logistic or financial reasons, but it is our preferred method.

Training

For hostage negotiators, training is a crucial part of the job and they understand that the more you train, the more comfortable you will feel in the situation and in turn the better the outcome.

From what I have seen in the user experience community, little or no time is spent training on the best practices of communicating with participants. Every so often a workshop is attended but that only happens a few times a year. If we take a page from the book of negotiating, we would learn that just a little bit of training on a regular basis will take us to a whole new level of success. Here are some modified exercises that we can use to polish our communication skills as researchers:

1. Communicate with new people all the time – every time you have a chance to meet someone new and learn about their life, do it. Ask questions about who they are, where they come from and what they’re like. When you start to feel more comfortable doing this, start pushing yourself and asking more intimate questions about their life such as “What keeps you up at night?” Play a little game with yourself where you try to learn as much as you can about a person in a short amount of time. Three things will happen when you do this. First, you will learn about boundaries and what you can ask and you should not ask. This doesn’t mean that you’ll necessarily insult people and then learn not to do it; it means that you will see what topics are challenging to people and how to pull that information out of them without upsetting them. You’ll also learn more about people in general and how they function in the world. You’ll get a better sense of behavior and start to see trends in people. Finally, you will make more friends in the process and that is always a nice outcome.
2. Learn to listen – A well-known question asks, “When someone is speaking to you, do you think about what they are saying or do you think about what you are going to say next?” This is a very important question when learning to communicate more effectively with others. When you are communicating with participants or friends, really listen to what they are saying. Listen to the emotions behind the words, look at body language, and ask questions about their responses if they are unclear. There are sometimes differences between what a user will answer and what they believe. Ask yourself what those statements say about the speaker as a person; it will enable you to discover the areas where people are most passionate.
3. Learn how to disclose – When talking with friends and new people, start disclosing about your life and observe how people respond. Most people will feel more comfortable sharing information if you share information also. You’ll be amazed at how quickly people will open up.
4. Learn to trust your gut – When working with a participant or talking with a friend, learn to listen to your instincts. There are times when you need to speak up, times when you need to bring the communication back on track and times when you need to let the other person just open up and talk about whatever they feel is important. Different types of communication are needed for different situations. If you train frequently, you will notice that your gut tells you what you need to do.

Conclusion

This article began with a scenario of a hostage situation. After the first 30 minutes, the hostage taker and negotiator were talking like old friends about food, sports, and pets. At two hours, the hostage taker confided in the negotiator about his relationship problems and issues that led to him losing his job. At two and a half hours, the young man sent his mother out of the house, despite her protests that she wanted to stay with her son. At four hours, the young man placed his empty gun in a bucket attached to a rope outside an open window, where it was retrieved by the police tactical team. At 7:50am, after five hours and fifty minutes of negotiation, the young man peacefully exited his home and surrendered to police. The negotiator followed up on all of his promises. He rode with the young man to the police station, allowed his mother to visit so that he could apologize and even made a statement on the young man’s behalf at his trial, resulting in a reduced sentence.

This process should be mirrored in a research session in a condensed period of time. After the first ten or fifteen minutes, the participant should feel like you know each other and feel pretty comfortable talking with you about your product or service. After about twenty minutes, the participant should understand the product and its goals. After thirty minutes, the participant should be discussing how the product would fit into his or her life. In order to achieve this, some form of communication training should be implemented. Researchers typically receive a great deal of training in research methods, statistics, human factors and elements of design, but little training on advanced communication. Researchers who really want to invest into their skills as a researcher should think about spending time and energy to learn effective communication methods.

Prototyping is a big deal right now. We get wrapped up in mailing list threads, new tools are released at an astonishing pace, books are being published, and articles show up on Boxes & Arrows. Clients are even asking for prototypes. But here’s the thing… prototyping is not a silver bullet.

There is no one right way to do it.

However, prototyping is a high silver content bullet. When aimed well, a prototype can answer design questions and communicate design ideas. In this article, I talk about the dimensions of prototype fidelity and how you can use them to choose the most effective prototyping method for the questions you need answered.

The dimensions of fidelity

A prototype’s fidelity has the most influence over its effectiveness. Fidelity simply refers to how realistic the prototype is. Most of the time when we talk about a “high-fidelity” prototype we are referring to a prototype that has some visual or industrial design applied to it. But that leaves out what’s most important to UX designers, what it’s like to actually work with the prototype!

Fidelity is multidimensional.

Not only can you have a prototype that looks like a realistic product, but you can also have a prototype that works like a realistic product. I call these dimensions of fidelity “visual fidelity” and “functional fidelity.” By varying your prototyping methodology along these two dimensions you can ensure that your prototyping effort is successful given your particular context. Let’s take a look at some examples.

A prototype can be as simple as a series of hand-sketched wireframes that flow together. This is a good example of a low visual fidelity prototype. These wireframes show layout and functionality but have no visual design. Take the same wireframes, integrate a visual design, and your prototype has a high visual fidelity. While you might think of them as being similar, these two prototypes are most effective in two different situations.

That same series of sketches is also a low functional fidelity prototype. Moving through screens drawn on paper is much different than working with the developed system. But if you render those sketches in HTML, JavaScript, & CSS, they have a high functional fidelity. Working with an interactive prototype is very similar to working with the developed system. Again, high- and low-fidelity prototypes are most effective in two completely different situations.

After spending all this time talking about fidelity, I want to share one of my favorite quotes on prototyping. Bill Buxton said this in his Interaction08 keynote:

There is no such thing as high or low fidelity, only appropriate fidelity.

Appropriate Fidelity

“Appropriate fidelity” refers to a level of prototype fidelity that allows you to achieve the goals you’ve set for doing a prototype in the first place. By varying the fidelity of your prototype along the dimensions of visual design and functionality, you make your prototype more effective at achieving some goals and less effective for others.

Low Visual and Low Functional Fidelity

Very low fidelity prototypes are extremely useful to UX designers. Why? They can be made swiftly, changed without repercussion, and still help visualize a concept. Low visual & functional fidelity prototypes are helpful at answering large structural questions. Here are some examples:

• Does the system have all the features required to support the user’s goals?
• Does the workflow make sense at a high level?
• Which UX concept works best?
• Coming to consensus on a UX concept with stakeholders, e.g.“Is this what you meant?”

Low Visual and High Functional Fidelity

In my own practice, this is the type of prototyping I do most often. What I make are interactive, HTML interactive wireframes. Everything is black, white, and gray, but the interactions are extremely close to what they’d be in the developed system. These types of prototypes are effective in many situations:

• Evaluating the usability of proposed designs for new systems
• Exploring isolated interactions as a proof-of-concept
• Validating UX design direction with stakeholders
• Validating the implementation of requirements with stakeholders
• Supplementing printed documentation for development teams
• Performing remote testing

Remote testing has become more and more important over the last several years. At Evantage, we do approximately 75% of our user testing remotely. It would be difficult for us to get good data about our designs for modern, highly interactive sites if we were limited to representing those designs using low-to-medium functional fidelity prototyping techniques such as clickable PDFs or interactive PowerPoint presentations.

I also want to expand on proof-of-concept testing. This technique supports creativity and innovation. By prototyping isolated interactions at a high functional fidelity and testing them with users, I can get really good data about whether that interaction works before I base an entire application around it. This allows me to explore my crazy ideas and find out if they are, in fact, crazy. But if it turns out that those ideas are actually pretty slick, I’ll know that and can release the design with confidence instead of with gritted teeth.

High Visual and Low Functional Fidelity

At first thought, these prototypes may not make much sense. Why bother making something look nice if it doesn’t work? Well, because how something looks can have a huge impact on how easy it is to use. A high visual and low functional fidelity prototype allows you to test that with users. You can print out screen images and do a paper prototype test with them, or you can image map some JPGs and do what I’ve heard termed a “slap and map” test from within a browser.

High Visual and High Functional Fidelity

High visual and functional fidelity prototypes are the Rolls-Royce of prototypes. They take more time and effort to build than a lower fidelity prototype and are correspondingly more complicated to manage. Most of the time, this extra cost isn’t worth it. But there are a few situations where it is:

• Evaluating the usability of proposed UX designs for an existing system
• Performing usability tests with non-savvy user groups
• Supplementing printed documentation for offshore development teams

Prototype testing is all about data, right? In the first two situations above, the prototype’s high visual fidelity reduces the confounding factors a wireframey prototype can introduce into test results, thus maintaining the quality of your data. In the third situation, the high visual fidelity helps minimize the design communication and interpretation problems inherent in offshore development.

Integrating Prototyping Into Your Design Process

What I’ve talked about so far has focused on the tactical, on how to prototype effectively to achieve specific goals. What I want to talk about now is more strategic. How can you integrate prototyping effectively into your design process?

First off, do what you’d do to begin any organizational change. Start small. Find a small project, express the value of prototyping and your interest in doing it, and do it. It would be best to start with something richly interactive though, as prototyping is more crucial the more interactive a system is. Of course, make sure you use a prototype of the right visual and functional fidelity for your purpose.

As you near completion of the prototype, make sure you walk through the prototype with the project’s stakeholders. Ask them if something like this was what they had in mind. This will impress them on two levels. First, people like to feel important, and you’re soliciting their opinions. Second, people like shiny things that move. The cool factor of prototyping will be difficult to resist. When these stakeholders are involved in future projects, it’s very likely they will actually request a prototype as a result of their first experience with you.

Once you get buy-in, you can start integrating prototyping into your process. But just like different methods of prototyping are more effective for answering certain questions, different business contexts call for different ways to integrate prototyping.

Corporate, Agile, Mature UX Practice

This situation is fast-paced and iterative, but as an employee (as opposed to a contractor or consultant) you have the opportunity to own the UX of your company’s products. In this situation, there are three points in the design process that prototyping can be of benefit.

• Low visual and functional fidelity prototypes can help select good UX concepts from several that you develop at the beginning of a project.
• High functional fidelity proof-of-concept prototypes can help develop those concepts into usable designs.

You can work with a dedicated prototyper to build a separate prototype using code that can be reused in the production system to build efficiencies into an Agile process.

Corporate, Waterfall, New UX Practice

In this situation, the organization might not be comfortable enough with UX design to support the development of multiple UX concepts. You might just have to begin developing the wireframes and prototypes to meet the organization’s need for documentation and measurable signs of progress. This situation relies heavily on the prototype for communicating and validating direction with project stakeholders, with user testing often not yet being a real possibility. Here’s how prototyping can help:

• High functional fidelity prototypes can help you communicate better with stakeholders and get their input on your direction
• These prototypes should also be used for user testing, if at all possible.
• Walk through the interactive prototype at the same time you walk through the printed documentation for the developers during handoff.

Consulting/Agency

When doing UX design for an external client, a lot of the magic is worked behind the scenes. The result is a process that is relatively unencumbered by internal politics. The challenge is to convey the importance of iterative prototyping to clients who sometimes feel like they’re paying for the same thing twice.

• Sketch two or three of your design concepts into simple, low visual and functional fidelity prototypes and test them to decide which to go with. At this stage, testing can be very informal and done with anyone capable of putting themselves in the user’s shoes (e.g., other UX designers, customer service staff, or product managers who used to be users).
• Build a small interactive prototype that shows the critical interactions, walk through it with stakeholders to validate your direction, then test with users.
• Revise the prototype based on the test results, flesh it out to support more holistic tasks, and test again.
• Revise the prototype and use it to supplement the paper documentation as you walk through both with the development team.

Just like with any other UX research or design tool, context plays a critical role in determining how effective prototyping will be for you. If you follow the simple guidelines above and prototype to an appropriate level of fidelity for your context, you will achieve your goals and improve your design. No firearms required.

Consider how video games are currently researched: market research-based focus groups and surveys dominate the landscape, measuring opinion and taste in a controlled lab environment, and largely ignoring players’ actual in-game behaviors. Behavior is obviously the most direct and unbiased source of understanding how players interact with the game—where they make errors, where they become irritated, where they feel most engaged. When Electronic Arts engaged Bolt|Peters to lead the player research project for Spore, we set out to do one better than the usual focus group dreck by coming at it from a UX research perspective.

SIMULATED NATIVE ENVIRONMENT RESEARCH

One overarching principle guided the design of this study: we would let the users play the game in a natural environment, without the interference of other players, research moderators, or arbitrary tasks. This took a good bit of planning. Usually, we prefer to use remote research methods, which allow us to talk to our users in the comfort of their own homes. Spore, however, was a top-secret hush-hush project; we couldn’t very well send out disks for just anybody to get their hands on. Instead, CEO Nate Bolt came up with what we call a “Simulated Native Environment.” For each of the ten research sessions, we invited six participants to our loft office, where they were seated at a desk with a laptop, a microphone headset, and a webcam. We told them to play the game as if they were at home, with only one difference: they should think-aloud, saying what ever is going through their mind as they’re playing. When they reach certain milestones in the game, they would fill out a quick touchscreen survey at their side, answering a few questions about their impressions of the game.

Elsewhere, Nate, the clients from EA, and I were stationed in an observation room, where we set up projectors to display the players’ gameplay, the webcam video, and the survey feedback on the wall, which let us see the players’ facial expressions alongside their in-game behaviors. Using the microphone headset and the free game chat app TeamSpeak, we were able to speak with players one-on-one, occasionally asking them what they were trying to do or to go a little more in depth about something they’d said or done in the game.

Doesn’t that sound simple? Actually, the setup was a little brain-hurting: we had six stations; each station needed to have webcam, gameplay, survey, and TeamSpeak media feeds broadcast live to the observation room – that’s 18 video feeds and 6 audio feeds, and not only did the two (that’s right, two!) moderators have to be able to hear the participants’ comments, but so did the dozen or so EA members. On top of that, everything was recorded for later analysis.

“The feedback we received from users wasn’t based on tasks we’d ordered them to do, but rather on self-directed gameplay tasks the users performed on their own initiative”

The important thing about this approach is the feedback we received from players wasn’t based on tasks we’d ordered them to do, but rather on self-directed gameplay tasks the players performed on their own initiative. We didn’t tell players outright what to do or how to do things in the game, unless they were critically stuck (which was useful to know in itself). The observed behavior and comments were more stream-of-consciousness and less calculated in nature.

The prime benefits to our approach were the absence of moderators, which mitigated the Hawthorne effect, as well as the absence of other participants, eliminating groupthink. Additionally, the players were more at ease: it’s hard to imagine these video outtakes (see below) being replicated in a focus group setting. Most importantly, they weren’t responding to focus questions–they were just voicing their thoughts aloud, unprompted, which gave us insight into the things they noticed most about the game, rather than what we just assumed were the most important elements.

OOPS, WE MESSED UP

Over the year-long course of the project, there was one incident which proved to us just how important it was to preserve the self-directed task structure of our research. Because of the multiphase progression of Spore, we believe it was important to carefully structure the sessions to give players a chance to play each phase for a predetermined amount of time, and in a set order as if they were experiencing the game normally.

Partway through the second session, we started having doubts: even though we weren’t telling players what to do within each phase, what if our rigid timing and sequencing is affecting the players’ engagement and involvement with the game?

To minimize this, between sessions, we made a significant change to the study design: instead of telling users to stop at predetermined intervals and proceed to the next phase of the game, we threw out timing altogether and allowed users to play any part of the game they wanted, for as long as they wanted, in whatever order they wanted. The only stipulation was that they should try each phase at least once. Each session lasted six hours spread over two nights, so there was more than enough time to cover all five phases, even without prompting users to do so.

Sure enough, we saw major differences in player feedback. We are unable to provide specific findings for legal reasons, but we can say that the ratings for certain phases consistently improved (as compared with previous sessions). Additionally, a few of the lukewarm comments players had made about certain aspects of the game seemed to stem from the limiting research format, rather than the game itself.

It became clear that when conducting game research, it was vitally important to stick to the actual realities of natural gameplay as much as possible, even at the expense of precisely structured research formatting. You have to loosen up the control a little bit; video games are, after all, interactive and fun. It makes no more sense to formalize a gameplay experience than it does to add flashy buttons and animated graphics to a spreadsheet application.

BRINGING GAME RESEARCH INTO THE HOME

There are a lot of ways to go with the native environment approach. Even with all efforts to keep the process as natural and unobtrusive as possible, there are still lots of opportunities to bring the experience even closer to players’ typical behaviors. The most obvious improvement is the promise of doing remote game research–allowing participants to play right at home, without even getting up.

Let’s consider what a hypothetical in-home game research session might look like: a player logs into XBox Live, and is greeted with a pop-up inviting him to participate in a one-hour user research study, to earn 8000 XBox Live points. (The pop-up is configured to appear only to players whose accounts are listed as 18 or older, to avoid issues of consent with minors.) The player agrees, and is automatically connected by voice chat to a research moderator, who is standing by. While the game is being securely delivered and installed to the player’s XBox, the moderator introduces the player to the study, and gets consent to record the session. Once the game is finished installing, the player tests the game for an hour, giving his think-aloud feedback the entire time, while the moderator takes notes and records the session. At the end of the session, the game is automatically and completely uninstalled from the player’s XBox, and the XBox Live points are instantly awarded to the player’s account.

Naturally, there are lots of basic infrastructure advances and logistical challenges to overcome before this kind of research becomes viable:

• Broadband penetration
• Participant access to voice chat equipment
• Online recruiting for games, preferably integrated into an online gaming framework
• Secure digital delivery of prototype or test build content
• Gameplay screensharing or mirroring

For many PC users, these requirements are already feasible, and for games with built-in chat and/or replay functionality, the logistics should already be much easier to meet. Remote research on PCs is already viable (and, in fact, happens to be Bolt|Peters’s specialty). Console game research, on the other hand, would likely require a substantial investment by console developers to make this possible; handheld consoles present even more challenges.

We expect that allowing players to give feedback at home, the most natural environment for gameplay, would yield the most natural feedback, bringing game evaluation and gameplay testing further into the domain of good UX research.

Spore Research: Outtakes from bolt peters on Vimeo.

Science of Fun from bolt peters on Vimeo.

Steps in the QTUT Process

• Step 1: Sales & Kickoff
• Step 2: Recruitment
• Step 3: Preparation
• Step 4: Testing
• Step 5: Analysis & Reporting

Testing

It’s testing day. You have successfully recruited enough participants for the first day, but you feel a bit of panic as you make the finishing touches before the first participant arrives. You have a rough but solid test script. You have five attentive stakeholders in the observation room ready to begin taking notes. Now you need to execute the test and you need to compile results as you go along.

Up to this point, the lack of time you had to plan and to refine your method creates a bit of a panic as you begin the testing phase. Often, we are working on the script until the very last second, incorporating changes from the stakeholders that they hand off when they arrive at our testing facility.

Early on the test day, I print out a screenshot of every important page and component (e.g., the primary navigation). I number these screenshots and then tape them above a large whiteboard that we keep in an “idea” room that is adjacent to our usability lab’s observation room. We use the whiteboard to keep track of issues and metrics across participants.

After you finish each participant session, immediately note changes that you need to make to your test script or the application. Then go talk with your stakeholders about the results. Here, the time that you have budgeted between sessions for discussion really pays off. If your stakeholders have been watching and taking notes, they are likely already talking about the results. They may also be already talking about potential fixes.

The whiteboard can be useful for focusing the discussion on issues. It’s often useful to set ground rules for the discussion. First, the discussion should focus on results and not solutions. It is important to manage your stakeholders by telling them to be patient and to let the results play out over several people before drawing conclusions. Second, as the person facilitating the study, you should lead the discussion on each topic by first summarizing your notes on the whiteboard. After we summarize a page, we ask for any additional feedback from the stakeholders and then quickly move on to the next page. Occasionally, we need to remind the stakeholders that since we have limited time between participants, we cannot dwell on any one finding.

You may think that stakeholders will not have much valuable feedback to add, but we have found that they often see things that we don’t because of their knowledge of the history of the application. For example, one client had just made a political decision to change a button label. Since the participants understood the new button name, we didn’t think to list it as a finding. But for the stakeholders, it was helpful to track it and mention it in the report.

After the first participant’s session, you will likely realize that some of your tasks and questions were not worded correctly. Through the whiteboard session, you may think of additional questions to ask participants, or you might even want to add, delete, or change tasks. Do not rely on your memory to do any of this. Instead, make the changes to your script and print out a new copy for the next participant.

At the end of your first day of testing, you should have a board full of findings, including some task completion data. If you test five participants per day, by the fifth participant trends are becoming clear. You may decide to stop a task because you already know the issues, which allows you to test other tasks that you couldn’t fit in. Discuss this at the end of the day so that you can make the necessary edits to your test script before you leave for the day.

In addition, if your stakeholders are already discussing potential fixes while you facilitate the test, it is important for you to be a part of that discussion. In their eagerness to fix things, our stakeholders occasionally solve the problem in such as way that a larger problem is created.

If you are in a fast-paced development environment, where developers are staying until 10 p.m. at night to make changes, your stakeholders may want to change things that night. Because of the potential to affect your testing, you should be very careful to attempt only easy fixes. You do not want to rework the navigation or radically alter a task flow overnight. However, you may want to change a button’s behavior or the text in a label.

As sessions progress, your whiteboard may start to fill up. Usually it is easy to condense the notes. We use a Post-It note for each issue and then start writing participant numbers on the note for each person that experiences the same issue (see Figure 1). We take pictures of the whiteboard throughout the process, prune non-issues, and type up results for tasks and questions that we have stopped using.

Figure 1: Whiteboard full of findings after two days of testing. Post-It notes contain reoccurring issues.

Finally, as you reach the last few sessions, increase the level of detail about potential recommendations. If possible, test these ideas with your participants. Get a sense from your stakeholders about what recommendations are feasible and cost effective on a short timeline, and which require long-term attention.

Testing Tips

• Keep the testing as simple as possible—for QTUT avoid technology such as eyetracking and anything else that makes your job more difficult or confusing.
• For changes to your tasks and questions, modify the test script and do not rely on your ability to remember the changes.
• Be willing to change questions and tasks. It is better to find and to fix the issues with your testing script early on.
• Remind your stakeholders that because of the aggressive timeline your test script is not perfect, so you may have to change tasks and questions on the fly for the first couple of participants.
• Keep a whiteboard in the observation room and use it to keep track of and to discuss problems with your stakeholders after each session.
• At the end of the day, summarize the results and discuss potential recommendations. Consider making a list of the most important findings from the day.
• Expect long work hours.
• Don’t duplicate effort by typing up the results before you have tested all of the participants; write up the results only when you have stopped testing a task or asking a question.

Analysis & Reporting

You have made it through two days of testing. The stakeholders have left and now you have to create a report that clearly communicates the issues and your recommended fixes.

The beauty of the whiteboard method is that your report becomes simply a summary of what you have already written on the whiteboard, including completion metrics, findings, and recommendations that have been vetted by key stakeholders.

Although the people who attended the sessions may understand the issues, you still need to translate them from shorthand whiteboard notations to a format that a wider audience can read. We use a presentation-style report because it is much faster to create than a report with lots of text.

Because time is important to the stakeholders, we classify recommendations as short-term and long-term fixes. Short-term fixes can be finished in a few days, or before the application is released. Long-term fixes require more time than your clients have, but they are severe enough that they need to be addressed eventually.

Our QTUT ends when we present the results to the stakeholders, which often includes several interested people who were not able to attend the sessions. We present our findings in person so that the stakeholders feel more comfortable asking questions about the test and so they can get immediate clarification on findings that have been inadequately described.

Analysis and Reporting Tips

• Use a presentation-style report format that does not require long text passages.
• If you will be presenting results to stakeholders who did not attend the sessions, give the necessary context to your findings.
• If you change recommendations that you discussed with your stakeholders, let them know in advance so that they do not feel undermined.
• Classify your recommendations in terms of the expected timeline for the fix.
• Take pictures of your whiteboard in order to preserve your notes quickly and safely, enabling you to write the report from another location.

And now this.

“Hi! So we have some executives coming in tomorrow to observe the test sessions.” This was the researcher phoning. He was pretty pleased that his work was finally getting some attention from management. I would have been, too. But. He continued, “I need you to [oh yeah, the Phrase of Danger] call up the participants and move some of them around. We really want to see the experienced guy and the novice back-to-back because Bob [the head of marketing] can only come at 11:30 and has to leave at 1:00.”

“Sure,” I say, “we can see if the participants can flex. But your sessions are each an hour long. And they’re scheduled at 9:00, 10:30, 12:00, and 2:00. So I’m not quite clear about what you’re asking us to do.”

“I’m telling you to move the sessions,” the researcher says, “so the experienced guy is at 11:30 and the novice is at 12:30. Do whatever else you have to do to make it work.”

“Okay, let me check the availability right now while we’re on the phone,” I say. I pull up the spreadsheet of participant data. I can see that the experienced guy was only available at 9:00 am. “When we talked with Greg, the experienced guy, the only time he could come in was 9:00 am. He’s getting on a plane at 12:30 to go to New York.”

“Find another experienced guy then.” What?!

Five signs that you’re dissing your participants

You shake hands. You pay them. There’s more to respecting participants? These are some of the symptoms of treating user research participants like lab rats:

They seem interchangeable to you.


If you’re just seeing cells in a spreadsheet, consider taking a step back to think about the purpose and goals of your study.

You’re focused on the demographics or psychographics.


If it’s about segmentation, consider that unless you’re running a really large study, you don’t have representative sample, anyway. Loosen up.

Participants are just a way to deliver data.


You’ve become a usability testing factory, and putting participants through the mill is just part of your life as a cog in the company machine.

You don’t think about the effort it takes for a person to show up in your lab.


Taking part in your session is a serious investment. The session is only an hour. But you ask participants to come early. Most do. You might go over time a little bit. Sometimes. It’ll take at least a half hour for the participant to get to you from wherever she’s coming from. It’ll take another half hour for her to get wherever she’s going afterward. That’s actually more than 2 hours all together. Think about that and the price of gas.

You don’t consider that these people are your customers and this is part of their customer experience.

You and your study make another touch point between the customer and the organization that most customers don’t get the honor of experiencing. Don’t you want it to be especially good?

They’re “study participants” not “test subjects.”

Don’t forget that you couldn’t do what you do without interacting with the people who use (or might use) your organization’s products and services. When you meet with them in a field study or bring them into a usability lab, they are doing you a massive favor.

Although you conduct the session, the participant is your partner in exploration, discovery, and validation. That is why we call them “participants” and not “test subjects.” There’s a reason it’s called “usability testing” and not “user testing.” As we so often say in the introductions to our sessions, “We’re not testing you. You’re helping us evaluate the .”

Throw away your screener: Tips on recruiting humans

I’m not kidding. Get rid of your screener and have a friendly chat with your market research people. Tell them you’re not going to recruit to match the market segments anymore. Why not? Because they usually don’t matter for what you’re doing. In a usability test, you focus on behavior and performance, right? So recruit for that.

Focus on behavior, not demographics


Why, if you’re testing a web site for movie buffs, will selecting for household income matter? What you want to know is whether they download movies regularly. That’s all. Visualize what you will be doing in the session, and what you want to learn from participants. This should help you define what you absolutely require.

Limit the number of qualifiers


Think about whether you’re going to compare groups. Are you really going to compare task success between people from different sized companies, or who have multiple rental properties versus only one, or different education levels? You might if you’re doing a summative test, but if most of your studies are formative, then it’s unlikely that selecting for multiple attributes will make a big difference when you’re testing 5 or 6 people in each audience cell.

Ask open-ended questions


Thought you covered everything in the screener, but fakers still got into your study? Asking multiple-choice questions forces people to choose the answer that best fits. And smart people can game the questionnaire to get into the study because they can guess what you’re seeking. Instead, ask open-ended questions: Tell me about the last time you went on a trip. Where did you go? Where did you stay? Who made the arrangements? You’ll learn more than if you ask, Of your last three trips taken domestically, how many times did you stay in a hotel?

Learn from the answers


You get “free” research data when you pay attention to the answers given to open-ended screening questions because now people have volunteered information about their lives, giving you more information about context in which you can make decisions about your study design and the resulting data.

Flex on the mix


If you use an outside recruiting firm, ask to review a list of candidates and their data before anyone gets scheduled. You know the domain better than the recruiters do. You should know who will be in the testing room (besides you). You should make the trade-offs when there’s a question about how closely someone meets the selection criteria.

Remember, we’re all human, even your participants. These steps will help you respect the human who is helping you help your company design better experiences for customers.

If you design user experiences for standards-based websites and applications (i.e. those built with XHTML, CSS, and JavaScript), there are several great reasons for adding XHTML prototyping to your UX tool kit. Perhaps you’ve found that traditional wireframes just aren’t sufficient and are looking for more powerful ways to explore and communicate design solutions. Perhaps your current practice is based on the traditional waterfall model (i.e. first creating wireframes, which are handed off to creative, who hand off comps to tech, and so forth), and you want to explore more contemporary methodologies, such as agile and iterative development. Regardless, a great way to embark on that journey is to start prototyping with XHTML.

So what does it mean to prototype with XHTML? Essentially, it’s the process of using the XHTML itself, and related technologies, to evolve and define your design solution. And what does an XHTML prototype look like? While, as we’ll see, that depends on where you are in your prototyping process, an XHTML prototype generally looks like any other web page built with XHTML, with some links or features perhaps being non-functional. In other words, anything you can build with XHTML, from consumer websites to enterprise applications, you can also prototype with XHTML. As we’ll see, there are numerous advantages to this approach compared to designing with wireframes or other prototyping tools.

An Iterative Process

While prototyping with XHTML isn’t tied to a specific design process, iterative development seems to effectively leverage its strengths. There are many reasons for this, but perhaps the most significant is that in both cases the prototype, and later the application itself, doubles as a specification. We’ll explore what that means in a bit, but first let’s walk through a suggested process for prototyping with XHTML.Let us start with an overview of the larger design process:

In this (iterative) methodology, rather than design the entire application before starting to build it, one designs and builds a unit of the application and then uses what has been built to inform and serve as a starting point for other application units. As with other design methods, the design work begins with sketching, which plays a particularly important role relative to prototyping.

Sketching: A Freeform Question

The term ‘sketching’ refers here to any freeform exploration unconstrained by a specific technology. This includes production of wireframes, which in this model are reframed, as it were, from specification artifact to refined sketch. When thought of, and presented to stakeholders, as sketches, its more natural to discard your wireframes once the design has evolved beyond them. This is usually after a prototype equivalent has been produced. With the design team I work with, we’ve found that when prototyping with XHTML, wireframes often became superfluous, and it’s more effective to go directly from sketch to prototype.

Prototyping: A Concrete Response

Prototyping has a counterpoint relationship to sketching. To paraphrase Bill Buxton, while sketches ask a question—“Is this a good design idea?”— prototypes provide a response. By making the idea manifest, prototypes force upon it the concrete realities and user experience idiosyncrasies of the actual production technology and offer a crisp verdict on the quality of what you dreamed up in drawings.

The Prototype/Build Relationship

When prototyping with XHTML, especially in an iterative model, the build and prototype become very intertwined. If you’re prototyping a new application or product, the XHTML prototype is essentially a rough draft of the actual application. However, when updating the design of an existing application, the production version can serve as the starting point for the prototype of the new solution.

Three Integrated Layers: Structure, Behavior, Foundation

The model for XHTML prototyping is based on the best practices model for actual site production: start by setting the foundation with XHTML, add a presentation layer with CSS, follow it by a behavioral layer using JavaScript then iterate.

Let’s start by looking at the structural layer.

Structure: Set the Page Foundation

The first step in production of the XHTML prototype is to create a structural foundation. Similar to how we create a wireframe, we start by representing the main content areas on the page, except we do so with text-based XHTML markup.

| If our sketch or wireframe looks like this | …our XHTML might look like this: |
|^. |^. …

…
(We’re only displaying the relevant snippet of the XHTML here.) |

Next, we add detailed content elements that have been defined, using the XHTML structure appropriate for the corresponding content.

| For example, if our detailed sketch looks like this | …we’d represent the list of account help topics as an unordered list (i.e. use the ul tag): |
|^. |^.
…

…
|

Continuing to add detailed content to the page, we have essentially produced a structured content inventory of the page. This serves as a foundation for the rest of the prototype production. While wireframes force us to represent a page’s information architecture within a specific layout, this is pure structure and hierarchy, and, in my opinion, represents the true information architecture of a web page.

By defining the information architecture directly in the XHTML, we can also easily define accessibility-specific attributes, such as being cognizant of how users traversing the page with a screen reader will experience the page, and order content blocks accordingly. Additionally, we can more easily define elements often overlooked when working with wireframes, such as effective use ofLabel tags in forms.

If one were to view the structural layer in a browser, it would essentially look like an unstyled web page, and would not be interesting to look at. Just as building foundations are not known for their aesthetic qualities, but instead for the impact their quality has on the building they support, so too will the quality of the page structure significantly impact the overall quality of the web page. In fact, that absence of style is a key advantage of working with XHTML.

Evolving the Presentation Layer

With a page structure in place, we are ready to focus on how content will be presented. Looking back at our sketches, we’ve already explored some layout concepts, which we can begin to apply to our content structure. The way that look and feel is developed and applied will vary widely from team to team. While you may choose to do your initial exploration of look and feel with design comps, especially if you are also developing an overall brand, it’s worthwhile to redefine comps similarly to how we previously redefined wireframes. Just like wireframes are great as sketches, design comps are great for initial exploration of look and feel. But the practice of fully developing the presentation layer away from the actual technology, and then cutting it up and applying it wholesale to a web page is like wallpapering a façade onto a building. It’s impossible to be aware of all the dynamic aspects of a web page when working in static illustration software. However, when prototyping with XHTML, you can leverage the power of rendering your design in the same way that it will be seen by users, and incrementally evolve page presentation based on this immediate and rich feedback.

Issues that don’t easily reveal themselves when working in illustration software will often be obvious. This includes issues related to your design and the browser viewport, from the basic question of if the design should center itself in the browser window, to more advanced issues, such as how to design for different window sizes and browser resolutions. For example, for small windows sizes, is it okay if some content disappears out of view, or should the design adapt to the window size? When look and feel is designed solely with illustration software, questions like this are often unexplored to the detriment of user experience.

Adding Behavior: Unreinventing the Wheel

When prototyping with XHTML, you are designing within the larger ecosystem of the web, which effectively becomes your always-up-to-date UI library. Instead of laboring over the design of a detailed piece of functionality, start by letting Google inform you if anyone else has designed and built something similar, and then use that as the starting point for your solution. This can include anything from date-pickers to web widgets to whatever cutting edge UI idea was just created. Additionally, prototyping with XHTML makes it easy to incorporate and simulate Web 2.0 functionality, such as embedded widgets and syndication. If you don’t know JavaScript, or whatever technology is being used, you can collaborate with your developer on integrating the solution. Of course, you’re not going to find a solution for all your design needs online. In those cases, go back to sketching and collaborating with your team.

Iteration: Discovery, Evolution

The true power of prototyping really emerges during iteration This is when users can interact with your prototype. On a recent project, we sketched out a solution in which users could drag videos from a library onto a playlist. Looking at the static illustrations, it seemed a simple and elegant idea. But when users were able to interact with the solution, dragging and dropping video thumbnails, they found that it was a pretty tedious activity, especially for large numbers of videos. In other words, the prototype allowed us to discover a design problem that went unnoticed when looking at a wireframe.

And therein lies a core problem with using static artifacts to communicate interactive solutions; they effectively force the user to prototype the solution in their imagination, where all solutions seem to function in glorious perfection. With XHTML, we minimize the cognitive leap that users need to make, allowing them to instead experience and respond to something nearly identical to the actual solution.

Once users provide feedback and the team begins work on the next iteration, another measure of the quality of the prototyping methodology comes into play: how rapidly are you able to iterate? The longer an iteration takes, the less valuable your prototype. When prototyping with XHTML, iterations can be incredibly fast, first because the prototype can be easily presented to users, since it’s usually just a question of posting your files and sending out a URL. Second, because XHTML is text-based, iterations such as text changes or basic functional updates can often be completed in just a few minutes. More advanced design updates usually don’t take more than a few hours of actual production time.

How XHTML Can Double as a Specification

One of the most powerful aspects of XHTML is that it is self-describing. The same XHTML markup that tells a browser what to display can also double as a specification for a developer. For example…

^.

^.

In buzzword-speak, the practice we are applying here is writing semantically meaningful markup, which means we are selecting tags and naming our IDs and Classes such that they communicate the meaning and function of the content they enclose.

Annotations Visible Only to Those Who Care About Them

Another advantage of using XHTML as a specification is that IDs and Class names can double as annotation references. In other words, the annotations for the content block with the ID “account-options” would appear under the heading “Account Options” in your specification.

Rather than obscure and clutter a page design by placing annotation callouts on top of it, a common practice when using wireframes, that may confuse and distract non-technical viewers, references are only in the markup view for developers who are interested in seeing them. And since the XHTML file itself is so richly informative, the actual annotations written tend to only be short bullet points.

More Standards, Less Noise

One of the biggest problems with wireframes is the lack of a standardized notation. In other words, my wireframes certainly don’t look anything like your wireframes. This means that visual designers and developers who use wireframes are continually relearning how to interpret our work, leading to noise between author and reader. To compensate for the lack of a standard, we have to create highly detailed wireframes, with often lengthy annotations that explain what our wireframes mean and how elements in them work. These, in turn, are collected in large specification documents that usually are so labor-intensive they become impossible to maintain. When they are no longer kept up-to-date, the team stops trusting and relying on them as the design specification, which leads to all kinds of bad things happening.

In contrast to wireframes, XHTML is a standardized notation, anyone who knows XHTML can read your document. More importantly, it is a language spoken fluently by a key target audience of your design documents, the developers. And those who don’t know or care about XHTML can view the part they do care about, the page design, by opening the document in a browser.

Using a standardized notation also means you are not confined to specialized wireframing or prototyping software, but can use anything from a simple text editor to the range of tools available for editing XHTML files. Also the compact syntax of XHTML, particularly compared to verbose wireframe annotations, combined with the fact that you are just typing in a text file, leaving it to a browser to deal with the visuals, allows you to work rapidly and efficiently.

A Small Amount of Knowledge Goes a Long Way

If you’re new to XHTML, you’ll discover that a small amount of knowledge goes a long way. Spend just a few hours following any of the innumerable online tutorials and you’ll be writing XHTML markup in no time. (Two great places to start are htmldog.com or w3schools.com) Better yet, rather than invest time learning the UX tool du jour, you deepen your understanding of the technology that realizes your design.

Dividing and Conquering

The redefining of a wireframe from a blueprint to a sketch has a domino effect on who does what and when in evolving the page or application design. After a rough page design has been sketched out, rather than have one team member toil away in isolation, wireframing detailed representations of each page design, this model takes a divide-and-conquer approach. On the team I work with, I might produce an initial cut of the XHTML and some of the CSS, while other team members build on that, updating the XHTML, adding more advanced CSS, as well as JavaScript. If the team as a whole conceives of a solution, why not also have the team as a whole design it? In other words, rather than creating one person’s vision of a team’s solution, why not have the entire team contribute their particular expertise? When working with XHTML, we can use the tight integration of CSS and JavaScript to allow team members to contribute their dimension of the design via a set of integrated artifacts.

Where To Go From Here

This has, of course, been a mere whetting of the appetite for anyone interested in prototyping with XHTML. If you are interested in exploring the methodology further, particularly if you currently follow a traditional waterfall-oriented process, I recommend a many-small-steps approach. In other words, prototype the methodology itself, working with your team on a small project, and then building on that. If your experience is anything like mine, you’ll find it an incredibly powerful addition to your UX toolbox — a more effective way to straddle that proverbial divide between user experience and technology.