Quick Turnaround Usability Testing, Part II

Written by: Paul Nuschke

In Part I, I discussed how to make the first three steps of Quick Turnaround Usability Testing (QTUT)—Sales & Kickoff, Recruitment, and Preparation—as short and efficient as possible. In Part II, I discuss the final two steps: Testing and Analysis & Reporting.

Steps in the QTUT Process

  • Step 1: Sales & Kickoff
  • Step 2: Recruitment
  • Step 3: Preparation
  • Step 4: Testing
  • Step 5: Analysis & Reporting

Testing

It’s testing day. You have successfully recruited enough participants for the first day, but you feel a bit of panic as you make the finishing touches before the first participant arrives. You have a rough but solid test script. You have five attentive stakeholders in the observation room ready to begin taking notes. Now you need to execute the test and you need to compile results as you go along.

Up to this point, the lack of time you had to plan and to refine your method creates a bit of a panic as you begin the testing phase. Often, we are working on the script until the very last second, incorporating changes from the stakeholders that they hand off when they arrive at our testing facility.

Early on the test day, I print out a screenshot of every important page and component (e.g., the primary navigation). I number these screenshots and then tape them above a large whiteboard that we keep in an “idea” room that is adjacent to our usability lab’s observation room. We use the whiteboard to keep track of issues and metrics across participants.

After you finish each participant session, immediately note changes that you need to make to your test script or the application. Then go talk with your stakeholders about the results. Here, the time that you have budgeted between sessions for discussion really pays off. If your stakeholders have been watching and taking notes, they are likely already talking about the results. They may also be already talking about potential fixes.

The whiteboard can be useful for focusing the discussion on issues. It’s often useful to set ground rules for the discussion. First, the discussion should focus on results and not solutions. It is important to manage your stakeholders by telling them to be patient and to let the results play out over several people before drawing conclusions. Second, as the person facilitating the study, you should lead the discussion on each topic by first summarizing your notes on the whiteboard. After we summarize a page, we ask for any additional feedback from the stakeholders and then quickly move on to the next page. Occasionally, we need to remind the stakeholders that since we have limited time between participants, we cannot dwell on any one finding.

You may think that stakeholders will not have much valuable feedback to add, but we have found that they often see things that we don’t because of their knowledge of the history of the application. For example, one client had just made a political decision to change a button label. Since the participants understood the new button name, we didn’t think to list it as a finding. But for the stakeholders, it was helpful to track it and mention it in the report.

After the first participant’s session, you will likely realize that some of your tasks and questions were not worded correctly. Through the whiteboard session, you may think of additional questions to ask participants, or you might even want to add, delete, or change tasks. Do not rely on your memory to do any of this. Instead, make the changes to your script and print out a new copy for the next participant.

At the end of your first day of testing, you should have a board full of findings, including some task completion data. If you test five participants per day, by the fifth participant trends are becoming clear. You may decide to stop a task because you already know the issues, which allows you to test other tasks that you couldn’t fit in. Discuss this at the end of the day so that you can make the necessary edits to your test script before you leave for the day.

In addition, if your stakeholders are already discussing potential fixes while you facilitate the test, it is important for you to be a part of that discussion. In their eagerness to fix things, our stakeholders occasionally solve the problem in such as way that a larger problem is created.

If you are in a fast-paced development environment, where developers are staying until 10 p.m. at night to make changes, your stakeholders may want to change things that night. Because of the potential to affect your testing, you should be very careful to attempt only easy fixes. You do not want to rework the navigation or radically alter a task flow overnight. However, you may want to change a button’s behavior or the text in a label.

As sessions progress, your whiteboard may start to fill up. Usually it is easy to condense the notes. We use a Post-It note for each issue and then start writing participant numbers on the note for each person that experiences the same issue (see Figure 1). We take pictures of the whiteboard throughout the process, prune non-issues, and type up results for tasks and questions that we have stopped using.

whiteboard

Figure 1: Whiteboard full of findings after two days of testing. Post-It notes contain reoccurring issues.

Finally, as you reach the last few sessions, increase the level of detail about potential recommendations. If possible, test these ideas with your participants. Get a sense from your stakeholders about what recommendations are feasible and cost effective on a short timeline, and which require long-term attention.

Testing Tips

  • Keep the testing as simple as possible—for QTUT avoid technology such as eyetracking and anything else that makes your job more difficult or confusing.
  • For changes to your tasks and questions, modify the test script and do not rely on your ability to remember the changes.
  • Be willing to change questions and tasks. It is better to find and to fix the issues with your testing script early on.
  • Remind your stakeholders that because of the aggressive timeline your test script is not perfect, so you may have to change tasks and questions on the fly for the first couple of participants.
  • Keep a whiteboard in the observation room and use it to keep track of and to discuss problems with your stakeholders after each session.
  • At the end of the day, summarize the results and discuss potential recommendations. Consider making a list of the most important findings from the day.
  • Expect long work hours.
  • Don’t duplicate effort by typing up the results before you have tested all of the participants; write up the results only when you have stopped testing a task or asking a question.

Analysis & Reporting

You have made it through two days of testing. The stakeholders have left and now you have to create a report that clearly communicates the issues and your recommended fixes.

The beauty of the whiteboard method is that your report becomes simply a summary of what you have already written on the whiteboard, including completion metrics, findings, and recommendations that have been vetted by key stakeholders.

Although the people who attended the sessions may understand the issues, you still need to translate them from shorthand whiteboard notations to a format that a wider audience can read. We use a presentation-style report because it is much faster to create than a report with lots of text.

Because time is important to the stakeholders, we classify recommendations as short-term and long-term fixes. Short-term fixes can be finished in a few days, or before the application is released. Long-term fixes require more time than your clients have, but they are severe enough that they need to be addressed eventually.

Our QTUT ends when we present the results to the stakeholders, which often includes several interested people who were not able to attend the sessions. We present our findings in person so that the stakeholders feel more comfortable asking questions about the test and so they can get immediate clarification on findings that have been inadequately described.

Analysis and Reporting Tips

  • Use a presentation-style report format that does not require long text passages.
  • If you will be presenting results to stakeholders who did not attend the sessions, give the necessary context to your findings.
  • If you change recommendations that you discussed with your stakeholders, let them know in advance so that they do not feel undermined.
  • Classify your recommendations in terms of the expected timeline for the fix.
  • Take pictures of your whiteboard in order to preserve your notes quickly and safely, enabling you to write the report from another location.

Prototyping with XHTML

Written by: Anders Ramsay

Illustrations by Leah Buley

If you design user experiences for standards-based websites and applications (i.e. those built with XHTML, CSS, and JavaScript), there are several great reasons for adding XHTML prototyping to your UX tool kit. Perhaps you’ve found that traditional wireframes just aren’t sufficient and are looking for more powerful ways to explore and communicate design solutions. Perhaps your current practice is based on the traditional waterfall model (i.e. first creating wireframes, which are handed off to creative, who hand off comps to tech, and so forth), and you want to explore more contemporary methodologies, such as agile and iterative development. Regardless, a great way to embark on that journey is to start prototyping with XHTML.

So what does it mean to prototype with XHTML? Essentially, it’s the process of using the XHTML itself, and related technologies, to evolve and define your design solution. And what does an XHTML prototype look like? While, as we’ll see, that depends on where you are in your prototyping process, an XHTML prototype generally looks like any other web page built with XHTML, with some links or features perhaps being non-functional. In other words, anything you can build with XHTML, from consumer websites to enterprise applications, you can also prototype with XHTML. As we’ll see, there are numerous advantages to this approach compared to designing with wireframes or other prototyping tools.

An Iterative Process

While prototyping with XHTML isn’t tied to a specific design process, iterative development seems to effectively leverage its strengths. There are many reasons for this, but perhaps the most significant is that in both cases the prototype, and later the application itself, doubles as a specification. We’ll explore what that means in a bit, but first let’s walk through a suggested process for prototyping with XHTML.Let us start with an overview of the larger design process:

In this (iterative) methodology, rather than design the entire application before starting to build it, one designs and builds a unit of the application and then uses what has been built to inform and serve as a starting point for other application units. As with other design methods, the design work begins with sketching, which plays a particularly important role relative to prototyping.

Sketching: A Freeform Question

The term ‘sketching’ refers here to any freeform exploration unconstrained by a specific technology. This includes production of wireframes, which in this model are reframed, as it were, from specification artifact to refined sketch. When thought of, and presented to stakeholders, as sketches, its more natural to discard your wireframes once the design has evolved beyond them. This is usually after a prototype equivalent has been produced. With the design team I work with, we’ve found that when prototyping with XHTML, wireframes often became superfluous, and it’s more effective to go directly from sketch to prototype.

Prototyping: A Concrete Response

Prototyping has a counterpoint relationship to sketching. To paraphrase Bill Buxton, while sketches ask a question—“Is this a good design idea?”— prototypes provide a response. By making the idea manifest, prototypes force upon it the concrete realities and user experience idiosyncrasies of the actual production technology and offer a crisp verdict on the quality of what you dreamed up in drawings.

The Prototype/Build Relationship

When prototyping with XHTML, especially in an iterative model, the build and prototype become very intertwined. If you’re prototyping a new application or product, the XHTML prototype is essentially a rough draft of the actual application. However, when updating the design of an existing application, the production version can serve as the starting point for the prototype of the new solution.

Three Integrated Layers: Structure, Behavior, Foundation

The model for XHTML prototyping is based on the best practices model for actual site production: start by setting the foundation with XHTML, add a presentation layer with CSS, follow it by a behavioral layer using JavaScript then iterate.

Let’s start by looking at the structural layer.

Structure: Set the Page Foundation

The first step in production of the XHTML prototype is to create a structural foundation. Similar to how we create a wireframe, we start by representing the main content areas on the page, except we do so with text-based XHTML markup.

| If our sketch or wireframe looks like this | …our XHTML might look like this: |
|^. |^.

My Account

Account options

Account details

Account Help


(We’re only displaying the relevant snippet of the XHTML here.)
|

Next, we add detailed content elements that have been defined, using the XHTML structure appropriate for the corresponding content.

| For example, if our detailed sketch looks like this | …we’d represent the list of account help topics as an unordered list (i.e. use the ul tag): |
|^. |^.


|

Continuing to add detailed content to the page, we have essentially produced a structured content inventory of the page. This serves as a foundation for the rest of the prototype production. While wireframes force us to represent a page’s information architecture within a specific layout, this is pure structure and hierarchy, and, in my opinion, represents the true information architecture of a web page.

By defining the information architecture directly in the XHTML, we can also easily define accessibility-specific attributes, such as being cognizant of how users traversing the page with a screen reader will experience the page, and order content blocks accordingly. Additionally, we can more easily define elements often overlooked when working with wireframes, such as effective use ofLabel tags in forms.

If one were to view the structural layer in a browser, it would essentially look like an unstyled web page, and would not be interesting to look at. Just as building foundations are not known for their aesthetic qualities, but instead for the impact their quality has on the building they support, so too will the quality of the page structure significantly impact the overall quality of the web page. In fact, that absence of style is a key advantage of working with XHTML.

Evolving the Presentation Layer

With a page structure in place, we are ready to focus on how content will be presented. Looking back at our sketches, we’ve already explored some layout concepts, which we can begin to apply to our content structure. The way that look and feel is developed and applied will vary widely from team to team. While you may choose to do your initial exploration of look and feel with design comps, especially if you are also developing an overall brand, it’s worthwhile to redefine comps similarly to how we previously redefined wireframes. Just like wireframes are great as sketches, design comps are great for initial exploration of look and feel. But the practice of fully developing the presentation layer away from the actual technology, and then cutting it up and applying it wholesale to a web page is like wallpapering a façade onto a building. It’s impossible to be aware of all the dynamic aspects of a web page when working in static illustration software. However, when prototyping with XHTML, you can leverage the power of rendering your design in the same way that it will be seen by users, and incrementally evolve page presentation based on this immediate and rich feedback.

Issues that don’t easily reveal themselves when working in illustration software will often be obvious. This includes issues related to your design and the browser viewport, from the basic question of if the design should center itself in the browser window, to more advanced issues, such as how to design for different window sizes and browser resolutions. For example, for small windows sizes, is it okay if some content disappears out of view, or should the design adapt to the window size? When look and feel is designed solely with illustration software, questions like this are often unexplored to the detriment of user experience.

Adding Behavior: Unreinventing the Wheel

When prototyping with XHTML, you are designing within the larger ecosystem of the web, which effectively becomes your always-up-to-date UI library. Instead of laboring over the design of a detailed piece of functionality, start by letting Google inform you if anyone else has designed and built something similar, and then use that as the starting point for your solution. This can include anything from date-pickers to web widgets to whatever cutting edge UI idea was just created. Additionally, prototyping with XHTML makes it easy to incorporate and simulate Web 2.0 functionality, such as embedded widgets and syndication. If you don’t know JavaScript, or whatever technology is being used, you can collaborate with your developer on integrating the solution. Of course, you’re not going to find a solution for all your design needs online. In those cases, go back to sketching and collaborating with your team.

Iteration: Discovery, Evolution

The true power of prototyping really emerges during iteration This is when users can interact with your prototype. On a recent project, we sketched out a solution in which users could drag videos from a library onto a playlist. Looking at the static illustrations, it seemed a simple and elegant idea. But when users were able to interact with the solution, dragging and dropping video thumbnails, they found that it was a pretty tedious activity, especially for large numbers of videos. In other words, the prototype allowed us to discover a design problem that went unnoticed when looking at a wireframe.

And therein lies a core problem with using static artifacts to communicate interactive solutions; they effectively force the user to prototype the solution in their imagination, where all solutions seem to function in glorious perfection. With XHTML, we minimize the cognitive leap that users need to make, allowing them to instead experience and respond to something nearly identical to the actual solution.

Once users provide feedback and the team begins work on the next iteration, another measure of the quality of the prototyping methodology comes into play: how rapidly are you able to iterate? The longer an iteration takes, the less valuable your prototype. When prototyping with XHTML, iterations can be incredibly fast, first because the prototype can be easily presented to users, since it’s usually just a question of posting your files and sending out a URL. Second, because XHTML is text-based, iterations such as text changes or basic functional updates can often be completed in just a few minutes. More advanced design updates usually don’t take more than a few hours of actual production time.

How XHTML Can Double as a Specification

One of the most powerful aspects of XHTML is that it is self-describing. The same XHTML markup that tells a browser what to display can also double as a specification for a developer. For example…

^.

^.

This markup Would be read as
^.

“This is the start of the header content block.”
   
^. XYZ Application “display the product name, which should link to the homepage”
   
^.

Signed in as Jane Smith (Editor)

;

^. “display user information, including the user’s role (or set of application permissions”

In buzzword-speak, the practice we are applying here is writing semantically meaningful markup, which means we are selecting tags and naming our IDs and Classes such that they communicate the meaning and function of the content they enclose.

Annotations Visible Only to Those Who Care About Them

Another advantage of using XHTML as a specification is that IDs and Class names can double as annotation references. In other words, the annotations for the content block with the ID “account-options” would appear under the heading “Account Options” in your specification.

Rather than obscure and clutter a page design by placing annotation callouts on top of it, a common practice when using wireframes, that may confuse and distract non-technical viewers, references are only in the markup view for developers who are interested in seeing them. And since the XHTML file itself is so richly informative, the actual annotations written tend to only be short bullet points.

More Standards, Less Noise

One of the biggest problems with wireframes is the lack of a standardized notation. In other words, my wireframes certainly don’t look anything like your wireframes. This means that visual designers and developers who use wireframes are continually relearning how to interpret our work, leading to noise between author and reader. To compensate for the lack of a standard, we have to create highly detailed wireframes, with often lengthy annotations that explain what our wireframes mean and how elements in them work. These, in turn, are collected in large specification documents that usually are so labor-intensive they become impossible to maintain. When they are no longer kept up-to-date, the team stops trusting and relying on them as the design specification, which leads to all kinds of bad things happening.

In contrast to wireframes, XHTML is a standardized notation, anyone who knows XHTML can read your document. More importantly, it is a language spoken fluently by a key target audience of your design documents, the developers. And those who don’t know or care about XHTML can view the part they do care about, the page design, by opening the document in a browser.

Using a standardized notation also means you are not confined to specialized wireframing or prototyping software, but can use anything from a simple text editor to the range of tools available for editing XHTML files. Also the compact syntax of XHTML, particularly compared to verbose wireframe annotations, combined with the fact that you are just typing in a text file, leaving it to a browser to deal with the visuals, allows you to work rapidly and efficiently.

A Small Amount of Knowledge Goes a Long Way

If you’re new to XHTML, you’ll discover that a small amount of knowledge goes a long way. Spend just a few hours following any of the innumerable online tutorials and you’ll be writing XHTML markup in no time. (Two great places to start are htmldog.com or w3schools.com) Better yet, rather than invest time learning the UX tool du jour, you deepen your understanding of the technology that realizes your design.

Dividing and Conquering

The redefining of a wireframe from a blueprint to a sketch has a domino effect on who does what and when in evolving the page or application design. After a rough page design has been sketched out, rather than have one team member toil away in isolation, wireframing detailed representations of each page design, this model takes a divide-and-conquer approach. On the team I work with, I might produce an initial cut of the XHTML and some of the CSS, while other team members build on that, updating the XHTML, adding more advanced CSS, as well as JavaScript. If the team as a whole conceives of a solution, why not also have the team as a whole design it? In other words, rather than creating one person’s vision of a team’s solution, why not have the entire team contribute their particular expertise? When working with XHTML, we can use the tight integration of CSS and JavaScript to allow team members to contribute their dimension of the design via a set of integrated artifacts.

Where To Go From Here

This has, of course, been a mere whetting of the appetite for anyone interested in prototyping with XHTML. If you are interested in exploring the methodology further, particularly if you currently follow a traditional waterfall-oriented process, I recommend a many-small-steps approach. In other words, prototype the methodology itself, working with your team on a small project, and then building on that. If your experience is anything like mine, you’ll find it an incredibly powerful addition to your UX toolbox — a more effective way to straddle that proverbial divide between user experience and technology.

Quick Turnaround Usability Testing

Written by: Paul Nuschke

It starts with any number of scenarios: Design and development have taken too long to produce a prototype, you need to release in three weeks, and you suspect there may be design flaws. You are trying to incorporate usability testing into an Agile development process. Or maybe you simply want to pare down your process to make it shorter and less expensive.

Completing usability testing quickly is a challenge anywhere but especially in consultancies, which have to overcome additional challenges, such as learning a new application. To assure success on these projects, I’ve developed a quick turnaround usability testing methodology (QTUT) that minimizes the time needed to complete testing. In Part I of this article, I discuss how to make the first three steps of QTUT—Sales & Kickoff, Recruitment, and Preparation—as short and efficient as possible. In Part II, I will discuss the final two steps: Testing and Analysis & Reporting.

Steps in the QTUT Process

Step 1: Sales & Kickoff
Step 2: Recruitment
Step 3: Preparation
Step 4: Testing
Step 5: Analysis & Reporting

Sales & Kickoff

A new client or a group within your company has approached you about doing usability testing. They need the results next week, which works out to six business days from today. What should you do?

Before a project kicks off, we typically have a number of discussions with the client to understand their goals and deadlines for our engagement. However, in QTUT, each day spent in the sales and kickoff process takes a day away from testing and analysis. To help our sales team conduct initial discussions efficiently, we’ve prepared a one-page guide outlining the do’s and don’ts of QTUT.

One critical “do” for our sales team is that they should discuss the method with the client and immediately set deadlines for our testing results. Starting with the final due date, an experienced tester can work backwards to determine the dates for testing, beginning recruitment, and finishing the test plan.

It’s also important to review expectations with the client. For example, current clients typically expect a certain level of quality in our deliverables. When we compress a week of work into one day, delivering a perfect document or presentation is impossible, so simply reviewing the timeline and discussing how we plan to shorten the process is extremely helpful.

Since the project must start very quickly, our sales staff and project team use part of the kickoff as a working session. During this working session, we develop specific goals, learn what types of results will be helpful, develop an initial list of testing tasks, and learn about the users that we need to test the application. We also compile a list of what we need from the client. Depending on the project, we may need:

  • Information about users for both recruiting and for writing the test script. For example, do people regularly use the application or do they only use it occasionally?
  • Training materials for the application and a subject matter expert who can answer questions.
  • Background information, feedback, or previous testing materials that give context to the current design or may help us to write screeners and test scripts.
  • Access to a stable application, especially when it is under development.

Sales & Kickoff Tips

  • Create a “Do’s & Don’ts” guide to help team members through the process.
  • Avoid clients who are rigid, who prefer not to participate in the process, or who want long reports.
  • Use the kickoff as a working session to learn about the participants and potential tasks.
  • Have a facilitator available with the domain knowledge needed to quickly learn the application.
  • Refine your methodology before you have a project.

Recruitment

Before starting the project, you called several of your favorite recruiters to see if they could meet the deadline. You’ve gotten information about participants during the kickoff meeting, and now you simply need to write a screener in less than two hours!

Recruiting quality participants is a big challenge. There are two key components to recruiting: writing the screener and scheduling the participants. The screener is a list of questions used to filter out participants who are not appropriate for your study. Scheduling involves calling potential recruits, asking the questions on the screener, and scheduling participants who qualify and are available.

Because scheduling is time-consuming, it’s critical to write the screener immediately; in fact, we often start and finish the screener immediately after the kickoff meeting. That’s why it’s important to get information about participants during this meeting.

The level of effort required to recruit depends upon whether you’ve recruited similar participants before and whether you handle the scheduling internally or externally. If you’ve recruited similar participants before, you can reuse the same screeners almost verbatim. To facilitate recruitment for new participant types, we’ve built a list of standard questions so that, at worst, we need to write only a few new questions. In addition, we use a screener template that has our facility procedures, location, and contact numbers; the facilitator need only fill in the specific test questions.

External schedulers. We typically use external schedulers, which is particularly helpful for QTUT, because it takes burden off the facilitator. I highly recommend contacting your external recruiters before agreeing to do a study to see if they can meet the timeline that you need. If at least one of your external recruiters cannot meet your timeline, do not hesitate to ask your client for assistance before agreeing to the study.

Internal schedulers. If you do your own scheduling for QTUT, you may have a list of participant leads in house, so writing a screener isn’t always necessary. However, depending on your timeline and your success rate when calling, it’s often necessary to ask your client or a co-worker to make the calls for you.

You might be tempted to schedule testing sessions very closely together in order to complete more sessions per day. However, we find it more efficient to leave time between sessions in order to quickly debrief and begin analyzing what we observed during the previous session (I’ll discuss this analysis method in more detail in Part II). You may also need the extra time to modify the testing script as needed before the next session.

Recruitment Tips

  • Start recruiting as soon as possible.
  • Create a screener template so that you only need to write the questions.
  • Reuse questions from past screeners.
  • If you do your own recruiting, ask a coworker to make the calls for you.
  • Budget more money than usual when using third party recruiting firms.
  • Screen carefully but avoid particularly restrictive screeners.
  • Schedule backup participants (“floaters”) to cover multiple time-slots in case a participant fails to arrive.
  • Leave time between participants to summarize results and to change the tasks as needed.

Preparation

You’ve started recruiting. You are familiar with the application type but you need to learn this particular application. You need to write the tasks and questions for your test script and you need to clear them with your client.

With recruiting underway, you can turn your attention to learning the application and writing the test script.

One critical note in our sales guide is to avoid aggressive timelines for applications in unfamiliar domains or those which require extensive training. Even if you’ve followed this advice, you will still likely need to learn certain aspects of the application.

It may seem obvious, but you cannot learn an application if you do not have access to it. Often the clients who come to us are still developing the application, so be sure to schedule adequate access to a stable version.

After you’ve had a chance to learn the application, writing tasks and questions for QTUT is similar to a normal study. We rely on client stakeholders more than usual because they help to prioritize testing needs and to identify key tasks. In QTUT, most clients are releasing soon so they’re more interested in finding easy-fix problems than in statistical data or qualitative feedback. Consequently, we tend to focus our effort on creating tasks vs. writing a lot of questions. You also need to consider what you can realistically summarize quickly. For example, if it takes two hours to compile error rates, then eliminate error tracking from your test plan.

We run pilot tests as early as possible because they help us more rapidly iterate and improve the script. Keep in mind that you probably will not have perfected your script when your first participant arrives, so you may need to modify slightly the tasks and questions between sessions.

Preparation Tips

  • Avoid gathering data that takes a lot of time to compile and analyze.
  • Use open-ended questions such as “Tell me about what you did yesterday at work” and allow time for discussion with the participant in order to learn about them during their session.
  • For tasks that require more information that you currently know about participants, use a series of questions to build more realistic tasks.
  • Use Likert-style scales to get data if you need it, and rely less on task completion data (as your tasks may often change).

Next time: Part II

In Part II, I will discuss how to facilitate the sessions, and I’ll describe a novel approach to analyzing and reporting the results.

Information Architecture for Audio: Doing It Right

Written by: Jens Jacobsen

Content today is increasingly delivered by audio both online and in the real world. We have radio shows and newscasts, and in recent years, podcasts, audio books and navigation/car assistance systems have been added to the field. Audio is more emotional, as sound effects and acoustic atmosphere enhance content to help deliver its messages. It also affords users the opportunity to interact with content while their hands and eyes are busy (i.e. when doing physical work, driving, walking, etc).

However, the inclusion of audio often results in usability issues that make it difficult for users to access and understand content. That is why we need new tools to organize linear content like audio. Luckily, a wide range of techniques employed in information architecture, journalism, usability engineering and interface design are available. All that’s required is the knowledge to combine them effectively. This article presents a practical framework for designing and implementing audio-based content, such as podcasts.

“There is no reason to over-estimate the importance of writing and thereby under-estimate other technologies of information processing.” Harald Haarmann in History of Writing.

The Problem with Audio

When using audio today, we face challenges similar to those of written text about a decade ago. During this time, information was being transferred from hand-held documents to the computer screen, without being optimized for the new online medium. Now the same mistakes are being repeated with audio. Existing text is read by a narrator, or worse, the text is speech-synthesized by a computer. Audio doesn’t function the same way as written text, so its execution is often poor. The main difference between printed text, be it on paper or on the computer screen, is that audio is linear. You can only consume it in a linear fashion and you have to listen to it at a given speed.


Figure 1: Part of the “Web Trend Map by Information Architects, Japan

For example, Figure 1 shows part of the famous “Web Trend Map by Information Architects Japan”:http://informationarchitects.jp/web-trend-map-2008-beta/. It’s an excellent example of how information can be displayed in a two-dimensional space. It’s not possible to use one-dimensional spoken text in the same way. When accessing audio, users have no idea how long the segment will last, unless this information is provided by the interface or the narrator states it at the beginning of the segment. Users only have a vague idea of where they are within the narration. If you don’t have any visual hints it’s difficult to determine how much time is left and what topics are going to be discussed. Finding specific content by rewinding or forwarding is difficult. In contrast, finding the next subsection within a text document is very simple. You can easily find a particular word on a page by scanning it or by using your browser’s find function.

Best Practices for Audio

When beginning an audio-related project, ask yourself whether audio is the right medium for your message. In some cases, text is a better choice and in other cases it’s video. Don’t use audio just because you can. If you are certain audio is the best choice, there are several fields to help inform how you implement it. The most important professions we can learn from include:

  • Information Architecture
  • Journalism
  • Educational Psychology
  • Usability Engineering
  • Interface Design

Information Architecture for Audio

The principles of information architecture are exactly what you need to create usable audio. Your approach to creating audio should be similar to developing a large website. In both scenarios you don’t want the user to get lost or overwhelmed by content. For any informational audio that is longer than a few minutes, follow these guidelines:

  • State the Length: Typically the user has no way to assess the length of the audio segment. Sometimes the length is provided by the interface, but not always accessible. For example, if you listen to a podcast with your MP3 player, it might be in your pocket so you don’t see the time and duration displayed on the device.

  • Give an Overview of the Structure: Informing users how the audio is structured helps them find the content they’re looking for. It also gives them the option to directly locate the information they’re most interested in.

  • Introduce the Topic: An introduction helps set the mood and prepares the listener for the content to come. In printed text, such an introduction might seem hackneyed, but with audio it’s good practice to describe a situation the listener knows from everyday life. It’s better not to jump right into the topic, but instead provide some information about it.

  • Provide Orientation from Time to Time: If the audio is longer than a few minutes, help the user form a mental model of the content by repeating its structure from time to time. Tell the user where they are within the content and give an overview of up-coming topics. For longer audio pieces, consider giving users the option to skip sections/chapters via the interface or offer content in segments.

Journalism for Audio

Radio has been around for more than a century, and most of the best practices from radio journalism are ideal for creating usable audio. Here are some of the most important points.

  • Keep it Short: Ideally, audio narration should be shorter than printed text covering the same subject. If you have three pages of printed text, don’t write three pages of text for the narrator. Since users are unable to easily scan audio content and must listen at the narrator’s speed, concentrate on the most important content.
  • Moreover, the sentences and the individual words should be kept short. It is much more difficult to comprehend a long, complicated sentence read aloud than to read it in print.

  • Repeat Often: Repetition is something you usually try to avoid with written text. With audio, however, it helps to get your point across if the most important facts are summarized and repeated. The key is a summary at the end of the audio. It’s also a good idea to repeat the main subjects or themes rather than referring to them by pronouns or synonyms. The text might seem strange when you read it, but as soon as you hear it, you will realize the audio is easier to follow.

  • Use Mental Pictures: Good journalistic audio sparks the listener’s imagination. It not only makes the piece more entertaining, it also helps the user understand and remember it. Try to create pictures in the listener’s mind. Describe what they might see and feel if they were in your place. For example, let them hear the sounds of the location where your story is set.

  • Take Advantage of the Possibilities: Different styles of speech, tone, speed and dialects can be used to create memorable audio. When the language is too formal, you lose credibility and narration is more difficult to understand.

  • Don’t Overuse the Thesaurus: This is another piece of advice from radio journalism. When you use overuse synonyms, you decrease comprehension. The listener has to decipher the synonyms while the narrator continues talking, so he might not understand some of the text. For example, when referencing Japan, avoid using the terms “Nippon” or “Land of the Rising Sun.”

Here is an example of how to structure an audio sequence:

1. Greeting
2. Introduction (i.e. audio length, what subjects/topics will be covered and how the user can interact)
3. First section of content
4. Describe the structure (i.e. summaries, repetition, overview and acoustic bumpers)
5. Repeat Steps 3 and 4 until all content is delivered
6. Conclusion (i.e. summary or what action users can take next)
7. Farewell

This structure is derived from the typical sequence of a radio show and has been successfully adopted by many podcasters.

Educational Psychology for Audio

Much research has been conducted on reader comprehension and written text, notably the work of Norbert Groeben from 1972 onward. Most of the results show that the techniques from information architecture and radio journalism cited above are also valuable for creating accessible content to be used in an academic setting.

  • Keep Short-term Memory In Mind: It is important to write short sentences and to repeat words rather than using synonyms.
  • Design Audio Content for Different Reading Speeds: Research shows that reading speed varies by individual, depending on age, familiarity with the subject, education and other factors, so it’s important to adapt the complexity and the reading speed of the narrator to your intended audience.

Usability Engineering for Audio

Because audio differs, some of the established techniques used in web development cannot be applied audio. Wireframes, card sorting exercises or eye tracking can be used to evaluate information architecture or interface design, but these techniques do not work for developing and testing audio content. Still, we can borrow from usability engineering when including audio:

  • Design for the Target Audience: It’s still uncommon to apply the techniques of user-centered design to audio, but do convince your design team to produce content for the users, not its creators.

  • Create Personas: Personas are the perfect method for representing your target audience, so use them.

  • Create Scenarios: Usage scenarios are a technique you can successfully apply when creating audio content. It is crucial to understand the user’s:
    • Environment (i.e. quiet or noisy)
    • Access Possibilities (i.e. Do users need to rely on their eyesight or hands right now? When driving or working out eyes and hands are mostly occupied.)
    • Mood (i.e. passive/reclined or active/leaning forward)
    • Expectations (i.e. entertainment or information)
    • Experience (with interface as well as with content)
  • Test With Users: If possible, test early versions with selected users from the target audience. Usability testing in a research lab is best, but informal tests are a good start.

  • Conduct Log File Analysis: Do your statistics. Look at which files are most frequently downloaded (and the least). Correlate the files with their content and then produce more of the successful content types.

  • Consider Users’ Goals & Tasks: Figure 2 shows that audio delivered over the web has a different level of interactivity than, say, just listening to the radio. Apart from listening on demand, users can forward or skip through audio. They may also be looking and interacting with other materials at the same time they are listening.

Fig1
Figure 2: Depending on the context, the amount of interactivity varies.

What’s more, knowing user’s expectations is crucial to creating the appropriate content (Figure 2). With audio this is more important because it is difficult to skip irrelevant information.

Interface Design for Audio

Finally, give careful consideration to the interface that provides access to and control of audio content. Again, the well-known principles of interface design apply. In general, give the user as many hints as possible about what to expect from the audio—before he even starts listening.

  • Provide a concise description of the content.
  • When linking, make it clear that an audio file is linked.
  • Explain how to locate content within the audio piece, if possible.
  • Include metadata (i.e. ID3 tags in MP3 files that are shown within the playback software, as well as on portable devices).


Figure 3: Podcast page of “The New Yorker website”:http://www.newyorker.com/online/2008/08/11/080811on_audio_grann?xrail

Figure 3 shows the podcast page of the The New Yorker magazine. Much of the information on how to subscribe to the podcast and how to download the audio file is in text. Some short links at the end of the paragraph might work better.


Figure 4: Metadata in iTunes

Above is a good example of metadata displayed via iTunes (Figure 4). Note the long description; it’s concise but not suitable for scanning.

Conclusion

Creating usable audio is not difficult when you follow a few simple rules. These mostly stem from the creation of usable content in the form of text. Information architecture, journalism, educational psychology, usability engineering and interface design provide plentiful tips for doing so. Most of the methods used in these fields can be applied to the creation of audio. To summarize, the main guidelines for usable audio are:

  • Write with your audience in mind.
  • Structure your content by providing an overview at the beginning and giving an introduction to longer audio pieces. Be sure to include a summary at the end.
  • Follow the rules of radio journalism for creating easily understandable narratives.
  • Rely on a familiar interface or put your design in front of users if you digress from a familiar practice.

If you follow these tips you will be able to create audio that is easily accessible, engaging and helps to communicate your message, not only intellectually but also emotionally. After all, emotional quality is one audio’s main advantages over text.