“How many sides do you see?”

“One,” I said.

He pulled the box towards his chest and turned it so one corner faced me.

“Now how many do you see?”

“Now I see three sides.”

He stepped back and extended the box, one corner towards him and one towards me.

“You and I together can see six sides of this box,” he told me.

—Eber Hampton (2002) The Circle Unfolds, p. 41–42

Creating a Learning Resource with Aboriginal Students

My graduate school thesis project was to create a learning resource for an Aboriginal literature course for Aboriginal students at the University of Alberta. This effort was an interesting challenge since it involved me—a non-Aboriginal designer—trying to design for Aboriginal students from multiple cultural backgrounds.

While navigating the expected cross-cultural design issues, I met some wonderful people and learned a great deal about the importance of letting those with whom you design guide the research and design process.

This daunting task left me more than a bit intimidated at the end of the day. However, I felt from the outset that if I took the time to get to know my design partners and if I took the time to examine how they could guide me, somehow we could be successful.

The resource was to be a website representing Aboriginal literatures from multiple cultures to enrich the learning experience for first-year Aboriginal students. My final project was a prototype design that I then tested in paper form.

The learning resource is based on materials used in English 114: Aboriginal Literature and Culture, a compulsory, full-year Aboriginal Canadian literature course in the Transition Year Program (TYP) at the University of Alberta. The TYP program is for Aboriginal undergraduate students and is designed to prepare them for admission into one of eight faculties at the University. Some students have extensive contact with their Aboriginal communities and heritage; some have little or no contact. Most students are from Canada, but there are students in the program from all over North America.

Extra tutorial support is included in every class, to aid in the transition to post-secondary study. The material used in English 114 comes from Aboriginal authors from diverse communities in North America. Many different forms of literature are studied, including poetry, prose, and oral literatures.

Resource objectives

There were no specific learning outcomes created for the resource. It was simply meant to help students explore and appreciate the literature—particularly oral literatures—through video, audio, images, and text. Both teachers and students felt that commenting on and discussing works on the blog portion of the site was important. Teachers felt that the materials and the blog would help students become more engaged with the material.

“I found I wasn’t comfortable, until the end, talking to the professor because for me, the work I was doing was so personal I was afraid of criticism—to think I didn’t know what I was doing. I was kind of shy.”—Jesse, student

I spent many months completing a literature review. I didn’t find much research on visual design practice for multiple cultures and genres. I interviewed students, all the instructors, the teaching assistant, and many other academics and professionals working in Aboriginal studies. I also audited courses in Aboriginal literature and attended as many classes as possible for the course for which I would design the website.

There were some pretty clear needs identified for the course that the instructors, students, and I felt might be helped by the use of digital media. One of the biggest benefits centered on the importance of oral literatures. Audio and video files would clearly benefit the study of oral literatures when storytellers could not be brought into the classroom. Other benefits were to:

• Help students refine independent research skills
• Encourage communication between students and between instructors and students by creating a forum
• Show media in class using the website
• Give students the opportunity to access materials outside of class
• Provide background materials to students who are unfamiliar with the material, benefiting students who hesitate to ask questions
• Provide a more “low stakes” context for students to explore concepts raised in class
• Allow students to seek out information
• on their own and explore issues they might be too embarrassed to raise in class
• Increase continuity between course sections and allow instructors to share resources
• Allow instructors to check in more often on students’ progress
• Give reticent students the chance to express opinions in the possibly less-intimidating forum
• Use multimedia to demonstrate the “webbed” nature of the texts by allowing students to see how issues intersect

Some pseudonymous student comments about the course:

“I think there wasn’t enough visual stuff, personally. I would like to see the places described in the story.”—Sarah, student

“We did have some overheads, but we couldn’t make out the overheads.” —Trish, student

“ …when the author reads it you are like, wow, is that ever powerful, I’ve got tingles. You can hear their emotion and what they are trying to portray with the literature.”—Christine, student

“Video would be awesome—tone and facial expression are really important.” —Jason, student

Importance of Visual Design

I obviously can’t summarize here all the project complexities or discuss all the wonderful things I learned throughout this process. One aspect of the research, however, reinforced for me the power of collaboration in the design process.

Refining needs and features for the project was challenging, but my most daunting task from the outset was how to create a visual design to meet the needs of the university course, the multiple cultures of the students, and also reflect the literature studied.

Students and instructors also felt the resource had to visually represent traditional and contemporary Aboriginal culture without using imagery that was specific to one culture or geographic region, but they also believed it needed to reference Western literary traditions because many of the authors on the syllabus referenced Western literary traditions in their literature. A daunting task to say the least.

Visual design is not of minor importance to Aboriginal cultures and the visual expression of those cultures.

While progress has been made, by various institutions, to foster understanding of Aboriginal cultures in Canada and the United States, images created by non-Aboriginal and Aboriginal designers found in mainstream media are not nearly as enlightened. Visual communication design creates effective communications that can “affect the knowledge, attitudes and behavior of people” (Frascara, 1997, p. 5). If designers can affect people with the communications they create, then they might be able in some way to prevent visual stereotyping of Aboriginal peoples and cultures.

It is easy to find examples of appropriation and stereotyping of imagery in mainstream media. Logos for the Atlanta Braves, Cleveland Indians, and Chicago Blackhawks are some highly visible examples of blatant stereotyping, but more subtle examples also exist. The popularity of the “New Age” spiritual movement is one such example of cultural appropriation that furthers the misrepresentation of Aboriginal cultures (Hulan & Warley, 1999–2000).

“ …you just can’t have a dreamcatcher a wagon and a chief on every page. There is a lot more to Aboriginal culture.” —Jason, Student

“ …the fact that there’s a million different medicine wheels if you look on the internet, it’s almost like a cliché or something, even though it has real cultural worth.” —Rob, Instructor

The “Aha” Moment

My plan for the visual design process had been to conduct semi-structured interviews with students and instructors and begin collecting ideas for visuals. I had hoped that from there we’d have some ideas to start working together on creating some participatory design sessions.

However, each time I came to discuss the visual tone of the resource and what should be included, it seemed we ended up talking about placement of menus and what types of information were desired for the website. Looking at websites or thinking about them made it very hard to get away from talking about features, functionality, and content. Not that those factors were not critical to shaping the visual design, but I wanted to discuss the visual tone and symbolic references that users felt might be important to include.

My thesis supervisor suggested I try to move away from the digital medium into the print medium to avoid the focus on features. Why not look at some other medium? You’re thinking this is the big “Aha” moment right? Not yet…

My plan was to try a session with five students reviewing book covers from Aboriginal authors and discussing what visual design appealed to them. From my own training and experience as a designer, I selected what I felt was a representative sample of different visual styles: Illustration, photography, typography, and the like.

Throughout my research, a critical theme was the importance of the land and landscape to Aboriginal cultures. I had this information, but I wasn’t really sure what to do with it. Any specific references to a geographic region might exclude some students or authors referenced in the curriculum. Students were from all over North America and had connections to a huge variety of landscape—prairies, mountain ranges, coastal regions, arctic regions, rain forest… This project had to be respectful to the students, teachers, and Aboriginal literatures studied. Imagery was already a very sensitive topic, so omitting any particular region or any one group was far from ideal.

During our sessions, students gravitated to abstract images of landscape that employed natural color palettes. Interestingly, some students selected these images because they felt the images represented both contemporary and traditional cultures. Few if any students selected imagery that had specific symbolism. There was great sensitivity to the use of symbols, partially because they may exclude certain groups but also because there has been a long history of offensive appropriation of Aboriginal symbols. Most students seemed to gravitate to those images that suggested the land in an abstract way.

For example, the book Transitions was selected as both a contemporary and traditional portrayal of Aboriginal culture. One student thought that because the stones were constantly evolving and were part of the land that they were representative of the contemporary and the traditional. Another student mentioned that the industrial feel of the rock landscape suggested the contemporary but that the rocks also suggested traditional elements of the “Grandfather, the stones of the earth.”

Monkey Beach and Transitions: Photography and landscape seemed most appealing to students.

I had already been thinking about using natural imagery, but the problem seemed to me to be the suggestion of one geographic region. Abstract representations of landscape seemed to solve my dilemma. It was suggestive of some type of landscape, but it wasn’t specific. When students suggested a color palette of earth tones, images of landscape, and photography instead of illustration, I had my a-ha moment.

I now had a direction for the visual design that was so simple yet would address a very complex issue. I could create an image that was respectful and addressed what was important to those involved but was universal enough not to exclude anyone.

Putting it all together

In the discipline of design—in sharp contrast to the modernist movement—there is a concern with how to tailor communications specifically to suit audience needs. McCoy (1995) calls this “narrowcasting” instead of “broadcasting.” With this concern comes the understanding that designers cannot make assumptions about cultural groups, even their own (Steiner & Haas, 1996).

I felt so incredibly challenged by this project not only because it meant I had to try to learn how to respectfully design with students and teachers from cultures that I was not familiar with but also because I could not understand how I could possibly design something that could be tailored to so many varied groups. My session with the students helped me realize how my own background led me to subconsciously dismiss the idea of finding some kind of universal solution since it would go against my own postmodern sensibilities.

Those sensibilities closed my mind to the possibility of universals that might be meaningful to the students and also be representative of the wide variety of literatures and cultures represented in the curriculum.

In short, without the different perspectives brought to the project by the students and teachers I was working with, I would probably never have come to the solution that I did. As I have moved forward with my career and my teaching and I see so many changes in the world of design, I think that more and more we may find ourselves in a position where we may have to challenge ourselves to find those universals. My takeaway from all this: try to practice a collaborative approach whenever possible to inform your work and allow yourself to see six sides of the box.

Works Cited

• Frascara, J., (1997). User-centred Graphic Design: Mass communications and social change. London: Taylor & Francis.
• Hulan, R., & Warley, L. (1999–2000). Cultural literacy, First Nations and the future of Canadian literary studies. Journal of Canadian Studies, 34.3–4, 59–73.
• McCoy, K. (1995). Graphic design in a multicultural world. How 10(2), 146–151.
• Steiner, H., & Haas, K. (1996). Design for the global village. Applied Arts, 11(3), 46–48 and 50–52.

The church had met a design challenge by constructing a narrative in an environment that advanced its messaging goals and met the goals of its audience. As the visitor moves from illustration to illustration depicting the famous trials and tribulations, they are moved both physically and emotionally.

The means of telling stories in spaces have advanced since then, of course. Display, audio, and sensor technology is getting cheaper and more ubiquitous, and many businesses and institutions are investing in integrated media environments to communicate with audiences.

Integrated media environments merge video and motion graphics, environmental and exhibit design, and interactive installations into a physical environment. The visitor moves through the space, taking in the story and the message.

Creating a successful integrated media environment involves design of every stripe (interior, 3D, graphic, UX, UI), as well as software development, hardware engineering, content creation, and curation.

The traditional approach toward applying media of any kind to a physical space has been to start with the space. You mark up a floor plan with a bubble diagram showing the storytelling opportunities, and then figure out what might best fill that space. That alcove could be perfect for an interactive table! That wall is crying out for a projection! Put a reader rail over there!

Instead, consider employing the same methodology that content strategy for the web has been using for years:

Diagram by Swanny Mouton

*Research and discovery*:
* First, understand both the messaging goals of the client and the experience goals of the audience. Understand this audience and their preconceived notions about the messaging.
* Determine the most intriguing content the client has (and what format it’s in), and what’s possible to acquire or produce within the general confines of the budget and timeframe.

*Concept model/planning*:
* Then build a solid conceptual base, with a core metaphor that can inspire both visual and content decisions later.
* Develop the narrative. What do you want the experience to be like? What do we want people to feel at each stage of the experience? What’s the transformation of understanding and mind we want the audience to feel?

*Materials evaluation*:
* At the same time, think about the space, technology opportunities, and other material constraints.

Only then can you pull them all together into a holistic environment that has the power to inspire, inform, and move people.

Of course, rolling out narratives in space continued into the twentieth century, especially in museums. Charles and Ray Eames (shown right) were innovators in this realm, designing and constructing carefully considered exhibits such as “Mathematica”:http://en.wikipedia.org/wiki/Mathematica:_A_World_of_Numbers…_and_Beyond, originally installed in 1961 at the California Museum of Science and Industry. The Eameses were essentially changing the information architecture of museum exhibits, telling a story about the world of numbers and math as a pathway of discovery. So they installed a long timeline wall on one side and a long image wall on the other. These linear, passive experiences flanked a gallery that included many participatory exhibits, including a light-bulb-cube multiplication machine, peep shows, and, famously, a large mobius strip with an arrow that moved on a track. Visitors bounced around in the center space that was anchored by the long walls. In the final exhibit, the thematic message and the visitor experience meshed perfectly, and it remains a seminal example of integrated interactive media.

Most storytelling environments such as these have involved primarily static graphics, passive media, or very simple interactions.

At “Second Story”:http://www.secondstory.com, the interactive media studio where I am a content strategist, designed and built a project for the University of Oregon that serves as a great example of how this kind of strategic thinking and planning can be applied to an integrated media environment with deep interactive content and space design.

The client came to us wanting to create a media gallery in the lobby of the new Ford Alumni Center. This building was to become the front door of the university, welcoming prospective students, alumni, and potential donors.

The final implementation, we knew, would need content, design, and user interfaces that spoke to everyone at once. We discussed the various audiences in depth, to understand their needs:
* College-bound seventeen-year-olds are intoxicated by possibility, without wanting to be overwhelmed. Some know (or think they know) exactly what they want out of their experience at UO, while others expect to explore.
* Returning alumni and potential donors want to be reminded of what makes the UO special, to reminisce about their college years, and to ensure that youth today get similar or better experiences.

Based on this and other research, we devised a visual metaphor that could resonate with the client and their audiences and act as a steady beacon for everyone involved. Taking inspiration from Oregon’s natural environment and the branding of the university, we envisioned the university as the bedrock beneath a river and the students as the water flowing past, subtly making their mark on the institution’s culture. Each student’s path through the river was fluid and determined entirely by the student. This strategic vision helped guide our design choices throughout the project.

The concept of the university as bedrock, with the students and faculty flowing through, was a source of inspiration in developing the concepts.

Our initial content survey showed that we were looking at materials covering the entire university, including:
* The identities of every alumnus in the school’s 140 years.
* Images and archival video going back almost that far.
* Imagery, video, and text from the website of every academic department, every sport in the athletic department, and every student club.

The content requirements for the Oregon Cascades prompted the development of a powerful CMS for the client.

The content volume was overwhelming, and the assets were stored in hundreds of locations online and off. In conversations with the client, we decided that they should manage the content post-installation, allowing it to grow over time and not stagnate. We proposed the development of a powerful content management system with a template-based front end that would allow the content to reflect the university as things changed – the experience would remain consistent while still being fresh.

The final front end for the project’s content management system. The CMS allows the client to keep the experience perpetually fresh.

With the visual metaphor and a vision of the content in hand, we started figuring out the story of the space. This being an alumni center, one of the anchoring elements was the long history of alumni that had come through the university. Another central point in our story was the multitude of available options at the university, which required an experience that allowed visitors to reach out and grab what interested them, at a glance. And, inevitably, we would need to serve announcements and list names of donors.

In looking at the physical space, we realized that there was no possible straight narrative, as people would wander in from multiple directions. Just as we might consider hierarchies on a web page, we thought about how visitors might engage with these stories from far away, closer, and within their interactions. The principle is the same, just extended into three dimensions.

Armed with a powerful story, we sketched the experience into the physical environment.

The placement of the story elements in the space evolved over time, but the core of the experience was consistent.

At this point, we had a holistic understanding of content, messaging, and strategy. We also understood the physical environment, as well as a powerful metaphor that could keep us on track. We designed concepts for three kinds of physical installations, each of which had its own set of communication goals, and took into account visitor pathways through the space:

* The Oregon Entry Wall standing near one of the entrances, presenting donor names flowing across the wall and displaying events and announcements.
* The Oregon Alumni Table showing information about every alumnus from the university’s history, and resembling a river that visitors can dip into.
* Nine Oregon Cascades that stand 14 feet tall, and evoke waterfalls. Six Cascades are digital interactive experiences that allow visitors to swipe through content. These are presented in unique form factors, with four large monitors stacked to the ceiling. Three Cascades are cases, displaying artifacts from the university’s history. The Cascades would be installed on tracks that allow university staff to alter their position.

This was, of course, just the beginning—we spent another year or so on design and development. The final product has a cohesive feel that joins the physical and the digital. We feel strongly that the strategic foundations we laid down early in the process were a crucial factor in the project’s success. You can read more about the project at http://secondstory.com/project/browse/featured-work/university-of-oregon.

In the final media environment, the Oregon Alumni Table is in the foreground, and the floor-to-ceiling Oregon Cascades are in the background.

Content strategists have been leading the charge on how to deliver a consistent message across channels. The advent of smaller, cheaper technology is making it possible for every surface to be a new channel. Sensors such as the Microsoft Kinect depth-sensing camera are removing the need for complex input devices or even touchscreens. The more that physical environments gain digital and interactive dimensions, the more important it is to provide clear, focused storytelling that has been considered carefully. The strategies that have become codified over time for the web and are helping us with the transition to the mobile web also provide us with a powerful framework to design great experiences in physical environments.

_Thanks to Michael Neault and Swanny Mouton for their assistance with this article._

This case study shows how the ComputerWeekly user experience team integrated with an agile development group. It’s important to note the methods we used do not guarantee getting the job done. People make or break any project. Finding and retaining good people is the most important ingredient for success.

The brief

In 2008, we were tasked with resurrecting a tired, old, and ineffective site. It was badly out of date, and the information architecture was decrepit to both users and search engines.

Our goals were:

1. Make content visible and easy to find
2. Create an enjoyable and valuable user experience so users would return
3. Increase page impressions to bring in ad revenue
4. Allow site staff to present more rich media content
5. Give the site more personality and interactivity

The UX team created personas from ethnographic studies, online surveys, and in-depth interviews with users. The data gave us a clear idea of the user’s needs and wants. We also gleaned data from analytics that told us where users engaged and where the bounce rates were highest.

At this point the development team maintained the site with an agile process. They created features for the new site in parallel to ongoing site maintenance, but this work was outside the normal maintenance sprints. The new site was considered as an entirely new project with a separate budget and scheduled into longer term.

Boundary Spanner

As the User Experience team gathered data key team members were recruited. The diagram below shows the key team members needed to produce this large scale site, their specific concerns, and their methodologies.

To bring these teams and disparate elements together requires a launch manager or ‘boundary spanner’. Rizal Sebastian wrote about boundary spanners in Design Issues in 2005¹. The boundary spanner needs to be aware of the individual issues the project faces. He need not know the detail but needs to know the cultural context of the collaborative environment.

Do people get on with each other? Are communication lines clear? Are there any personality clashes on the team. To throw developers, interface designers, business analysts, SEO experts, and usability guys together and expect them all to gel is optimistic but unlikely. It also requires those people devote all their time to just one project and that is unrealistic for a large operation where several projects are underway simultaneously.

They are more than a project manager because the user— and not the project—is at the heart of their concerns. He is responsible for delivering and continually developing a quality product. He is not just monitoring the features on a checklist. The user is at the center of all decisions on the design and development of the site. This is the only way you can ensure the user will be heard throughout product development – to employ somebody who listens to user voices and never forgets what they said. They must also ensure that SEO and business requirements are satisfied, and a well-defined strategy is in place. The boundary spanner owns and clearly communicates the vision until the whole team understands.

The boundary spanner’s strength is that they are core to the product and not a department or team known for their skillset (like a UX team for example). In many cases it is a product manager, but in this case it was the website editor who was responsible for synchronizing the teams.

Defining a process

To assist this user focused approach, the IAs produced set of deliverables that ensured the launch manager’s vision could be realized and developed.

Diagramming the process using a concept model engaged key stakeholders with the project by communicating the vision of what the UX team would achieve with the speed and clarity of an elevator pitch.

Information gathering

A content audit revealed broken links, redundant content, and poor usability plagued the site. It also revealed how much content became lost the second it moved from the home page. The high value research papers were impossible to find.

30 interviews, 20 ethnographic studies, and 950 responses to an online survey. created six personas. With the content audit, personas, and business objectives (what we wanted them to do on our site), we began creating the taxonomy.

Analytics

During this project we were very fortunate to work alongside the web analytics manager who provided insight into user behavior, including high bounce rates from visitors arriving from search engines. He also provided a scorecard that showed where the site failed in terms of traffic and user engagement. We knew what users were searching for, and pretty quickly could tell why they were not finding content we knew we had.

By looking at web metrics we were understood usage patterns and popular and unpopular areas of the site. The depth of information enabled us to quickly formulate a plan.

Persona driven taxonomy

As we knew our user base were industry experts, we also knew their vocabulary was related to specific areas of their markets.

The taxonomy was created by gathering as many industry sources (websites, journals, white papers), breaking these down into unique elements, and clustering these elements together to form categories for our content.

The CW taxonomy formed the basis of the navigation, content categorization, and highlighted areas for future development. It also ensured our search results served up related content in context.

We defined four main areas by looking at the community of users.

News was an obvious requirement, defined by their particular area of interest within the sector. The need for knowledge was evident, and we created an in-depth research area where case studies and white papers could be easily accessed. Tools and services, RSS, email news alerts, and newsletters reflected user needs to be kept up to date and in tune with their specialization.

Finally, although the CW community was secretive and did not divulge information amongst their peers, they were very interested in expert opinions. This need gave rise to much more integrated blog postings.

Interface development

The navigation scheme defined the elements of the page the users would use to move to other areas in the site. It clarified the naming of those items on the page.

We then considered the prioritization of information and content for each page, and this facilitated the production of wireframes that represented the culmination of all research, showed the interface and interactions for elements on the page, and were a working document that changed as we iterated the design.

Core and Paths

We used Are Halland’s method for ‘designing findability inside out.’²

• Prioritize and design the core – Satisfy user goals using prioritized content and functionality.
• Design inward paths to the core – Consider how users will arrive at the page from search engine results, facets, menus, search, RSS, aggregation, email, etc.
• Offer relevant outward paths from the core – Ensure that the site delivers both user and business goals through clear calls to action and completion of interaction tasks.

For CW.com, we focused on the cores for the home page, a channel level homepage, and a news article page and looked at key content such as lead news story and the editor’s picks or the best from the web aggregated from external sources. The key functionality and supporting content also had to be included and prioritized on the page.

Next we considered the inward paths, which are the channels that our users are likely to utilize to arrive at the page.

Inward paths included search engines, blogs, bookmarks, syndication, aggregators, RSS, and email subscriptions. Considering inward paths helped us focus on the marketing channels we needed to drive users to the relevant type of page. It also focused on the keywords and themes that users are likely to use and helped us optimize pages for search and online marketing campaigns.

Finally we designed the outward paths that helped users complete online tasks for our business objectives.

These outward paths include:

• Newsletter sign-up
• Inline links to related articles to drive page consumption
• Sharing, printing or emailing of news articles
• Related content types such as video or audio
• Stimulating community participation in forums or blogs
• Contextual navigation to aggregated content or the editors best bets
• Subscription to an RSS feed
• Prioritizing the content



Once the wireframes had been approved, the content was organized so the most commercially valuable and user focused content was pushed to the top of the page. As the design went through user testing, certain elements changed, as with any iterative process, but through team collaboration, the solution remained true to the initial vision from concept to design delivery.

The development cycle

The wireframes were handed over to creative, and they began designing the interface and graphic elements. The development group released some functional elements to the old website before the relaunch.

These agile methods allowed the old site to feel the benefits of the new widgets. However, as the site changed so radically in the new design, we still had to release the site in an old style ‘big-bang’ manner. This is perhaps where agile has its problems as a methodology for new launches. It’s focus on many small releases is a great tool to implement incremental changes but not for a completely new site.

As the html flat pages were passed to the team, the SEO requirements were defined and built into the site. By the time the site launched it, had been through four major pieces of testing.

A holistic solution

Providing user experience deliverables like the concept map and wireframes ensured more comprehensive requirements were delivered to the design and development team. This approach addressed marketing, editorial, sales, and business requirements next to the needs and wants of the user. The vision was aligned with an achievable delivery from the IT team that ensured we delivered the site we wanted to give the user.

The core IA work enabled the new site to be future-focused and versatile. The structure and design of good sites should be able to adapt to change.

User-centered design and agile can work alongside each other but what is more important is having people who can tie all the loose strands of a website design and development cycle together. The concept map, wireframes and the IA strategy document that listed the rationale behind the design decisions helped ensure the product vision was correctly communicated to the development team, so the product that was developed through their agile processes was in line with the overall product vision.

¹http://www.mitpressjournals.org/doi/abs/10.1162/0747936053103020? cookieSet=1&journalCode=desi

²http://www.slideshare.net/aregh/core-and-paths-designing-findability-from-the-inside-and-out

How It Started: The Quick Fix

With the recent outbreak of H1N1, Santa Clara County’s official public flu information site was taken down by the surge in web traffic. To help relieve the demand, the Stanford SIE Program, a Stanford University group that develops technology for social change, stepped in literally within hours of the interruption to create an ad hoc backup site using Google sites, so people could still access the critical info.

This is the version of the site they originally posted, using Google Sites’ standard WYSIWYG editing tools:

Stanford’s original stopgap design

After the site went live, Stanford trained the Santa Clara County maintain it and add their own information. Santa Clara County needed to have site that could handle the traffic and get the information out as quickly as possible—which is to say that there wasn’t a whole lot of time to think about design.

This experience made it clear that it would be valuable to create a well-designed, easy-to-edit template for local governments to distribute information in case of emergencies—not just H1N1, but any public hazard, including floods, earthquakes, wildfires, and so on.

Bill contacted us in late October with the original draft of the website. Since it was important to make the site available as soon as possible to deal with the ongoing H1N1 outbreak, so the timeline we had for design recommendations was really brief—just a few days. With that in mind, we got to work.

Spotting the Problems

Because of the layout restrictions, our design evaluation focused primarily on information design. We had two main issues with the early design, along with a handful of usability tweaks here and there.

1. Lack of visual hierarchy.


With two columns of equal width and mostly indistinguishable boxes filled with text, it was hard to tell at a glance what information was urgent, time-sensitive, or recently added.



2.
Big chunks of info, no
 organization

The info wasn’t grouped into meaningful categories—there wasn’t much visual or spatial distinction between contact info, prevention info, calls to action, and so on, making it difficult to zero in on information even if you know what you were looking for. Also, the info came in big blocks of unscannable prose, and deep reading is the last thing you want to do when you’re trying to learn about the tsunami headed your way.




3. It didn’t anticipate the distinct needs of the most critical user
segments.




There was plenty of good info on the site, but it was never clear who a given piece of info was for—you couldn’t scan the page headers and think, “Yeah, there’s what I’m looking for”. Instead it had a “general audience” feel to it; the info didn’t take into account that different groups might have different needs and different levels of urgency.



4. Buried info.


Vital info on vaccines, symptoms, and SMS / Twitter updates was absent from the front page entirely, lurking deep in the navigation.

Our Recommendations

To keep editing simple for the local government end-users who would be using the template, we had to stick to using the WYSIWYG Google Sites editor, which meant no custom CSS and very little control over layout. We also had literally a single day to make our recommendations and synthesize them into a first-draft mockup—the result wasn’t pretty, but got our main info design recommendations across loud and clear.

Our first stab at redesigning the H1N1 template

Redesign Goal #1: Prioritize information according to the urgency
of visitor need.

Our design takes into account that there are different “general public” user segments with different goals, and that there are tiers of urgency and priority. From most-to-least urgent, we identified these segments:
* People infected with the flu: Immediate help / contact info
* People worried that they might have the flu: Self-diagnosis
* People concerned about catching and/or spreading the flu: Preventative measures and vaccine info)
* People just curious, staying informed: Information about travel restrictions, public response, news updates, deep flu info

The structure of the site was altered to serve each of these segments:
# We added a page-width alert box that would be displayed to convey urgent, time-sensitive info—this is a common feature on many of Google’s sites, as well as CNN.com.
# A yellow-shaded box to give the highest priority group, currently affected individuals, clear instructions on what to do.
# The left-column contains info on self-diagnostic and preventative measures for at-risk or concerned individuals.
# The top-right contains info on vaccines; below is links to deep info, research, and update notifications. Though the Google Sites template didn’t allow us to resize the right column, we noted that it should be made smaller than the left column.
# The left sidebar navigation was reserved for redundant quick links to most important info, as well as links to pages for specially affected individuals and organizations.

Redesign Goal #2: Reduce block text down to easy-to-scan lists
and chunks.
Cut the bureaucratic BS.

We broke down the block text to keep from overwhelming users with too much difficult-to-scan information upfront. Lists were kept to under 8 items long, unless they broken down into categorized sub-lists; text was kept under five lines. All information that couldn’t be condensed in this way was moved to lower-level pages, and linked from
higher-level pages.

Users don’t need to know what the mission statement and goals of the organization are; they just want to know if and how they are personally affected, and what they can do in case they are affected.

Redesign Goal #3: Use informative headers that directly address
user goals, and which give all users clear next steps.

All types of visitor (e.g. directly affected, at risk, concerned, just curious, administrative, medical, etc.) should be able to tell by the header if that information is “for them”. We tweaked the headers to make it clear what kind of info you could find in each section. We also made it clear what, if anything, each user segment needed to do:
* Immediately affected individuals are given step-by-step instructions on how to deal with their
emergency situation(s).
* At-risk individuals are given step-by-step information on preventative, precautionary, and self-
diagnostic measures.
* Individuals seeking non-urgent information can be given supplementary external information
resources (by phone, online, and downloadable / printable) and means to stay updated (by email,
text, RSS, Twitter).
* Urgent contact info is immediately visible in the right sidebar.

The First Revision

After we sent over our recommendations and mockup, Bill sent us a nice note a day or two later:

You’ve convinced us that we have to completely rethink and redesign the site from scratch, so
your style guidelines come at a very good time. I can’t thank you enough for opening our eyes to
how we can do this in a much better way. I think we can create a site that works much better than
the original site.

…and then a few days after that, Stanford sent a revised version over to Google, who worked with the design firm OTT Enterprises to create two new designs with some added visual design flourishes.

Unfortunately we don’t have permission to show you this revision yet (working on it), but it wasn’t bad—certainly cleaner and better organized, easier to scan, less dense. There was, however, a large distracting green gradient background, some empty space in the sidebar columns, a few stock photos, and a muddled color palette (green, blue, yellow, and gray).

Our second batch of suggestions focused mostly on visual design and layout. Just a few of them:

Visual Design

* Get rid of the gradient background; it’s distracting and confuses the emphasis on different parts of the site, since it runs behind multiple different elements.
* Get rid of the green coloring, which is tertiary to the blue and yellow. Instead, use several shades of blue as the primary color and a little light beige or light grey as a secondary trim. Blue signifies authority, calmness, trustworthiness, etc., which are of course appropriate here. Notice how almost every major government agency’s website (including the CDC) uses dark blue and gray as the main colors.
* Remove the stock images, including the CDC and Flu.gov widget images, which look like ads. The phone and email icons are fine, however.
* Rather than images, make the content headers stand out with size and strong typography—”make the content the focus” is an old web design adage, and the content, in this case, is the flu information. Here are a bunch of sites that use typography, font size, whitespace, and bold layout to create emphasis, using few images [list of a bunch of websites].

Layout

* Header and upper-page content takes up way too much space—note that the important info (”If you or your child…”) doesn’t begin until more than halfway down the screen. Compress.
* I like how Design #2 places the alert and contact info in the sidebar; in Design #1 the sidebar is wasted space. This frees up space to move important info (Vaccine and How to Care for Someone With The Flu) up to the right side.
* This design consists mostly of links to deeper pages, but there’s definitely room to put more specific, useful info right on the homepage: symptoms, preventative measures, vaccine info (see our original design)
* Get rid of the Contents box.

The Results

Stanford started over once again, aided by our style guide and input from OTT Enterprises. After further iterations in Google Sites, they handed it over to the Google visual designers, and here’s the before-and-after:

Google Sites template, super rush draft

Google Sites Public Health Template 1.0

Can you do better?

As with all things on the web, the template is a design-in-progress, and will be improved as time goes on. Stanford SIE is looking for feedback on the design, so here’s where you can send feedback for the Public Health template and the All Hazards template. Since these Google Sites templates are available to everyone, you can even make your own design edits and mock up improvements.

Or if you just think it’s great and you just want to use it yourself, here’s the complete list of links:

Google Sites Templates blog post

Public health sites:

Template
Example site
User guide

All hazard sites:

Template
Example
User guide
Stanford SIE site (we’re credited here!)

Note: Nate and Tony’s book on remote testing, “Remote Research”:http://www.rosenfeldmedia.com/books/remote-research/, will be published by Rosenfeld Media in 2010.

The Balance of Power

There are a wide variety of uses for Wikis and a level of interest in using them that’s matched by an extensive range of Wiki software. Wikis introduce to the Internet a collaborative model that not only allows, but explicitly encourages, broad and open participation. The idea that anyone can contribute reflects an assumption that both content quantity and quality will arise out of the ‘wisdom of the crowd.’

There are, however, negative effects of this extreme openness. One problem is the deliberate vandalism of Wiki pages. Another is that even those with no destructive intent may yet degrade the quality of a Wiki’s content through lack of knowledge or skill. Anyone can write nonsense as though it were fact. Anyone can accidentally delete useful information. Someone with half-baked knowledge of grammar may change all the “its” to “it’s.” Of course, someone more knowledgeable may notice the problem and fix it … but then again maybe they won’t.

Wikis can impose various forms of control to protect against these risks, including user registration, moderation, enforced stylistic rules, and imposing prescribed topic structures and page layouts. These types of control, however, are typically seen as contrary to the basic Wiki concept.

Consequently, one of the central tensions when managing a Wiki is between centralized control and anarchy. In the public arena, the balance of power tends towards anarchy, but in a corporate environment a more centralized approach is often required.

In this article I describe one application of the Wiki way to a common corporate process and extract some guidelines for the effective use of Wikis in that context. In particular, I am seeking insight from this case study into the “balance of power” tension.

The example on which these reflections are based is a project within the software company CorVu [1] to improve the technical knowledge base related to the products we sell. Like many companies, CorVu has extensive knowledge of its own products and a desire to make that knowledge available to customers. A major block to achieving that desire has been a lack of people with the time to either record the internal knowledge or to fashion the knowledge into a customer-ready format. We needed to spread the load so that a broad range of developers, tech writers, professional service consultants and others could all contribute what time and knowledge they had to a shared goal. Our hope was that a process built around several Wiki sites would facilitate this collaborative approach.

There’s no guarantee, of course, that lessons learned in that context will transfer to others. But without documented cases such as this one, any theorizing about the balance of power issue is just speculation.

Three contexts for a Wiki

To start with, it is important to clarify the key differences between three contexts in which Wikis are used: public, team and enterprise Wikis. [2].

Public Wikis

By a “public Wiki,” I mean one where any Internet user can read and contribute to the collaborative effort. It may be that editing content is restricted to a registered user group (as is the case with Wikipedia), but anyone can register. Consequently, the size of the contributing community is potentially huge, there is a high level of anonymity, and the contributors do not typically relate to each other outside the confines of the Wiki.

In this context, very little centralized control is evident. You typically find some explicit guidelines for contributors, either formulated by the founders/hosts, or as an evolving page edited by the contributors themselves. There is also an implicit understanding of etiquette and an implied social contract that comes with joining the “community.” But in the end, anyone can edit anything … and anyone else can un-edit it. This is the essence of anarchy: not that anything goes, but that what goes depends on peer acceptance. In an anarchy, it is not the case that there is no control; rather, the control is exerted by peers (around the edges) rather than by an authority (in the centre).

Requiring registration prior to participation does not alter the anarchistic nature of the process. Registration has numerous benefits, not least of which is that contributors can be recognized and gain respect for their contributions. Registration may also increase the sense of belonging because it reflects each contributor’s conscious choice to join the community. That sense of belonging is essential to any viable anarchy.[3]

Moderation, on the other hand, inevitably moves the balance of power towards the centre. Moderation invests some users with the power to limit the contributions of other users. While moderation is sometimes seen as necessary in order to combat vandalism and dissension, this imposition of authority denies the libertarian aspirations of most public Wikis.

Team Wikis

A “team Wiki” is one where the people who read and contribute all belong to the same team or work-group. Perhaps the R&D team uses the Wiki to record evolving product specifications; or the members of a local church collaboratively documents its history; or a class of students collates the results of a research project. Membership of the team predates and takes precedence over membership of the Wiki community. A person joins the team and as a by-product may be requested or required to use the Wiki. The number of people participating tends to be small and the contributors are likely to relate to each other outside the context of the Wiki.

In contrast to public Wikis, where self-selection guarantees that the vast majority of users are technically savvy and keen to be involved, the people contributing to a team Wiki may not be doing so voluntarily or with much enthusiasm. It may well be a required part of their work that they would prefer to avoid. The need to make the Wiki as easy as possible to use becomes even more important in this context. This includes clear navigation and an effective search function, but more than anything else it means a simple, familiar user interface for editing text. Many team Wikis fail simply because the potential contributors refuse to learn Wiki markup or to use a non-wysiwyg editor.

In this context, registration is essential, but moderation is not. The restrictions on who can contribute protect against vandalism and, because the collaborators have pre-existing relationships and a common commitment to a higher cause, the community operates with a trust model. In fact, apart from the restrictions on membership, a team Wiki is unlikely to impose much control at all over contributions. Standards, structures, and conflicts will be resolved using the organization’s normal processes outside the Wiki. The collaborators will discuss and vote, or demand and threaten, or just do what the boss says, without that process being explicitly controlled by mechanisms within the Wiki.

Enterprise Wikis [4]

When it comes to implementing Wikis across a large enterprise such as a global corporation, a new set of concerns affect the balance of power. Management wisdom is required to maximize participation while keeping business objectives clearly in sight.

In my experience, it is rare that a single Wiki site within an enterprise is open to contributions by any employee. Where this is the case, moderation is likely to be required because of the large numbers of contributors who have no direct accountability to each other. The concerns at the enterprise level relate to how numerous organizational Wikis within the enterprise can be integrated into the IT infrastructure and how the use of Wikis can most effectively support corporate goals.

Rather than allow the proliferation of diverse Wiki projects throughout the enterprise, IT management is more likely to select the Wiki software that everyone is to use and perhaps host all instances centrally. It may be that some IT managers are “control freaks,” but there are good reasons for standardizing on Wiki software:

• Risk. If many work groups host their own Wiki using their own choice of software, there is a significant risk of knowledge loss. It is hard to guarantee that each work group will secure the Wiki adequately or ensure appropriate disaster recovery. What happens if the work group’s server dies? Will they have an adequate backup procedure? What happens if the work group’s IT expertise leaves the company? Will the knowledge of how to run the Wiki be passed on to the remaining team? What happens if the Wiki software no longer operates when the server’s operating system is upgraded? Centralized Wiki management can avoid such problems.
   
• Support. Most Wiki software is easy to learn (at least to us!), but some are certainly easier to learn than others. In a context where many employees participate in multiple Wikis within the enterprise, training and user frustration can be reduced by using the same software for all the Wikis.
   
• Cost. Centralized IT management can also reduce the total cost of ownership of Wiki projects. That may be counter-intuitive given that most Wiki software is free. But the costs of running a Wiki include the cost of the hardware that hosts the Wiki, the time it takes to manage the Wiki (installation, user admin and support, backup, etc.) and the time it takes to teach people how to use the system. Although these costs may be small for each work group, the total across the enterprise can be substantial, and can be reduced by standardization and centralization.

In this context, the balance of power swings inevitably towards centralized control. The challenge is how to do so without stifling the free and creative contributions that are essential to a Wiki’s success.

The CorVu case study

The company I work for, CorVu, started using Wikis within its R&D group back in 2000 using the original WikiWikiWeb software. The project described below was based on MoinMoin, but we have also used DoKuWiki and have since standardized on Confluence.

CorVu produces software that assists other enterprises to implement their strategy and to track their performance against that strategy over time. CorVu has a variety of channels for making its internal product knowledge available to its customers, but the product functionality grows at a faster rate than the Tech Writers can keep up with. Apart from the fundamental description of each feature, a complex assortment of configuration details need to be documented – performance optimization, best-practice implementation techniques, interactions with third-party software, etc. A lot of knowledge at that level resides with the Professional Services team rather than the Product Development team. Often, the people with the knowledge do not have the time nor the writing skills to record it, and the people with the responsibility to deliver documentation to the customers do not have the knowledge. There’s nothing uncommon about that problem!

Since the goal of capturing and disseminating quality technical documentation requires collaboration, I thought that a Wiki might help. So we set up two independent Wikis to capture knowledge from two different groups of employees, and a third so that customers could access a sanitized version of that knowledge.

I’m not putting my own case forward as the paradigm of success. In fact, although the project yielded a significant improvement in capturing internal knowledge, we have not yet achieved the final goal of effectively disseminating that knowledge to our customers.

Figure 1. Knowledge capture and dissemination using three Wikis

In this project, there was a clear business goal and a centrally-driven process to attain that goal. The Professional Services and Customer Wikis were seeded with pages that provided a structure for delivering accurate and accessible content to customers. While the ability to contribute was widespread, there were explicit “rules of engagement” around user registration, topic naming, page layout templates, content categorization, and navigation.

Although there was a degree of central control, we tried to balance that with encouragement for broad-based collaboration–otherwise, why use a Wiki? The distinction that guides this balance is between structure and content. Although the structure is imposed centrally, content is generated by a diverse range of people in a way that promotes openness, the recognition of contributors, editing of any content without fear of criticism, and shared responsibility for quality.

Since the quality of the documentation exposed to our customers is crucial, the process includes a QA step that is uncommon for Wikis. We did not want to constrain all contributors to adhere to strict grammar, spelling and style rules. Instead we left the knowledge capture stage free from those restrictions and used technical writers to edit the content before its dissemination to customers.

It may seem strange that we would use a Wiki to publish non-editable information, but this is a testament to the versatility of the software. Wikis provide a very fast means of building a web site, whether collaboration is the intention or not. In our case, we use one Wiki site to capture knowledge from one group of people and another Wiki site to disseminate the information to a different group of people. With regard to my categorization of Public, Team and Enterprise Wikis, the “Customer Wiki” is a hybrid: it is built by a specific team and hosted within an enterprise infrastructure in order to publish in the public arena. A more traditional approach to software documentation would have been to repackage the knowledge into some other HTML or PDF format for customer consumption. But the maintenance of that dichotomy would have been far more onerous than copying between two parallel Wikis.

Embedding Wiki tools across an enterprise is an organizational change project and as such requires appropriate planning and project management, along both technical and cultural dimensions. I won’t go over those generic processes, nor repeat suggestions for Wiki adoption that are documented in places like WikiPatterns. But drawing from CorVu’s experience, I will highlight some advice for project managers in the enterprise Wiki context.

People

1. Seek patronage at the highest possible level. That is, find a person with as much power within the enterprise as possible who will sponsor the project. The sponsor may do no more than ‘give the nod’ to your work, but that invests you with the authority to draw on other people’s time. In CorVu’s case, the CEO himself was a key supporter.
2. Enthuse a champion. This needs to be a person who is well respected, who will lead by example, and in doing so enthuse others. The champion will need to be able to put a lot of time into the project and will often be the primary contributor to the Wiki, especially at the beginning. In our case, that turned out to be myself.
3. Identify the group of people who can be expected to generate the majority of the Wiki content. These are typically subject matter experts. Discuss with them the value of writing down what they know or Wiki-izing what they have already written.
4. Identify anyone whose participation is mandatory. Is there a key political player or subject matter expert who absence from the project will cause others to think, “Well, if she’s not involved, I’m certainly not going to waste my time?”
5. Since our goal was to create a knowledge base for external consumption, it was important that the content generated by subject matter experts was checked for both accuracy and readability in the same way as other customer documentation. Consequently, the people involved in the project needed to include professional technical writers.

Tools

There are many different Wiki software tools in the market (Wiki Matrix lists over 100) but most are not adequate for an enterprise rollout. CorVu’s experience suggests that an enterprise Wiki requires at least the following:

1. Administration tools to manage a large number of users, with integration to enterprise security mechanisms (e.g. LDAP and single sign-on).
2. Separately secured spaces for different knowledge areas.
3. Effective management of attachments that includes versioning and a built-in search function that indexes the attachments.
4. Integration with other enterprise software such as portals, business intelligence, and content management systems.
5. Many contributors in an enterprise context will be non-technical. This makes it essential that the Wiki has a familiar, WYSIWYG editing mode rather than forcing users to learn some Wiki markup language.
6. An assortment of non-functional requirements such as good reputation, reference sites, some assurance of product longevity, and the availability of support.

Generating participation

All Wikis stand or fall based on whether an active community is formed. You can’t achieve the ‘wisdom of the crowd’ unless you have an active crowd. The means of achieving that across an enterprise are somewhat different from public Wikis.

1. Build a critical mass of contributors. Since the contributors are employed by the enterprise, it is possible to make the Wiki part of people’s responsibilities. At CorVu we found this to be imperative. Unlike a public Wiki (where there are many people who contribute huge amounts of time as a hobby), in a work context (where everyone is probably too busy already), this isn’t going to happen. So write it into job descriptions. Get managers to send emails to their staff saying that one hour a week should be spent writing up their knowledge on the Wiki. Arrange a seminar on how to use the system. Use the company newsletter to promote the value of the project.
2. Build a critical mass of topics. To be used, the site must be useful. To generate traffic to the site, make the most frequently required information available on the Wiki first, and make the Wiki the only source for that information. In CorVu’s case, for example, one significant page stored the latest product release information. When any software version was moved from internal QA to Beta, or from Beta to General Release, this page was updated. Once people learn that the Wiki contains a lot of useful information they will look there for answers to start with rather than wasting someone else’s time by phoning or emailing questions.
3. Send links rather than information. Set an expectation that when anyone is asked for some detailed information, the response should be a link to a Wiki page. If the information has not yet been Wiki-ized, don’t type a lengthy answer in an email; instead, spend an extra minute typing it into a Wiki page.
4. Provide recognition and rewards. As with most Wikis, the best way to encourage participation in the long term is to ensure that the efforts of the contributors are valued. This is easier in team and enterprise Wikis than in public Wikis because the contributors are known. Wiki pages can indicate explicitly who the primary authors were. There can also be rewards within the enterprise beyond the boundaries of the Wiki. For instance, some employees may have components of their annual review linked to their involvement in Wikis.

Our experience with Wikis at CorVu has been very positive and gives encouraging signs about the future potential of this approach to shared document workspaces. There are multiple offerings that meet enterprise IT standards, and the tools currently available are robust, simple to administer, simple to use, and inexpensive. The CorVu case also shows that enterprise Wikis can be used not only for internal purposes, but also as a means of publishing information to external stakeholders.

By putting minimal central control in place an enterprise can gain significant benefit from this simple technology, including improved knowledge capture, reduced time to build complex knowledge-based web sites, and increased collaboration. Although enterprise Wiki use requires a greater degree of centralized control than public Wikis, this need not impinge on the freedom to contribute that is the hallmark of a Wiki approach. The balance of power is different in an enterprise context, but fear of anarchy should not prohibit Wiki adoption.

Nevertheless, I predict that Wikis will disappear over the next 5 to 10 years. This is not because they will fail but precisely because they will succeed. The best technologies disappear from view because they become so common-place that nobody notices them. Wiki-style functionality will become embedded within other software – within portals, web design tools, word processors, and content management systems. Our children may not learn the word “Wiki,” but they will be surprised when we tell them that there was a time when you couldn’t just edit a web page to build the content collaboratively.

Consider how video games are currently researched: market research-based focus groups and surveys dominate the landscape, measuring opinion and taste in a controlled lab environment, and largely ignoring players’ actual in-game behaviors. Behavior is obviously the most direct and unbiased source of understanding how players interact with the game—where they make errors, where they become irritated, where they feel most engaged. When Electronic Arts engaged Bolt|Peters to lead the player research project for Spore, we set out to do one better than the usual focus group dreck by coming at it from a UX research perspective.

SIMULATED NATIVE ENVIRONMENT RESEARCH

One overarching principle guided the design of this study: we would let the users play the game in a natural environment, without the interference of other players, research moderators, or arbitrary tasks. This took a good bit of planning. Usually, we prefer to use remote research methods, which allow us to talk to our users in the comfort of their own homes. Spore, however, was a top-secret hush-hush project; we couldn’t very well send out disks for just anybody to get their hands on. Instead, CEO Nate Bolt came up with what we call a “Simulated Native Environment.” For each of the ten research sessions, we invited six participants to our loft office, where they were seated at a desk with a laptop, a microphone headset, and a webcam. We told them to play the game as if they were at home, with only one difference: they should think-aloud, saying what ever is going through their mind as they’re playing. When they reach certain milestones in the game, they would fill out a quick touchscreen survey at their side, answering a few questions about their impressions of the game.

Elsewhere, Nate, the clients from EA, and I were stationed in an observation room, where we set up projectors to display the players’ gameplay, the webcam video, and the survey feedback on the wall, which let us see the players’ facial expressions alongside their in-game behaviors. Using the microphone headset and the free game chat app TeamSpeak, we were able to speak with players one-on-one, occasionally asking them what they were trying to do or to go a little more in depth about something they’d said or done in the game.

Doesn’t that sound simple? Actually, the setup was a little brain-hurting: we had six stations; each station needed to have webcam, gameplay, survey, and TeamSpeak media feeds broadcast live to the observation room – that’s 18 video feeds and 6 audio feeds, and not only did the two (that’s right, two!) moderators have to be able to hear the participants’ comments, but so did the dozen or so EA members. On top of that, everything was recorded for later analysis.

“The feedback we received from users wasn’t based on tasks we’d ordered them to do, but rather on self-directed gameplay tasks the users performed on their own initiative”

The important thing about this approach is the feedback we received from players wasn’t based on tasks we’d ordered them to do, but rather on self-directed gameplay tasks the players performed on their own initiative. We didn’t tell players outright what to do or how to do things in the game, unless they were critically stuck (which was useful to know in itself). The observed behavior and comments were more stream-of-consciousness and less calculated in nature.

The prime benefits to our approach were the absence of moderators, which mitigated the Hawthorne effect, as well as the absence of other participants, eliminating groupthink. Additionally, the players were more at ease: it’s hard to imagine these video outtakes (see below) being replicated in a focus group setting. Most importantly, they weren’t responding to focus questions–they were just voicing their thoughts aloud, unprompted, which gave us insight into the things they noticed most about the game, rather than what we just assumed were the most important elements.

OOPS, WE MESSED UP

Over the year-long course of the project, there was one incident which proved to us just how important it was to preserve the self-directed task structure of our research. Because of the multiphase progression of Spore, we believe it was important to carefully structure the sessions to give players a chance to play each phase for a predetermined amount of time, and in a set order as if they were experiencing the game normally.

Partway through the second session, we started having doubts: even though we weren’t telling players what to do within each phase, what if our rigid timing and sequencing is affecting the players’ engagement and involvement with the game?

To minimize this, between sessions, we made a significant change to the study design: instead of telling users to stop at predetermined intervals and proceed to the next phase of the game, we threw out timing altogether and allowed users to play any part of the game they wanted, for as long as they wanted, in whatever order they wanted. The only stipulation was that they should try each phase at least once. Each session lasted six hours spread over two nights, so there was more than enough time to cover all five phases, even without prompting users to do so.

Sure enough, we saw major differences in player feedback. We are unable to provide specific findings for legal reasons, but we can say that the ratings for certain phases consistently improved (as compared with previous sessions). Additionally, a few of the lukewarm comments players had made about certain aspects of the game seemed to stem from the limiting research format, rather than the game itself.

It became clear that when conducting game research, it was vitally important to stick to the actual realities of natural gameplay as much as possible, even at the expense of precisely structured research formatting. You have to loosen up the control a little bit; video games are, after all, interactive and fun. It makes no more sense to formalize a gameplay experience than it does to add flashy buttons and animated graphics to a spreadsheet application.

BRINGING GAME RESEARCH INTO THE HOME

There are a lot of ways to go with the native environment approach. Even with all efforts to keep the process as natural and unobtrusive as possible, there are still lots of opportunities to bring the experience even closer to players’ typical behaviors. The most obvious improvement is the promise of doing remote game research–allowing participants to play right at home, without even getting up.

Let’s consider what a hypothetical in-home game research session might look like: a player logs into XBox Live, and is greeted with a pop-up inviting him to participate in a one-hour user research study, to earn 8000 XBox Live points. (The pop-up is configured to appear only to players whose accounts are listed as 18 or older, to avoid issues of consent with minors.) The player agrees, and is automatically connected by voice chat to a research moderator, who is standing by. While the game is being securely delivered and installed to the player’s XBox, the moderator introduces the player to the study, and gets consent to record the session. Once the game is finished installing, the player tests the game for an hour, giving his think-aloud feedback the entire time, while the moderator takes notes and records the session. At the end of the session, the game is automatically and completely uninstalled from the player’s XBox, and the XBox Live points are instantly awarded to the player’s account.

Naturally, there are lots of basic infrastructure advances and logistical challenges to overcome before this kind of research becomes viable:

• Broadband penetration
• Participant access to voice chat equipment
• Online recruiting for games, preferably integrated into an online gaming framework
• Secure digital delivery of prototype or test build content
• Gameplay screensharing or mirroring

For many PC users, these requirements are already feasible, and for games with built-in chat and/or replay functionality, the logistics should already be much easier to meet. Remote research on PCs is already viable (and, in fact, happens to be Bolt|Peters’s specialty). Console game research, on the other hand, would likely require a substantial investment by console developers to make this possible; handheld consoles present even more challenges.

We expect that allowing players to give feedback at home, the most natural environment for gameplay, would yield the most natural feedback, bringing game evaluation and gameplay testing further into the domain of good UX research.

Spore Research: Outtakes from bolt peters on Vimeo.

Science of Fun from bolt peters on Vimeo.

From guerilla theatre to testimonials, posters to graphic instructions, users have benefited from alternative communication methods, particularly in situations where education or cultural barriers make it difficult for people to access services important to their well-being and safety. In some cases, the comic book format has been used as a way to help people get access to critical legal help. This case study from my time as a Publication Manager at the Legal Services Society (LSS) of British Columbia (BC) could inspire the use of comics outside the development process.

The Situation

BC has over 253 First Nations tribes (known as “Native Americans” in the United States), which is about one-third of all First Nations in Canada. Seven of Canada’s eleven unique native language families are located exclusively in BC. When BC joined Confederation (Canada) in 1871, the provincial policy of the day did not recognize aboriginal title to the land, so no treaties were signed with the First Nations unlike in other provinces.

Instead, the federal government made it a criminal offence for a First Nation to hire a lawyer to pursue land claims settlements, and removed a generation of children to residential schools, where many were abused and traumatized. As a result, many tribes were left in an ongoing state of economical and social upheaval, with rampant unemployment, social problems, and poverty.

The Legal Services Society (LSS) in BC is the provincial agency that provides legal aid to poor and marginalized residents of the province. Prior to the crippling budget cuts the government imposed in the late 1990s, LSS also provided public legal education material to people who didn’t quite qualify for legal aid but certainly needed it. They may not have been quite poor enough, or they were poor enough, but legal aid didn’t cover their particular problem.

LSS knew that solving some of the smaller problems up front would keep them from escalating into larger problems – problems that would then qualify them for legal aid, but also would be devastating for their lives.

In 1995, the LSS asked its Publishing Program, where I was the manager, to collaborate with them on some self-help material for First Nations women. The Native Services Department wanted to help these women understand their rights in the area of family law, specifically around the issue of spousal violence. Based on the number of women who came to social service agencies for help, LSS recognized that there were a number of issues that were not well understood and, if caught early, could be addressed to prevent larger legal problems.

The agency decided that it was within its mandate to produce a publication for this population segment, and the two departments began the process of creating the publication that would eventually be called Getting Out: Escaping Family Violence.

Why the Comics Format?

LSS produced all publications collaboratively. In this case, the two departments explored different formats, and ultimately chose the comic form. Comics’ graphical format could draw low-literacy women to pick up information off a publication rack. LSS had previously done one other publication in comic book format, which had worked for that audience.

The issue of family violence was a sensitive one, and the LSS had to be sure that the audience would not consider the graphical format of the publication condescending. To take the pulse of those who would use the publication, we conducted several focus groups in places where women would gather for learning (e.g. literacy, friendship, and women’s centers).

We used an approach that combined outreach, usability testing, literacy skills improvement, immigration intake, and legal education. We’d bring food and beverages, humbly ask questions, and be the learners instead of the teachers. Particularly with an all-women’s group, it was important to do something based around food. Participants would often bring their children, and they would ask us questions and giggle over our perspectives.

For this publication, a comic book format seemed to be a natural. The literacy levels in First Nations communities have been cited as being significantly lower than the general population, particularly in rural areas. Conveying the nuances of family law to a low-literacy population segment was one challenge; another was understanding specific cultural references that could be missed or become “localization” barriers.

Considerations similar to those for producing publications in different languages apply to those being translated from “majority culture English” to “minority culture English,” or same-language localization, so to speak. There may not be a language difference, strictly speaking, but significant dialectic differences apply, graphics are very culturally-specific, and emphasis differs between cultures. In this instance, we had to localize our content to make it relevant to our First Nations audience and not concern ourselves about whether the publication resonated with other people sitting in a legal aid waiting room.

Elements of Development

The commitment of the LSS to create effective material for our users extended to all aspects of the publication process.

Authoring–The content creation was undertaken by seeking out a subject matter expert in the topic area, usually a lawyer or case worker in one of the field offices. The author gathered profiles, based on cases from offices around the province, and distilled the important legal information that went into the publication. For this publication, I hired a television screenwriter named “Candis Callison”:http://www.cwy-jcm.org/en/aboutus/board/callison who was from the Tahltan band of First Nations to provide an authentic voice for the comic book.

Editorial–The editorial process was done in-house. For this project, the process included editing the script to fit the comic book genre. I also worked with the artist to ensure that the number of panels would fit the booklet format, and the dialog would fit the panels. Once the substantial edit was done, in-house staff did the copy edit. Then the Native Services lawyer, also First Nations, reviewed the publication for legal accuracy.

Production–As positions opened in the department, I was able to hire more culturally and ethnically diverse employees so that, eventually, we were able to produce and proofread material for diverse cultures and languages. (We produced material for recent immigrants, as well, in Chinese, Farsi, Spanish, Punjabi, and Vietnamese.) The new staffed helped greatly during the back-translation, where a publication is translated back into English to ensure translation integrity. In this case, the back-translation was not for language, but to ensure that cultural references were effective.

Art, A Critical Element

An LSS employee was friends with a budding artist named “Brian Jungen”:http://en.wikipedia.org/wiki/Brian_Jungen who was of Dunne-za (a First Nations tribe) and Swiss background. His artwork provided authentic visuals for the initial book. His work now hangs in the Vancouver Art Gallery, amongst other places, and I like to think he looks back fondly on the project.

How The Book Came About

The structure of the book took shape as the artist and I divided the script into chunks to fit the drawings, and then the drawings as necessary. As the Publishing Program manager, I took on the role of substantive editor for the writing and graphics. I also worked with the artist to figure out how to get exactly enough panels to fit the amount of print space allotted.

The structure of the book needed to be in multiples of four pages — minus both covers, the copyright page, and the title pages — and couldn’t exceed 8 pages of actual panels, to control costs. The story had to stay coherent within these constraints and couldn’t focus on the local color at the expense of delivering the legal message. All of that took quite a bit of balancing to keep the interest, use the right level of language, and keep the key legal phrases that would be important for someone to know. In the end, it worked.

Much like any other localized material, we had the material checked by a lawyer to ensure that no legal concepts were compromised during the “translation,” and then the material was tested with audiences to determine effectiveness. The Native Services Department fieldworker took copies of the storyboards out on a road trip to band offices and friendship centers.

We held our breath until word came back through the “moccasin grapevine” that the results were well received. This feedback loop was critical because it provided the opportunity to incorporate any changes that came up from the test.

In the End…

The publication story line opens with a guy in a plaid shirt (it has to be a plaid shirt) having an altercation with his wife. Then they’re at a pow-wow in a truck (it has to be a pick-up truck) where she warmly greets an old (male) friend. Then they’re at a party where he’s being abusive to his wife. By the end of the publication, the wife has identified that his verbal and physical violence is not acceptable, gotten a restraining order by following a few simple steps, and taken some basic legal steps without incurring huge legal costs.

One of the dialog bubbles states “… If he tries to do any of these things, we will arrest him again for breach of bail,” and then explains what the term “breach of bail” means. Another panel explains that, “If you live in a rural community far from the cities, Crown counsel [a prosecuting attorney] travels from community to community. You may have a different lawyer at the trial.”

The “insider” cultural perspectives made me feel a bit of a voyeur, but that very characteristic was what made it so effective. The 8.5” x 11” saddle-stitched booklet was immediately identifiable on the publication rack by its distinctive graphics. Also, the title, Getting Out, reflected the vernacular used by women in the community caught in situations of domestic violence so it was an instantly recognizable phrase.

The agency ran a modest print run of the publication, partly to contain printing costs in case of waste, and partly to gauge reaction to the publication. The booklets were distributed to legal aid offices, band offices, and other social service agencies where women were likely to go when they found themselves in marital distress. Offices and agencies were notified of its availability, and I mentioned it in passing during a radio interview.

The demand for the publication soon depleted the initial print run, and another was requested. The frontline workers liked the format, and handed it out to the women who didn’t quite qualify for legal aid but who clearly wouldn’t be able to afford a lawyer. Gathering post-production metrics was not a strong point at LSS, but by the measure of popular opinion, it was a winner, and the exercise was repeated with a companion publication entitled The Ministry Took My Kids, about parental rights when children are apprehended by social services.

In Part I of We Tried to Warn You, three themes were developed:
# Organizations as wicked problems,
# The differences of failure leverage in small versus large organizations, and
# The description of failure points

These should be considered exploratory elements of organizational architecture, from a communications information architecture perspective. While the organizational studies literature has much to offer about organizational learning mechanisms, we find very little about failure from the perspective of product management, management processes, or organizational communications.

Researching failure is similar to researching the business strategies of firms that went out of business (e.g., Raynor, 2007). They are just not available for us to analyze, they are either covered-up embarrassments, or they become transformed over time and much expense into “successes.”

In The Strategy Paradox, Raynor describes the “survivor’s bias” of business research, pointing out that internal data is unavailable to researchers for the dark matter of the business universe, those that go under. Raynor shows how a large but unknowable proportion of businesses fail pursuing nearly perfect strategies. (Going concerns often survive because of their mediocre strategies, avoiding the hazards of extreme strategies).

A major difference in the current discussion is that organizational failure as defined here does not bring down the firm itself, at least not directly, as a risky strategy might. But it often leads to complete reorganization of divisions and large projects, which should be recognized as a significant failure at the organizational level.

One reason we are unlikely to assess the organization as having failed is the temporal difference between failure triggers and the shared experience of observable events. Any product failure will affect the organization, but some failures are truly organizational. They may be more difficult to observe.

If a prototype design fails quickly (within a single usability test period), and a project starts and fails within 6 months, and a product takes perhaps a year to determine its failure – what about an organization? We should expect a much longer cycle from originating failure event to general acknowledgement of failure, perhaps 2-5 years.

There are different timeframes to consider with organizational versus project or product failure. In this case study, the failure was not observable until after a year or so of unexpectedly weak sales, with managers and support dealing with customer resistance to the new product.

However, decisions made years earlier set the processes in place that eventuated as adoption failure. Tracing the propagation of decisions through resulting actions, we also find huge differences in temporal response between levels of hierarchy (found in all large organizations).

Failures can occur when a chain of related decisions, based on bad assumptions, propagate over time. These micro-failures may have occurred at the time as “mere” communication problems.

In our case study, product requirements were defined based on industry best practices, guided by experts and product buyers, but excluding user feedback on requirements. Requirements were managed by senior product managers and were maintained as frozen specifications so that development decisions could be managed. Requirements become treated as-if validated by their continuing existence and support by product managers. But with no evaluation by end users of embodied requirements – no process prototype was demonstrated – product managers and developers had no insight into dire future consequences of product architecture decisions.

Consider the requisite timing of user research and design decisions in almost any project. A cycle of less than a month is a typical loop for integrating design recommendations from usability results into an iterative product lifecycle.

If the design process is NOT iterative, we see the biggest temporal gaps of all. There is no way to travel back in time to revise requirements unless the tester calls a “show-stopper,” and that would be an unlikely call from an internal usability evaluator.

In a waterfall or incremental development process, which remains typical for these large-scale products – usability tests often have little meaningful impact on requirements and development. This approach is merely fine-tuning foregone conclusions.

Here we find the seeds of product failure, but the organization colludes to defend the project timelines, to save face, to maintain leadership confidence. Usability colludes to ensure they have a future on the job. With massive failures, everyone is partly to blame, but nobody accepts personal responsibility.

h2. The Roles of User Experience

Figure 1. Failure case study organization – Products and project timeframes. (View figure 1 at full-size.)

As Figure 1 shows, UX reported to development management, and was further subjected to product and project management directives.

In many firms, UX has little independence and literally no requirements authority, and in this case was a dotted-line report under three competing authorities. That being the case, by the time formal usability tests were scheduled, requirements and development were too deeply committed to consider any significant changes from user research. With the pressures of release schedules looming, usability was both rushed and controlled to ensure user feedback was restricted to issues contained within the scope of possible change and with minor schedule impact.

By the time usability testing was conducted, the scope was too narrowly defined to admit any ecologically valid results. Usability test cases were defined by product managers to test user response to individual transactions, and not the systematic processes inherent in the everyday complexity of retail, service, or financial work.

* Testing occurred in a rented facility, and not in the retail store itself.
* The context of use was defined within a job role, and not in terms of productivity or throughput.
* Individual screen views were tested in isolation, not in the context of their relationship to the demands of real work pressures – response time, database access time, ability to learn navigation and to quickly navigate between common transactions.
* Sequences of common, everyday interactions were not evaluated.

And so on.

The product team’s enthusiasm for the new and innovative may prevent listening to the users’ authentic preferences. And when taking a conventional approach to usability, such fundamental disconnects with the user domain may not even be observable.

Many well-tested products have been released only to fail in the marketplace due to widespread user preference to maintain their current, established, well-known system. This especially so if the work practice requires considerable learning and use of an earlier product over time, as happened in our retail system case. Very expensive and well-documented failures abound due to user preference for a well-established installed base, with notorious examples in air traffic control, government and security, medical / patient information systems, and transportation systems.

When UX is “embedded” as part of a large team, accountable to product or project management, the natural bias is to expect the design to succeed. When UX designers must also run the usability tests (as in this case), we cannot expect the “tester” to independently evaluate the “designer’s” work. The same person in two opposing roles, the UX team reporting to product, and restricted latitude for design change (due to impossible delivery deadlines) – we should consider this a design failure in the making.

In this situation, it appears UX was not allowed to be effective, even if the usability team understood how to work around management to make a case for the impact of its discoveries. But the UX team may not have understood the possible impact at the time, but only in retrospect after the product failed adoption.

We have no analytical or qualitative tools for predicting the degree of market adoption based on even well-designed usability evaluations. Determining the likelihood of future product adoption failure across nationwide or international markets is a judgment call, even with survey data of sufficient power to estimate the population. Because of the show-stopping impact of advancing such a judgment, it’s unlikely the low-status user experience role will push the case, even if such a case is clearly warranted from user research.

h2. The Racket: The Organization as Self-Protection System

Modern organizations are designed to not fail. But they will fail at times when pursuing their mission in a competitive marketplace. Most large organizations that endure become resilient in their adaptation to changing market conditions. They have plenty of early warning systems built into their processes – hierarchical management, financial reports, project management and stage-gate processes. The risk of failure becomes distributed across an ever-larger number of employees, reducing risk through assumed due diligence in execution.

The social networks of people working in large companies often prevent the worst decisions from gaining traction. But the same networks also maintain poor decisions if they are big enough, are supported by management, and cannot be directly challenged. Groupthink prevails when people conspire to maintain silence about bad decisions. We then convince ourselves that leadership will win out over the risks; the strategy will work if we give it time.

Argyris’ organizational learning theory shows people in large organizations are often unable to acknowledge the long-term implications of learning situations. While people are very good at learning from everyday mistakes, they don’t connect the dots back to the larger failure that everyone is accommodating.

Called “double loop learning,” the goal is learn from an outcome and reconfigure the governing variables of the situation’s pattern to avoid the problem in the future. (Single-loop learning is merely changing one’s actions in response to the outcome). Argyris’ research suggests all organizations have difficulties in double-loop learning; organizations build defenses against this learning because it requires confrontation, reflection, and change of governance, decision processes, and values-in-use. It’s much easier to just change one’s behavior.

h2. What can UX do about it?

User experience/IA clearly plays a significant role as an early warning system for market failure. Context-sensitive user research is perhaps the best tool for available for informed judgment of potential user adoption issues.

Several common barriers to communicating this informed judgment have been discussed:

* Organizational defenses prevent anyone from advancing theories of failure before failure happens.
* UX is positioned in large organizations in a subordinate role, and may have difficulty planning and conducting the appropriate research.
* UX, reporting to product management, will have difficulty advancing cases with strategic implications, especially involving product failure.
* Groupthink – people on teams protect each other and become convinced everything will work out.
* Timing – by the time such judgments may be formed, the timeframes for realistic responsive action have disappeared.

Given the history of organizations and the typical situating of user experience roles in large organizations, what advice can we glean from the case study?

Let’s consider leveraging the implicit roles of UX, rather than the mainstream dimensions of skill and practice development.

h3. UX serves an Influencing role – so let’s influence

User experience has the privilege of being available on the front lines of product design, research, and testing. But it does not carry substantial organizational authority. In a showdown between product management and UX, product wins every time. Product is responsible for revenue, and must live or die by the calls they make.

So UX should look to their direct internal client’s needs. UX should fit research and recommendations to the context of product requirements, adapting to the goals and language of requirements management. We (UX) must design sufficient variability into prototypes to be able to effectively test expected variances in preference and work practice differences. We must design our test practices to enable determinations from user data as to whether the product requirements fit the context of the user’s work and needs.

We should be able to determine, in effect, whether we are designing for a product, or designing the right product in the first place. Designing the right product means getting the requirements right.

Because we are closest to the end user throughout the entire product development lifecycle, UX plays a vital early warning role for product requirements and adoption issues. But since that is not an explicit role, we can only serve that function implicitly, through credibility, influence and well-timed communications.

UX practice must continue to develop user/field research methods sensitive to detecting nascent problems with product requirements and strategy.

h3. UX is a recursive process – let’s make recursive organizations as well

User experience is highly iterative, or it fails as well. We always get more than one chance to fail, and we’ve built that into practices and standards.

Practices and processes are repeated and improved over time. But organizations are not flexible with respect to failure. They are competitive and defensive networks of people, often with multiple conflicting agendas. Our challenge is to encourage organizations to recurse (recourse?) more.

We should do this by creating a better organizational user experience. We should follow our own observations and learning of the organization as a system of internal users. Within this recursive system (in which we participate as a user), we can start by moving observations up the circle of care (or the management hierarchy if you will).

I like to think our managers do care about the organization and their shared goals. But our challenge here is to learn and perform from double-loop learning ourselves, addressing root causes and “governing variables” of issues we encounter in organizational user research. We do this by systematic reflection on patterns, and improving processes incrementally, and not just “fixing things” (single-loop learning).

We can adopt a process of socialization (Jones, 2007) rather than institutionalization, of user experience. Process socialization was developed as a more productive alternative to top-down institutionalization for the introduction of UX practices in organizations introducing UX into an intact product development process.

While there is strong theoretical support for this approach (from organizational structuration and social networks), socialization is recommended because it works better than the alternatives. Institutionalization demands that an organization establish a formal set of roles, relationships, training, and management added to the hierarchy to coordinate the new practices.

Socialization instead affirms that a longer-term, better understood, and organizationally resilient adoption of the UX process occurs when people in roles lateral to UX learn the practices through participation and gradual progression of sophistication. The practices employed in a socialization approach are nearly the opposite (in temporal order) of the institutionalization approach:

# Find a significant UX need among projects and bring rapid, lightweight methods to solve obvious problems
# Have management present the success and lessons learned
# Do not hire a senior manager for UX yet, lateral roles should come to accept and integrate the value first
# Determine UX need and applications in other projects. Provide tactical UX services as necessary, as internal consulting function.
# Develop practices within the scope of product needs. Engage customers in field and develop user and work domain models in participatory processes with other roles.
# Build an organic demand and interest in UX. Provide consulting and usability work to projects as capability expands. Demonstrate wins and lessons from field work and usability research.
# Collaborate with requirements owners (product managers) to develop user-centered requirements approach. Integrate usability interview and personas into requirements management.
# Integrate with Product Development. Determine development lifecycle decision points and user information required.
# Establish User Experience as process and organizational function
# Provide awareness training, discussion sessions, and formal education as needed to fit UX process.
# Assessment and renewal, staffing, building competency

We should create more opportunities to challenge failure points and process breakdowns. Use requirements reviews to challenge the fit to user needs. Use a heuristic evaluation to bring a customer service perspective on board. In each of those opportunities, articulate the double-loop learning point. “Yes, we’ll fix the design, but our process for reporting user feedback limits us to tactical fixes like these. Let’s report the implications of user feedback to management as well.”

We can create these opportunities by looking for issues and presenting them as UX points but in business terms, such as market dynamics, competitive landscape, feature priority (and overload), and user adoption. This will take time and patience, but then, its recursive. In the long run we’ll have made our case without major confrontations.

h2. Conclusions

Scott Cook, Intuit’s Founder, famously said at CHI 2006: “The best we can hope to bat is .500. If you’re getting better than that, you’re not swinging for the fences. Even Barry bonds, steroids or not, is not getting that. We need to celebrate failure.”

Intelligent managers actually celebrate failures – that’s how we learn. If we aren’t failing at anything, how do we know we’re trying? The problem is recognizing when failure is indeed an option.

How do we know when a project so large – an organizational level project – will belly-up? How can something so huge and spectacular in its impact be so hard to call, especially at the time decisions are being made that could change the priorities and prevent an eventual massive flop? The problem with massive failure is that there’s very little early warning in the development system, and almost none at the user or market level.

When product development fails to respect the user, or even the messenger of user feedback, bad decisions about interface architecture compound and push the product toward an uncertain reception in the marketplace. Early design decisions compound by determining architectures, affecting later design decisions, and so on through the lifecycle of development.

These problems can be compounded even when good usability research is performed. When user research is conducted too late in the product development cycle, and is driven by usability questions related to the product and not the work domain, development teams are fooled into believing their design will generalize to user needs across a large market in that domain. But at this point in product development, the fundamental platform, process, and design decisions have been made, constraining user research from revisiting questions that have been settled in earlier phases by marketing and product management.

h2. References

Argyris, C. (1992). On organizational learning. London: Blackwell.

Howard, R. (1992). The CEO as organizational architect: an interview with Xerox’s Paul Allaire. Harvard Business Review, 70 (5), 106-121.

Jones, P.H. (2007). Socializing a Knowledge Strategy. In E. Abou-Zeid (Ed.) Knowledge Management and Business Strategies: Theoretical Frameworks and Empirical Research, pp. 134-164. Hershey, PA: Idea Group.

Raynor, M.E. The strategy paradox: Why committing to success leads to failure (and what to do about it). New York: Currency Doubleday.

Rittel, H.W.J. and Weber, M.W. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4, 155-169.

Taleb, N.N (2007).The Black Swan: The Impact of the Highly Improbable. New York: Random House.

There are many kinds of failure in large, complex organizations – breakdowns occur at every level of interaction, from interpersonal communication to enterprise finance. Some of these failures are everyday and even helpful, allowing us to safely and iteratively learn and improve communications and practices. Other failures – what I call large-scale – result from accumulated bad decisions, organizational defensiveness, and embedded organizational values that prevent people from confronting these issues in real time as they occur.

So while it may be difficult to acknowledge your own personal responsibility for an everyday screw-up, it’s impossible to get in front of the train of massive organizational failure once its gained momentum and the whole company is riding it straight over the cliff. There is no accountability for these types of failures, and usually no learning either. Leaders do not often reveal their “integrity moment” for these breakdowns. Similar failures could happen again to the same firm.

I believe we all have a role to play in detecting, anticipating, and confronting the decisions that lead to breakdowns that threaten the organization’s very existence. In fact, the user experience function works closer to the real world of the customer than any other organizational role. We have a unique responsibility to detect and assess the potential for product and strategic failure. We must try to stop the train, even if we are many steps removed from the larger decision making process at the root of these failures.

h2. Organizations as Wicked Problems

Consider the following scenario: A $2B computer systems integrator provider spends most of a decade developing its next-generation platform and product, and spends untold amounts in labor, licenses, contracting, testing, sales and marketing, and facilities. Due to the extreme complexity of the application (user) domain, the project takes much longer than planned. Three technology waves come and go, but are accommodated in the development strategy: Proprietary client-server, Windows NT application, Internet + rich client.

By the time Web Services technologies matured, the product was finally released as a server-based, rich client application. However, the application was designed too rigidly for flexible configurations necessary for the customer base, and the platform performance compared poorly to the current product for which the project was designed as a replacement. Customers failed to adopt the product, and it was a huge write-off of most of a decade’s worth of investment.

The company recovered by facelifting its existing flagship product to embrace contemporary user interface design standards, but never developed a replacement product. A similar situation occurred with the CAD systems house SDRC, whose story ended as part two of a EDS fire sale acquisition of SDRC and Metaphase. These failures may be more common that we care to admit.

From a business and design perspective, several questions come to mind:
* What were the triggering mistakes that led to the failure?
* At what point in such a project could anyone in the organization have predicted an adoption failure?
* What did designers do that contributed to the problem? What could IA/designers have done instead?
* Were IA/designers able to detect the problems that led to failure? Were they able to effectively project this and make a case based on foreseen risks?
* If people act rationally and make apparently sound decisions, where did failures actually happen?

This situation was not an application design failure; it was a total organizational failure. In fact, it’s a fairly common type of failure, and preventable. Obviously the market outcome was not the actual failure point. But as the product’s judgment day, the organization must recognize failure when goals utterly fail with customers. So if this is the case, where did the failures occur?

It may be impossible to see whether and where failures will occur, for many reasons. People are generally bad at predicting the systemic outcomes of situational actions – product managers cannot see how an interface design issue could lead to market failure. People are also very bad at predicting improbable events, and failure especially, due to the organizational bias against recognizing failures.

Organizational actors are unwilling to acknowledge small failures when they have occurred, let alone large failures. Business participants have unreasonably optimistic expectations for market performance, clouding their willingness to deal with emergent risks. We generally have strong biases toward attributing our skills when things go well, and to assigning external contingencies when things go badly. As Taleb (2007)1 says in The Black Swan:

bq. “We humans are the victims of an asymmetry in the perception of random events. We attribute our success to our skills, and our failures to external events outside our control, namely to randomness. We feel responsible for the good stuff, but not for the bad. This causes us to think that we are better than others at whatever we do for a living. Ninety-four percent of Swedes believe that their driving skills put them in the top 50 percent of Swedish drivers; 84 percent of Frenchmen feel that their lovemaking abilities put them in the top half of French lovers.” (p. 152).

Organizations are complex, self-organizing, socio-technical systems. Furthermore, they can be considered “wicked problems,” as defined by Rittel and Webber (1973)2. Wicked problems require design thinking; they can be designed-to, but not necessarily designed. They cannot be “solved,” at least not in the analytical approaches of so-called rational decision makers. Rittel and Webber identify 10 characteristics of a wicked problem, most of which apply to large organizations as they exist, without even identifying an initial problem to be considered:

# There is no definite formulation of a wicked problem.
# Wicked problems have no stopping rules (you don’t know when you’re done).
# Solutions to wicked problems are not true-or-false, but better or worse.
# There is no immediate and no ultimate test of a solution to a wicked problem.
# Every solution to a wicked problem is a “one-shot operation”; because there is no opportunity to learn by trial-and-error, every attempt counts significantly.
# Wicked problems do not have an enumerable set of potential solutions.
# Every wicked problem is essentially unique.
# Every wicked problem can be considered to be a symptom of another [wicked] problem.
# The causes of a wicked problem can be explained in numerous ways.
# The planner has no right to be wrong.

These are attributes of the well-functioning organization, and apply as well to one pitched in the chaos of product or planning failure. The wicked problem frame also helps explain why we cannot trace a series of decisions to the outcomes of failure – there are too many alternative options or explanations within such a complex field. Considering failure as a wicked problem may offer a way out of the mess (as a design problem). But there will be no way to trace back or even learn from the originating events that the organization might have caught early enough to prevent the massive failure chain.

So we should view failure as an organizational dynamic, not as an event. By the time the signal failure event occurs (product adoption failure in intended market), the organizational failure is ancient history. Given the inherent complexity of large organizations, the dynamics of markets and timing products to market needs, and the interactions of hundred of people in large projects, where do we start to look for the first cracks of large-scale failure?

h2. Types of Organizational Failure

How do we even know when an organization fails? What are the differences between a major product failure (involving function or adoption) and a business failure that threatens the organization?

An organizational-level failure is a recognizable event, one which typically follows a series of antecedent events or decisions that led to the large-scale breakdown. My working definition:

“When significant initiatives critical to business strategy fail to meet their highest-priority stated goals.”

When the breakdown affects everyone in the organization, we might say the organization has failed as whole, even if only a small number of actors are to blame. When this happens with small companies, such as the start-up I worked with early in my career as a human factors engineer, the source and the impact are obvious.

Our company of 10 people grew to nearly 20 in a month to scale up for a large IBM contract. All resources were brought into alignment to serve this contract, but after about 6 months, IBM cut the contract – a manager senior to our project lead hired a truck and carted away all our work product and computers, leaving us literally sitting at empty desks. We discovered that IBM had 3 internal projects working on the same product, and they selected the internal team that had finished first.

That team performed quickly, but their poor quality led to the product’s miserable failure in the marketplace. IBM suffered a major product failure, but not organizational failure. In Dayton, meanwhile, all of us except the company principals were out of work, and their firm folded within a year.

Small organizations have little resilience to protect them when mistakes happen. The demise of our start-up was caused by a direct external decision, and no amount of risk management planning would have landed us softly.

I also consulted with a rapidly growing technology company in California (Invisible Worlds) which landed hard in late 2000, along with many other tech firms and start-ups. Risk planning, or its equivalent, kept the product alive – but this start-up, along with firms large and small, disappeared during the dot-bomb year.

To what extent were internal dynamics to blame for these organizational failures? In retrospect, many of the dot-bombs had terrible business plans, no sustainable business models, and even less organic demand for their services. Most would have failed in a normal business climate. They floated up with the rise of investor sentiment, and crashed to reality as a class of enterprises, all of them able to save face by blaming external forces for organizational failure.

h2. Organizational Architecture and Failure Points

Recognizing this is a journal for designers, I’d like to extend our architectural model to include organizational structures and dynamics. Organizational architecture may have been first conceived in R. Howard’s 1992 HBR article “The CEO as organizational architect.” (The phrase has seen some academic treatment, but is not found in organizational science literature or MBA courses to a great extent.)

Organizations are “chaordic” as Dee Hock termed it, teetering between chaotic movement and ordered structures, never staying put long enough to have an enduring architectural mapping. However, structural metaphors are useful for planning, and good planning keeps organizations from failing. So let’s consider the term organizational architecture metaphorical, but valuable – giving us a consistent way of teasing apart the different components of a large organization related to decision, action, and role definition in large project teams.

Let’s start with organizational architecture and consider its relationships to information architecture. The continuity of control and information exchange between the macro (enterprise) and micro (product and information) architectures can be observed in intra-organizational communications. We could honestly state that all such failures originate as failures in communications. Organizational structure and processes are major components, but the idea of “an architecture,” as we should well know from IA, is not merely structural. An architectural approach to organizational design involves at least:

• *Structures*: Enterprise, organizational, departmental, networks
• *Business processes*: Product fulfillment, Product development, Customer service
• *Products*: Structures and processes associated with products sold to markets
• *Practices*: User Experience, Project management, Software design
• *People and roles*: Titles, positions, assigned and informal roles
• *Finance*: Accounting and financial rules that embed priorities and values
• *Communication rules*: Explicit and implicit rules of communication and coordination
• *Styles of interaction*: How work gets done, how people work together, formal behaviors
• *Values*: Explicit and tacit values, priorities in decision making

Since we would need a book to describe the function and relationships within and between these dimensions, let’s see if the whole view suffices.

Each of these components are significant functions in the organizational mix, all reliant on communication to maintain its role and position in the internal architecture. While we may find may have a single communication point (a leader) in structures and people, most organizational functions are largely self-organizing, continuously reified through self-managing communication. They will not have a single failure point identifiable in a communication chain, because nearly all organizational conversations are redundant and will be propagated by other voices and in other formats.

Really bad decisions are caught in their early stages of communication, and become less bad through mediation by other players. So organizations persist largely because they have lots of backup. In the process of backup, we also see a lot of cover-up, a significant amount of consensus denial around the biggest failures. The stories people want to hear get repeated. You can see why everyday failures are easy to catch compared to royal breakdowns.

So are we even capable of discerning when a large-scale failure of the organizational system is immanent? Organizational failure is not a popular meme; employees can handle a project failure, but to acknowledge that the firm broke down – as a system – is another matter.

According to Chris Argyris (1992), organizational defensive routines are “any routine policies or actions that are intended to circumvent the experience of embarrassment or threat by bypassing the situations that may trigger these responses. Organizational defensive routines make it unlikely that the organization will address the factors that caused the embarrassment or threat in the first place. (p. 164)” Due to organizational defenses most managers will place the blame for such failure on individuals rather than the consequences of poor decisions or other root causes, and will deflect critique of the general management or decision making processes.

Figure 1 shows a pertinent view of the case organization, simplifying the architecture (to People, Process, Product, and Project) so that differences in structure, process, and timing can be drawn.

Projects are not considered part of architecture, but they reveal time dynamics and mobilize all the constituents of architecture. Projects are also where failures originate.

The timeline labeled “Feedback cycle” shows how smaller projects cycled user and market feedback quickly enough to impact product decisions and design, usually before initial release. Due to the significant scale, major rollout, and long sales cycle of the Retail Store Management product, the market feedback (sales) took most of a year to reach executives. By then, the trail’s gone cold.

Figure 1. Failure case study organization – Products and project timeframes. (View figure 1 at full-size.)

Over the project lifespan of Retail Store Management, the organization:

• Planned a “revolutionary” not evolutionary product
• Spun off and even sequestered the development team – to “innovate” undisturbed by the pedestrian projects of the going concern
• Spent years developing “best practices,” for technology, development, and the retail practices embodied in the product
• Kept the project a relative secret from rest of the company, until close to initial release
• Evolved technology significantly over time as paradigms changed, starting as an NT client-server application, then distributed database, finally a Web-enabled rich client interface.

Large-scale failures can occur when the work domain and potential user acceptance (motivations and constraints) are not well understood. When a new product cannot fail, organizations will prohibit acknowledging even minor failures, with cumulative failures to learn building from small mistakes. This can lead to one very big failure at the product or organizational level.

We can see this kind of situation (as shown in Figure 1) generates many opportunities for communications to fail, leading to decisions based on biased information, and so on. From an abstract perspective, modeling the inter-organizational interactions as “boxes and arrows,” we may find it a simple exercise to “fix” these problems.

We can recommend (in this organization) actions such as educating project managers about UX, creating marketing-friendly usability sessions to enlist support from internal competitors, making well-timed pitches to senior management with line management support, et cetera.

But in reality, it usually does not work out this way. From a macro perspective, when large projects that “cannot fail” are managed aggressively in large organizations, the user experience function is typically subordinated to project management, product management, and development. User experience – whether expressing its user-centered design or usability roles – can be perceived as introducing new variables to a set of baselined requirements, regardless of lifecycle model (waterfall, incremental, or even Agile).

To make it worse (from the viewpoint of product or requirements management), we promote requirements changes from the high-authority position conferred by the reliance on user data. Under the organizational pressures of executing a top-down managed product strategy, leadership often closes ranks around the objectives. Complete alignment to strategy is expected across the entire team. Late-arriving user experience “findings” that could conflict with internal strategy will be treated as threatening, not helpful.

With such large, cross-departmental projects, signs of warning drawn from user data can be simply disregarded, as not fitting the current organizational frame. And if user studies are performed, significant conflicts with strategy can be discounted as the analyst’s interpretation.

There are battles we sometimes cannot win. In such plights, user experience professionals must draw on inner resources of experience, intuition, and common sense and develop alternatives to standard methods and processes. The quality of interpersonal communications may make more of a difference than any user data.

In Part II, we will explore the factors of user experience role, the timing dynamics of large projects, and several alternatives to the framing of UX roles and organizations today.

Introduction

One of the challenges faced most often by those of us in the field of usability is finding good data about user behavior quickly, accurately, and, in most cases, cheaply. In an environment where many stakeholders question the return on investment in usability, some in the industry have developed interesting ideas aimed at gathering user data. One such idea is the analysis of server log files to gather information about user behavior. On the surface, it is easy to understand the gravitation towards server logs: They’re supposedly a data source which portrays what people are doing on a site. Server logs supposedly show what people click on, which pages they view, and how they get from page to page.

Unfortunately, practitioners who espouse such methods seem to lack important technical knowledge regarding the nature of the web, the Hypertext Transfer Protocol (HTTP) and the process of caching within networks, proxies, ISPs, and browsers. These technical details greatly limit the types and quality of information that can be retrieved from server logs.

In addition to the technical limitations of server log file analysis, without information regarding exactly what the user expects to find and why he makes the choices he makes, there’s no way for us to know whether he was successful in his quest and whether that quest was satisfying. Ultimately that is the usability information we seek.

Server log files are inappropriate for gathering usability data. They are meant to provide server administrators with data about the behavior of the server, not the behavior of the user. The log file is a flat file containing technical information about requests for files on the server. Log file analysis tools merely assemble them in a conjecture-based format aimed at providing insight into user behavior. In the commentary below, I will explain why the nature of the web, the HTTP Protocol, the browser, and human behavior make it impossible to derive meaningful usability data from server logs.

First, some technical background information is needed.

What is a Server Log File?

Server traffic logs are files generated by the server in order to provide information about requests to the server for data. When a computer connects to a site, the computer, browser, and network will deliver some data to the site’s server itself to create a record that a file was requested. Here’s what an entry into a log file looks like:

86.42.132.114 – - [31/Oct/2005:18:15:16 -0500] "GET /styles/style.css HTTP/1.1" 200 5194 "http://www.example.com/links/links.php?cat=css" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7"

The format above is from an Apache log. Depending on the type of server the site is on, the log entries may look different. Thousands (or even hundreds of thousands) of entries such as the one above are placed into a plain text file, called the server log.

The above log entry includes the following information:

1. IP address of the requesting computer: 86.42.132.114. This is not the user's IP address, but rather the address of the host machine they've connected to.
2. Date and time of the request: [31/Oct/2005:18:15:16 -0500]. That's October 31, 2005 at 6:15:16pm and the time zone is 5 hours behind GMT, which is Eastern Standard Time in the USA (this is because the server is in that time zone, not the user.)
3. The full HTTP request: "GET /styles/style.css HTTP/1.1"
   1. Request method: GET
   2. Requested file: /styles/style.css
   3. HTTP Protocol version: HTTP/1.1
4. HTTP Response Code: 200. This particular code means the request was ok.
5. Response size: 5194 bytes. This is the size of the file that was returned.
6. Referring document: http://www.example.com/links/links.php?cat=css. The links.php file is referring to its embedded style sheet.
7. User-Agent String (Browser & Operating system information):"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7". The user’s computer is using Firefox browser on Windows XP and the language is set to English – US.

Of all of the information in the log entry, only the time and date, HTTP Request, and the response information should be regarded as accurate. The IP address, referrer, and user-agent string should be regarded as unreliable, as it can be faked in some way by the user. For example, the user has the option with Netscape 8 to publicly identify the browser as Internet Explorer during the setup process and many other browsers offer this option in their “Options” or “Preferences” menus as well.

Analysis Tools

Many organizations use an analysis program to parse server log files so that they’re much easier to understand. Imagine trying to cull anything of value from a huge text file of entries like the one above when hundreds of thousands (or even millions) of entries are present in the log file! Essentially, the analysis tool treats the log file as a flat-file database, processes it, and generates the “statistics” that are discussed throughout the rest of this commentary. In other words, “Web Analytics” software does little more than provide its own interpretation of data contained in the log file which, as stated above, could be at least partially faked.

[Note: I realize that some analytics software gather data by other means than parsing log files and may in fact contain some features meant to overcome one or more of the criticisms I outline throughout this article. I do not discuss such programs, primarily because there is little consistency between them and ultimately they are just as poor at gathering real usability data as analytic tools which parse log files.]

How the Web Works

In order to understand how Web server log files work and why they are considered inappropriate for measuring usability, it is important to get a brief background on how the web itself works.

The HTTP Protocol

HTTP, Hypertext Transfer Protocol, is “A protocol used to request and transmit files, especially webpages and webpage components, over the Internet or other computer network.” [1] HTTP dictates the manner in which computers, servers, and browsers transfer data over the Web. HTTP is known as a “stateless” protocol, meaning that an HTTP connection to a site is not continuous. The steps involved in requesting a Web page are:

1. The user requests a page by following a link or typing an address in the browser’s address bar.
2. The browser requests a page.
3. The server responds by delivering the page, which is displayed in the user's browser window.
4. The connection between the user’s computer and the Web server is severed. At this point, the transaction between the user and the website is considered "done" as far as the server is concerned.

Every request for a new page on the server initiates these steps. In fact, this process occurs for every element requested on that page. Therefore, if there is a page with 10 images and a style sheet, the above routine will repeat 12 times (1 page + 1 style sheet + 10 images = 12 requests).

The reasoning behind all this is important to keep in mind, because in the early days of the Web, computers were slow. Servers were slow and networks were slow as well. This slowness was compensated for in two ways: by opening and closing connections for each request as described above (rather than as one big stream), and by "caching" the data as well.

Caching

Caching is defined as “local storage of remote data designed to reduce network transfers and therefore increase speed of download. The cache is a ‘storeroom’ where the data is kept.” [2] To put it more simply: Computers store data in a cache on the user’s computer so that they can get to that data again without downloading it each time it’s needed. For the Web this is important because, as mentioned above, the historical slowness of the Web connections and computers was a severe limitation on the overall usability of the Web.

Caching, even with the speed of today’s Web connections and the power of modern personal computers, enhances performance. For example, in a very simple site where most pages have only two images and a style sheet, each page generates only four requests – again, 1 page + 1 style sheet + 2 images = 4 requests. Without caching, every page viewed on the site would require the server to send four files to the computer when only one is really necessary: the new document. This not only goes for the individual elements embedded into the page, but also the page itself. If a user visits a page, then another, then wants to return to that first page, the computer should not have to download the same page again when it has already been downloaded once. Even with the increasing number of broadband users, if everything had to be downloaded repeatedly for every page viewed, the computer, the server, and the Internet would get bogged down transferring and receiving this enormous volume of redundant data.

Who caches?

Everyone caches. Every browser of every user throughout the world has a cache generated while browsing. Default settings for the cache are set rather high by browser manufacturers in an attempt to increase the performance (and therefore usability) for the user. The screen capture below shows the default cache size for Internet Explorer 7 as 124 Megabytes:

As a browser’s cache fills up, more and more pages which the user views frequently (such as on their favorite sites) will be retrieved right from cache, rather than arriving via new requests to the server.

In addition, almost all corporate and institutional networks have caches (most users would experience a cache like this at their job), as do almost all Internet Service Providers (ISP), such as AOL or MSN. The larger the network or ISP, the larger the cache.

A quote from Stephen Turner, developer of Analog Stats, explains it this way: “This means that if I try to look at one of your pages and anyone else from the same ISP has looked at that page, the cache will have saved it, and will give it [the page] out to me without ever telling you [the source of the web page] about it. (This applies whatever my browser settings). So hundreds of people could read your pages, even though you’d only sent it out once.”

Why cache?

Caching saves on bandwidth and hardware needs, and provides overall usability for everyone on the web. This means HTTP is really more complicated than the previous description of four steps. It now becomes:

Caching Wreaks Havoc on Statistics

From the image displayed above, we begin to realize the problems with looking at server logs. There may never be a connection between the user’s computer and the site’s server in order to fulfill the user’s request to see the page.

Caching is definitely a good thing. Its importance for the overall usability of the Web cannot be overstated. But for understanding log file analysis, the purpose of this article, caching makes all site traffic stats completely inaccurate. To further invalidate the stats, the most popular pages of the site are likely to be cached more often, thus creating more problems in getting accurate log data.

Caching’s Effect on Margin-of-Error

It’s reasonable to take the position that we don’t need 100% accurate logs to get reliable data. We just need an acceptable margin of error. If the stats in question are from a small sized sample (read as: a less popular site with little traffic), then our margin of error in these statistics will be very high. To overcome the problem of a large margin of error, a larger sample size (a more popular site) might seem appropriate. Unfortunately, the more popular the site, the more likely it is that the site will be cached by networks and proxies, and the more likely it is that AOL users will be swapping Internet Protocol (IP) Addresses with each other (more on that later). Sites with large numbers of visitors may never achieve an acceptable margin of error because so much caching occurs that the data are not accurate.

Usability Data Cannot Be Found In Log Files

It is helpful to remember that the ONLY things that log files can report are regarding requests to the server. So if the user’s network or ISP has a cached copy of your page, then there is no request. If there was no request, then no log entry is created. If no log entry is created, the user’s desire for that page is not accounted for in the stats.

Usability Data: Who Is Coming To Your Site?

As demonstrated in the beginning of this article, log entries do not include any demographic data about a user. The log entry cannot tell the user’s age, sex, race, education, experience with computers and the Internet, experience with the organization’s product or service, or any of the other information usability practitioners typically include when generating a “persona.” The closest thing to “who” information is the logged IP address.

In the log entry example provided in the beginning of this article, 86.42.132.114 is an address owned by Eircom IP Networks Group from Dublin Ireland. But even this does not equate to definitive information about the identity of the user. Eircom’s own web site says they offer services to businesses and individuals. So, this could be a home surfer, a secretary, or the president of a company – all three of whom could have very different demographic qualities and reasons for coming to the site. The only possible assumption one could make is that the person came from somewhere in Ireland. But even that may not be true. The IP address recorded in the log file is the IP address of the host machine that the user was connected to when they accessed the site. The user could actually be very far away from that host machine or could even be using a program which “anonymizes” him, hiding his location, and making him seem like he’s in an entirely different country. For usability purposes, there simply is no way of knowing anything accurate about “who” is visiting the site – not even where he is located.

Usability Data: What Information is Requested?

Log files cannot tell how many actual times a page from the site was viewed because caching causes some "traffic" to not get counted. Even if we pretend that the page counts are accurate, the bigger issue is that "what information they're requesting" provides no data of value for usability.

• Log files don't indicate what information users want.
• Log files don't indicate what information users expected to find.
• Log files don't indicate whether that request fulfilled the user's needs.
• Log files don't indicate whether that information was easy to find.
• Log files don't indicate whether that information was easy to use once users found it.

At most, a measure of page requests can indicate that users thought they could get what they wanted there, but we still don't know exactly what 'it' was they wanted. Without information about what they wanted, there's no way of knowing whether their page view satisfactorily gave the users what they came for.

Furthermore, some page views could be artificially inflated by users going from page-to-page looking for something they cannot find. Page views of some pages could also be inflated as users drill down into a site's architecture as they seek what they really want. No matter whether the request is a success or failure, pages in the site could easily seem to have large requests even if those pages are just on the way or in the way.

Let’s imagine that the user of a newspaper site wants a medicine-related story published June 20, 2004 and that page is located at: Archives -> 2004 -> June -> 20th -> Medicine -> Story [goal]. Further, let's imagine the user's response is to follow the path above until the June 20th edition has been found. In doing so, this increases the page views to five pages to find the story in question.

If a large number of users who peruse the archives of the site use a similar scheme, in the overall traffic measure they have the affect of "artificially" increasing hits to all of the pages at the top levels as user after user branches off of them. Ultimately that data does not provide usability information. The measure of a site's usability is whether the user succeeded in doing that which he set out to do and whether he felt it was an easy thing to do. Page requests have no direct bearing on any true aspects of usability and indicate neither success nor satisfaction.

Usability Data: Can Log Files Indicate How People Navigate on a Site?

Another misunderstanding is found in claims that log files can describe “how people navigate” on a site. As before, the problem with this claim lies in the issue of caching. Even in a best-case scenario the only requests that would be counted would be requests for pages the user had not seen already. Caching means that anytime the user goes back to a page they’ve already seen, that page view is not counted. So, with that understanding, how can log files tell us “how people navigate”?

Tracking a participant's usage of the Sears.com website's tool section, we saw this example played out in real time. The pages visited were as follows:

1. Home
2. Tools
3. Air Compressors
4. Automotive Air Tools
5. Drills
6. Craftsman 1/2 in. Professional One-Touch Drill
7. "Back" to Drills – comes from cache
8. Chicago Pneumatic 3/8 in. Angle Drill
9. "Back" to Drills – comes from cache
10. Craftsman 1/2 in Heavy Duty Reversible Drill
11. "Back" to Drills – comes from cache
12. "Back" to Automotive Air Tools – comes from cache
13. Grinders
14. Craftsman 7 in. Angle Grinder
15. "Back" to Grinders – comes from cache
16. Craftsman 1/4 in Die Grinder kit
17. "Back" to Grinders – comes from cache
18. Craftsman Die Grinder
19. "Back" to Grinders – comes from cache
20. "Back" to Automotive Air Tools – comes from cache
21. Sanders
22. Craftsman Dual Action Sander
23. "Back" to Sanders – comes from cache
24. Chicago Pneumatic Dual Action Sander
25. "Back" to Sanders – comes from cache
26. Craftsman High Speed Rotary Sander

As the page list above demonstrates, as a user browses through the site, the number of actual requests may very well diminish because more and more pages are getting cached as the user navigates through the site. Thus, as they browse and eventually return to pages they've already seen their requests for those pages are not counted in the server's log file. Using log files to track a user's visit as a way to tell how people navigate would result in numerous gaps between what pages the log files indicate were visited and what pages the user actually did visit. So, log files are not a way to identify trends in how visitors navigate.

Moreover, what usability data could a list of pages visited provide? Even if there was no caching anywhere, all we'd have is data about a series of requests that a user made during a visit to the site. To repeat we are left with the following stark realities:

• The user's path does not tell us what the user wanted.
• The user's path does not tell us what information they expected to find.
• The user's path does not tell us whether that request fulfilled their needs.
• The user's path does not tell us whether that information was easy to find.
• The user's path does not delineate distractions that interfered with an initial purpose.
• The user's path does not tell us whether that information was easy to use once it was found.

Usability Data: What Element(s) Did the User Click?

Some sources state that you can determine which element (link, icon, etc.) a visitor clicked on the page to go to the next page. There is simply no component of web sites, web servers, or server log analysis programs which would tell exactly which link/ icon/ button the user clicked to navigate.

Instead, analytics programs "guess" at this information by interpreting the referring pages for a request. For example, if a user follows a link to the site's "News" page, and the referring document was the "Home" page, then the analytics program will locate the "News" link on the "Home" page and say that's where the user clicked. This data is rendered completely unreliable in any instance where there are two links on the page that go to the same destination.

There are those who've proposed methods using DOM Scripting or Ajax to gather this information. Unfortunately, such proposed methods typically involve means which would cause duplicate requests to the server. Such methods – while interesting – are inappropriate for a production environment as these duplicate requests are likely to cause performance problems for the site. These methods would be excellent for a very brief A/B test, but long-term data gathering in such a manner should be avoided.

Usability Data: How Long Did the User View the Page(s)?

Caching again makes it impossible to reliably generate such a statistic. Using the trip to Sears.com as an example again, we can see numerous times when our user visits a new page, and then returns to a page he has already seen, then a new page again. What can happen is that, as far as the logs are concerned, the time the server thinks the user spends viewing page(s) will be artificially increased because it isn’t counting the time he spent re-reading a page he has already seen. The traffic analysis tool then makes an assumption that the gap between new requests (because the other pages were in cache) is the time that was spent on the previous page. This can make it seem like users are spending longer on the pages than they really are, and not actually counting the pages on which they actually spent their time.

If you could tell how long the user "viewed" the page, what conclusion could you draw?

When our user was on the Sears.com web site looking for a part for his air compressor, he realized that he also needed a new pressure gauge because the old one wasn't working. So, he went to Sears.com and clicked on "Parts" on the left. The user then arrived at a new site for the Sears Parts Store. As soon as he got there, he saw that he could enter the actual part number for the gauge itself rather than browsing for it. He then left his computer, went to the basement, and spent approximately the next 5 minutes going through a stack of owner's manuals to get the part number for the gauge. Returning, he entered the part number, went about finding the part on the site and continued with his visit.

With the web server logs as the only data about what our user did, what conclusion could be drawn? As far as the server logs go, he spent 5 minutes at the home page of the site before going to the page listing the part.

• Does that 5 minute page view time mean he had difficulty understanding the page? Or,
• Does that mean he found the page particularly interesting?

Of course, neither is true. He wasn’t even at the computer for those five minutes. But the server logs don’t register, “User walked away to look for something.” Server logs don’t know whether the user went to get a cup of coffee, walk the dog, write something down, print something off, or left the house to go shopping. How long a user spends on a page means nothing without additional information about what the user was doing, why the user was doing it, and whether the experience was successful and satisfying. If someone isn’t with the user, there is no way to know what is contributing to the length of time users spend between new requests.

Usability Data: When Do Visitors Leave Your Site?

Server log analysis tools report the last page the user has requested during their visit as their "exit point." These programs establish a certain amount of time that they assume to be long enough to constitute a single user’s session. Most analysis tools allow the server administrator to set what is called a "Visit Timeout" after which point if there is no more activity from that IP address, it regards the user’s visit as over. Any additional activity from that IP address will be regarded as a new visit and likely to be counted as a "repeat visit" by the analysis tool. Most analysis tools’; timeouts are set, by default, to 20 or 30 minutes. Even if there weren’t any issues with caching, the analysis tools make an assumption that a user’s interaction with the site is over when, in the user’s mind, it might not actually be over. If our user’s trip to find the owner’s manual had taken 31 minutes, Sears.com would have registered him as a repeat visit, but in his mind it was the same event. If the logs can’t reliably tell use when the user left the site, we can't possibly come to any conclusion about why they left the site, and that is really what we really want from that data.

Stats Do Not Represent How Many Visits (or Visitors) the Site Had

Essentially, everything said above means any statistics from log files about number of "visits" are going to be unreliable. If we’re faced with caching issues and visit timeouts, that means we can’t expect to get a reliable statistic of how many visitors have come by or how many visits have happened. In fact, Stephen Turner, developer of Analog Stats refuses to generate visit stats for this exact reason.

Stats Cannot Tell You Where the Visitors Came From or Where They Entered

Because the site's pages may or may not be cached, a user may have viewed several pages on the site before actually needing to request a page from the server. The server logs will show that first request as the user's entry point, which may not be the user's actual entry point. This is all the more true for visitors who frequently come from bookmarks or who have the site entered as their "home" page, for these first pages are very likely to be cached in the user's browser. The site's most loyal visitors may very well be skewing the statistics by not generating an accurate count of these popular entry pages. Instead, data might make it look like other pages are more popular than they really are.

Further complicating this matter, referrer data may not even be available. If the site is running under the secure HTTPS protocol (as should be the case for web stores and any page which contains a form to collect information), referrer data will be unavailable, as this is a feature of that secure protocol. Even without the secure connection, many browsers on the market do not pass referrer data. Even in cases where they do, this capability can often be turned off in the browser as a feature intended to protect users' privacy. A user could also be using a "de-referrer" service such as UltiMod to hide their referrer.

Stats Cannot Measure Users' Online Success

There are those who would claim that stats can tell you how "successful" the site's users are. Their claim is that items purchased, files downloaded, and information viewed are concrete indicators of user success. By stating this, they're making an assumption that every user comes to a site with a singular, specific goal in mind – buy something or look for some very specific information – and then leave.

Realistically, nobody uses the Web only to complete a singular task on each site they visit. Most users do not come to a site for only one purpose and leave when they're done. The user's goal may be as well-defined as "order a new gauge for my air compressor" or as open-ended as "look at all the neat tools I could buy if I won the lottery". Frequently, a user will come for one reason and stay for a completely different one. In fact, most organizations (should) hope that users do exactly that. Amazon.com is the undisputed champion of facilitating this type of interaction.

In the screen cap above, you can see that while the user is viewing a specific watch, Amazon is recommending alternate items to view in case the user has determined that the item he’s viewing does not fit his needs. If Amazon assumed that the user only had one goal (look for a specific item and buy it), then Amazon would miss out on a ton of sales, cross-sales, and follow-up sales. What if the user found the specific watch he was after, but decided not to buy it? Amazon loses money. However, by offering alternatives, they support users who don’t have only one specific goal.

Website Success Cannot Be Measured Online Even If Such Stats Were Possible

Different types of users have different goals, many of which can not be measured online. Based on the author’s personal experience as well as surveys conducted by the Pew Internet & American Life Project, there are three different types of users:

• Users who prefer to deal with you in person. They would rather come in to a brick and mortar location and transact their business and/or gather their information face-to-face.
• Users who prefer to deal with the organization over the phone. They would rather call than spend all that time traveling to and fro. The telephone is close to them, dialing is easy, the interaction feels comfortable, and they still enjoy the benefit of speaking with someone who can give them personal attention.
• Users who prefer to deal with the organization online. The Internet is always there regardless of how early or late it is, and they do not need or want the assistance of another human.

We shouldn't assume, however, that people strictly stay in one of the categories above at all times. Surveys show that people will often switch between these modes based upon a variety of factors such as the type of product, their level of expertise, proximity of the brick and mortar location (if one exists), their level of desire, and their overall goal. One study by Pew indicates that even people who frequently make purchases online are not more likely to do things like manage their investments online than those who don't buy online as frequently. [3] The reason the survey respondents gave for this different approach is easy to understand. They say that handling their entire financial well-being online isn't exactly the same as buying a book, CD, or gift for someone.

Generally, people will choose what type of interaction they want based on what is needed and available for their particular situation and how comfortable they are doing it online. If a user only goes to the Financial Consultant Locator Tool on the Solomon Smith Barney web site to find out where to visit the local office, then subsequently has SSB handle his investments, then the site was successful – but the log files do not track that. Likewise, saying "X amount of people visited the Locator Tool" is certainly no measure of success. Some people, after visiting that page, may decide not to go. Other people might feel the nearest consultant was too far away. Further other people might not be able to use the Locator Tool. Measuring requests to the Financial Consultant Locator Tool – which is simply an imaginary "end-point" – is no measure of success without knowing what the user's criteria for "success" actually is, and without also knowing what contributed to any failures. In the case I've just outlined, it is certainly not the site's fault if the nearest consultant is too far away. In that case, the Locator Tool would have performed perfectly. The lack of generated business may still be considered a “failure,” but not one attributable to the site.

Tracking Usage Trends Provides No Useful Data

Some people who recognize the weaknesses of web stats still argue that the log files can be used to look for trends in site usage. Unfortunately, this is also untrue. Again, caching is just as much of a spoiler here as it is elsewhere. The problem is compounded by the fact that network administrators and ISPs are constantly working to improve their system's performance. Their end-goal is to ensure their networks run quickly and efficiently for the benefit of their customers. This could mean that they could suddenly choose to place a larger (or smaller) amount of data in their cache and may choose different items to cache as well. Such activity could result in vast swings in the number of requests your site receives and in the number of the hosts making requests.

The figure above is a "Monthly history" generated by AWStats for a semi-popular personal site. Notice that traffic in August, September, and October is almost double that of April, May, June, and July? Despite the apparent evidence, absolutely nothing changed about the site at that time. Nothing was added and nothing was taken away. This could happen if some network administrator or ISP tweaked how their system was caching pages, resulting in more requests to the site.

Actions by a site's own management or the management of the organization can also change trend data. Anytime a site's structure changes, grows, or shrinks, the "trends" will as well. If the organization adds a new section and puts a big announcement about it on the home page, there will be a surge in traffic to that area of the site. If a new product is announced in the company's newsletter or on commercials then the site's overall traffic will grow. If the settings on the search engine are tweaked, pageviews may increase in some sections and decrease in others. Ignore the site for several months and the traffic will lull. Completely redesign the site or re-organize the content, and "trends" are no longer trends. Merely by managing the site, an organization invalidates the usefulness of the trend data by creating variation in the site. The more frequently new content is added to the site, the more opportunities exist for variation in the “trends.”

The Special Case of AOL

AOL further complicates statistics by assigning new IP addresses to their users in mid-session. Since AOL is the largest online service in the world, AOL is often a site's largest source of users. Since server log analysis tools often rely on the visitor’s IP address and/or hostname as a unique identifier, this means that a site’s logs can show data attributed to multiple "users" from AOL that actually belong to one person. Depending on how much time the person spends on the site, one AOL user may look like dozens of users.

One person, in a post to the Usenet Newsgroup alt.www.webmaster posted: "Here's a section of my access log that shows an AOL user requesting one page, followed by requests for the images on that page:" (edited for privacy)

195.93.21.98 – - [15/Mar/2006:12:44:37] "GET /xxxx/…
195.93.21.42 – - [15/Mar/2006:12:44:37] "GET /images/…
195.93.21.3 – - [15/Mar/2006:12:44:37] "GET /images/…
195.93.21.36 – - [15/Mar/2006:12:44:37] "GET /images/…
195.93.21.36 – - [15/Mar/2006:12:44:38] "GET /images/…
195.93.21.99 – - [15/Mar/2006:12:44:38] "GET /images/…
195.93.21.68 – - [15/Mar/2006:12:44:38] "GET /images/…
195.93.21.135 – - [15/Mar/2006:12:44:38] "GET /images/…
195.93.21.73 – - [15/Mar/2006:12:44:38] "GET /images/…
195.93.21.38 – - [15/Mar/2006:12:44:38] "GET /images/…
195.93.21.132 – - [15/Mar/2006:12:44:38] "GET /images/…
195.93.21.137 – - [15/Mar/2006:12:44:38] "GET /images/…
195.93.21.137 – - [15/Mar/2006:12:44:38] "GET /images/…
195.93.21.69 – - [15/Mar/2006:12:44:38] "GET /images/…
195.93.21.34 – - [15/Mar/2006:12:44:38] "GET /images/…
195.93.21.106 – - [15/Mar/2006:12:44:38] "GET /images/…
195.93.21.72 – - [15/Mar/2006:12:44:38] "GET /images/…
195.93.21.130 – - [15/Mar/2006:12:44:38] "GET /images/…

As you can see above, one visitor registers 16 different IP addresses in the log by requesting only one page. This is because the individual requests for each image increase the number of requests significantly, therefore seeming to indicate multiple users. Caching confuses traffic data on how long people view the pages, but AOL confuses it even more. AOL's use of dynamic IP addresses changing mid-session muddles everything in a site's traffic statistics. AOL even admits to the challenges this creates for site statistics. Located on AOL's Webmaster FAQ, they state:

Q. Can I use the IP address of the request to track a member's access to my site?
A. No. Because AOL uses proxy servers to service the requests made by members, webmasters see the IP address of the server, not the Dynamically Assigned Host Address (DAHA) of the member in their web site log files. The problem with trying to use the IP address to track access is that there may easily be multiple members assigned to a proxy server. All of the member requests would appear to be coming from one member if you assumed a relationship between member and IP address. In addition, members may be reassigned to a different proxy server during a session.

Problems created by AOL's dynamic IP addressing practice can include:

• The analytics tool may show many more users and visits than the site actually had.
• In the case of AOL, it may also show less users than the site actually had. As they state above, multiple users could potentially use the same IP address.
• The data on entry and exit points will be unreliable. The more AOL users the site has, the less reliable this data. Users will look like they've left when they've simply gotten a new IP address. Their next page view will make them look like a new user.
• The data on length of visit and time on each page will also be unreliable.

Conclusion – Server Log Analysis Is an Unreliable Tool for Usability

It is recommended that an organization not spend extensive amounts of time and money to gain usability data from server logs. An organization would be better served by hiring an experienced human factors engineer to perform an expert review or conduct a formal study with users. The results would be much quicker, more accurate, and more informative.

[1]. HTTP. (n.d.). The American Heritage® Dictionary of the English Language, Fourth Edition. Retrieved September 30, 2007, from Dictionary.com website: http://dictionary.reference.com/browse/HTTP

[2]. Cache Planet Science. Retrieved September 30, 2007 http://www.scienceyear.com/about_sy/index.html?page=/about_sy/help/glossary.html

[3].Pew Internet and American Life Project. http://www.pewinternet.org/PPF/r/131/report_display.asp