Google, Stanford, and The Government Fight Swine Flu

by:   |  Posted on

Bolt | Peters recently collaborated with a team at Stanford University on designing the Google Sites template for local governments to use as a backup to deliver information on the H1N1 outbreak, and also disasters and emergencies in general. The goal was to create a template that was well laid-out, easy for non-techie local governments to edit and update with content, and conveyed the most important information to different audiences.

Swine Flu info template


How It Started: The Quick Fix

With the recent outbreak of H1N1, Santa Clara County’s official public flu information site was taken down by the surge in web traffic. To help relieve the demand, the Stanford SIE Program, a Stanford University group that develops technology for social change, stepped in literally within hours of the interruption to create an ad hoc backup site using Google sites, so people could still access the critical info.

This is the version of the site they originally posted, using Google Sites’ standard WYSIWYG editing tools:

Stanford's original stopgap design
Stanford’s original stopgap design

After the site went live, Stanford trained the Santa Clara County maintain it and add their own information. Santa Clara County needed to have site that could handle the traffic and get the information out as quickly as possible—which is to say that there wasn’t a whole lot of time to think about design.

This experience made it clear that it would be valuable to create a well-designed, easy-to-edit template for local governments to distribute information in case of emergencies—not just H1N1, but any public hazard, including floods, earthquakes, wildfires, and so on.

The team contacted us in late October with the original draft of the website. Since it was important to make the site available as soon as possible to deal with the ongoing H1N1 outbreak, so the timeline we had for design recommendations was really brief—just a few days. With that in mind, we got to work.

Spotting the Problems

Because of the layout restrictions, our design evaluation focused primarily on information design. We had two main issues with the early design, along with a handful of usability tweaks here and there.

First draft of Google template

1. Lack of visual hierarchy.

With two columns of equal width and mostly indistinguishable boxes filled with text, it was hard to tell at a glance what information was urgent, time-sensitive, or recently added.

Big chunks of info, no

The info wasn’t grouped into meaningful categories—there wasn’t much visual or spatial distinction between contact info, prevention info, calls to action, and so on, making it difficult to zero in on information even if you know what you were looking for. Also, the info came in big blocks of unscannable prose, and deep reading is the last thing you want to do when you’re trying to learn about the tsunami headed your way.

3. It didn’t anticipate the distinct needs of the most critical user

There was plenty of good info on the site, but it was never clear who a given piece of info was for—you couldn’t scan the page headers and think, “Yeah, there’s what I’m looking for”. Instead it had a “general audience” feel to it; the info didn’t take into account that different groups might have different needs and different levels of urgency.

4. Buried info.

Vital info on vaccines, symptoms, and SMS / Twitter updates was absent from the front page entirely, lurking deep in the navigation.

Our Recommendations

To keep editing simple for the local government end-users who would be using the template, we had to stick to using the WYSIWYG Google Sites editor, which meant no custom CSS and very little control over layout. We also had literally a single day to make our recommendations and synthesize them into a first-draft mockup—the result wasn’t pretty, but got our main info design recommendations across loud and clear.

First revision of template
Our first stab at redesigning the H1N1 template

Redesign Goal #1: Prioritize information according to the urgency
of visitor need.

Our design takes into account that there are different “general public” user segments with different goals, and that there are tiers of urgency and priority. From most-to-least urgent, we identified these segments:
* People infected with the flu: Immediate help / contact info
* People worried that they might have the flu: Self-diagnosis
* People concerned about catching and/or spreading the flu: Preventative measures and vaccine info)
* People just curious, staying informed: Information about travel restrictions, public response, news updates, deep flu info

The structure of the site was altered to serve each of these segments:
# We added a page-width alert box that would be displayed to convey urgent, time-sensitive info—this is a common feature on many of Google’s sites, as well as
# A yellow-shaded box to give the highest priority group, currently affected individuals, clear instructions on what to do.
# The left-column contains info on self-diagnostic and preventative measures for at-risk or concerned individuals.
# The top-right contains info on vaccines; below is links to deep info, research, and update notifications. Though the Google Sites template didn’t allow us to resize the right column, we noted that it should be made smaller than the left column.
# The left sidebar navigation was reserved for redundant quick links to most important info, as well as links to pages for specially affected individuals and organizations.

Redesign Goal #2: Reduce block text down to easy-to-scan lists
and chunks.
Cut the bureaucratic BS.

We broke down the block text to keep from overwhelming users with too much difficult-to-scan information upfront. Lists were kept to under 8 items long, unless they broken down into categorized sub-lists; text was kept under five lines. All information that couldn’t be condensed in this way was moved to lower-level pages, and linked from
higher-level pages.


Users don’t need to know what the mission statement and goals of the organization are; they just want to know if and how they are personally affected, and what they can do in case they are affected.

Redesign Goal #3: Use informative headers that directly address
user goals, and which give all users clear next steps.

All types of visitor (e.g. directly affected, at risk, concerned, just curious, administrative, medical, etc.) should be able to tell by the header if that information is “for them”. We tweaked the headers to make it clear what kind of info you could find in each section. We also made it clear what, if anything, each user segment needed to do:
* Immediately affected individuals are given step-by-step instructions on how to deal with their
emergency situation(s).
* At-risk individuals are given step-by-step information on preventative, precautionary, and self-
diagnostic measures.
* Individuals seeking non-urgent information can be given supplementary external information
resources (by phone, online, and downloadable / printable) and means to stay updated (by email,
text, RSS, Twitter).
* Urgent contact info is immediately visible in the right sidebar.

The First Revision

After we sent over our recommendations and mockup, a member of the team sent us a nice note a day or two later:

You’ve convinced us that we have to completely rethink and redesign the site from scratch, so
your style guidelines come at a very good time. I can’t thank you enough for opening our eyes to
how we can do this in a much better way. I think we can create a site that works much better than
the original site.

…and then a few days after that, Stanford sent a revised version over to Google, who worked with the design firm OTT Enterprises to create two new designs with some added visual design flourishes.

Unfortunately we don’t have permission to show you this revision yet (working on it), but it wasn’t bad—certainly cleaner and better organized, easier to scan, less dense. There was, however, a large distracting green gradient background, some empty space in the sidebar columns, a few stock photos, and a muddled color palette (green, blue, yellow, and gray).

Our second batch of suggestions focused mostly on visual design and layout. Just a few of them:

Visual Design

* Get rid of the gradient background; it’s distracting and confuses the emphasis on different parts of the site, since it runs behind multiple different elements.
* Get rid of the green coloring, which is tertiary to the blue and yellow. Instead, use several shades of blue as the primary color and a little light beige or light grey as a secondary trim. Blue signifies authority, calmness, trustworthiness, etc., which are of course appropriate here. Notice how almost every major government agency’s website (including the CDC) uses dark blue and gray as the main colors.
* Remove the stock images, including the CDC and widget images, which look like ads. The phone and email icons are fine, however.
* Rather than images, make the content headers stand out with size and strong typography—”make the content the focus” is an old web design adage, and the content, in this case, is the flu information. Here are a bunch of sites that use typography, font size, whitespace, and bold layout to create emphasis, using few images [list of a bunch of websites].


* Header and upper-page content takes up way too much space—note that the important info (”If you or your child…”) doesn’t begin until more than halfway down the screen. Compress.
* I like how Design #2 places the alert and contact info in the sidebar; in Design #1 the sidebar is wasted space. This frees up space to move important info (Vaccine and How to Care for Someone With The Flu) up to the right side.
* This design consists mostly of links to deeper pages, but there’s definitely room to put more specific, useful info right on the homepage: symptoms, preventative measures, vaccine info (see our original design)
* Get rid of the Contents box.

The Results

Stanford started over once again, aided by our style guide and input from OTT Enterprises. After further iterations in Google Sites, they handed it over to the Google visual designers, and here’s the before-and-after:

Google Sites template, super rush draft

Google Sites Public Health Template 1.0

Can you do better?

As with all things on the web, the template is a design-in-progress, and will be improved as time goes on. Stanford SIE is looking for feedback on the design, so here’s where you can send feedback for the Public Health template and the All Hazards template. Since these Google Sites templates are available to everyone, you can even make your own design edits and mock up improvements.

Or if you just think it’s great and you just want to use it yourself, here’s the complete list of links:

Google Sites Templates blog post

Public health sites:

Example site
User guide

All hazard sites:

User guide
Stanford SIE site (we’re credited here!)

Note: Nate and Tony’s book on remote testing, “Remote Research”:, will be published by Rosenfeld Media in 2010.

Researching Video Games the UX Way

by:   |  Posted on

Video games are often overlooked in the scope of usability testing simply because, in a broad sense, their raison d’etre is so different than that of a typical functional interface: fun, engagement, and immersion, rather than usability and efficiency. Players are supposed to get a feeling of satisfaction and control from the interface itself, and in that sense, interaction is both a means and an end. The novelty and whimsy of the design occasionally comes at the expense of usability, which isn’t always a bad thing—that said, video games still have interfaces in their own right, and designing one that is easy to-use and intuitive is critical for players to enjoy the game.

Consider how video games are currently researched: market research-based focus groups and surveys dominate the landscape, measuring opinion and taste in a controlled lab environment, and largely ignoring players’ actual in-game behaviors. Behavior is obviously the most direct and unbiased source of understanding how players interact with the game—where they make errors, where they become irritated, where they feel most engaged. When Electronic Arts engaged Bolt|Peters to lead the player research project for Spore, we set out to do one better than the usual focus group dreck by coming at it from a UX research perspective.


One overarching principle guided the design of this study: we would let the users play the game in a natural environment, without the interference of other players, research moderators, or arbitrary tasks. This took a good bit of planning. Usually, we prefer to use remote research methods, which allow us to talk to our users in the comfort of their own homes. Spore, however, was a top-secret hush-hush project; we couldn’t very well send out disks for just anybody to get their hands on. Instead, CEO Nate Bolt came up with what we call a “Simulated Native Environment.” For each of the ten research sessions, we invited six participants to our loft office, where they were seated at a desk with a laptop, a microphone headset, and a webcam. We told them to play the game as if they were at home, with only one difference: they should think-aloud, saying what ever is going through their mind as they’re playing. When they reach certain milestones in the game, they would fill out a quick touchscreen survey at their side, answering a few questions about their impressions of the game.

Elsewhere, Nate, the clients from EA, and I were stationed in an observation room, where we set up projectors to display the players’ gameplay, the webcam video, and the survey feedback on the wall, which let us see the players’ facial expressions alongside their in-game behaviors. Using the microphone headset and the free game chat app TeamSpeak, we were able to speak with players one-on-one, occasionally asking them what they were trying to do or to go a little more in depth about something they’d said or done in the game.

Doesn’t that sound simple? Actually, the setup was a little brain-hurting: we had six stations; each station needed to have webcam, gameplay, survey, and TeamSpeak media feeds broadcast live to the observation room – that’s 18 video feeds and 6 audio feeds, and not only did the two (that’s right, two!) moderators have to be able to hear the participants’ comments, but so did the dozen or so EA members. On top of that, everything was recorded for later analysis.

“The feedback we received from users wasn’t based on tasks we’d ordered them to do, but rather on self-directed gameplay tasks the users performed on their own initiative”

The important thing about this approach is the feedback we received from players wasn’t based on tasks we’d ordered them to do, but rather on self-directed gameplay tasks the players performed on their own initiative. We didn’t tell players outright what to do or how to do things in the game, unless they were critically stuck (which was useful to know in itself). The observed behavior and comments were more stream-of-consciousness and less calculated in nature.

The prime benefits to our approach were the absence of moderators, which mitigated the Hawthorne effect, as well as the absence of other participants, eliminating groupthink. Additionally, the players were more at ease: it’s hard to imagine these video outtakes (see below) being replicated in a focus group setting. Most importantly, they weren’t responding to focus questions–they were just voicing their thoughts aloud, unprompted, which gave us insight into the things they noticed most about the game, rather than what we just assumed were the most important elements.


Over the year-long course of the project, there was one incident which proved to us just how important it was to preserve the self-directed task structure of our research. Because of the multiphase progression of Spore, we believe it was important to carefully structure the sessions to give players a chance to play each phase for a predetermined amount of time, and in a set order as if they were experiencing the game normally.

Partway through the second session, we started having doubts: even though we weren’t telling players what to do within each phase, what if our rigid timing and sequencing is affecting the players’ engagement and involvement with the game?

To minimize this, between sessions, we made a significant change to the study design: instead of telling users to stop at predetermined intervals and proceed to the next phase of the game, we threw out timing altogether and allowed users to play any part of the game they wanted, for as long as they wanted, in whatever order they wanted. The only stipulation was that they should try each phase at least once. Each session lasted six hours spread over two nights, so there was more than enough time to cover all five phases, even without prompting users to do so.

Sure enough, we saw major differences in player feedback. We are unable to provide specific findings for legal reasons, but we can say that the ratings for certain phases consistently improved (as compared with previous sessions). Additionally, a few of the lukewarm comments players had made about certain aspects of the game seemed to stem from the limiting research format, rather than the game itself.

It became clear that when conducting game research, it was vitally important to stick to the actual realities of natural gameplay as much as possible, even at the expense of precisely structured research formatting. You have to loosen up the control a little bit; video games are, after all, interactive and fun. It makes no more sense to formalize a gameplay experience than it does to add flashy buttons and animated graphics to a spreadsheet application.


There are a lot of ways to go with the native environment approach. Even with all efforts to keep the process as natural and unobtrusive as possible, there are still lots of opportunities to bring the experience even closer to players’ typical behaviors. The most obvious improvement is the promise of doing remote game research–allowing participants to play right at home, without even getting up.

Let’s consider what a hypothetical in-home game research session might look like: a player logs into XBox Live, and is greeted with a pop-up inviting him to participate in a one-hour user research study, to earn 8000 XBox Live points. (The pop-up is configured to appear only to players whose accounts are listed as 18 or older, to avoid issues of consent with minors.) The player agrees, and is automatically connected by voice chat to a research moderator, who is standing by. While the game is being securely delivered and installed to the player’s XBox, the moderator introduces the player to the study, and gets consent to record the session. Once the game is finished installing, the player tests the game for an hour, giving his think-aloud feedback the entire time, while the moderator takes notes and records the session. At the end of the session, the game is automatically and completely uninstalled from the player’s XBox, and the XBox Live points are instantly awarded to the player’s account.

Naturally, there are lots of basic infrastructure advances and logistical challenges to overcome before this kind of research becomes viable:

  • Broadband penetration
  • Participant access to voice chat equipment
  • Online recruiting for games, preferably integrated into an online gaming framework
  • Secure digital delivery of prototype or test build content
  • Gameplay screensharing or mirroring

For many PC users, these requirements are already feasible, and for games with built-in chat and/or replay functionality, the logistics should already be much easier to meet. Remote research on PCs is already viable (and, in fact, happens to be Bolt|Peters’s specialty). Console game research, on the other hand, would likely require a substantial investment by console developers to make this possible; handheld consoles present even more challenges.

We expect that allowing players to give feedback at home, the most natural environment for gameplay, would yield the most natural feedback, bringing game evaluation and gameplay testing further into the domain of good UX research.

Spore Research: Outtakes from bolt peters on Vimeo.

Science of Fun from bolt peters on Vimeo.