CHI 2002 Summary
Minneapolis, Minnesota—April 20-25, 2002
CHI, the annual conference for ACM’s special interest group on computer-human interaction (SIGCHI) was, as usual, packed with information, research findings, and hotly debated theories. In this article I’ll try to cover the events and topics that were most interesting to me as well as the issues that stirred up the most intense conversations during breaks or at social gatherings afterwards.
Tutorial and workshops
The first few days of the conference schedule were dedicated to tutorials that allow attendees to hone their skills, and workshops where participants share ideas and experience related to specific topics. I attended a couple of excellent tutorials: one by Deborah Mayhew on how to establish a usability organization and another by Rolf Molich on advanced usability testing. I also went to User Interface Engineering’s Monday night tutorial that covered their latest research findings.
Experience Design Forum
The CHI2002/AIGA Experience Design Forum was a new joint event that ran concurrently with the tutorials and workshops on Sunday and Monday. Since I was attending tutorials on those days, I didn’t get a chance to experience this rather groundbreaking event that brought the design and usability communities together. It was rather cool to see folks like Clement Mok, someone I wouldn’t expect to see at CHI, mingling at the Sunday night networking reception. Everyone I talked to indicated that the ED Forum was a pretty good experience and that they hoped it would continue to evolve in the future.
Plenary: David Brin (or “I’ll show you mine if you show me yours”)
David Brin is a science-fiction author whose novels include Earth, The Postman, Startide Rising and Uplift War. The Postman was made into a movie in 1998 with Kevin Costner.
Brin’s theory is that in order to avoid an Orwellian “Big Brother” type of future where a select few “watchers” have information on everyone else, it’s better that everyone can watch everyone else. With new technologies like tiny surveillance cameras, no one can expect to keep everything private, and transparency is the only workable solution. He did say that there would need to be limits on the kinds of information that is publicly available and drew the line at things like medical history. As you can imagine, such a controversial idea really got people talking.
Another major point in Brin’s talk was that the human race must rely on criticism in order to avoid major pitfalls in the future. He feels that criticism, sometimes in the form of fiction, helps to prevent misuse of science and technology. Brin said films like Dr. Strangelove and On The Beach helped us see the dangers of nuclear war. Likewise, he thought Orwell’s 1984 has helped prevent a Big Brother society. People definitely took up Brin’s banner of criticism as they moved into the technical sessions, freely taking issue with speakers and panel members when differences of opinions came up.
The theme of the conference was “Changing the world, changing ourselves.” I’m usually not one for “themes”, but I think Brin’s opening helped attendees think about how we might change the world and ourselves.
Panel: CHI @ 20 (or “Clean cars, white rats and people”)
The purpose of the first panel was to discuss the past and future of HCI, usability and related fields (I’ll refer to it as “the usability field”). Mitch Waldrop, biographer of JCR Licklider, gave a brief history on Licklider who was an early pioneer in personal computing and cognitive psychology. Licklider started or greatly influenced many early computing companies and labs like BBN, ARPA, and Xerox PARC.
Stu Card said he thought HCI has been a major success, but that we’re backsliding. He also said we need to build things and start by establishing “good foundations”—I took this to mean a strong basis of research that is well accepted. Marilyn Tremaine said HCI needs to change its attitude. She said current HCI courses focus too much on new techniques, and that students often don’t know how to apply the concepts they learn to new situations. She also seemed to be calling for a renewed focus on basic concepts and fundamental research, outlining these five things she thinks we need to do:
- To reward replication
- Frameworks for designers
- A set of “white rats”
- Theory of evaluation methods
- To stop being cool
Ben Shneiderman said SIGCHI needs to be more in the public eye like other computing disciplines. We need to work more closely with the press, government officials and policy-makers as well as the public. He echoed Norman in saying that we need to move out of the role of critic.
Don Norman poked a few barbs at ACM (SIGCHI’s parent organization), saying it shouldn’t be the “Association for Computing Machinery”—it’s not about machines, but rather, about people, he said. He pointed out that people don’t really care about interfaces, but about what they can do.
Norman also noted the constant struggle for a better title saying, “we tried interaction design, but now that means ‘hey my web site flashes’.” He said that, in reality design is done by teams, and no one person needs to cover both the “quantitative and aesthetic” aspects.
Card pointed out that some of the best systems designed in the past came from an environment where the people that led design ran the whole project. He noted that whoever gets to the design step first in a product development cycle has the most power. For instance, in jet design, the fuselage design is usually done first based on airfoil work, and that will usually preclude any changes to the fuselage design based on the need for more space inside the cockpit.
The general theme across the panel was that usability, as a field, needs to focus on its visibility, adding value to business by better integrating with the design process, and really proving our theories.
Panel: Formative Evaluations (or “What’s wrong with usability testing today?”)
A formative evaluation is an assessment of a product while it is being developed. In contrast, a summative evaluation is one that is conducted at the end of a product’s development cycle. This panel focused on usability testing as a type of formative evaluation.
You’ll note that the panel was pretty hard on usability testing as a method, but they didn’t seem to have any magical better solutions either. This was one of my favorite sessions since it really made me think about how to improve the usability tests I conduct.
Jared Spool said that when they first started doing usability tests at Digital, they didn’t know what they were doing and made a lot of mistakes. He said “now, we think we know what we’re doing,” but he said, we have no documented cases showing that usability tests make a difference from a business standpoint. I found this a bit ironic, since Spool’s firm (UIE) sells research reports largely based on usability testing they conduct.
Bob Bailey said he started doing usability testing in 1970, and that tests are still run in the same basic fashion. He said that, during usability tests, we collect a lot of preference data and incorrectly draw performance conclusions from it. He also said that testing turns up a lot of false positives—that up to 50 percent of “problems” found during testing really aren’t usability problems.
Rolf Molich pointed out that the hardest part of usability testing is communicating the results to the design/development team that can make the proper design change. He said that reports alone don’t do any good, but that we have to build consensus within the project team.
Spool said that, from a practical standpoint, there is no “summative evaluation” because there is no “end of cycle.” Jared joked that “end of cycle is when the dot com finally closes its doors.” Otherwise, product design typically iterates—therefore post-implementation evaluations can be used to inform the next release. He said that, in UIE’s research, the products that were the most usable were delivered by teams that think in terms of three or four releases, not just one release.
Audience questions at large sessions at CHI 2002 were submitted on index cards and the session moderators selected what questions to ask the panel members.
In response to an audience question about automated test tools, the panel had much more optimistic outlook than I expected. Dumas said that most automated tools gather data at too low of a level and the data can be very messy to deal with. He thought it would contain more preference data than performance data (though most of the vendors I talked to said just the opposite). He mentioned that there’s a study soon to be published that looked at remote testing versus lab testing. Bailey cautioned that automated tools might move us toward a more experimental process, and that might present the issue of finding too many “problems” that don’t really make a difference.
Rolf Molich was hesitant to offer opinions on automated tools. He said “let’s go test them.” This was one of the themes I heard Rolf repeat many times during the conference—basically that we need to quit offering and following opinions and go prove something true, false, better, worse, or whatever. Rolf said that automated tools might be a kind of better mousetrap, but we need independent tests, not vendor sponsored tests, to prove whether automated tools are really better than traditional usability testing.
Bailey said one benefit of automated tools is that they allow us to work with a much larger sample size for a given test. He also said we still haven’t validated many of our other methods. Spool jumped on this point and provided an analogy: In the 1850’s doctors used leeches as treatment for a broken leg—they didn’t set broken legs. According to Spool, we don’t know if the methods we have today are “leeching or leg-setting practices.”
Molich then echoed something I’d heard Deborah Mayhew say in her tutorial. He said we should use usability testing for what it’s good for: showing developers what a real usability problem is. We need to move teams toward taking preventative measures, like following heuristics, rather than simply evaluating after the fact, he said.
After providing some healthy introspection on the topic of usability testing, the panel was asked to provide a few comments on the future of usability testing and any positives they see ahead.
Bailey commented that he thinks Molich’s CUE studies are excellent (I have to agree). These studies compare test finding of multiple teams testing the same web sites. The results of the studies clearly point out some areas that anyone conducting tests should focus on as well as some common pitfalls.
Bailey also said “people need to understand how hard usability really is.” Echoing Molich, he added, “We’ve been saying we know the answers, and we don’t know the answers, but we’re turning the corner.”
Dumas summed up much of the conference, saying, “we’re in a questioning phase, and we need that.” Bailey said that the goal of experiments, usability testing and engineering is “to seek the truth.” It seemed as though that’s what the panel was asking everyone in the usability field to do: to seek the truth—in research, in practice and in each other.
Molich noted that practitioners need usable research from research organizations. Bailey pointed to Spool’s UIE and the Software Usability Research Lab (SURL) at Wichita State University as two groups that are putting out usable research. But, he said we need further research to validate and confirm research findings.
PARC—Still cool after all these years
At one of the papers presentations I attended, PARC was showing off an experimental browser it developed called “Popout Prism.” Popout Prism provides an enhanced document overview alongside the web page that the user if browsing.
Essentially, it looks like a thumbnail of the whole web page (not just the visible part) in a window pane to the left side of the browser window. The user can scroll quickly by clicking anywhere on the thumbnail image.
What really makes Prism “pop” though is the popout concept it employs. The user can enter specific keywords they’re looking for, and Prism creates large colorful boxes with the keywords in them (“popouts”) wherever those keywords exist in the web page. The popouts are shown in both the browser window and in the thumbnail. They disappear after a few seconds in the main browser so that they don’t obscure the view of content – but they reappear whenever the user scrolls again so the user can find the information they were looking for.
Prism is very slick. You can download it yourself and play with it. It doesn’t have all the features of a standard browser, but I can definitely see this type of interface integrated into browsers, and possibly other software like email packages.
Certification anyone?
Certification of usability professionals was a hot topic at CHI this year. A Special Interest Group (SIG) was held on the topic, and there was some talk about Human Factors International’s certification program they were promoting in the weeks right before the conference. At my local Usability Professionals Association-Minnesota chapter meeting, which was held at CHI, Julie Nowicki, Chair of the UPA Professional Development Committee, talked about her involvement in an international consortium on certification for usability practitioners. The consortium is currently evaluating the feasibility of such certification. The proposed certification would cover user-centered design as described in ISO 13407.
I didn’t get a lot of details, because I had to leave early, but it sounds like this will be in the works for some time. It will also be presented at the national UPA conference in July.
Ask Jakob
Jakob Nielsen, affectionately called “the Great Dane” by some, was invited to answer questions from CHI attendees, and Nielsen provided some rather spirited answers. I’ll try to summarize some key points rather than go through the whole Q&A blow-by-blow.
- It’s usually aiming for insight for design impact, not statistical accuracy, so you don’t need lots of users.
- Experienced usability professionals can assess how severe a problem is and don’t need to see the same problem with 100 users.
- Coverage is always an issue. You won’t cover 100,000 pages with 20 users. Need to focus on high-level things like navigation and style of content.
- You’re better off studying more things rather than more users.
On web application vs. web pages:
- With applications, user typically selects a particular application as the one they want to use. For example, a particular game, extranet, etc. With web pages, often there are lots of other pages available.
- Tasks are different—you run into problems if you treat applications the same as web pages (from a design standpoint).
- It doesn’t make sense to convert all legacy applications to web applications. Whenever he hears that a company wants to covert existing applications (e.g. a call center) to web applications he thinks, “There goes a million dollars of productivity.”
On new ethical questions in usability:
- Sometimes have to be careful what you report publicly, so it isn’t used against users/consumers. For example a 1996 study showed users clicked ads that looked like error dialog boxes—but it wasn’t reported then to avoid promoting bad advertising practices.
- Nielsen showed a clip of an interview he did with CNN Headline News where he was warning parents that kids are very likely to click on ads, thinking they are content. He pointed out that a lengthy report had to be boiled down to a very simplified message.
On aesthetics:
- Good interaction design is a combination of science and art.
- We need to get business to understand the art part of it.
- The need for aesthetics is easier to communicate by example.
- Aesthetics and usability aren’t inherently in conflict, but you can create conflict if you only emphasize one at the expense of the other. (Hmm, so does useit.com over-emphasize usability at the expense of aesthetics? I know a few folks who would love to hear that question answered.) He did contrast useit.com and MTV.com, saying that their different purposes prescribe different designs.
On “expert reviews”:
- Expert reviews aren’t opinion based, but rather experience based.
- Need to be done using general heuristics, more specific guidelines, and experts’ own test-based guidelines. Have to have all three as well as multiple experts.
- Not an exact science—it’s a design process.
- It’s a “very fuzzy type of thing to design user interfaces,” and there’s no one right answer.
On new ad formats:
- Expanding ads (for example, where a car drives around the screen over content) aren’t unethical, but are “definitely evil.”
- Popup ads—my screen is my castle
- When overlaying content, user has to wait for the ad to get out of the way.
- The web is “barely sufferable” already; we can’t make it worse. If the web gets more annoying to use, users will use it less.
The biggest problems facing the HCI discipline:
- The lack of qualified staff.
- 50 million user interface designs out there, but only 50,000 qualified people. Need 50 million different solutions—design isn’t a one-size-fits-all.
- Need to figure out how to improve the efficiency of usability staff and software design processes. Guidelines are a partial solution.
- New challenges coming up. For example, mobile, collaboration, integration of multiple devices, etc.
 |
| For more information: |
| |
|
| Lyle Kantrovich is a User Experience Architect with Cargill, Inc. He blogs his thoughts on usability, web design, information architecture and user experience practices at Croc o’ Lyle. |
|