This intensive camp gives you the practical skills you need to recruit participants, conduct hands-on testing with real users, and turn your findings into action items for the design team.

===============================================

Seasol

MLS

My impression from this review is not so much the (scandalous!) idea that NN/g practices slipshod methodology… what I have learned is that measuring usability ROI is quite often nearly impossible. In particular the “mitigating factors” section pulled the rug out from under almost every ROI calculation/model I’ve ever heard of (there are few usability changes that are not accompanied by simultaneous business, design, technological, and/or marketing changes). This article leaves me even more skeptical about *any* consultant that claims that usability can be statistically tied to revenue… wearing what Jesse James Garrett called “the lab coat”.

ROI for usability may be no easier to prove than, say, ROI for any other “production quality” choice in any other industry, from the strength of the threads used to sew a t-shirt together to the amount of sawdust added to a fast-food hamburger patty.

Is the fudging of ROI metrics really that uncommon in the usability world? Or is it, as I beleive, far and away the norm?

Or, for that matter, is crappy ROI methodology really that uncommon in other industries? ROI may be a business concept that is abused universally, not just by usability pundits.

My firm sells ourselves as ‘experienced and smart’, not necessarily as ‘impartial and scientific’. We are user experience designers, not research scientists. This article only confirms my suspicion that most of what we do in this industry is difficult to measure with a large degree of scientific rigor. There is always a huge space where innovative and clever people fill in the gaps.

I’m not saying that it is *always* impossible to measure usability ROI, but I think this article is suggesting that it is a immensely more difficult task than many of us imagine — and that it is, indeed, often totally impossible. ROI numbers analysis can *inform* but will never replace smart business and design sensibilities.

The article’s opening statements, which I think are meant to be facetious, end up seeming provide the best existing rationale for investment in usability:
“Of course you’ll sell more products if they’re more usable! Or you’ll decrease costs because of heightened productivity! Exactly how much will you profit? I don’t know, but don’t you want to build the best product you can anyway?”

(I had my doubts about this article before I read it, seeing that one of its authors is a partner at a firm that competes with NN/g. But after reading it these doubts were dispelled.)

(I also am chuckling at the idea that there may be tons and tons of similar “evidence” DISproving NN/g’s thesis: where a ton of money was spent on usability and yet sales remained flat, or even decreased dramatically.)

-Cf

I think the goal of Nielsen/Norman Group’s report was to make a general assessment that would inspire the reader to go and interact with the available sites. Merholz’s and Hirsch’s review reads as though they got caught up in having paid for such a light report. I had to ask myself, “Did they forget to actually go and interact with the available featured web sites?”

To be fair, I recognize other influencers were taken for granted. What about brand experience? Did usability recommendations add to brand perception? In some of the case studies, you can connect it. Having roots in software engineering, Nielson/Norman Group has a tendency of distancing itself from “how” functionality can heighten brand awareness (versus user experience). So, Yes… you could argue that brand experience was slightly discounted by the Nielson/Norman Group report.

Anyway, Usability is about user interaction/perception and it seems Merholz and Hirsch was reacting to some of the limitations of a flat report. To their credit, I agree with their points concerning the report’s case study selection and measurement process. Further, I think that a more tactical approach with fewer sites would have allowed more design context to be established. With this context, I think several of Merholz’s and Hirsch’s issues would have been addressed.

So, I have to ponder what both sets of Arthur’s would answer: Tactically (2 to 3 focused sites) or Strategically (10 to 15 broad vision sites), How would you address a report on usability and ROI?

I think the goal of Nielsen/Norman Group’s report was to make a general assessment that would inspire the reader to go and interact with the available sites. Merholz’s and Hirsch’s review reads as though they got caught up in having paid for such a light report. I had to ask myself, “Did they forget to actually go and interact with the available featured web sites?”

To be fair, I recognize other influencers were taken for granted. What about brand experience? Did usability recommendations add to brand perception? In some of the case studies, you can connect it. Having roots in software engineering, Nielson/Norman Group has a tendency of distancing itself from “how” functionality can heighten brand awareness (versus user experience). So, Yes… you could argue that brand experience was slightly discounted by the Nielson/Norman Group report.

Anyway, Usability is about user interaction/perception and it seems Merholz and Hirsch was reacting to some of the limitations of a flat report. To their credit, I agree with their points concerning the report’s case study selection and measurement process. Further, I think that a more tactical approach with fewer sites would have allowed more design context to be established. With this context, I think several of Merholz’s and Hirsch’s issues would have been addressed.

So, I have to ponder what both sets of arthur’s would answer: Tactically (2 to 3 focused sites) or Strategically (10 to 15 broad vision sites), How would you address a report on usability and ROI?

There are others, of course.

If this is your way of offering to be a (volunteer) copyeditor on the (all-volunteer) staff of this (no-fee) publication, we’ll be glad to have the extra pair of hands. Just say the word and we’ll add you to the roster.

In answer to Christopher Fahey’s question about the abuse of ROI in other contexts, this also happens with vendors of Customer Management Systems (CMS). The ROI claims for these systems often don’t pass accounting scrutiny.

I suspect that the ROI argument for usability itself does not have a positive ROI (in terms of its ultimate effectiveness). In Seattle we are fortunate to have an artistically acclaimed opera company that is also consistently on-budget (even this year when many arts organization are struggling with deficits). Over the years, it has built a diverse audience base (including many folks in the 20’s and 30’s) in part through its education and outreach programs. Opera education programs are costly and have almost zero associated revenue. How does Seattle Opera justify a program with no short-term return? They have a clear, strategic vision about audience enjoyment and are willing to invest long-term in that vision. Maybe that’s the type of thinking that needs to be needs to be present in an organization before usability become institutionalized: that no amount of quantitative data will convince an organization of the value of usability if that organization isn’t strategically focused on providing a high-quality customer experience.