Comments on: Measuring the Success Of a Classification System

By: donnamaurer

donnamaurer — Wed, 30 Nov -0001 00:00:00 +0000

A good enhancement & neat visualisation of outcomes!

By: michaelbeavers

michaelbeavers — Wed, 30 Nov -0001 00:00:00 +0000

This is great…I like that you also keep testing results visually represented with only three colors. Any more than that would require too much thinking and analysis (think of the American homeland defense color scheme). I hope practitioners try your method and report back in this comment string.

By: northcoast_ryan

northcoast_ryan — Wed, 30 Nov -0001 00:00:00 +0000

thank you so much for outlining this. the final spreadsheet really helps show clients the value of the different iterations. I will apply my own flavor to it and report the results!

By: henrikpp

henrikpp — Wed, 30 Nov -0001 00:00:00 +0000

at first glance this looks like a great way of improving your ia iteratively. but do the results get better because you adapt to the test users or the other way around?

By: cwingrav

cwingrav — Wed, 30 Nov -0001 00:00:00 +0000

Nifty. One additional success metric to apply would be the levels of the hierarchy traversed to get to each item. I.e., average “clicks” to finish each scenario in each redesign. The success rate and “click” count together would really be compatible with each other. I’ll have to give this a go in the future.

By: iainbarker

iainbarker — Wed, 30 Nov -0001 00:00:00 +0000

Chad,
So far I’ve always tried to avoid making the user focus too much on their success/failure – I’ve just given them two attempts, but it sounds like what you are suggesting could work. Let me know how it goes.

By: dawnbuie

dawnbuie — Wed, 30 Nov -0001 00:00:00 +0000

Hey Ian – did you accidentally repeat the first photo of the index card after this text:

2. On another set of index cards, write and number around 15 common information-seeking tasks. One task per index card.

By: iainbarker

iainbarker — Wed, 30 Nov -0001 00:00:00 +0000

Thanks Dawn – hopefully that will be corrected very soon.

By: mdmswim

mdmswim — Wed, 30 Nov -0001 00:00:00 +0000

Iain, Very nice article. I work in the Process Improvement arena, specifically with CMMI, and capturing measures and managing to them is integral to achieving high level certifications. Measuring the improvement of websites, and proving it, can be tricky. Your spreadsheets provide a simple and understandable way of showing this.

How many people do you interview (ie. is there a magic number), and is it important that the same people be interviewed throughout the process?

By: iainbarker

iainbarker — Wed, 30 Nov -0001 00:00:00 +0000

Mike,
Thanks for your comment. So far I’ve only used the technique to provide data to support what would otherwise be an entirely subjective assessment of the success/failure of the classification system.

The limited budgets/timescales I typically work within have only enabled me to run one or two days of sessions within each iteration. This means I’ve spoken to between 15 to 30 people, each attempting 10 to 15 tasks. I am not a statistician, but I am guessing that these numbers won’t produce statistically valid data (I am always careful to point this out when using this technique to communicate to clients). Sadly I can’t advise as to what kind of numbers you would need to involve to provide statistically valid data – maybe someone else can help there?

As for repeatedly using the same people, I have always had the luxury of a large pool of potential users, and have always used new users for each iteration. Obviously this could cause some to question the comparison between iterations, but I’ve never had this problem with the clients I’ve worked with. Even if a client wants to include some repeat participants, I would always argue for some new participants with each iteration.

My preference for using new participants is primarily so they don’t take ownership of any solutions/recommendations they may give during the sessions. Also I find that from a political perspective, it is often better to be able to say that 100 people participated in the creation of the classification system, rather than just 15.

I am sure that a statistician may well be wincing at my response – if one is out there reading and wants to recommend how a more rigourous approach could be applied I’d like to hear from them.