The first time I read it, I thought you meant you offered only two choices in the hierarchy, even if there were something like six. Now I’m thinking you mean, you only asked twice before moving on.

I’m wondering one other thing: what about one person running this card-based test? I know with usability testing, that is not ideal even for a discount style test, since so much is going on, but does it make more sense for this technique?

I think I could manage to do this with one person, but I have always been lucky and had someone to give me a hand. The tricky bit with doing it alone would be recording the scenario & choice and re-bundling the cards for the next scenario – this may break the flow of the discussion a bit (but probably not much). It would be tricky to note comments at the same time.

Thanks for sharing this great technique! The examples were very helpful. The method seems to be a bit of a twist on a closed card sort — kind of a closed card sort mixed with a cognitive walkthrough.

It seems it would be very useful, especially with large taxonomies. A closed sort requires the participant to deal with lots of content examples, finding what category they’d put each into. Your technique is scenario based which is more natural – usually, most end users will use a hierarchy to find information rather than categorize content using it.

For more on cognitive walkthroughs, see:

Usability Evaluation with the Cognitive Walkthrough
http://www.acm.org/sigchi/chi95/proceedings/tutors/jr_bdy.htm

or

http://jthom.best.vwh.net/usability/cognitiv.htm

I think this may be better than card-sorting. Lyle makes an excellent point that this exercise is natural to a user. It seems more realistic than telling a user to sort a pile of cards (something they normally don’t do online or offline).

It also seems like this method is much more applicable than card-sorting. I’ve found that card-sorting really requires a larger-scale site with lots of content. I assume this method could be used on small or large sites as long as they’re are at least two levels “deep”, right?

I hope to implement this new method soon. Again, very innovative method and a good article.

I think this may be better than card-sorting. Lyle makes an excellent point that this exercise is natural to a user. It seems more realistic than telling a user to sort a pile of cards (something they normally don’t do online or offline).

It also seems like this method is much more applicable than card-sorting. I’ve found that card-sorting really requires a larger-scale site with lots of content. I assume this method could be used on small or large sites as long as they’re are at least two levels “deep”, right?

I hope to implement this new method soon. Again, very innovative method and a good article.

Kyle – you are correct – the method could be used on anything at least 2 levels deep, and I do get people to think out loud.

Actually I sometimes do something similar informally for just top-level stuff as well – when someone asks me where I would put something, I give them a related scenario and ask where they would look. Answers their own question usually.

this was an extremely useful article indeed! i work as an analyst programmer, who also gets to do some usability/IA type work if i push hard enough *grin*

we are currently in the process of providing a category browsing interface for our company’s knowledge base, and after having done an open card-sort exercise to assist with initial grouping ideas, we are now testing our proposed taxonomy/hierarchy using your method.

it seems to be working very well (and users certainly enjoy participating), though i do have one question..

the ‘by eye’ method of analysis as outlined in your example seems to work well with smaller numbers of scenarios, but the set we’re currently using has 50-odd cards to be sorted. i’ve tried to keep the number to a minimum, but we have a very wide range of categories and subjects to be validated.

is there any automated analysis method / software that you or anyone else might be able to recommend? for the open card-sort testing we used ezsort, which worked well, but it’s not applicable to ‘known item’ exercises such as this.

or is it the case that we’re just using too many scenarios for this method?

any feedback gratefully received, we’re still relatively new at this and eager to learn more ;o)

cheers,
chris

I still used the spreadsheet as described in the article, printed it out, stuck it together and looked at it on a big desk (or on the floor!). It would be worth probably shading alternate rows so you can follow a row or column across.

I’m very visual, so this works well for me. Let me know if it doesn’t work for you, and I’ll see if I can think of another way to do it