I often found that the cost/effort of running usability tests like this prevented companies I’ve been at from doing enough of these types of usability tests. To that end, I created a “quick and dirty” way to usability test against a specific target market, that only costs $15 per tester, and gives you results in less than 24 hours. Check it out, I’d love any comments – http://EasyUsability.com.

Doug Breaker
Founder – http://EasyUsability.com

The main problem with switching tasks halfway through is the low N on some issues. It requires some intuition to determine which issues are likely to project to a larger audience, which are not, and which require more evidence. My decision is usually based on the severity of the issue (persistence, impact, and frequency) and the type of participant. If it’s a minor problem, then I might mention it with other low priority issues. If it is a big problem, like a mislabeled button that prevents 25-50% of people from ever finding a webpage, then I might include it with high-priority fixes. The participant type matters if the participant is unreliable or only one type of participant find the issue.

In quick turnaround studies, you have an additional constraint: the release timeline. If the issues can be quickly fixed then you do it. If it is going to take a long time, you delay it.

A few comments of my own:

Regardless of how tight your deadline, I think it is an absolute must to at least run one or two pilot tests on internal users. The panic you describe, and the likelihood of having to change tasks (which could compromise your metric data) could be greatly reduced. More importantly, you lower the risk of potentially wasting a participants time (a big nono if you are testing an executive / VP user).

I very much liked the whiteboard strategy, I’ve found that when running a study of my own, stakeholders often run into the room after a session has ended and bounce a million ideas off of me… perhaps I’ll try maintaining a living storyboard throughout the process.

One other thing is, if you find that this is going to be the kind of usability study where developers are likely going to make changes between days of testing, and tasks are likely going to be altered… consider doing a rapid Iterative style usability test . In this kind of test, metrics are charted out in iterations based on changes that happen to the UI that affect specific tasks. My company found this technique to be a powerful tool which blends nicely with a team’s agile development style. Our site has some PDFs on the topic.

–Etan
UX engineer
EchoUser Inc

I think you should call your service EasySurvey or something else. That isn’t a usability test at all.