Testing Incentives: The Best Way to Pay

Posted by

What do users want? We all know the answer to that one. Users want products that are simple, yet powerful, yet inexpensive. They want to go to your website and find the information they need in the blink of an eye, whether they browse or search. They want to use your software to accomplish their tasks quickly and easily, without even being aware of your UI.

But how about those users in your lab? Are they, like you, simply doing their part to make the world a better, more usable place? Do they realize how valuable their feedback is, and nobly offer it freely to all for the betterment of mankind? Or are they just in it for the money?

Survey

Recently, I informally surveyed the members on a popular usability list serv to see how other usability professionals compensate their participants. I heard from 37 respondents. Here’s how you compensate your participants, by the numbers:

  • Cash – 18
  • Gift certificate – 17
  • Gift cards / gift checks – 10
  • Freebies – 9
  • Nothing – 7
  • Checks – 6
  • Other – 6

It Depends

It doesn’t take a mathematician to see that the responses above don’t add up to 37. Usability professionals typically use more than one method. In fact, the average usability professional used about two methods (1.97), with some using as many as five. Less than half (42%) cited only one method.

Why so many different methods? Well, it depends, of course – on the context, on the users, on the kind of test. “Our compensation depends upon the situation,” says Doug Beck, with Agilent. That’s further explained by Bob Virzi, of Verizon Labs, “We use a variety of options depending on the type of study we are doing.” Sue Heim added, “It depends upon who the users are.” Others were even more detailed, citing one method for in-person and another for remote tests, one method for customers and a different one for employees, and so on. In fact, the why’s and wherefore’s are much more interesting than the mere numbers.

Cash

“Cash is king,” says Ron Perkins, of DesignPerspectives. Bob Virzi points out that cash is the “preferred method, especially by participants.” This is an important point, detailed by Ted Sienknecht:

Nothing has the impact that cold cash does. While many corporations throw up more red tape about disbursing cash, believe me, the user appreciates it more (no issues with stores not accepting it, no befuddled clerks, no balance to check, no card to lose/misplace, no fees/non-use penalties).

Other usability professionals point to cash’s supreme flexibility. “Obviously, as long as cash is still accepted everywhere,” jokes Lyle Kantrovich, of Human Factors International, “it’s the most flexible.” Chauncey Wilson, with MathWorks, Inc, continues the joke: “Cash is generally the best alternative. Cash can be used almost anywhere.” Ted offers a concrete example of what this can mean for usability professionals: “Since cash is liquid, you can easily divert any leftover cash due to no-shows to appropriate uses ([which is] much more difficult with gift cards).”

Cash is not, however, without its issues. Cold, hard cash can be somewhat risky. As consultant Carolyn Snyder points out, “It can’t be replaced if lost or stolen.” Susan Farrell shared a particularly close call:

I flew over the weekend to the UK for a study and needed to have something like £2000 before 8 a.m. on Monday. (My fabulous recruiter … met me with cash at the testing venue and billed it back to me.)

Others pointed to the fact that cash is simply not allowed at their company. Several pointed to particular reasons for this, such as the Sarbanes-Oxley Act or the fact that government employees are often prohibited from accepting gifts of any kind. Chauncey cedes the point, but encourages people to fight on: “Companies, educational, and government institutions are often reluctant to give out cash, but it is worth investing the time in getting a good process set up for getting cash for participants.”

Overall, cash may be the best alternative. Ash Donaldson, of Produxi, summarizes the issue well: “It’s instant gratification [for users] and very little hassle for me.”

Gift Certificates

Gift certificates ran a very close second in my survey. They seemed to be the method of choice when doing remote studies. Note that this includes not just tests, but surveys, phone interviews, and journal/diary efforts. Employees seem to be good candidates for gift certificates as well.

Survey respondents mentioned several different kinds of gifts certificates. Those from Amazon seemed the most popular, but iTunes, Starbucks, and movie tickets all had their proponents. Of Starbucks, Carl Myhill, with GE, waxes positively poetic:

That made me think how much I liked getting a Starbucks card as a gift…. Starbucks seems to worry a lot about customer experience. It’s the first place I ever went where I could use a credit card without signing my name or typing my pin. Wow, it was almost like Starbucks actually trusts me or something. OK, I’m generally pretty anti huge corporation and like to support the little guy, but when I get wowed by a corporation who trusts me, phew, just get me a chai latte!

Gift certificates also, however, have their drawbacks. As with cash, there is some risk involved. “If you use the paper version,” notes Chauncey Wilson, “you must inform the person that the paper certificate is just like money, and if they lose it, you cannot replace it.”

as long as cash is still accepted everywhere — it’s the most flexible.

Chauncey also points to lack of choice as an issue: “Movie cards work well in some places, but not everyone is a movie fan.” Using her own experience as a guide, Leanne Waldal, of Otivo, offers a possible solution, “I love the idea of a choice of gift cards because a snobby foodie-person like me would really hate to be given a Starbucks or Jamba Juice card, but would love an iTunes gift card. ”

Lyle Kantrovich goes one further, recommending a gift certificate that can be used at multiple locations:

One option I haven’t heard mentioned is a service like Hallmark’s Premiere Choice Awards® – where you basically can give a gift certificate (or split it into multiple certificates) that can be used at many different stores (physical, catalog, and online). Recipients can redeem their award for gift certificates at over 350 stores. Of course, there’s a little more cost involved, but I think it makes it more likely that the recipient will enjoy what you give them. My previous employer gave these certificates for employee recognition, so I used them a few times, and it was a great experience. Better than getting cash from my employer.

Though these more flexible gift certificates are ideal for employees, they may also be useful for customers.

Gift Cards / Gift Checks

Similar to Hallmark’s Premiere Choice Awards® are gift cards, which also includes gift checks. They operate very similarly to the Premiere Choice Awards®, but can be used almost anywhere. Examples include American Express® Gift Cheques and Gift Cards, and Visa Gift Cards.

Several survey respondents pointed to the cards’ portability. Carolyn Snyder explains how she “buys a bunch at the start of the project so we have them on hand.” Bob Virzi likes them when he needs to “take a pile of compensation to a study in another city.”

Others use them for remote users. Carolyn notes that “they can work very well for remote users, when you’re not able to hand them cash. At one of my clients, we’ve used them for several studies involving phone interviews, and they’ve worked fine.” Kirk Doggett, with VistaPoint, also points to the fact that they can be purchased online and that they “can be associated with a purchase requisition for accounting purposes.”

Unfortunately, gift cards and checks also have their problems. Ironically, these issues seem to revolve around the cards’ usability. Paul Sherman, of Sage Software, has a fair amount to say about that user experience, both for the participant and the usability professional:

Both have big drawbacks (for the participants, as well as for us). For the participant, they’re a big PITA (pain in the …). Some checkout clerks go into brainlock at the sight of an Amex gift check. Some won’t give change off it. The Visa gift cards are annoying because you have to call to determine your balance. And if your purchase is over your limit, you have to split the item cost across the Visa card and some other form of payment. As for us, we get one or two no-shows for a study, and end up accumulating Amex checks and Visa cards in varying denominations. And did I mention that the Visa cards start auto-debiting a $2.50 per month service charge after six months? If I had my druthers, I’d just cut a check.

Leanne Waldal shares her own less than optimal user experience with this method:

I received a Visa $100 “gift” card as a gift once, and I ended up throwing it away because I couldn’t figure out how to use it. Stores wouldn’t accept it unless I purchased the “right” amount. I had to call some number and register it and give up all sorts of personal information. It wasn’t worth it. It’s horrible that it’s called a “gift card” when it imposes so much on the recipient.

Unfortunately, getting compensated isn’t supposed to be part of the usability test. It sounds like gift cards and checks in same cases may be a poor solution to the problem of compensation.

Freebies

Call it what you want – goodies, swag, booty, boodle – freebies remain a popular option. This is especially the case when there are limited funds to offer cash, checks, or other forms of real money.

Traditional items like T-shirts, coffee mugs, caps, and pens are popular, typically emblazoned with the company logo. Though some respondents thought that users appreciate this sort of thing, they really function more as a thoughtful gesture more than anything else.

In a different category are company products, which may be of real interest to the user. This can mean software, upgrades, and hardware. Barb Hernandez, of TechSmith, points to the difference: “We offer real cool swag.”

Still, Susan Farrell wraps up the appeal of the typical freebie well: “Everyone needs cash, but not everyone needs another bag with another company logo on it.”

Nothing

A surprising number of professionals (almost one in five) say they offer no compensation. This is done – overwhelmingly – because tests involve employees. A couple of survey respondents cited company policy in this regard. Others just cited lack of funds and the availability of warm bodies.

Several respondents noted that, even though their users weren’t receiving any compensation, recruiting was not an issue. Tom Suther noted, though, that this isn’t always the case:

We have our real users and do not need to look much beyond our staff to perform usability or Human Factors work. Since we build things for the staff, they are captive and willingly volunteer as part of their job to produce the best that we can. Yes, we are in a luxury state in this regard.

Not compensating users is much rarer when trying to recruit customers or the general public. I only heard from one user who admitted to this. Kim Moroni, (who didn’t indicate a company association), sounds like she shares the same lucky situation as Tom:

Typically, our customers value the opportunity to participate in the evaluations early enough in our product/application design that they volunteer. We still have more participants sign up than we can include. I guess our primary “compensation” is providing customers with a real voice in improving our products.

Checks

A final major category is checks. Mostly, though, checks are condemned. “Never checks,” says Tara Bazler, of Indiana Univ. “A check is definitely second choice,” echoes Ron Perkins.

Why are checks so universally loathed? Mostly, it has to do with timeliness. Carolyn Snyder tells a story about “one client who tried to go through their A/P department and have checks mailed, but there was no way to make the process happen in less than 30 days. Given that we’d told the users to expect a check within two weeks, we had some apologizing to do.”

My company also uses checks. In addition to timeliness, we have also had problems with the check requisition forms users must fill out. Sometimes, users forget to fill out all the fields on the form, forcing us to follow up with them. Our form also requires users to give us their Social Security number, which often raises privacy and security alarms. We’ve also found there is many a slip ‘twixt the cup and the lip, with the form or check or both getting mislaid or misdirected. Finally, whenever we work with users from a recruiting agency, the idea that they won’t receive cold hard cash immediately after the study leaves them dumbfounded, and can even hurt the recruiting effort.

Other Incentives

Several usability professionals got a little more creative. One particularly interesting suggestion was to do a lottery. Instead of getting a coffee mug, the user’s name goes into a drawing for a Bluetooth®. Instead of $40, the user gets a chance to win $200.

Another strategy was simply allowing a recruiting agency to take care of everything. As Bob Virzi puts it, “Sometimes we use an outside agency to recruit and pay participants, and I don’t give a hoot how they incent [sic] them, so long as they show up and meet the requirements.”

And a final strategy cited by more than one respondent was to offer copies of the report. For a consulting firm like Forrester or Giga, this can really be valuable.

Other Considerations

The survey respondents also shared some very valuable advice on issues other than the pros and cons of particular methods. For example, Chauncey Wilson points to setting expectations: “You need to carefully spell out your incentives when you recruit and how the person will get them and what the limitations are on the incentives.” Based on my own experience, users who are told they will receive cash, then get a check, won’t be happy.

Something similar needs to be done at the end of the test, especially if the compensation is something less straightforward than cash or a check. For these kinds of payment, instructions may be necessary. Here’s what Carolyn Snyder does for gift checks:

At the end of the interview we ask if they have any questions about how to use the gift check. We provide a link to the FAQ page that explains how to use them, in both the thank-you email and the letter accompanying the gift check itself.

In Summary

There are many ways to compensate a user. And which method you select depends a lot on the situation. Still, I would recommend cash for in-person studies and gift certificates for remote ones. Even if we can’t guarantee that we’ll solve all of the users’ problems, at least we can put a smile on the face of those with whom we actually interact.

8 comments

  1. I think that cash is really the best incentive for users in a lab setting–hands down ($75-$100 per lab hour, depending on the market). However, there is an intangible compensation with many respondent populations that I’m not sure how to get around: the need to share an opinion, be heard, and interact with people.

    Regardless of the city or the screening demographic or behavior, one thing I’ve never been able to shake is the professional respondent. These are the people for whom part of their livlihood depends on being a research subject, or a primary outlet for social interaction. We also see that many respondents aren’t “heard” in other aspects of their lives, and it can be quite sad. I realize this post is not entirely related to your article, Cliff, but could make a good story idea if any researchers on B&A have found a good way around it–or if it is really necessary to find a way around it.

  2. I have run unto some issues with using any incentive with government employees. Due to some recent scandals in our area with local and state officials, there is reluctancy to except anything from an outside entity. Luckily, participants have been willing to participate sans compensation. Did you receive any response from your survey that had tactics for compensating government employees?

    I appreciate this article and will look into some of the online gift certificates referenced for some upcoming remote testing. Thanks!

  3. With the cost of gas these days, it can be difficult getting anyone into their cars for anything other than cash. I have no issues with this, but the problem has always been with finding the appropriate compensation amount for tests of varying sunstance. For instance, there are times when a usability test may last 90-minutes and dive deeply into multiple tasks, etc. Other times, we need to quickly test users (30-minutes) on a single (yet mission-critical) bit of interaction. In both cases users had to travel the same distance and expend the same amount of fuel. The only variant was the amount of time spent with actual test partcipation. Is it fair to compensate the 30-minute test subject the same as the 90-minute test subject?

    My experience has been that compensating test subjects for varying lengths of tests is akin to buying a sofa set. A loveseat, though fully 1/3 smaller than a couch, is just a wee bit cheaper. This is because it’s the ends of each piece (round-trip transportation) that are costly to make, not the cushion space between them.

  4. Another practical topic to consider: the security of your incentive method.

    Cash and gift cards have a level on anonymity that goes with them from test to use. Once those incentive items disappear from your lab, it’s hard to know who really has them, or who really spent them. Of course, it’s not difficult to make sure your participants do get paid, however, it is sometimes difficult to make sure only true participants get the incentive.

    Specifically, I was once leading a team when one of our members was quietly stealing gift cheques meant for participant compensation. Because of the ease in “repurposing” gift cheques, and also due to the laxity in our bookkeeping, this continued for over a year, and added up to several thousand dollars.

    Surely, a stricter accounting policy would have helped, but personalized checks would have also eliminated the problem.

  5. Great article, Cliff.

    Our rule of thumb is to compensate test participants around NZ$50 per hour in the form of a food voucher; for example: a gift certificate to a restaurant or grocery store. Food is always appreciated and, well, most people need it 🙂

  6. Can anyone tell me if offering incentives to increase survey response rate is prohibited by law in countries outside the U.S.? Our research firm is telling us that this is illegal practice in all countries outside the U.S., but I find that hard to believe.

    Thank you.

  7. Mary,

    I don’t know about all countries outside the U.S., but I did run into this problem working with remote usability participants in Germany. Apparently compensating them (even the small amounts for a usability study, and even in non-cash equivalents) would lead to complicated tax issues, so we were unable to do it. We finally determined that it would be OK to send each participant a Christmas gift (our study was in the fall) of the new software release that they had helped test.

    –Kirsten

    http://www.dynamicdiagrams.com

Comments are closed.