Testing Incentives: The Best Way to Pay

What do users want? We all know the answer to that one. Users want products that are simple, yet powerful, yet inexpensive. They want to go to your website and find the information they need in the blink of an eye, whether they browse or search. They want to use your software to accomplish their tasks quickly and easily, without even being aware of your UI.

But how about those users in your lab? Are they, like you, simply doing their part to make the world a better, more usable place? Do they realize how valuable their feedback is, and nobly offer it freely to all for the betterment of mankind? Or are they just in it for the money?

Survey

Recently, I informally surveyed the members on a popular usability list serv to see how other usability professionals compensate their participants. I heard from 37 respondents. Here’s how you compensate your participants, by the numbers:

• Cash – 18
• Gift certificate – 17
• Gift cards / gift checks – 10
• Freebies – 9
• Nothing – 7
• Checks – 6
• Other – 6

It Depends

It doesn’t take a mathematician to see that the responses above don’t add up to 37. Usability professionals typically use more than one method. In fact, the average usability professional used about two methods (1.97), with some using as many as five. Less than half (42%) cited only one method.

Why so many different methods? Well, it depends, of course – on the context, on the users, on the kind of test. “Our compensation depends upon the situation,” says Doug Beck, with Agilent. That’s further explained by Bob Virzi, of Verizon Labs, “We use a variety of options depending on the type of study we are doing.” Sue Heim added, “It depends upon who the users are.” Others were even more detailed, citing one method for in-person and another for remote tests, one method for customers and a different one for employees, and so on. In fact, the why’s and wherefore’s are much more interesting than the mere numbers.

Cash

“Cash is king,” says Ron Perkins, of DesignPerspectives. Bob Virzi points out that cash is the “preferred method, especially by participants.” This is an important point, detailed by Ted Sienknecht:

Nothing has the impact that cold cash does. While many corporations throw up more red tape about disbursing cash, believe me, the user appreciates it more (no issues with stores not accepting it, no befuddled clerks, no balance to check, no card to lose/misplace, no fees/non-use penalties).

Other usability professionals point to cash’s supreme flexibility. “Obviously, as long as cash is still accepted everywhere,” jokes Lyle Kantrovich, of Human Factors International, “it’s the most flexible.” Chauncey Wilson, with MathWorks, Inc, continues the joke: “Cash is generally the best alternative. Cash can be used almost anywhere.” Ted offers a concrete example of what this can mean for usability professionals: “Since cash is liquid, you can easily divert any leftover cash due to no-shows to appropriate uses ([which is] much more difficult with gift cards).”

Cash is not, however, without its issues. Cold, hard cash can be somewhat risky. As consultant Carolyn Snyder points out, “It can’t be replaced if lost or stolen.” Susan Farrell shared a particularly close call:

I flew over the weekend to the UK for a study and needed to have something like £2000 before 8 a.m. on Monday. (My fabulous recruiter … met me with cash at the testing venue and billed it back to me.)

Others pointed to the fact that cash is simply not allowed at their company. Several pointed to particular reasons for this, such as the Sarbanes-Oxley Act or the fact that government employees are often prohibited from accepting gifts of any kind. Chauncey cedes the point, but encourages people to fight on: “Companies, educational, and government institutions are often reluctant to give out cash, but it is worth investing the time in getting a good process set up for getting cash for participants.”

Overall, cash may be the best alternative. Ash Donaldson, of Produxi, summarizes the issue well: “It’s instant gratification [for users] and very little hassle for me.”

Gift Certificates

Gift certificates ran a very close second in my survey. They seemed to be the method of choice when doing remote studies. Note that this includes not just tests, but surveys, phone interviews, and journal/diary efforts. Employees seem to be good candidates for gift certificates as well.

Survey respondents mentioned several different kinds of gifts certificates. Those from Amazon seemed the most popular, but iTunes, Starbucks, and movie tickets all had their proponents. Of Starbucks, Carl Myhill, with GE, waxes positively poetic:

That made me think how much I liked getting a Starbucks card as a gift…. Starbucks seems to worry a lot about customer experience. It’s the first place I ever went where I could use a credit card without signing my name or typing my pin. Wow, it was almost like Starbucks actually trusts me or something. OK, I’m generally pretty anti huge corporation and like to support the little guy, but when I get wowed by a corporation who trusts me, phew, just get me a chai latte!

Gift certificates also, however, have their drawbacks. As with cash, there is some risk involved. “If you use the paper version,” notes Chauncey Wilson, “you must inform the person that the paper certificate is just like money, and if they lose it, you cannot replace it.”

as long as cash is still accepted everywhere — it’s the most flexible.

Chauncey also points to lack of choice as an issue: “Movie cards work well in some places, but not everyone is a movie fan.” Using her own experience as a guide, Leanne Waldal, of Otivo, offers a possible solution, “I love the idea of a choice of gift cards because a snobby foodie-person like me would really hate to be given a Starbucks or Jamba Juice card, but would love an iTunes gift card. ”

Lyle Kantrovich goes one further, recommending a gift certificate that can be used at multiple locations:

One option I haven’t heard mentioned is a service like Hallmark’s Premiere Choice Awards® – where you basically can give a gift certificate (or split it into multiple certificates) that can be used at many different stores (physical, catalog, and online). Recipients can redeem their award for gift certificates at over 350 stores. Of course, there’s a little more cost involved, but I think it makes it more likely that the recipient will enjoy what you give them. My previous employer gave these certificates for employee recognition, so I used them a few times, and it was a great experience. Better than getting cash from my employer.

Though these more flexible gift certificates are ideal for employees, they may also be useful for customers.

Gift Cards / Gift Checks

Similar to Hallmark’s Premiere Choice Awards® are gift cards, which also includes gift checks. They operate very similarly to the Premiere Choice Awards®, but can be used almost anywhere. Examples include American Express® Gift Cheques and Gift Cards, and Visa Gift Cards.

Several survey respondents pointed to the cards’ portability. Carolyn Snyder explains how she “buys a bunch at the start of the project so we have them on hand.” Bob Virzi likes them when he needs to “take a pile of compensation to a study in another city.”

Others use them for remote users. Carolyn notes that “they can work very well for remote users, when you’re not able to hand them cash. At one of my clients, we’ve used them for several studies involving phone interviews, and they’ve worked fine.” Kirk Doggett, with VistaPoint, also points to the fact that they can be purchased online and that they “can be associated with a purchase requisition for accounting purposes.”

Unfortunately, gift cards and checks also have their problems. Ironically, these issues seem to revolve around the cards’ usability. Paul Sherman, of Sage Software, has a fair amount to say about that user experience, both for the participant and the usability professional:

Both have big drawbacks (for the participants, as well as for us). For the participant, they’re a big PITA (pain in the …). Some checkout clerks go into brainlock at the sight of an Amex gift check. Some won’t give change off it. The Visa gift cards are annoying because you have to call to determine your balance. And if your purchase is over your limit, you have to split the item cost across the Visa card and some other form of payment. As for us, we get one or two no-shows for a study, and end up accumulating Amex checks and Visa cards in varying denominations. And did I mention that the Visa cards start auto-debiting a $2.50 per month service charge after six months? If I had my druthers, I’d just cut a check.

Leanne Waldal shares her own less than optimal user experience with this method:

I received a Visa $100 “gift” card as a gift once, and I ended up throwing it away because I couldn’t figure out how to use it. Stores wouldn’t accept it unless I purchased the “right” amount. I had to call some number and register it and give up all sorts of personal information. It wasn’t worth it. It’s horrible that it’s called a “gift card” when it imposes so much on the recipient.

Unfortunately, getting compensated isn’t supposed to be part of the usability test. It sounds like gift cards and checks in same cases may be a poor solution to the problem of compensation.

Freebies

Call it what you want – goodies, swag, booty, boodle – freebies remain a popular option. This is especially the case when there are limited funds to offer cash, checks, or other forms of real money.

Traditional items like T-shirts, coffee mugs, caps, and pens are popular, typically emblazoned with the company logo. Though some respondents thought that users appreciate this sort of thing, they really function more as a thoughtful gesture more than anything else.

In a different category are company products, which may be of real interest to the user. This can mean software, upgrades, and hardware. Barb Hernandez, of TechSmith, points to the difference: “We offer real cool swag.”

Still, Susan Farrell wraps up the appeal of the typical freebie well: “Everyone needs cash, but not everyone needs another bag with another company logo on it.”

Nothing

A surprising number of professionals (almost one in five) say they offer no compensation. This is done – overwhelmingly – because tests involve employees. A couple of survey respondents cited company policy in this regard. Others just cited lack of funds and the availability of warm bodies.

Several respondents noted that, even though their users weren’t receiving any compensation, recruiting was not an issue. Tom Suther noted, though, that this isn’t always the case:

We have our real users and do not need to look much beyond our staff to perform usability or Human Factors work. Since we build things for the staff, they are captive and willingly volunteer as part of their job to produce the best that we can. Yes, we are in a luxury state in this regard.

Not compensating users is much rarer when trying to recruit customers or the general public. I only heard from one user who admitted to this. Kim Moroni, (who didn’t indicate a company association), sounds like she shares the same lucky situation as Tom:

Typically, our customers value the opportunity to participate in the evaluations early enough in our product/application design that they volunteer. We still have more participants sign up than we can include. I guess our primary “compensation” is providing customers with a real voice in improving our products.

Checks

A final major category is checks. Mostly, though, checks are condemned. “Never checks,” says Tara Bazler, of Indiana Univ. “A check is definitely second choice,” echoes Ron Perkins.

Why are checks so universally loathed? Mostly, it has to do with timeliness. Carolyn Snyder tells a story about “one client who tried to go through their A/P department and have checks mailed, but there was no way to make the process happen in less than 30 days. Given that we’d told the users to expect a check within two weeks, we had some apologizing to do.”

My company also uses checks. In addition to timeliness, we have also had problems with the check requisition forms users must fill out. Sometimes, users forget to fill out all the fields on the form, forcing us to follow up with them. Our form also requires users to give us their Social Security number, which often raises privacy and security alarms. We’ve also found there is many a slip ‘twixt the cup and the lip, with the form or check or both getting mislaid or misdirected. Finally, whenever we work with users from a recruiting agency, the idea that they won’t receive cold hard cash immediately after the study leaves them dumbfounded, and can even hurt the recruiting effort.

Other Incentives

Several usability professionals got a little more creative. One particularly interesting suggestion was to do a lottery. Instead of getting a coffee mug, the user’s name goes into a drawing for a Bluetooth®. Instead of $40, the user gets a chance to win $200.

Another strategy was simply allowing a recruiting agency to take care of everything. As Bob Virzi puts it, “Sometimes we use an outside agency to recruit and pay participants, and I don’t give a hoot how they incent [sic] them, so long as they show up and meet the requirements.”

And a final strategy cited by more than one respondent was to offer copies of the report. For a consulting firm like Forrester or Giga, this can really be valuable.

Other Considerations

The survey respondents also shared some very valuable advice on issues other than the pros and cons of particular methods. For example, Chauncey Wilson points to setting expectations: “You need to carefully spell out your incentives when you recruit and how the person will get them and what the limitations are on the incentives.” Based on my own experience, users who are told they will receive cash, then get a check, won’t be happy.

Something similar needs to be done at the end of the test, especially if the compensation is something less straightforward than cash or a check. For these kinds of payment, instructions may be necessary. Here’s what Carolyn Snyder does for gift checks:

At the end of the interview we ask if they have any questions about how to use the gift check. We provide a link to the FAQ page that explains how to use them, in both the thank-you email and the letter accompanying the gift check itself.

In Summary

There are many ways to compensate a user. And which method you select depends a lot on the situation. Still, I would recommend cash for in-person studies and gift certificates for remote ones. Even if we can’t guarantee that we’ll solve all of the users’ problems, at least we can put a smile on the face of those with whom we actually interact.