Thanks for a thorough look at this subject. I agree that the system used for rating should be created with the particular product or service in mind, as there are variables appropriate to one that will not be appropriate to others. This is one reason why the effort of the reviewer, especially whether he or she has taken the time to write clear and comprehensive comments in addition to any numeric rating system that is available, is so important to the person viewing the rating later. As you noted, one customer may give a product a low rating due to a particular expectation he or she had for it, while another may give the same product a high rating based upon a completely different expectation. Having access to comments or an explanation helps the prospective customer discern the importance of certain reviews to his or her decision.

One “genre” of rating systems for which this idea is applicable is the dining-related ratings site (like yelp.com or foodbuzz.com). I’ve often noted that I might read a disparaging review that gives low marks for having a limited menu and think, “that sounds like my kind of place- why would someone consider a limited menu a bad thing?” Interestingly, many users only need one not-so-good dining experience to prompt them to submit a negative review, whereas positive reviews might come from long-time loyal customers. Given that any restaurant’s service can be affected by many unpredictable factors, a measured reviewer should take into consideration whether it is their first visit, whether their order was a specialty of the restaurant, the experience of the waiter, etc. Thankfully, most websites dedicated to rating particular services would have considered specific attributes (i.e. wait time, menu scope, wait staff, etc. for restaurants, as apposed to format, page count, dimensions, etc. for books) in order to make sure that the overall presentation of any given business/product is, in general, accurate. As you mentioned, this can become a problem on sites like ebay.com, where the user is in full control over the portrayal of what is being sold. Perhaps Amazon.com has found a good fusion in this regard in the way it allows users to sell used versions of books already in their inventory?

Chris

However, to follow on to David Shen’s comment, I wish you’d included a bit more discussion on how the risk management perspective can instruct the design of the ratings display. Some sites are using histograms to provide a bit more richness for interpreting the rating. But to David’s point, even if I can see the distribution, how do I know if I am more like the group who gave 4 stars, or the group who gave 5? And how do I relate the difference between 4 and 5 to my purchase risk?

Also related is the brilliant Bernardo Huberman’s research, in particular his work with voting sites
http://www.hpl.hp.com/research/idl/papers/opinion_expression/
His Parc Forum talk is very enlightening as well
http://www.parc.com/cms/get_article.php?id=703

A couple of things occurred to me while I read this article:

“Real Names increases likelihood of Responsible Content”
I was struck by your comment about Amazon’s use of “Real Names” to promote social responsibility in the rating system. This assumes that the old saw that anonymity on the web fosters poor behavior. I believed the same thing until I saw a few studies that have shown that ‘real names’ does not do much to improve online social graces. I guess the take away is if others perceive that a rating by a user with a ‘real name’ is more reliable, so much the better but in terms of data collection, it may not make much difference.

One research report – http://www.guardian.co.uk/technology/2007/jul/12/guardianweeklytechnologysection.privacy

“Finding an Appropriate Rating Instrument”
Your point about appropriate rating systems is important. Everyone has “5 stars” on the brain – from iTunes to Amazon to Netflix and beyond. It’s great for a simple interface design but often it’s woefully inappropriate for the context it’s being used for. Some things are just too complicated to reduce to a single representative number.

I recently had a similar conversation concerning Likert scales on political surveys. Many of the hot-topic issues are so nuanced that a 5 point scale doesn’t really allow me to express my thoughts accurately ( Do I “somewhat support” abortion? What does that even mean?).

Creating an appropriate measuring instrument is just as important as making the decision to ask for rating feedback in the first place.

I’m curious about “defensive rating” and how eBay solved it… is rating still limited to registered, signed-in members, but they can now rate anonymously? Or does rating no longer require sign-in? Thanks.