One nit I have is with your weights. 1x, 2x, and 3x seems awfully extreme. For most projects, cost will be the least flexible factor, so business value will get a 3x for all their ratings. While I would agree with the principle – low flexible on cost means higher focus on high business value requirements – I think a smaller multiplier would make more sense. Like I said, this is a nit.

On a more fundamental level, I wonder how this would impact one of the major issues that I deal with – the difficulty in getting small improvements into plan. The user value and business value of small improvements is always small, and while this is balanced by high technical ease, it still sounds like the user experience team would need to “cook the books” to artificially raise the user value ratings for the small improvements to make them rise to the top. The other potential issue is that a release should be about more than a grab bag of independent requirements – the release should have themes. If a requirement is critical to a release theme, then it should rise in importance. Maybe the way to fix both the “small problem” and “theme” issues is to add a grouping mechanism to the process so that requirements are not considered in isolation.

Regardless, I enjoyed the article, and think this process has a lot of merit.

Regardless, I think the best use for this type of approach is in establishing a common understanding upon which the team can have more objective, and hopefully more reasonable, discussions between the interested parties. Any tool that might help avoid some of the “he said/she said” arguments that can occur when individual teams are working under different constraints is worth using in some capacity. Might be really useful for projects where the business, design, and development teams don’t share a common timezone…

UPA Presentation Slides, for those interested:
http://www.usabilityprofessionals.org/usability_resources/conference/2006/Marine-Driving-Product-Design.pdf

As for the scoring – When I was a consultant, cost typically led the way. Now-a-days, as an internal IA, the desire to stay competitive or a focus on usability (especially on the tail-end of an unflattering survey or market study) often outweigh the dollars if it come down to what stays in and what gets cut. To that point I don’t agree with the absolutes that User and Business value will always be small. That may be an attribute of a particular environment.

Don’t take as writ that scoring weights should always be constrained to X1,2 & 3. If, as you’re experiencing, there’s a sort of “Built-in” weight, due to some individual slant, you can use the scoring to counter those environmental inequities. It may be cooking the books but it should be done toward levelling the playing field. That said, the process relies on a certain amount of fair play from all the stakeholders. The public nature of the effort can help engender that. Unfortunately, that won’t always be the case. If someone with enough influence is dead sure their way is the right way -all irrefutable evidence to the contrary, there’s not a lot you can do.

Maybe the piece doesn’t go far enough into how you can extend the results of the excercise. You can take a really large list of requirements and use this method for part of it or apply it to the whole thing and use it to chop up the work into iterations. You can introduce other metrics that might be mandates in your particular environment. All these things can some into play when it’s time to make the final decisions about “what comes first”.

I don’t want to put this method across as more than a way to apply that first level of characterization to a chaotic list of “Stuff” that brings the right voices to the table earlier rather than later. The real value isn’t in the number you come up with for each feature or story. It’s in the collaberative nature of doing it, the understanding it yields and the down-stream benefits of establishing a base-line that establishes the balance you want to see between business, technology and design. It’s about getting the right people to the table in the first place.

I have always been heavily influenced by the book “Information Architecture for Designers,” by Peter Van Dijck (link below). Our firm always walks through a strategic planning process with the client where we first develop 1) business goals that the project needs to achieve, and then define 2) user goals of why someone would visit the website. We then prioritize each goal and generate a list of 3) potential features that would accomplish each goal.

The problem with our current system is that personal opinion can still play a large role in deciding if a feature effectively fulfills a business or user goal. I think the next logical step in our process should be to numerically rate each potential feature as Adam has displayed here. For us, at least, I believe this faceted feature analysis will still be one step of several rather than a end-all be-all solution.

Information Architecture for Designers:
http://www.amazon.com/exec/obidos/ASIN/2880467314

In my experience though, Terry is right to sound some notes of caution. The risks of this approach that I’ve found are:

1. You will be seen as a sort of list-keeping Nazi (“That would seem like an excellent idea, but with a low business value my spreadsheet says no”). Worse, you might be seen as spuriously justifying bad ideas (“I think the user value of a splash page is a 10 – we gotta do it man!”).

2. You risk losing the big picture in disjointed minutiae. (“If we can justify a lower business value rating for a link-rich footer and a send-a-friend feature, then we might just be able to include a Flash movie on the home page for the same budget”) Design is, after all, about big pictures in the end.

3. You stake too much on the clarity of the requirements up front. Unless you are completely sure that everyone has the same understanding of what the requirements are, you can’t very well apply a rating to them. Falling into the trap of wording requirements to suit your design objectives is also rather tempting.

But with those risks in mind, I would say it’s a good approach in some (possibly limited) circumstances. Like all techniques, do it with yours eyes open.

That said, this effort is specifically designed to temper the kinds of extremes like your example of the “splash-page=10″ scenario. (For me it was about finding a graceful way to put the “interactive Rockette paperdoll” applet in the…uhm…proper perspective.)

To your point about the big picture; that’s one of those constant vigilance issues that we’ll always labor under throughout the entire life of a project. I wish I had a nickle for every time I looked up and realized that the default display of a drop-down list box had just claimed 20 minutes of my life that I’ll never get back. These days it’s become an integral part of how I operate with my teams to be the one to keep asking if we’re really keeping our eye on the ball.

Your last point brings up another item that probably should have been mentioned in the article which is that unknowns are still relevent in this process particularly where technical ease is concerned. Sometimes they just don’t know how easy/difficult, fast/time-consuming something is going to be and that’s all they can give you. However, if you can determine that there’s high user and business value, it’s enough to say “Let’s factor-in the due diligence necessary to establish the tech effort. By the same token, if the other values are low, you can probably conclude that the discovery effort isn’t required at that time.

There are TONS of similar exercises in which we’ve all taken part. Common sense dictates that you have to do something when you begin with an unqualified list of requirements. This is a mash-up. You mention design purpose and design objectives and while those concerns are the ones closest to my heart, I realized that there are other people whose business and technical objectives are just as near and dear to theirs and justifyably so.

This exercise isn’t meant to put UX in the driver’s seat. It’s meant to get UX into the balance when you’re determining how to move forward so you’ll spend less time moving backward later on.

It’s worked very well for us with a strong project manager keeping us on track and supporting the appropriate meetings. The “what we did well” column at the end of each iteration far outweigh the “what we didn’t do so well” column from all team member perspectives, including development, product owner, IA, design and technical analyst.

I think it’s in the way the user stories are defined. In our case, that meant a conscious effort to remove design/UI language such “a checkbox option” or “a drop down list” that helped define the skeleton under the muscle.

Like you said, it removes the politics, frees the UI in a way I have not experienced in a multi-role team environment and tames the value business creates for certain visual aspects and interactions by putting them into the proper perspective.

I don’t think that giving rates from 1-5 eliminates all subjective or political choices. Maybe it’s more clear giving importance for product releases and deliverables: “(-1) not important, (0) neutral, (1) important”. This give you a list of items rated by importance: the features to build on product version 01, product version 02, and so on, keeping your product development on track. Also, if needed, each team may justify the item importance given based on another document or spec pre-aproved.

Do you have a particular experience with that? What do you think?

Thanks.