Thanks,
Chris.

Glad you like the article. PO stands for product owner, which is what they call the person in charge of the backlog in Scrum. They should be the person making sure you have a list of stories to start with. The PO should be your closest partner on any Scrum project as a UX person. If not you’ll find things will go poorly.

Sorry about omitting that definition, I worked with the B&A editors in a very agile fashion on this post. There are a few rough edges still, but we thought it was a MVP or “minimum viable post”…at least I hope so

If you want to learn more about the product owner role or Scrum in general, I recommend Mike Cohn’s book “Succeeding with Agile” or Roman Pichler’s book “Agile Product Management with Scrum.” Roman has a blog at http://www.romanpichler.com/ and that’s a great place to start. I found Mike’s book so good I took his Certified Scrum Product Owner course and it was time well spent.

I’m currently facing the situation highlighted in this article on a project I just came into. I tried implementing Scrum, but realized that this project should be UX-driven. It’s an interesting challenge because we really lack UX experience on our team, so we’ve gone external and hired a UX Designer. That solves the lack of experience to a degree, but the project, in terms of organization and process from UX to Development, is still in a bit of a flux, so I’m really eager to try this UXI Matrix out.

You mention UX metrics often and I agree that they are very important in order to make informative decisions. I’m really new to UX and I’m wondering where and how would I gather these metrics? Google Analytics or similar metrics systems (i.e., KISSMetrics)? I’m also particularly curious how you track task completion rates. Any help would be great appreciated.

Thanks,

John

I’ll let Jon respond to your comment and answer your questions about metrics, but you mentioned you implemented Scrum and later realized the project should be UX -driven.

This struck me as odd since Scrum only defines how the team meets, communicates, and prioritizes work. It doesn’t gvern whether a project is engineering- (what can we build?), market- (what is everyone else building?), or UX-driven (what do customers want us to build?).

A minor miscommunication or failure to communicate on my part. Sorry if that has caused you any confusion or frustration. The team was following an ad-hoc approach to development, but to be perfectly honest, there was no process. I put some process into place (i.e., Scrum). Now, our current challenge is integrating UX into Scrum. I hope that clears things up.

Let me answer the question about metrics first and then I’ll comment on the idea of “UX-driven” projects. They’re actually related related topics.

There are many types of UX metrics that you can collect, and it’s important to focus on the right ones for your situation. These should clearly map to the stories you list in your backlog and your overall business goals. Task completion is sort of the gold standard in design research. Simply put, think of task completion as verifying that the intended user can use what you built to satisfy the story/requirement.

Before selecting a tool, think about what you want to do with it. Web analytics tools like those you mention allow you to track site traffic and provide low cost ways to get metrics like time on site, or % of users that click a link. Other tools like UserZoom make it easy to test task completion rates, which map nicely to functionality described in user story format.

Make sure whatever metrics you track are actionable. It’s one thing to know nobody visits a page, another to know why. Task completion rates can even be measured without any special tools at all just by watching users try to use the site in person. In fact this might be a good place to start. You could write an entire book on this topic, and many have, a good one is Measuring the UX by Tullis & Albert.

This leads to the idea of being “UX-driven.” It’s a common misconception that UX is just about coming up with UI designs, but it’s much more than that. It’s a holistic viewpoint of the design process. The UXI Matrix supports that. Right now, if you tried to use it to track what stories you have UI designs for, how would you know if these designs were any good? Using the UXI Matrix as shown above will help you prioritize UX work (both design and research) and have meaningful discussions about what’s not getting done, and how well you are doing.

I appreciate the attention to detail in measuring and validating user stories. Are you seeing agile process as inclusive of the product planning stage? I don’t see that and just to clarify, I wouldn’t expect it either in the iterative design- in- production phase.

What is often confusing and lacking in these discussions and agile formulas is the recognition that the early gathering of user goals and user requirements is fundamentally a different process, where user research and PO’s work hand in hand th ideate and evnision product. That is not a development process. Until you have your general initiative and research done (including such things as personas), you can’t very well chunk out the individual stories for development. I think that is the problem with taking one ideology and trying to make it fit for the whole product lifecyle (which is much wider than the product software development life cyle, for which I think agile can work very well).

Perhaps you are making those assumptions ( that product planning has already occurred). A lot of folks though are bundling it all into one big blob…I’d like to hear your take on this.

For anyone that has been doing this for years in enterprise environments, and, are looking for some ways to do the “incremental improvement” thing, I can poitn you towards some more good info.

1.Larry Constantine and Activity Centered Design for Agile. Excellent for closed domains like the US Healthcare system.
2. Jeff Patton and Storymaps. I worked with Jeff on internal project 5 years ago which gave rise to looking at how Indi Youngs mental mapping could be applied to writing flat requirements backlogs that could be modeled any way you like to create meaning (aka Storymaps).
3. Behavior Driven Development. Some tools include rBehave, now part of rSpec. Has value in software projects where a comprehensive UX requirements matrix can drive the actual test criteria. Cool stuff for those mired in massive complexity with hundreds or thousands of programmers,

I really enjoyed your article particularly the way your characterizations translate to the sort of quantifiable metrics that Product Owners and Execs like to see and use in decision making. I have several questions:

– I don’t see much about how this plays in the broader project environment. I see why different stakeholders should value it but how do you get past the perception that UX is a strategic asset as well as a delivery asset?

- How has it typically been received by Product Owners or Project Managers? What level of education have you had to provide in order to simply be allowed to drive this method? I can see this being easier to implement as a consultant vs.an internal practitioner.

- How integrated or exclusive is this exercise? It looks like it resides entirely within UX for the things that inform it except for the PO Impact column. Do you have to defend it against Product & Dev bias? I think there are some predictable challenges there.

– How do you accommodate efforts where the empirical constraints like budget or time skew away from a more considered UX effort? (I know you’re not supposed to time-box an Agile effort but it’s not unheard of.)

I’m really intrigued by the opportunities this presents and I’m already trying to identify where we might use this method. It still looks like there are some common and widespread objections to overcome. I’ve had a great deal of experience and contention just trying to get people to agree that User Value is a separate consideration that needs to be factored-in on the same level as Business Value or Technical Risk. You make a great distinction between consumer driven products vs industrial client-server applications. You also note the need to account for UX lead-time when sequencing. I’d like to hear more about how the groundwork was laid in order to clear the way for this method to do its work.

Cheers,

-AP