Regardless of how the data is captured, it is unrealistic to expect to glean usability data from analytics. Without knowing what task a specific user is attempting to do, there’s no way of judging whether they’ve actually done it. Analytics tools only record requests. They can’t record whether that request did or did not satisfy a user’s need – only the user themselves can tell you that.

While your in-depth analysis does really well to point out the limitations of analytics I think the idea that one would replace analytics with a usability expert or vice-vera is a wrong. They both serve different purposes and both have their faults and I really can’t see how they are directly interchangeable.

I also think a lot of the analyitcs software around these days adds a lot more value and are ‘smarter’ than what it used to be.

I see analytics as another tool that adds weight to certain directions and provides valuable trend and user system data.

In the IA/Usability field I tend to feel there is far too many idelogical skews for and against certain approaches (I have been/am guilty of it myself) and feel that collecting all the information (both qualatative and quantative) possible that time/cost allows will be far more helpful in coming up with a suitable direction and solution.

Striking out the only available source of full scale system/user qualatative data (despite the limitations) seems a little strong.

Flexibility and a broad perspective is always the best approach and that nothing should be ruled out (unless it really is stretching the boundries of usefulness).

But depsite these points the article is a useful one as it does point out a log of specific analytic limitations I wasn’t fully aware of.

Cheers.

Look at the flip-side as well. Is testing a single user (or a couple of users) in a traditional usability test perfect, statistically speaking? Not remotely. Using a couple of users to generalize to thousands or tens of thousands is wildly unreliable, especially since those users may not even be representative. Does that mean the technique is useless? Of course not. In fact, using these techniques in tandem helps reduce noise even further: if you take the hypotheses from testing users individually and reference them against the clues you get from aggregate data, you’ll end up with information that is all the richer and more valuable.

My grad advisor used to say that there were three levels of understanding statistics: (1) knowing just enough to be dangerous, (2) knowing enough to play by the rules, and (3) and knowing when the rules don’t matter. I’ve met statisticians who can tear apart absolutely any real-world analysis someone could ever run, and they’re all level 2 people. The problem is, if you carry that too far, you can’t get anything done. I think it’s important to look at any tool objectively, know its faults, but then move forward and try to extract the best value you can. Web analytics aren’t a new, unproven tool, and while many people misuse them, I also know some flat-out brilliant people deriving real value from them for both clients and end-users.