...
Everyone is happy until several weeks after the site is released. The analytics show that certain parts of the site that expected heavy traffic are getting very little, while the web-feedback channel is choked with users complaining they can’t find this or that on the new site. (They love the new look and feel, but it’s much harder to find things now that everything has changed.) This feedback persists for several months.
"Way too much work to find what I want. Maps used to be all in one place, but now they're scattered all over the place." - a disgruntled site visitor
Eventually, a consultant is brought in to do some usability testing on the site and recommend fixes. One of the major issues she finds is that the site is organized in a way that makes sense to the project team, but not to some of their major audiences. Some of this can be fixed with simple terminology changes, but some of it will require fundamental changes to the site structure, which will in turn require a good deal of content rework.
No one (including Tom) wanted this result, but now that they know about it, it can be fixed in release 2 (if they can get funding for it).
"We were hoping to add a stolen-bike registry next, but that will have to wait until we solve some basic navigation issues." - site owner
Maria and the empirical method
...
Instead of finalizing the structure then and there, she wants some proof that a topic-based tree is better than her initial idea of an audience-based tree. So she creates an audience-based tree as well, and decides to test them against each other. (The project sponsor happens to prefer the audience scheme, so this is an additional reason to get objective data on both before deciding.)
She takes Maria spends a week to run running side-by-side tree tests (one for each tree), and gets about 100 people to try out each tree:
...
The results are revealing – the second topic-based tree actually performs much better, except for a few tasks where the first audience-based tree wins:
...
| Metric | Topic tree | Audience tree |
|---|---|---|
| Success rate | 72% | 55% |
| Directness | 77% | 64% |
| Average speed | 14 secs | 18 secs |
Maria reviews the results of the test with the team, and they agree to go with the second tree, but incorporating some elements of the first tree that worked particularly well. This is the structure they build the site with, and when they run a usability test on the alpha version, it performs well except for a few minor changes that they can make before the site is released.
...
Once the site is live, the analytics show the expected areas of traffic. A few users complain about not being able to find things because they’ve been moved, but these complaints dwindle after the first month as users become familiar with the new structure.
"Took a bit to get used to, but it's now much easier to find what I want." - a satisfied site visitor
Maria, her project team, and upper management are all happy with the new site, so they find it easy to get funding to do the exciting things they have in mind for release 2.
- quote from project manager about the cool stuff they can now do in v2
...
:
"Great feedback from users, and traffic has jumped too. The committee was happy to give us the go-ahead for the exciting stuff in release 2." - site owner
The moral of our story
Lest you think this is a straw-man example, over the years we've seen an astonishing number of websites are created using Tom’s single-design genius method (or something very close to it). The Toms of the design world may be talented, but they are often curiously reluctant to use empirical tools to test their designs before the website ships. And the odds of them getting everything right the first time are very low. That translates into a high risk for the organization.
...