...
- John is from the school of genius design. He believes that he creates great websites because he is a talented designer and his past work was praised.
- Maria is from the school of empirical design. She believes she is a good designer, but she knows that she will not get everything right the first time, so she uses a variety of research tools to inform (and correct) her designs as she goes.
John and the genius method
Having studied the content and talked to the site’s internal stakeholders, John thinks a lot, then opens a new spreadsheet and creates a text tree of possible headings and subheadings, based on the various types of activities that the site offers.~ss
- ss of activity-based tree
...
A day or two later, another idea occurs to him, so he creates another tree based on different types of audiences:~ss
- ss of audience-based tree
...
He adds a few questions here and there where he needs to check ideas and terminology with the other project members.
...
Everyone is happy until several weeks after the site is released. The analytics show that certain parts of the site that expected heavy traffic are getting very little, while the web-feedback channel is chocked with users complaining they can’t find this or that on the new site. (They love the new look and feel, but it’s much harder to find things now that everything has changed.) This feedback persists for several months.~example
- example of customer feedback
...
Eventually, a consultant is brought in to do some usability testing on the site and recommend fixes. One of the major issues she finds is that the site is organized in a way that makes sense to the project team, but not to one of their two major audiences. Some of this can be fixed with simple terminology changes, but some of it will require fundamental changes to the site structure, which will in turn require a good deal of content rework.
No one (including John) wanted this result, but now that they know about it, it can be fixed in release 2. New features will have to wait until later, if they can get funding.~quote
- quote from project manager about having to redesign navigation in v2
...
Maria and the empirical method
Like John, Maria starts by studying the content and talking to the site’s internal stakeholders. She also asks about any existing user research, and receives the results of a site survey done the previous year.
She has some ideas on how to structure the content, but she also knows that she is not the target audience, so she decides that she needs to get some user input to help generate structure ideas. She runs an open card sort online using some representative content, and discovers that most users actually organize the cards according to activity, not according to audience. Her initial hunch was wrong, but it’s easy to change direction this early in the game.~ss
- ss of online card sort and/or results
...
She then opens a new spreadsheet and creates a text tree of possible headings and subheadings, based on the various types of activities that the site offers. (John also did this.)
She then creates a second tree that is also activities-based, but uses a different organizing method for the next level down (based on another finding she picked up from the card sort).~ss
- ss of two alternative tree ideas in spreadsheet
...
Like John, Maria checks her ideas with the project team and revises some terminology accordingly. Most of her team prefers her first tree to her second, which confirms her own preference.
Instead of finalizing the structure then and there, she wants some proof that the first tree is the best, and to see if anything else needs revising from a user’s point of view. She takes a week to run side-by-side two tree tests (one for each tree), and gets about 100 people to try out both tree ideas:~ss
- ss of tree testing participant session and pie-tree results for that task
...
The results are revealing – the second tree actually performs better, except for a few tasks where the first tree wins:~ss
- ss of overall scores from each tree
...
Maria reviews the results of the test with the team, and they agree to go with the second tree, but incorporating some elements of the first tree that worked particularly well. This is the structure they build the site with, and when they run a usability test on the alpha version, it performs well except for a few minor changes that they can make before the site is released.~example
- example of good customer feedback about easy to find stuff
...
Once the site is live, the analytics show the expected areas of traffic. A few users complain about not being able to find things because they’ve been moved, but these complaints dwindle after the first month as users become familiar with the new structure.
Maria, her project team, and upper management are all happy with the new site, so they find it easy to get funding to do the exciting things they have in mind for release 2.~quote
- quote from project manager about the cool stuff they can now do in v2
...
The moral of our story
In our experience, an astonishing number of websites are created using John’s genius method (or something very close to it). The Johns of the design world may be talented, but they are often curiously reluctant to use empirical tools to test their designs before the website ships. And the odds of them getting everything right the first time are very low. That translates into a high risk for the organization.
...
It doesn’t take a genius to recommend the latter approach. 
...
Next: 2:
...