...
- Did it perform well, and all we have to do is tweak a few things?
- Did it perform just OK, and some parts need reorganizing?
- Did it perform poorly, and we need to discard it and look at other options?
Overall success rate
No Not surprisingly, the most important thing to look at is success rate – how many participants chose the correct answer, across all tasks?
...
But we do need to start from somewhere. In our experience, over hundreds of tree tests, the following rough markers have emerged for trees of average size and complexity: ~are these just success rate, or composite?
- 0-50 – The tree needs to be completely rethought or discarded.
Trying to tweak it will only bring it up to “mediocre”. - 50-65 – The tree needs substantial revisions.
If your analysis reveals specific problems (and it should), and you think you can fix them, you should be able to revise this tree to perform well. - 65+ - The tree is effective, but may need minor revisions.
Your participants are finding the correct answer at least two-thirds of the time, so the tree is doing its job well, and only needs tweaking.
| Note |
|---|
| A high score doesn’t mean “no revisions needed”. We’ve never run a tree test where everything worked so well that we couldn’t improve it a bit more. There are always a few lower-scoring tasks that suggest further improvements.note |
- how TT scores relate to UT scores, why 65+ is good, and the role of visual design, nav aids, content, etc. in eventual success rates. Lisa Fast’s graph?
...
What the overall success rate doesn't tell you is how much the success rate of the individual tasks varied. For example, a 60% overall score may mean that all tasks hovered around 60%, or that half your tasks were 90% and half were 30%. To find out, you need to drill down to the task level - see Task success rate later in this chapter.
Comparing tree-test scores to usability-test scores
People are often surprised that we consider 65+ to be a "good" score. Shouldn't the bar be set at 80 or 90?
Effective trees don't usually score higher than 80 because we're testing a top-down text-only tree with no other aids. Our participants are making choices without the benefit of:
- other navigation aids such as see-also links, featured links, and multi-level browsing (e.g. mega menus)
- visual design - chunking of links/content, and emphasis on more important links/content
- content that explains headings using decoration text, hover text, etc.
Once we refine our text tree to be effective (i.e. perform well in tree testing), we should then be able to add these other design elements to further improve the findability of items in our website.
| Tip |
|---|
| In our experience, success rates from the final website tend to be 20-30% higher than the scores we see in tree testing. |
Lisa Fast at Neo Insight has written an ~interesting blog post comparing tree-test scores to usability-test scores. Here's a graph showing what she found:
Overall directness (backtracking)
...
- Treejack calculates its overall score as a weighted average of success rate and directness (at a 3:1 ratio), but does not include speed in its calculations. ~no longer provides a total score?
- UserZoom?
...
Next: Analyzing by task