...

Once we get a score for a task, we naturally want to know why it scored like that:

...

This one comes from InternetNZ, a non-profit Internet advocacy group in New Zealand. When they redesigned their website, they ran several tree tests to check the effectiveness of the new structure. Here’s a task that performed fairly well:

As we can see from the task summary, almost three quarters of our users found a correct answer, with very little backtracking. That’s good to know, but it’s not enough – we need to see where they went, both for the correct and the incorrect cases.

Most tools give us a way to examine the click paths for a given task. We used Treejack for this study, which offers a “pie tree” diagram showing where participants went: 

For this example, let’s concentrate on the paths, not so much the pies themselves:

...

Notice also that there was very little “leakage” at the top level (the first click from the home page). Only 1 participant out of 63 made a wrong turn at the start. We’ll discuss the importance of first clicks in What they clicked first later in this chapter.

But tree tests are not all sunshine and lollipops. Some of our tasks will probably look like this one, again from InternetNZ: 

The correct answer in under Policies, in a section called Jurisdiction, but 87% of our participants failed to find it. Only 3% gave up, so where did all the others go?

The answer is that they went everywhere. Graphically, this is what “everywhere” looks like: 

Quite a mess. But what can we learn from it?

...

For example, let’s look at the Projects by name section more closely:

A large chunk of participants came here, and the correct answer (Jurisdiction) was waiting for them, but most didn’t choose it (partly because it’s more abstract than the other topics).

...

The graph makes another problem obvious too – the “leakage” at the top level of the tree:

Right at the start, as their first click, a large fraction of the participants ran off in all directions. This is not what we intend when we design a site structure. It suggests that either:

...

We can also see scattering when we view the results as a spreadsheet. The vertical cluster of cells shows that, for a single task, participants chose a wide variety of subtopics under the correct topic.

Studying the click paths of a single task gives us insights into how the tree performed for that particular task. But it’s dangerous to draw conclusions based on only one scenario. What we need to do is look for supporting evidence from the other tasks in our study – see Finding patterns among tasks later in this chapter. 

...

The best way to govern this is to agree on some consistent criteria for correctness ahead of time - see Identifying correct answers in Chapter 7.

...

Next: What they clicked first