How many tasks overall?

As you might expect, there is no single definitive answer to how many tasks you should include in your tree test. The number will vary depending on several factors, including:

The size of your tree
Larger trees typically need more tasks to ensure adequate coverage.
The complexity and variability of the tree
If you have many sections that reuse the same general structure (e.g. several product subtrees that have very similar layouts), then you may not need to test each one.
Your confidence in the tree
If you’re just making minor changes to a tree that has tested well before, you may not need to present many tasks. On the other hand, if it’s a completely redesigned, unconventional, or contentious tree design, you’ll definitely want to “put it through the ringer” by presenting a larger number of tasks.

As a starting point, though, the following numbers are typical of the tree tests that we run:

Size	# of items	# of tasks (overall)
Small tree	1-250	8-12
Medium tree	250-500	10-20
Large tree	500+	15-25

If you have fewer tasks than this, check your coverage of the tree (described in Mapping tasks to the tree later in this chapter). There may be important parts of the tree that you’re missing.

If you have more tasks than this:

Check that each task is answering a specific question you have about the site structure. If it isn’t pulling its weight, or if other tasks already address that question, consider removing it.
Remember that each task takes time to write (up front) and time to analyze (later), and that you’ll need a bigger participant pool to get all those questions answered (see below).

How many tasks per participant?

Give each participant 8-10 tasks. More is risky.

No matter how many tasks you create overall, there is a (relatively low) limit to how many tasks you should ask each participant to do. This limit stems from two factors:

The time and effort required by each participant
The learning effect of browsing the same tree repeatedly

Participant effort

If you ask a participant to do 8–10 tasks in a tree test, that typically works very well – they try a small number of tasks (each one a bit different), they see the tree a few times (but not too often), and they finish in 5 minutes or so, so they’re not tired, bored, or grumpy at the end. They got a short, challenging exercise that was different (and more fun) than a traditional survey, and they leave happy (and probably willing to say yes to the next study you ask them to do).

That’s fine if you’re testing a small tree, where it’s easy to whittle your task list down to 10 or so. But suppose you’re testing a big tree and you’ve decided that you need 25 tasks to get adequate coverage of the parts that need validating.

If you make each participant do 25 tasks, neither you nor they will be happy, for several reasons:

When you invite them to the tree test, you’ll have to tell them it will take 15-20 minutes, not 5. That may drastically cut down the number of respondents you get.
After about 10 tasks, they may start to get tired of the exercise, and not try as hard, perhaps skipping more tasks, perhaps picking answers that will just get the test over with sooner. In any case, task fatigue will likely change their behavior, which puts your results in question.
Fatigue and boredom also means they may be less likely to take part in future studies. This is a particularly important consideration if you have a small pool of participants to begin with.
The learning effect becomes a real problem (see below).

The learning effect

The other problem with giving each participant a lot of tasks is the “learning effect”; as they browse the tree for each successive task, they start learning the structure.

This is not a bad thing in itself. After all, users visiting a website for more than a few minutes will likely learn the overall navigation to some extent.

However, in tree testing, we’re asking them to find things over and over, without anything else to look at except the structure, so they come to know it better than they would when they visit the real site. (Most users don’t arrive at a site with 10 successive things to look for.)

While some learning is inevitable, we can do two things to minimize its effect on our results:

Randomize the task order
By putting the tasks in a random order per participant, we make sure that earlier tasks don't "help" later tasks across all participants. See Randomizing the order of tasks below.
Limit the number of tasks per participant
If we present each participant with 25 tasks, they’re going to know the tree very well by the end, and this will unrealistically improve their performance on the later tasks. Limiting each participant to fewer tasks (say, 8-10) reduces the learning effect to something closer to what they would learn using the real website.

Setting the number of tasks for each participant

To let you test a large number of tasks overall without overburdening participants, most tree-testing tools let you specify how many of the tasks are shown to each person.

For example, suppose you have a medium-sized tree with 20 tasks that cover it to your satisfaction. That’s too many tasks to give each participant – they’ll get bored or tired, and their tree “learning” will pollute your results. They will be happier (and you’ll get sounder results) if you give them 8-10 tasks each.

This does mean that you’ll need to recruit more participants to get the same number of responses per task. In the example above, if we had 20 tasks overall and showed 10 to each participant, we would need about 100 participants to get 50 responses per task. For more on this, see How many participants? in Chapter 9.

paper

Randomizing the order of tasks

Most tree-testing tools give you the option of randomizing the order of tasks that are shown to the participants.

If you don't randomize tasks, each participant will see the same tasks in the same order:

Participant 1	Participant 2	Participant 3
Task 1	Task 1	Task 1
Task 2	Task 2	Task 2
Task 3	Task 3	Task 3

If you do randomize tasks, each participant will get the tasks in a different (random) order:

Participant 1	Participant 2	Participant 3
Task 1	Task 2	Task 3
Task 2	Task 3	Task 1
Task 3	Task 1	Task 2

For most studies, you should randomize the order of tasks.

Why randomize tasks? Because we want to reduce the learning effect on our results:

If we ask all participants to do the tasks in the same order, then they are likely to do better on tasks presented late in the test because they have learned at least some of the tree by then (by browsing it repeatedly). This gives late tasks an unfair advantage.
By putting the tasks in a random order per participant, we make sure that task 15 (for example) is sometimes shown late, sometimes shown early, and sometimes shown in the middle. Mixing up the task order negates ensures that no task gets an advantage by always appearing late in the test.

warm-up/training task - see TJ's "don't randomize first task" option

Next: Mapping tasks to the tree