You should randomize tasks

Most tree-testing tools give us the option of randomizing the order of tasks that are shown to the participants.

If we don't randomize tasks, each participant will see the same tasks in the same order:

If we do randomize tasks, each participant will get the tasks in a different (random) order:

Reasons for randomizing tasks

Why randomize tasks? Because we want to reduce the learning effect on our results:

• If we ask all participants to do the tasks in the same order, then they are likely to do better on tasks presented late in the test because they have learned at least some of the tree by then (by browsing it repeatedly). This gives late tasks an unfair advantage.

• By putting the tasks in a random order per participant, we make sure that task 15 (for example) is sometimes shown late, sometimes shown early, and sometimes shown in the middle. Mixing up the task order negates ensures that no task gets an advantage by always appearing late in the test.

Not randomizing the first task

Some online tools provide an option to not randomize the first task; that is, the first task in our list is always shown first to participants, even if the other tasks are randomized.

The reason for this option is that most participants who do a tree test have never done one before, and no matter how good our pre-test instructions are, they may not fully understand how the test works until they've done a task.

The idea here is to provide our participants with a "warm-up" or "training" task, which we can exclude from our later analysis. (Participants may use this first task to click around the tree and explore a bit, even if they know the correct answer, so the results may be dodgy and therefore omitted from our scoring.)

What makes a good first "training" task?

• The task is not too hard.
  We don't want participants to be intimidated or dismayed at how hard our warm-up task is.

• The most likely paths go at least 2 clicks down the tree.
  We want participants to see that the tree is multi-leveled, and is only shown one branch at a time.

• The results for this task are not important to our study.
  Because this is a "throw-away" task, we want to pick something that we don't really care about (because we'll be ignoring this task's results later).

Next: Letting participants skip tasks