Most tree-testing tools give us the option of randomizing the order of tasks that are shown to the participants.
If we don't randomize tasks, each participant will see the same tasks in the same order:
| Participant 1 | Participant 2 | Participant 3 |
|---|---|---|
| Task 1 | Task 1 | Task 1 |
| Task 2 | Task 2 | Task 2 |
| Task 3 | Task 3 | Task 3 |
If we do randomize tasks, each participant will get the tasks in a different (random) order:
| Participant 1 | Participant 2 | Participant 3 |
|---|---|---|
| Task 1 | Task 2 | Task 3 |
| Task 2 | Task 3 | Task 1 |
| Task 3 | Task 1 | Task 2 |
For most studies, we should randomize the order of tasks.
Why randomize tasks? Because we want to reduce the learning effect on our results:
- If we ask all participants to do the tasks in the same order, then they are likely to do better on tasks presented late in the test because they have learned at least some of the tree by then (by browsing it repeatedly). This gives late tasks an unfair advantage.
- By putting the tasks in a random order per participant, we make sure that task 15 (for example) is sometimes shown late, sometimes shown early, and sometimes shown in the middle. Mixing up the task order negates ensures that no task gets an advantage by always appearing late in the test.
- warm-up/training task - see TJ's "don't randomize first task" option
Next: Letting participants skip tasks