...
- 30 participants will start showing patterns in your the results, but it will be hard to know what to do with “small effects” because you we don’t have enough participants to know if these are real effects or just outliers.
- 50-100 participants will make the patterns much clearer, and you’ll we’ll be able to identify which results are significant and which can be discarded as noise.
- Hundreds of participants give diminishing returns, and you’re we’re potentially “using up” participants who might be better employed as fresh participants in a subsequent round of testing.
For a more rigorous look at how many participants you we should aim for, see this MeasuringUsability article on tree testing.
Counting by user group
Most products/websites have more than one major type of user. In Which part of the tree? in Chapter 6, for example, we saw that the Shimano website has 3 user groups – cyclists, anglers, and rowers.
If your our study is covering several user groups, you’ll we’ll ideally want about 50 participants for each group. That way, you can wecan filter the results by user group and still have enough data to see clear patterns in your the results.
(You’ll We’ll also need a way of identifying which participants belong to which user group. We often do this with survey questions – see Adding survey questions in Chapter 8.)
Some user groups are more important than others, and your our pool of participants is often limited, so we try to get more participants from your our major groups. If you we end up with too few participants of a less important group, that’s something your the project team can probably live with.
...
The other factor that affects how many participants you we need is how many tasks (out of the total) that each participant does.
| Tip |
|---|
| If each participant is only asked a subset of your tasks, you’ll we’ll need proportionally more participants. |
...
- Each participant does all tasks.
Suppose you we have 10 tasks in your our tree test – that is, 10 things that you we want your our participants to find, and this provides adequate coverage of the important parts of your the site tree.
Suppose you we decide that each participant should do all 10 tasks. This is a reasonable number because, as we saw in How many tasks? in Chapter 7, 10 tasks makes for a quick test and minimizes the learning effect.
Because each participant is doing all the tasks, you we would simply aim for 50 participants of each user group. - Each participant does half the tasks.
Suppose now that you we actually wrote 20 tasks, perhaps because the tree is large and 10 tasks just wasn’t enough to test everything you we wanted to cover.
If you we asked each participant to do all 20 tasks, the number-of-participants answer is the same – about 50 per user group.
However, we saw in the Tasks chapter that it’s not a good idea to ask participants to do that many tasks: it takes too long, they get bored or tired, and they are more likely to “learn” the tree (which skews your the results).
If you we did the prudent thing and asked each participant to do 10 tasks, that’s half the tasks in the test, so you will we would need twice the number of participants (about 100) to get each task “hit” by the 50 youwe're aiming for.
In the end, the formula for this is simple: if you we divide your our total number of tasks by the number per participant, this gives you us a multiplier for how many participants you we need. For example, if you we have 20 tasks total and you we ask 10 per participant, this would be (20 ÷ 10) = 2, so you’ll we’ll need 2 times the normal number of participants. If you we wanted 50 responses per task (the minimum we recommendrecommended), that means youwe'll need to get (50 X 2) = 100 participants in total.
...