“How many people do we need to get?”

This is the #1 question we hear when we help clients run online studies. And because we’re consultants, our stock answer is “It depends.” 

The simple answer is:

The more sophisticated answer is:

• 30 participants will start showing patterns in your results, but it will be hard to know what to do with “small effects” because you don’t have enough participants to know if these are real effects or just outliers.
  
  
• 50-100 participants will make the patterns much clearer, and you’ll be able to identify which results are significant and which can be discarded as noise.
  
  
• Hundreds of participants give diminishing returns, and you’re potentially “using up” participants who might be better employed as fresh participants in a subsequent round of testing.
  
  

Most products/websites have more than one major type of user. In Which part of the tree? in Chapter 6, for example, we saw that the Shimano website has 3 user groups – cyclists, anglers, and rowers.

If your study is covering several user groups, you’ll ideally want about 50 participants for each group. That way, you can filter the results by user group and still have enough data to see clear patterns in your results.

(You’ll also need a way of identifying which participants belong to which user group. We often do this with survey questions – see Adding survey questions in Chapter 8.)

Some user groups are more important than others, and your pool of participants is often limited, so try to get more participants from your major groups. If you end up with too few participants of a less important group, that’s something your project team can probably live with.

The other factor that affects how many participants you need is how many tasks (out of the total) that each participant does.

To understand this, let’s consider two cases:

• Each participant does all tasks.
  Suppose you have 10 tasks in your tree test – that is, 10 things that you want your participants to find, and this provides adequate coverage of the important parts of your site tree.
  Suppose you decide that each participant should do all 10 tasks. This is a reasonable number because, as we saw in How many tasks? in Chapter 7, 10 tasks makes for a quick test and minimizes the learning effect.
  Because each participant is doing all the tasks, you would simply aim for 50 participants of each user group.
  
  
• Each participant does half the tasks.
  Suppose now that you actually wrote 20 tasks, perhaps because the tree is large and 10 tasks just wasn’t enough to test everything you wanted to cover.
  If you asked each participant to do all 20 tasks, the number-of-participants answer is the same – about 50 per user group.
  However, we saw in the Tasks chapter that it’s not a good idea to ask participants to do that many tasks: it takes too long, they get bored or tired, and they are more likely to “learn” the tree (which skews your results).
  If you did the prudent thing and asked each participant to do 10 tasks, that’s half the tasks in the test, so you will need twice the number of participants (about 100) to get each task “hit” by the 50 you're aiming for.
  
  

In the end, the formula for this is simple: if you divide your total number of tasks by the number per participant, this gives you a multiplier for how many participants you need. For example, if you have 20 tasks total and you ask 10 per participant, this would be (20 ÷ 10) = 2, so you’ll need 2 times the normal number of participants. If you wanted 50 responses per task (the minimum we recommend), that means you'll need to get (50 X 2) = 100 participants in total.

Next: Different user groups