Page Comparison

...

“How many people do we need to get?”

This is the #1 question we hear when we help clients run online studies. And because we’re consultants, our stock answer is “It depends.” 

The simple answer is: aim

The more sophisticated answer is:

• 30 participants will start showing patterns in your results, but it will be hard to know what to do with “small effects” because you don’t have enough participants to know if these are outlier results that can be ignoredreal effects or just outliers.
  
  
• 50-100 participants will make the patterns much clearer, and you’ll be able to identify which results are significant and which can be discarded as noise.
  
  
• Hundreds of participants give diminishing returns, and you’re potentially “using up” participants who would might be better employed as fresh participants in a subsequent round of testing.
  
  

...

The other factor that affects how many participants you need is how many tasks (out of the total) that each participant does.

To understand this, let’s consider two cases:

• Each participant does all tasks.
  Suppose you have 10 tasks in your tree test – that is, 10 things that you want your participants to find, and this provides adequate coverage of the important parts of your site tree.
  Suppose you decide that each participant should do all 10 tasks. This is a reasonable number because, as we saw in How many tasks? in Chapter 7, 10 tasks makes for a quick test and minimizes the learning effect.
  Because each participant is doing all the tasks, you would simply aim for 50 participants of each user group.
  
  
• Each participant does half the tasks.
  Suppose now that you actually wrote 20 tasks, perhaps because the tree is large and 10 tasks just wasn’t enough to test everything you wanted to cover.
  If you asked each participant to do all 20 tasks, the number-of-participants answer is the same – about 50 per user group.
  However, we saw in the Tasks chapter that it’s not a good idea to ask participants to do that many tasks: it takes too long, they get bored or tired, and they are more likely to “learn” the tree (which skews your results).
  If you did the prudent thing and asked each participant to do 10 tasks, that’s half the tasks in the test, so you will need twice the number of participants (about 100) to get each task “hit” by enough peoplethe 50 you're aiming for.
  
  

So, In the end, the formula for this is simple: if you divide your total number of tasks by the number per participant, this gives you a multiplier for how many participants you need. For example, if you have 20 tasks total and you ask 10 per participants is participant, this would be (20 ÷ 10) = 2, so you’ll need 2 times the normal number of participants. If you wanted 50 responses per task (the minimum we recommend), that means you'll need to get (50 X 2) = 100 participants in total.

...

Next: Different user groups

...