While a tree test is running, it’s good practice to log in every day or two and see how things are coming along.

We all hope that our studies attract lots of participants, of each type we ask for, and that the results show how much better our new tree is than the old one.

While that does sometimes happen, more often we find that things are not quite running to plan. Here are the problems we encounter most when we check the progress of a study.

Low participant numbers are the most common worry for any online study.

If you’re halfway through your test period and you only have a quarter of the participants you planned for, you may need to work harder to find people.

• Email: If you only sent an initial batch of email invitations, send another batch.
  
  
• Web ads: Re-check that your ad is very visible and presents a concise, attractive proposition. Consider putting it on more web pages and (if possible) on more websites to get more views.
  
  
• Social media: Don’t be afraid to repeat your invitation on your social networks halfway though the testing period.
  
  
• Incentive: If you suspect that your incentive is not big enough (and you’ve done everything else you can to boost your responses), consider increasing it. If your management reduced your planned incentive because they thought it was excessive, you may want to revisit that decision with them (with your data in hand).
  
  
  

When we target several user groups in a single tree test, sometimes we get lots of people from group A and B, but hardly any from group C. If you included a survey question that identified the participant’s user group, you can check that now to see if any groups are lagging and need more recruiting effort.

Obviously, the best way to boost a certain group’s numbers is to invite more of that group.

• If you have group-specific email batches that you haven’t sent yet, that’s the easiest thing to do.
  
  
• If you can place an ad on websites that this group frequents, that should also help boost your numbers.
  
  
• If this group is likely to have some kind of organization that they belong to (a trade association, meet-up group, special-interest forum, etc.), you may want to approach the organization’s administrator and ask for help.
  
  
  

Earlier we talked about splitting users randomly among tests. Usually this is an even split (e.g. two tests would each have a 50% chance of being selected), but sometimes we find that, halfway through the test period, test A has two thirds of the responses for some reason.

Whether you used code or a set of arbitrarily split links (first name A-M, first name N-Z, etc.), you can change this partway through the test to even up the numbers. For example, you may change the split so that test A now gets 30% of the clicks, while test B (the one that’s lagging) gets 70%.

Besides the number of participants, the other big thing you’re sure to check is the scoring – how well your tree is performing overall, and how individual tasks are doing.

Very often, you’ll be surprised (and appalled) by how low your interim scores are. Some part of the low scores will be justified – especially in a first-round test of a new tree, parts of that tree will simply not work well for your participants. Testing simply lets you identify the parts that need rethinking.

However, we also find that interim scores are often lower than expected because:

• Some tasks may be confusing or misleading.
  This is especially likely if you didn’t properly pilot your test. Some tasks are hard to phrase clearly without giving away the answer, but remember that a confusing task is a problem in your study, not necessarily a problem in the tree itself. You shouldn’t change the wording during the test, but you should revise in your next round of testing.
  
  
• Some correct answers aren’t marked as “correct”.
  After doing hundreds of tree tests, we still run into this wrinkle all the time. When we set up each task, we try to mark all the correct answers for it. However, in a large tree, each task may have several correct answers, and it’s likely we’ll miss a few.
  Because of this, a good testing tool should let you (as the test administrator) change which answers are correct for each task, either while the test is running or afterward when you’re doing your analysis. We often find that test scores go up substantially when we do this post-test correct. For more on this, see Chapter ~, Analyzing Results.
  
  
  

Ideally, a high task score means that you did your job well when you created the tree.

Unfortunately, it can also mean that you included a “giveaway” word (and didn’t spot it during piloting). If you did, then this isn’t a fair measure of the real-word effectiveness of your tree.

Again, you shouldn’t edit the task’s wording while the test is running (unless you spot it very early); fix it in the next round.

You may find that lots of participants start your study, but many drop out before they finish it.

You’ll always have some drop-off (it’s the nature of online studies), but if it exceeds about 25%, you should investigate.

• At the explanation page
  If your web ads or email invitations link to an explanation page, you can use web analytics to compare how many visit that page to how many actually start the tree test itself. A large drop-off here indicates that your explanation page is either confusing, hard to scan, or the “start” link is not obvious. (Is it above the fold?)
  
  
• During the test itself
  If a person makes it to the tree test itself, try to find out where they drop out. (Most testing tools do a poor job of helping you in this regard.)
  
  If it’s during the welcome/instructions stage, they may be finding these pages confusing, too long, or simply not what they expected. You can check this by trying the test with a few people in person to see where the problem lies.
  
  If they drop out during the tasks, it could be caused by having too many tasks (seeing “1 of 26” is daunting), presenting tasks that are confusing (“uh oh, this is just too hard”), or simply because this is the first time they’ve done a tree test and they’re not sure what to do. (Better instructions may help, but some people will leave no matter how well you explain it.)

Next: Keeping stakeholders informed