...
- Some tasks may be confusing or misleading.
This is especially likely if you didn’t properly pilot your test. Some tasks are hard to phrase clearly without giving away the answer, but remember that a confusing task is a problem in your study, not necessarily a problem in the tree itself. You shouldn’t change the wording during the test, but you should revise in your next round of testing. - Some correct answers aren’t marked as “correct”.
After doing hundreds of tree tests, we still run into this wrinkle all the time. When we set up each task, we try to mark all the correct answers for it. However, in a large tree, each task may have several correct answers, and it’s likely we’ll miss a few.
Because of this, a good testing tool should let you (as the test administrator) change which answers are correct for each task, either while the test is running or afterward when you’re doing your analysis. We often find that test scores go up substantially when we do this post-test correctcorrection. For more on this, see Cleaning the data in Chapter 12.
...
- At the explanation page
If your web ads or email invitations link to an explanation page, you can use web analytics to compare how many people visit that page to how many actually start the tree test itself. A large drop-off here indicates that your explanation page is either confusing, hard to scan, or the “start” link is not obvious. (Is it prominent and above the fold?) - During the test itself
If a person makes it to the tree test itself, try to find out where they drop out. (Most Unfortunately, most testing tools do a poor job of helping you in this regard.)
If it’s during the welcome/instructions stage, they may be finding these pages confusing, too long, or simply not what they expected. You can check this by trying the test with a few people in person to see where the problem lies.
If they drop out during the tasks, it could be caused by having :- Having too many tasks (seeing “1 of 26” is daunting)
- Presenting tasks that are confusing (
- “Forget this, it's just too hard”)
- Or simply because this is the first time they’ve done a tree test and they’re not sure what to do. (Better instructions may help, but some people will leave no matter how well you explain it.)
...
Next: Keeping stakeholders informed
...