It’s not just the tree that may need revising after we get our first round of results. Sometimes the tasks themselves need rework.

Even though we work hard to write clear and unambiguous tasks, the results sometimes show that a task was not clearly understood, or was clearly misunderstood. Because most tree testing is done by the participant alone, they can’t ask for clarification when they have trouble understanding a task.

• example of misunderstood task

If we’re doing another round of testing, we should revise these murky tasks so we can get higher-quality results. This makes before-and-after comparisons harder, but comparisons are less important than clear primary findings about the tree.

• example of revising a task

We should also reconsider tasks that (almost) everyone got right. It’s great that participants easily found what they were looking for (after all, that’s our Big Goal), but first let’s make sure those tasks weren’t “gimme’s”. Did we give away the answer by careless word matching, or by phrasing the question in the same browsing sequence that the participant would follow in the tree? (For more on these problems, see Writing a good task in Chapter 7.)

Even if the task passes these tests, we still may want to replace it because (unless we change the parts of the tree that it tests) we won’t learn much from it in later testing. It works, and we should move on. This is especially true if we find that another part of the tree is either not performing well or is not being tested enough; it may be better to replace our “golden” task with one that tells us more about what needs improving. ~but replacing tasks will make before/after comparisons harder to make, so is this really a good idea? Just add the more useful tasks and keep the golden ones?

• example

If we revise tasks, we must also remember to check the correct answers for those tasks. Our revisions may have changed what we should accept as correct. This is especially true if our revisions were not just minor rewording, but actually changed the meaning or purpose of the task itself.

Next: Tuning survey questions