...

We should also reconsider tasks that (almost) everyone got right. It’s great that participants easily found what they were looking for (after all, that’s our Big Goal), but first let’s make sure those tasks weren’t “gimme’s”. Did we give away the answer by careless word matching, or by phrasing the question in the same browsing sequence that the participant would follow in the tree? (For more on these problemspitfalls, see Writing a good task in Chapter 7.)

Even if the task passes these tests, we still may want to replace it because (unless we change the parts of the tree that it tests) in the next round. Why? Because we won’t learn much from it in later testing. It works, and we should move on.

This is especially true if we find that another part of the tree is either not performing well or is not being tested enough; it may be better to replace our “golden” task with one that tells us more about what needs improving. ~but replacing tasks will make before/after comparisons harder to make, so is this really a good idea? Just add the more useful tasks and keep the golden ones?

•  example

Note that there are two major downsides with replacing high-scoring tasks in a later round of testing:

• The overall success rate may go down.
  If we replace a task that scored 90%, the task that replaced it is unlikely to score that high, so our average score will go down. This makes the results harder to explain to the project team and management.
  
  
• Before/after comparisons will be harder to make.
  When we keep the tasks the same between tests, we can make apples-to-apples comparisons. If we start replacing tasks, we can no longer make broad comparisons across tests.
  
  

If these factors are important to our study, we may want to avoid replacing high-scoring tasks. If they're not so important (that is, if our need for specific answers to specific questions outweighs these broader considerations), then replacing high-scoring tasks becomes a way of re-focusing the study on the parts of the tree that need more testing.

If we revise tasks, we must also remember to check the correct answers for those tasks. Our revisions may have changed what we should accept as correct. This is especially true if our revisions were not just minor rewording, but actually changed the meaning or purpose of the task itself.

For more, see Identifying correct answers in Chapter 7.

...

Next: Tuning survey questions

...