...

We should also reconsider tasks that (almost) everyone got right. It’s great that participants easily found what they were looking for (after all, that’s our Big Goal), but first let’s make sure those tasks weren’t “gimme’s”. Did we give away the answer by careless word matching, or by phrasing the question in the same browsing sequence that the participant would follow in the tree? (For more on these problemspitfalls, see Writing a good task in Chapter 7.)

Even if the task passes these tests, we still may want to replace it because (unless we change the parts of the tree that it tests) in the next round. Why? Because we won’t learn much from it in later testing. It works, and we should move on.

This is especially true if we find that another part of the tree is either not performing well or is not being tested enough; it may be better to replace our “golden” task with one that tells us more about what needs improving. ~but replacing tasks will make before/after comparisons harder to make, so is this really a good idea? Just add the more useful tasks and keep the golden ones?

•  example

Note that there are two major downsides with replacing high-scoring tasks in a later round of testing:

• Your overall success rate may go down.
  If you replace a task that scored 90%, the task that replaced it is unlikely to score that high, so your average score will go down. This makes your results harder to explain to your project team and management.
  
  
• Before/after comparisons will be harder to make.
  When you keep the tasks the same between tests, you can make apples-to-apples comparisons. If you start replacing tasks, you can no longer make broad comparisons across tests.
  
  

If these factors are important to your study, you may want to avoid replacing high-scoring tasks. If they're not so important (that is, if your need for specific answers to specific questions outweighs these broader considerations), then replacing high-scoring tasks becomes a way of re-focusing your study on the parts of your tree that need more testing.

...