...
Note that there are two major downsides with replacing high-scoring tasks in a later round of testing:
- Your The overall success rate may go down.
If you we replace a task that scored 90%, the task that replaced it is unlikely to score that high, so your our average score will go down. This makes your the results harder to explain to your the project team and management. - Before/after comparisons will be harder to make.
When you we keep the tasks the same between tests, you we can make apples-to-apples comparisons. If you we start replacing tasks, you we can no longer make broad comparisons across tests.
If these factors are important to your our study, you we may want to avoid replacing high-scoring tasks. If they're not so important (that is, if your our need for specific answers to specific questions outweighs these broader considerations), then replacing high-scoring tasks becomes a way of re-focusing your the study on the parts of your the tree that need more testing.
...