How many rounds of testing?

The time, effort, money, and participants it will take to develop our site tree depends partly on how many rounds of testing we’re intending to do. More rounds usually means a better result (as we would expect), but there are also diminishing returns to consider.

In Putting it all together in Chapter 3, we recommended a "full-fat" process with 3 rounds of testing:

Because of budget or time constraints, this is often cut down to 2 rounds:

The first round of testing shows us where our tree is doing well (yay!) and where it needs more work. So we make some thoughtful revisions. Careful, though, because even if the problems we found seem to have obvious solutions, we still need to make sure our revisions actually work for users, and don’t cause further problems.

The good news is, it’s dead easy to run a second test, because it’s just a small revision of the first one. We already have the tasks and all the other bits worked out, so it’s just a matter of making a copy of the test (in whatever tool we’re using), pasting in our revised tree, and hooking up the correct answers. In an hour or two, we’re ready to pilot it again (to err is human, remember) and then send it off to a fresh batch of participants.

There are two possible outcomes here:

• Our fixes are spot-on, the participants find the correct answers more frequently and easily, and our overall score climbs. We could have skipped this second test, but confirming that our changes worked is both good practice and a good feeling. It’s also something concrete to show the boss.

• Some of our fixes didn’t work, or (given the tangled nature of IA work) they worked for the problems you saw in round 1, but now they’ve caused more problems of their own. Bad news, for sure, but better that we uncover them now in the design phase (when it takes a few days to revise and retest) instead of further down the track when the IA has been signed off and changes become painful.

Note that Round 1 combines the “before” and “after” testing, because most of our clients have a good idea of where the weaknesses are in their existing tree. If we don’t, the full 3-round approach described above is recommended; this can be combined with an open card sort to help generate ideas for the revised structure.

On some larger and more complex trees, additional revision rounds may be needed to confirm we have solved the major issues we uncover.

For planning, this means that we need to:

• add the desired # of rounds into our project schedule

• determine how we will get enough fresh participants for each round

Next: Which trees will we test?