Making several site trees compete against each other is great, but at some point, we need to reduce them down to a single high-performing tree.
Once we’ve run our first round of tree tests, on 2 or 3 of the most promising trees we thought up, we analyze the results. Typically we find:
After the first round of tests, we either have:
We can now run a second round of tests to see if our revisions did indeed make things better:
In a perfect world, we would keep testing until we had a perfect tree, but there is never time or budget enough for that. We typically only do more than 2 rounds of testing if there are important parts of the tree that are still not performing well enough.
If we were expecting tree testing to be a one-off trick – build a tree, test it, and we’re done – it may be alarming that we recommend several rounds of testing, with several trees, winnowing and refining them until we get a single high-performing tree.
What makes this approach feasible is that we now have mature testing tools that are both:
The combination of cheap and fast changes how we should approach design. Instead of doing a single round of deluxe in-person testing (or worse, no testing at all), we can do several cheap online tests in less time and for less money.
There are cases where in-person testing is the way to go, particularly for complex interactions, or for when there are only a small number of participants available. But for most projects, several rounds of lightweight tests are a better bang for the buck.
And that means we can go wide at the start, and go deep through to the end.
Next: Putting it all together