Size does matter

If we’re tree-testing on paper, the size of our tree obviously matters because of the physical effort involved. Most designers would balk at the idea of writing index cards for a tree 7 levels deep with 1000 items or more. They would immediately look for ways to cut their tree down to something more manageable for testing purposes.

We might think that online testing gets rid of this problem. After all, web apps can now handle vast amounts of data without bogging down, so why not test the whole tree?

The main problem with tree-testing very large structures is not technical; it’s about the participant’s experience of the test. If they have to search through a huge tree (either wide, or deep, or both), it’s going to take them longer for each task, and they’re more likely to get tired of the test sooner. That means more abandoned sessions (“Ack, I’m only halfway through this”) and less chance that they’ll participate in future online studies for us (“I finished, finally, but I’m not doing one of those again”).

We can counter this by asking each participant fewer questions, but that requires us to have more participants in total, which can be a real problem if we’re targeting very specific types of users.

If we prune large trees for testing, we can provide a better participant experience without giving up much in the way of useful findings. Ideally what we’re looking for is a study that is easy to pitch to users (“Do our 5-minute survey – win an iPad!”), quick for them to complete, and leaves them in a mood to say yes to the next study we cook up.

The other reason to consider pruning large trees is the effort needed to prepare the tree. If we can cut a few hundred entries out of our tree, those are entries that don’t have to be reviewed and cleaned up. If we’re testing several trees, the saved time really adds up.

How big is the tree?

If we’re testing a small or medium-sized tree (typically less than 500 items), we will normally test the whole tree; it’s not big enough to cause any unwanted test effects. We still need to check the tree for specific headings to include/exclude/massage (see the rest of this chapter), but we can safely skim or skip the rest of this particular section.

If the tree is large (in our experience, that means 500-1000 items), we have two options:

• Test the whole tree – This is usually easiest from our point of view (no pruning required), but can degrade the participant’s experience as described earlier.

• Test a “pruned” version of the tree – We may be most interested in how well the top few levels of our site work, in which case we can limit the test tree to the top 3 or 4 levels. Participants should then have time to do a larger number of tasks in a shorter total time.
  We can also cut the tree down to size by selectively pruning branches that are not important to our test.
  Lastly, we can opt to test only a particular subtree of the whole tree – e.g., the Sports section of an newspaper site. Here again, there are trade-offs, so it depends on what we’re really looking for.
  Each of these methods is explained below.

Finally, if our tree is very large (more than 1000 items), it’s almost guaranteed that we’ll need to prune it to keep our participants’ efforts from becoming onerous.

Pruning levels

The simplest way to cut a large tree down to a more manageable size is to cut everything below a certain level. That is, if we have a tree 5 or 6 levels deep, cut it off at 3 or 4 levels instead (in this example, the topics above the red area):

Open

This is easy, but we must be careful – we’re giving up something by axing the lower levels. In the full tree, we would have been able to see if participants could follow the full path down to the exact answer we intended (and if not, where they strayed off the path). With a “short” tree, we’ll only find out if they were able to get to the right general area.

This sounds like a big trade-off, but it really depends on the goals of our particular test. If we’re mainly concerned with the high-level organization of the site, losing the paths through the lower levels may not matter so much, and this type of horizontal pruning may be the best return for our effort.

If, however, we do want to know how well our lower levels perform too, then we should consider the other methods described below.

Pruning unnecessary branches

Instead of pruning straight across a given level of the tree, we may want to be more selective about what we cut. There’s no hard-and-fast rule that says we need to have an equal number of levels across all sections of our site. If there’s a section that we don’t intend to test (either because it gets very little traffic, or we already know that it works well), we can chop it off short, while leaving other sections (the ones we really want to test) at their full depth of topics.

For example, suppose we’re testing a new tree for a large corporate site, and we’ve decided to focus our testing on the sections that handle selling products and supporting them. Those sections might go 4, 5, or even 6 levels deep, which will give us some detailed results.

Suppose we also have an About Us section that’s equally deep. If we’re not really interested in testing it (perhaps because that section gets infrequent traffic), we could cut it off at, say, level 2 (at topics like Company History, Staff, Our Vision, Jobs, etc.).

Open

We may also want to do some selective pruning after we’ve written our tasks (covered in Chapter 7). If there’s a section of our site that none of our tasks come anywhere near, we can probably chop its lower sections without affecting the results. While it won’t save users time during the test (we don’t expect them to visit that section anyway), it will save us time preparing the tree.

Testing subtrees

If we have a very large tree (perhaps thousands of items), a more drastic way to cut it down is to only test a single section of it.

Suppose again that we’re testing a large corporate site like Sony or IBM, with many different business divisions, or perhaps a broad government agency with many different departments:

Do we test the entire tree, or just part of it?

Let’s take the easy case first. If we want to test several parts of the big tree, we’re going to create tasks that use those sections, so we’ll clearly need to keep all the sections, at least down to a certain level. If we want to cut down the tree, we’ll need to prune levels across the board or selectively, as described above.

If we’re really just testing a single section of the big tree, it may still be a good idea to include all the top-level sections anyway, to see if participants can find their content amid the “clutter” of other divisions/departments. In this case, we can safely chop off the sections we’re not interested in at level 2 or 3, because we don’t expect much traffic there (hopefully none at all). Note: don’t cut those sections off at level 1, because participants will quickly figure out that those are just stubs and not where the real action is supposed to be.

For example, if we want to test Robotics, we could abbreviate the other divisions:

In the end, we should only test a subtree by itself if:

• We don’t intend to test content in other parts of the big tree, and

• We’re very sure that participants would choose our subtree every time for each of the tasks we give them, or that they would go to our subtree directly in the real site, bypassing the parent site entirely.

A real-life example makes this clearer. Suppose we’re testing the Shimano website. Shimano is best known for cycling products (bike pedals, gears, etc.), but they also make equipment for fishing and rowing.

Here is their top-level navigation:

If we were testing pure cycling tasks (look for bike shoes, check a warranty, read a product FAQ), we could just test the Cycling subtree (with Shimano Cycling as the root of the tree), because we’d be 99% sure that no one would choose the other sections shown here – they clearly have nothing to do with cycling.

If, however, one of our tasks involved finding the story of how Shimano got started in the bike industry, now it’s not so cut and dried. We could imagine some participants going to the Corporate section to find the company’s history. In that case, we might be best to include these top-level sections, include the full Cycling section (obviously), most or all of the Corporate section, and the top level or two of the Fishing and Rowing sections.

If we were writing tasks that involved both cycling and fishing, then clearly we would have include all these top-level sections, but we could safely cut the Rowing section short. The bigger problem in this case would be that we’re asking each participant to try both cycling and fishing tasks, and a cyclist (for example) may not know anything about fishing, so their fishing results would be suspect. For more on setting different tasks for different user groups, see Different tasks for different user groups in Chapter 7.

Next: Which headings to include/exclude?