If you’re tree-testing on paper, the size of your tree obviously matters because of the physical effort involved. Most designers would balk at the idea of writing index cards for a tree 7 levels deep with 1000 items or more. They would immediately look for ways to cut their tree down to something more manageable for testing purposes.

You might think that online testing gets rid of this problem. After all, web apps can now handle vast amounts of data without bogging down, so why not test the whole tree?

The main problem with tree-testing very large structures is not technical; it’s about the participant’s experience of the test. If they have to search through a huge tree (either wide, or deep, or both), it’s going to take them longer for each task, and they’re more likely to get tired of the test sooner. That means more abandoned sessions (“Ack, I’m only halfway through this”) and less chance that they’ll participate in future online studies for you (“I finished, finally, but I’m not doing one of those again”).

You can counter this by asking each participant fewer questions, but that requires you to have more participants in total, which can be a real problem if you’re targeting very specific types of users.

If we prune large trees for testing, we can provide a better participant experience without giving up much in the way of useful findings. Ideally what we’re looking for is a study that is easy to pitch to users (“Do our 5-minute survey – win an iPad!”), quick for them to complete, and leaves them in a mood to say yes to the next study we cook up.

The other reason to consider pruning large trees is the effort needed to prepare the tree. If you can cut a few hundred entries out of your tree, those are entries that don’t have to be reviewed and cleaned up. If you’re testing several trees, the saved time really starts to add up.

If you’re testing a small or medium-sized tree (typically less than 500 items), you can normally test the whole tree; it’s not big enough to cause any unwanted test effects. You still need to check your tree for specific headings to include/exclude/massage (see the rest of this chapter), but you can safely skim or skip the rest of this particular section.

If your tree is large (in our experience, that means 500-1000 items), you have two options:

• Test the whole tree – This is usually easiest from your point of view (no pruning required), but can degrade the participant’s experience as described earlier.
  
  
• Test a “pruned” version of the tree – You may be most interested in how well the top few levels of your site work, in which case you can limit your test tree to the top 3 or 4 levels. Participants should then have time to do a larger number of tasks in a shorter total time.
  
  You can also cut your tree down to size by selectively pruning branches that are not important to your test.
  
  Lastly, you can opt to test only a particular subtree of the whole tree – e.g., the Sports section of an newspaper site. Here again, there are trade-offs, so it depends on what you’re really looking for.
  
  Each of these methods is explained below.
  
  

Finally, if your tree is very large (more than 1000 items), it’s almost guaranteed that you’ll need to prune your tree to keep your participants’ efforts from becoming onerous.

The simplest way to cut a large tree down to a more manageable size is to cut everything below a certain level. That is, if you have a tree 5 or 6 levels deep, cut it off at 3 or 4 levels instead:

• diagram of pruning levels

This is easy, but be careful – you’re clearly giving something up by axing the lower levels. In the full tree, you would have been able to see if participants could follow the full path down to the exact answer you intended (and if not, where they strayed off the path). With a “short” tree, you’ll only find out if they were able to get to the right general area.

This sounds like a big trade-off, but it really depends on the goals of your particular test. If you’re mainly concerned with the high-level organization of your site, losing the paths through the lower levels may not matter so much to you, and this type of horizontal pruning may be the best return for your effort.

If, however, you do want to know how well your lower levels perform too, then you should consider the other methods described below.

Instead of pruning straight across a given level of the tree, you may want to be more selective about what you cut. There’s no hard-and-fast rule that says you need to have an equal number of levels across all sections of your site. If there’s a section that you don’t intend to test (either because it gets very little traffic, or you already know that it works well), you can chop it off short, while leaving other sections (the ones you really want to test) at their full depth of topics.

For example, suppose you’re testing a new tree for a large corporate site, and you’ve decide to focus your testing on the sections that handle selling products and supporting them. Those sections might go 4, 5, or even 6 levels deep, which will give you some detailed results.

Suppose you also have an “About Us” section that’s equally deep. If you’re not really interested in testing it, you may want to cut it off at, say, level 2 (at topics like Company History, Staff, Our Vision, Careers, etc.).

• diagram of pruning “About Us”

You may also want to do some selective pruning after you’ve written your tasks (covered in Chapter ~). If there’s a section of your site that none of your tasks come anywhere near, you can probably chop its lower sections without affecting your results. While it won’t save users time during the test (you don’t expect them to visit that section anyway), it will save you time preparing the tree.

If you have a very large tree (perhaps thousands of items), a more drastic way to cut it down is to only test a single section of it.

Suppose again that you’re testing a large corporate site like Sony or Motorola, with many different business divisions, or perhaps a broad government agency with many different departments.

• diagram or ss of huge tree

Do you test the entire tree, or just part of it?

Let’s take the easy case first. If you want to test several parts of the big tree, you’re going to create tasks that use those sections, so you’ll clearly need to keep all the sections, at least down to a certain level. If you want to cut down the tree, you’ll need to prune levels across the board or selectively, as described above.

If you’re really just testing a single section of the big tree, it may still be a good idea to include all the top-level sections anyway, to see if participants can find their content amid the “clutter” of other divisions/departments. In this case, you can safely chop off the sections you’re not interested in at level 2 or 3, because you don’t expect much traffic there (hopefully none at all). Note: don’t cut those sections off at level 1, because participants will quickly figure out that those are just stubs and not where the real action is supposed to be.

In the end, you should only test a subtree by itself if:

• You don’t intend to test content in other parts of the big tree, and
  
  
• You’re very sure that participants would choose your subtree every time for each of the tasks you give them, or that they would go to your subtree directly in the real site, bypassing the parent site entirely.
  
  

An example makes this clearer. Suppose we’re testing the Shimano website. Shimano is best known for cycling products (pedals, gears, etc.), but they also make equipment for fishing and rowing.

Here is their top-level navigation:

If you were testing pure cycling tasks (look for bicycle shoes, check a warranty, read a product FAQ), you could just test the “Cycling” subtree (with “Shimano Cycling” as the root of the tree), because you’d be 99% sure that no one would choose the other sections shown here – they clearly have nothing to do with cycling.

If, however, one of your tasks involved finding the story of how Shimano got started in the bike industry, now it’s not so cut and dried. You could imagine some participants going to the “Corporate” section to find the company’s history. In that case, you might be best to include these top-level sections, include the full Cycling section (obviously), most or all of the Corporate section, and the top level or two of the Fishing and Rowing sections.

• diagram of selectively pruned Shimano tree

If you were writing tasks that involved both cycling and fishing, then clearly you would have include all these top-level sections, but you could safely cut the Rowing section short. The bigger problem in this case would be that you’re asking each participant to try both cycling and fishing tasks, and a cyclist (for example) may not know anything about fishing, so their fishing results are going to be suspect. For more on setting different tasks for different user groups, see Writing Tasks - ~.

Next: Which headings to include/exclude?