Analyzing by task

The whole point of running a tree test is to find out which parts of the tree work well and (more urgently) which parts don’t, for the most common and critical activities that our users do on the site.

We largely determine this by:

examining the results of individual tasks
looking for patterns among tasks
~~comparing tasks between different trees we're testing

Task success rate

The most important metric for a task

Where they went

Inspecting high- and low-success tasks

What they clicked first

Did participants agree on where to start, or did they scatter?

Directness – where they backtracked

Directness scoring, backing up from incorrect vs. correct paths

Task speed - where they slowed down

Task times vs. click times, and why they took longer

Where they gave up

Looking for patterns in skipped tasks

Confidence

Were they sure of their answer, or doubtful?

Dealing with outliers

When to ignore strange results from a few participants

Finding patterns among tasks

The most common patterns to look for

Next: Analyzing by branches