Page Comparison

...

Very few people will wait until the test is done before peeking at the results. We generally take a look once we’ve hit 10 20 or 20 30 respondents, just to see how things are going.

Often, they’re going a lot worse than we expected. For example, we may have been hoping for a success rate in the 60s or 70s, but the early results often come in around at 40 or even less.

•  example of initially low scores
•  ~Meridian original scores - tree 0 = 36%, tree 1 = 40%, tree 2 = 35%
•  ~Meridian corrected scores - tree 0 = 46%, tree 1 = 43%, tree 2 = 47%

...

Don’t get depressed, though. It’s likely that the data will need some cleaning up before it will accurately show what really happened out thereFor example, we ran three tree tests for an electricity company. We expected a low score for the existing site tree, but we also got low scores for the new trees we had designed:

While this was lower than we expected across the board, we told ourselves not to panic. From experience, we knew that the original data needs cleaning up before it accurately shows what happened.

There are a few common reasons for this:

...

• Choosing the same item at each level (often the first or last item)
  
  
• Going down the same path for every task
  
  
• Choosing nonsense paths for every task
  Careful with this one, because what you consider a “nonsense path” might have made sense to them. Only suspect those who do this for a large number of tasks.
  

•  example

...

If we find a session with a lot of this kind of garbage, we delete it, and if we are doing a prize draw for this study, we remove that participant from the draw. This is not a behavior to encourage.

...

And remember to download another local copy of the revised results, for safekeeping.

For the power-company study we described above, when we recalculated the scores after adding some missed correct answers, our results changed substantially (although they were still lower than we would have liked):

...

Next: Sharing the data