| Table of Contents | ||
|---|---|---|
|
...
The great thing about online testing tools is that, not only are the results compiled and partially analyzed for us, they are usually available instantly, even while the test is still “live”.
...
For example, we ran three tree tests for an electricity company. We expected a low score for the existing site tree, but we also got low scores for the new trees we had designed:
...
| Existing tree | New tree 1 | New tree 2 | |
|---|---|---|---|
| Uncorrected score | 36% | 40% | 35% |
...
This is also useful if the tool we’re using goes offline, either temporarily or (eventually) permanently. In the unlikely case that we need to go back and look at the raw numbers a year or two from now, we’ll want to have our own offline copy of the data.
Removing garbage sessions
...
We can normally weed out the latter, and reduce some of the former.
Going too fast
The first thing we look at is sessions that were done too quickly. Most tools track the total time taken for each participant, and this becomes a good way to weed out garbage data.
...
If we find a session with a lot of this kind of garbage, we delete it, and if we are doing a prize draw for this study, we remove that participant from the draw. This is not a behavior to encourage.
...
| Note |
|---|
Some tools let us “exclude” sessions from the analysis. This is what we think of as a “soft” deletion; the session is removed from the analysis, but it’s not actually deleted, so we can get it back later if we change our minds. If our tool offers exclusions, we recommend using them to clean up the data instead of actually deleting the data outright. |
Skipping too many tasks
Another way that some participants speed through a test is by skipping tasks.
...
For the remaining few, we review their click paths (like we did above for those who went too fast), look for the same indicators, and delete/exclude those that look guilty.
Being wrong way too often
...
Again, we review their click paths as described above, look for the same indicators, and mete out our justice accordingly.
Updating correct answers
One more way that we “clean up” the data is changing the correct answers for our tasks.
...
For the power-company study we described above, when we recalculated the scores after adding some missed correct answers, our results changed substantially (although they were still lower than we would have liked):
| Existing tree | New tree 1 | New tree 2 | ||
|---|---|---|---|---|
| Uncorrected score | 36% | 40% | 35% | |
| Corrected score | 46% | 43% | 47% |
...
...
Next: Sharing the data