Running a pilot also helps you spot technical problems with your study. By this, we mean issues that are caused by the supporting hardware (computers, networks, etc.) and software (operating systems, browsers, etc.), not by your test content.
Here are a few technical gotchas to be wary of:
- Spam blockers (for email invitations)
If you’re emailing invitations, be careful that they are not triggering the spam blockers built into the recipients’ email systems. While you can never be sure what will trigger a spam block, the fact that you’re asking your invitees to click a link in the email (and are offering some kind of reward to participate) makes this something to check with popular email systems such as Gmail, Yahoo Mail, Outlook.com, and so on. You may want to set up accounts with the most popular email providers just so you can send to your own addresses as a spam test. - Computers vs. phones/tablets
Most people may be doing your study on a conventional computer, but some will try doing it on a tablet or even on a smartphone.
Make sure that your testing tool works on these devices. If it doesn’t support a particular platform, make sure you mention that in your invitation (or web-ad explanation page). - Old (or odd) web browsers
Most testing tools will work in any modern browser that supports web standards (Chrome, Firefox, Opera, recent versions of Internet Explorer, etc.).
Old versions of browsers (notably Internet Explorer) are still commonly found in large organizations, so if you’re targeting these users, check with the tool vendor to see if they support them (or, failing that, if the tool warns the user to try a newer browser).
Mobile browsers seem to vary more than their desktop counterparts when it comes to handling online tests. You may want to try your test on a handful of the most popular browsers on iOS, Android, and so on. - Firewalls
Most home firewalls (usually built into routers or anti-virus software) are unlikely to interfere with your tree test, but some corporate firewalls might. Certain large organizations have very strict firewalls that have (in our experience) played merry hell with our studies.
If you are running a study that targets users in specific large organizations, and you get reports from those users that they can’t do your study, check with your tool vendor to see if they can help solve the problem. If they can’t (and these problems can be hard to pinpoint), you may need to ask users in that organization to try the study from a different location (e.g. from home).
Next: Revising the test