Checking for technical problems

Running a pilot also helps us spot technical problems with the study. By this, we mean issues that are caused by the supporting hardware (computers, networks, etc.) and software (operating systems, browsers, etc.), not by the test content.

Here are a few technical gotchas to be wary of:

• Spam blockers (for email invitations)
  If we’re emailing invitations, be careful that they are not triggering the spam blockers built into the recipients’ email systems. While we can never be sure what will trigger a spam block, the fact that we’re asking our invitees to click a link in the email (and are offering some kind of reward to participate) makes this something to check with popular email systems such as Gmail, Yahoo Mail, Outlook.com, and so on. We may want to set up accounts with the most popular email providers just so we can send to our own addresses as a spam test.

• Computers vs. phones/tablets
  Most people may be doing the study on a conventional computer, but some will try doing it on a tablet or even on a smartphone.
  Make sure that the testing tool works on these devices. If it doesn’t support a particular platform, we should mention that in the invitation (or web-ad explanation page).

• Old (or odd) web browsers
  Most testing tools will work in any up-to-date browser that supports web standards (Chrome, Firefox, Internet Explorer, Edge, etc.).
  Old versions of browsers (Internet Explorer in particular) are still commonly found in large organizations, so if we’re targeting these users, we need to check with the tool vendor to see if they support them (or, failing that, if the tool warns the user to try a newer browser).
  Mobile browsers seem to vary more than their desktop counterparts when it comes to handling online tests, so we try the test on a handful of the most popular browsers on iOS and Android.

• Firewalls
  Most home firewalls (usually built into routers or anti-virus software) are unlikely to interfere with a tree test, but some corporate firewalls might. Certain large organizations have very strict firewalls that have (in our experience) played merry hell with our studies.
  If we're running a study that targets users in specific large organizations, and we get reports from those users that they can’t do the study, we check with the tool vendor to see if they can help solve the problem. If they can’t (and these problems can be hard to pinpoint), we may need to ask users in that organization to try the study from a different location (e.g. from home).

Next: Revising the test