...

Whenever you we recruit participants, you we are looking to get a representative sample of your our actual (or desired) users.

You We may even go to a fair bit of effort to get (or exclude) specific types of users, by querying customer databases, posting invitations to specific user forums, and so on.

In the end, though, all recruiting is imperfect; you’ll we’ll miss some users you we were hoping to get, and you’ll we’ll get some that you we were hoping to miss.

It is important, nonetheless, to try to identify any selection bias in your our recruiting, so you we can take that into account when you we analyze your our results, or when you we do your our next study.

From Wikipedia’s article:

Selection bias is the selection of individuals, groups or data for analysis in such a way that proper randomization is not achieved, thereby ensuring that the sample obtained is not representative of the population intended to be analyzed.

...

In other words, certain recruitment methods may yield a skewed selection of participants, rather than the representative sample that you we normally want.

For example, suppose that you we only use a web ad on your our site to recruit for your our study. Only people who visit your our site in the next few days (the duration of your our recruitment) will see the ad. This means that:

• You We are ignoring customers who don’t use your our website. (For many businesses, such as banks, this may be a big chunk of customers.)
  
  
• You We are more likely to get people who visit your our site frequently (say, several times a week), but less likely to get people who use your our site once a month (e.g. to check their bill).
  
  
  

...

• Web ads only get web users
  If you we only use web ads to recruit users, then (by definition) you’re we’re only getting those users who visit your our website. While this is OK for many studies, it does ignore those customers who use other channels instead of the website. If you we need offline users too, you’ll we’ll need to find another way to recruit them.
  
  
• Customer email lists only get existing customers
  If you we only use a customer list to email invitations, you we are missing prospective customers (a potentially valuable audience) and ex-customers (who are often good sources of honest feedback).
  
  
• ~more sources of bias?
  
  
  

You We may decide that a given selection bias is acceptable; in our example above, you we may only want customers who visit your our website, so this implicit selection actually serves as a useful screening mechanism.

However, it’s important that you we consider what kind of selection bias each recruiting method adds to your our study. Then, you we can either:

• Try to reduce that bias, and/or
  
  
• Acknowledge the bias and take it into account when analyzing your our results and presenting your the findings.
  
  

The most common way to reduce selection bias is to use several different types of recruitment. For example, instead of just running a web ad (which only yields site visitors), you we could use customer lists to reach those customers who don’t use the website.

Another (generally less effective) way to reduce bias is to modify the single recruitment method you we are using. If you we only use a web ad, for example, you we could post that ad on several different websites. You We will still only get website visitors, but some of them will be people who have not visited your our website.

While you we will probably never eliminate all bias from your our studies completely, these steps should help minimize it so you we can be reasonably confident in your our results.

...

Next: Coordinating audiences and channels

...