...

Whenever we recruit participants, we are looking to get a representative sample of our actual (or desired) users.

...

It is important, nonetheless, to try to identify any selection bias in our recruiting, so we can take that into account when we analyze our results, or when we do our next study.

From Wikipedia’s article:

Selection bias is the selection of individuals, groups or data for analysis in such a way that proper randomization is not achieved, thereby ensuring that the sample obtained is not representative of the population intended to be analyzed.

...

In other words, certain recruitment methods may yield a skewed selection of participants, rather than the representative sample that we normally want.

...

• We are ignoring customers who don’t use our website. (For many businesses, such as banks, this may be a big chunk of customers.)
  
  
• We are more likely to get people who visit our site frequently (say, several times a week), but less likely to get people who use our site once a month (e.g. to check their bill).
  
  
  

Here are some common causes of selection bias:

• Web ads only get web users.
  If we only use web ads to recruit users, then (by definition) we’re only getting those users who visit our website. While this is OK for many studies, it does ignore those customers who use other channels instead of the website. If we need offline users too, we’ll need to find another way to recruit them.
  
  
• Customer email lists only get existing customers.
  If we only use a customer list to email invitations, we are missing prospective customers (a potentially valuable audience) and ex-customers (who are often good sources of honest feedback).
  ~more sources of bias?.
  
  
• Daytime in-person studies often get lots of retired people and students.
  These two groups are easier to schedule during typical business hours, but may skew your user sample.
• Commericial customer panels may be skewed to certain demographics.
  We've used commercial panels that were heavily skewed to consumers rather than businesses, or had a larger-than-normal representation of lower socio-economic groups.
  This is usually easy to solve by being careful in our selection criteria, because panels typically offer lots of ways to select the participants you need.
  
  
  

We may decide that a given selection bias is acceptable; in our example above, we may only want customers who visit our website, so this implicit selection actually serves as a useful screening mechanism.

...

The most common way to reduce selection bias is to use several different types of recruitment. For example, instead of just running a web ad (which only yields site web visitors), we could also use customer lists to reach those customers who don’t use the website.

...

While we will never eliminate all bias from our studies, these steps should help minimize it so we can be reasonably confident in our results. 

...

Next: Coordinating audiences and channels