I have two sets of surveys, one of which is from a random probability sample. The other, in which participants self-selected has unusually high scores across the measures. Is there a way to take the former survey as a framework to correct for the latter? All measures were on a 5-point Likert scale.
Hi Sumiran,
There is no ideal solution here. This first thing that I would do is compare the characteristics of your two samples (i.e., demographic characteristics). If there are variables that are very different you can control for these in your analyses. You can also look at the distributions for both samples. We know that your self-selected sample will be negatively skewed. I'd look at the standard deviations for both samples as well. If your random sample is large your observed scores should approximate the population. This may give some insights into possible controls as well. Here is an article on selection bias that might be of some help:
https://file.scirp.org/pdf/OJEpi_2015070913284831.pdf
Let me know how things turn out. Best wishes,
Nick
Nick makes good points, if you must combine the surveys. But scope permitting, I would de-couple them, ignore the quantitative data from the more [we think] biased one and only use the qualitative data, if any, for color commentary. Still it's worth looking at where the higher ratings weren't so high, where there was more divergence, etc. and use those insights directionally in your analysis.
Of course, the random sample one may still differ from the population under study due to selection biases of its own, so it behooves you to look at the demography and other secondary data (purchases, visits, clicks, whatever you have) around the study population and sample groups anyway, to see what caveats you may need to address.
- Badges
- Science topic
- Similar topics
- Correction