I am running a survey in which I am gathering responses from 2 sources. I would like to understand how I can ensure validity of this approach, since respondents from these sources have different incentive structures.
(One source is from a panel service that offers each respondent $20 per response. The other is from a local organization, where each respondent is entered in a raffle to receive an Amazon gift card.)
Are there any standard statistical approaches to check for bias due to data coming from different sources like this? For example, would testing for significant differences in parameter estimates make this approach acceptable?
You can check difference in mean by application of Independent t Test. If the difference between the two sample is insignificant it means incentive plan does not affect the response pattern.
By the central limit theorem one can estimate that the mean of the data is close to the "true population" (unbiased) if they have a very large sample size. Since you are afraid that your incentives might be causing a resulting bias, I think that might be difficult to tell if there is a bias mathematically, since both data sets could be uniquely biased. Basically, when estimating population parameters there is some logical guess work on the part of the statistician where they make some decisions of what to consider and what to not. This can verge on unethical, so in this territory tread carefully and with plenty of motivations and considerations of problems with your ideas.
An example of this is hypothesis testing, where the statisticians first move is to decide an alpha (confidence interval = 1- alpha) that has a huge effect in whether they accept or reject the null hypothesis.
A great place to start with data would be to graph both as histograms, and get a feel for their distribution. Note that sometimes unexpected differences between your $20 set and Amazon gift card set could be positive motivators for you deciding on a better incentive for taking the survey. In this case, I do not think the data sets are fit for explicit comparison as if they are from the same test, given that a significant portion of the test is different. You may be encouraging a different group of people with different answers for each set. Therefore, I would consider them as different random variables (i.e. X = set motivated by Amazon, Y = set motivated by $20 per response), and not as different replications of the same test. A "two sample"equation might be what you're looking for, since this will find the difference between each of the samples means.
Going beyond what Jeevan Jyoti suggested, you can use a regression approach to include a dummy variable to test for mean differences (which is what the t-Test would do) and interaction terms to check for differences in slopes. This will allow you to test whether the same model fits both subsamples. (Alternatively, you could accomplish the same thing with a moderation model in a Structural Equation Model.)
Not really valid as they are not random samples. You can check the distribution of key demographic variables against a known population such as the census to see how your sample stacks up against the population you are sampling from. If it matches then you can have more trust in whatever statistical tests you do, but it is still problematic.
- Badges
- Science topic