There are several points that should be addressed:

1) What do you mean with "I decided to draw two random samples from the non-participants list"? Why two samples?

2) Bootstrapping does not mean to draw a random sample from your sample and look at it, but to caclulate the relevant teststatistic, here your differences between the groups, many times and inspect the distribution of this teststatistic.

3) There is not THE bootstrapping procedure, it is a name for a couple of slightly different apporaches, like percentile bootstrap, bias corrected and accelerated bootstrap... have a look at robust estimation techniques described by Rand Wilcox in numerous publications.

4) Bootstrapping is most often used if you expect that some assumptions are not met, so how does your data look like? Is it expected that your sample is for example highly skewed?

5) Another issue is your sample size: with such huge samples I would expect that even very small differences between the groups will be statistically significant. Therefore it may be questioned if a inferential test will answer your question. Do you really expect very small differences between the groups and are these very small differences relevant? If your smallest effect size of interest (SESOI) is larger than what your sample size is able to detect, you may end up with a significant result, but which has no practical relevance. Therefore, assess, report and interpret the effect sizes, not only the p-values or confidence intervals.

There are samples and SAMPLES. The bootstrap algorithm should guarantee the comparison is statistically correct. Best wishes, David Booth

Thank you David Eugene Booth, did you mean bootstrapping the non-participant data and then comparing it to the participant group? Or must I also bootstrap the participant data?

Would you consider the following approach reasonable:

For each group (participants vs non-participants)

1) resample with replacement (bootstrapping)

2) repeat 5000 times to build a distribution of means

3) Carry out a t-test to assess the difference between those two distributions

If not, how would you recommend me to go ahead?

Thanks & regards,

MS

There are several points that should be addressed:

1) What do you mean with "I decided to draw two random samples from the non-participants list"? Why two samples?

2) Bootstrapping does not mean to draw a random sample from your sample and look at it, but to caclulate the relevant teststatistic, here your differences between the groups, many times and inspect the distribution of this teststatistic.

3) There is not THE bootstrapping procedure, it is a name for a couple of slightly different apporaches, like percentile bootstrap, bias corrected and accelerated bootstrap... have a look at robust estimation techniques described by Rand Wilcox in numerous publications.

4) Bootstrapping is most often used if you expect that some assumptions are not met, so how does your data look like? Is it expected that your sample is for example highly skewed?

5) Another issue is your sample size: with such huge samples I would expect that even very small differences between the groups will be statistically significant. Therefore it may be questioned if a inferential test will answer your question. Do you really expect very small differences between the groups and are these very small differences relevant? If your smallest effect size of interest (SESOI) is larger than what your sample size is able to detect, you may end up with a significant result, but which has no practical relevance. Therefore, assess, report and interpret the effect sizes, not only the p-values or confidence intervals.

In addition,. To Dr Dueessing's excellent reply you might look at what is available at z- library. Several fine references are available there(google z-library). Best wishes, David Booth

Thank you Rainer Duesing for your reply.

Correction, s1= ~2000 and s2= ~12000

I ran welch's test, given unequal sample sizes, for comparing the group means. However, I was asked if bootstrapping was not a better option. Since I had not used bootstrapping before, I was only relying on information available online.

I will check for the effect size, which I had not before.

Mari Szommer wrote:

I now have a list of 2000 participants and 13000 non-participants. I decided to draw two random samples from the non-participants list.

I had thought of comparing the three groups using ANOVA to test for differences among means.

Sorry if I missed something, but means on what variable(s)? What information do you have about the 13000 (or 12000?) non-respondents?

Also, Rainer Duesing makes an excellent point about tiny irrelevant differences being statistically significant when you have such large sample sizes.

HTH.

Hi @Bruce Weaver,

I have variables like no. of orders, average basket value ($), share of private labels in the no. of orders et cetera.

Yes, I get the point that @Rainer Duesing is making and will consider the effect size

With your sample size i doubt that the results would change if you bootstrap.

You would need to compute the mean of the resamples for every group then compare them

I don't think bootstrap can really help in inference. The amount of information contained in a sample (or dataset) is "fixed", it cannot be increased by bootstrap or any manipulation methods. The use of bootstrap is merely to make the analysis sounds "sophisticated". Besides, if you have a large sample, the results with and without bootstrap would be about the same.

Stefano Nembrini ,do you mean that if I resample, say 5000 times, I calculate (for both groups) the mean of each (x1, x2, ..., x500) sample and compare them?

Yes, so when you compute the difference you would still get a distribution with 5000 points, then you should plot it and compute mean/median/ confidence interval, whatever you need to summarize the results

in R:

difference=replicate(5000,sample(group1,replace=T)-sample(group2,replace=T))

Thank you for the reply, Stefano Nembrini Would you know how to do that in SAS?