For instance, in the table below, a GLM (ANOVA) Tukey posthoc bootstrapped 1000 times gives me a confidence interval like this. The columns are mean difference, bias, standard error, and the bounds of the CI.
http://i.stack.imgur.com/kPPDj.png
However, without a bootstrap with a Tukey multiple comparison, I get this. The columns are mean difference, standard error, significance, and the CI as the last two columns.
http://i.stack.imgur.com/RR3CF.png
The assumptions are all met (normally distributed and homogeneous variances), and there is an n = 10 per group, for a total of n = 60.
Which one do I believe?
(I used SPSS to perform the analysis and bootstrap)
Straightforward significance test estimates how likely is the set of data you obtained, assuming that the null-hypothesis is correct. When you do bootstrapping, the above question of likelihood is already decided upon: you assume that the next 1000 random samples will have exactly the same characteristics as the first one--i.e. you assume the above likelihood is 100% (irrespective of the null)--in order to establish more precisely what those characteristics of the first sample are.
Now, what you believe is always a function of your assumptions. This case is no different.
I'm not familiar with this type of output but it looks like you have pairwise comparisons for five treatments compared to A17S. Is that all you are interested in. What is the overall significance of your ANOVA (to provide information on whether it is then valid to check the pairwise comparisons.
Are you really only interested in identifying treatments that are statistically different compared to A17S? If so then the bootstrapping shows all five treatments as being different from A17S. Without bootstrapping, the analysis still indicates that A10T and FVBT are statistically significantly different from A17S (their confidence intervals do not include 0.
Choosing between the two methods is a function of your assumptions as Alex Chadyuk has said. I would go with the more conservative approach (in this case non-bootstrapping). But you can check residuals of the model to see if the analysis assumptions hold well (incl. normality and homogeneity of residuals)
It is interesting issue because when I was using bootstrapping in mediation, p values did not change, but standard errors slightly went up... so it did not affect significance at all.
My earlier answer was more on the practical interpretation of the analysis and making sure the right thing is being analysed rather than much on the bootstrapping and non-bootstrapping comparison..
I think the issue in this case appears to be that the A17T difference has a higher standard error than the other four treatments in the boot strapping. Thus the boot strapping is considering this to be a true reflection of the "population". Pradeep Harish said that the variances are homogeneous. Is that a reasonable assumption? Does the A17T result actually have a different variance? I suggest that in this case, the assumption holds okay (similar se's in the GLM output).
I would look at the individual results for A17T - is there an outlier/error etc in that treatment?
The sample size of 10 per treatment is sufficient for bootstrapping but would we reasonably assume that A17T results are more variable than the other treatments?
So that comes back to what your assumptions are.
I am not sure how important this is but bootstrapping in mediation is based on the assumption that sample is representative to the population it is pertaining to.
Yes that's relevant. For a small sample size this is a problem but above 8 would expect to be fine. Maybe there is more jeopardy in five samples that an issue shows up. Or was the sample taken in the bootstrapping. For whatever reason the bootstrapping is giving a result that I wouldn't rely on.
- Badges
- Science topic
- Similar topics
- Social Psychology
- Attitude