I used adonis to test the difference between groups/categories based on the 16s high-throughput sequencing data. I got a very significant result (p = 0.001), but the R2 is very low (< 0.1). I know this means this model only explains less than 10 percent of my data. From the PCoA plot, there were some overlaps between the groups. Do these overlaps contribute to the low R2? I'm wondering if I can still say that the difference between groups is significant?
I also did ANOSIM and the results are similar. The statistic value is very low (~0.1) but p-value is 0.001. Is this the same problem I got when doing adonis?
Some discussion groups said that there is no relationship between R2 and p-value. Doesn't it matter however small the R2 is?
The conclusions based on p-values need be adhered to. R2 analyses have their own shortcomings and are generally avoided.
Haitao -
R-square can be a bit misleading, but p-values are very often far more so. A p-value will be made smaller with a larger sample size, regardless of effect size (the difference between "truth" and any hypothesis). It is common that people set a "significance" level of 0.05 regardless of sample size and what effect size might be enough to matter to you. That is perhaps the worst statistical practice that has ever been perpetrated upon the research communities. If "Big Data" had been practical decades ago, people may have fallen into the equally bad habit of using, say, 0.0001 everywhere. (A type II error analysis matters.)
Perhaps the following may be of interest:
https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS
I do not know your application, but I imagine it possible to withhold some of your data for testing. Then you could predict for the withheld dependent variable data, and see how well your regression performs in those cases. (You could research "model validation.") You might look at estimated variances of prediction errors (which though only designed to estimate variance in predicted y, is also impacted by bias due to the way sigma is estimated). The square root of this in SAS PROC REG, for example, is STDI.
Best wishes - Jim
Article Practical Interpretation of Hypothesis Tests - letter to the...
Haitao,
low r-square means that you have a lot of variation that is unexplained in your model. The p-value means that despite the relative mediocrity of the model, the differences you observe still reject the null hypothesis. You should try to account for these unexplained variations and see whether your sampling design is e.g. nested, or if other variables (environmental) could be added. A first step to do that is to plot residuals vs. variables not in the model (plot id, altitude, block etc...). Good luck.
Were you able to resolve this Haitao Wang? How did you go about it? I am having a similar problem.
- Badges
- Science method