I have the following problem with statistics (imposed by a journal reviewer). I have to use Benjamini-Hochberg correction for false discovery rate. In our work we tested three separate hypotheses: (1) Bird abundance is dependent on habitat type, (2) bird number is higher in January than in December and (3) there is interaction between habitat type and month. In order to test these hypotheses I built generalized linear models (GLM) where all three hypotheses are tested in one model. However, I have 50 bird species to test these hypotheses on thus I built 50 GLMs. My question is: should I use P-values for calculation of Benjamini-Hochberg correction derived for a specific hypothesis of only (50 p-values for 50 species), or 150 values derived from general linear models for 50 species (3 hypotheses x 50 models)? Personally, I believe I should use 50 values for a specific hypothesis. Moreover, as I mentioned, all three hypotheses are tested in one statistical test.
Any idea or different point of view?
p.s. Let's omit the problem with very high number of tests and power.
Interesting question you pose. I tend to agree with the reviewer in this case simply because there are 150 hypotheses evaluated by the paper and so it makes sense to me to talk about the false discovery rate for all hypotheses in the paper rather than evaluate the false discovery rate for 50 at a time. Because the way BH correction works (per hypotheses) I don't think it should raise the bar too much; it is not at all like Bonferroni. Another thing to consider is changing over to a random effects model with species as a random effect but that is probably unappealing to you this late in the cycle.
Hi,
Thanks a lot for your suggestions. I am still not quite sure how to solve above problem.
My doubts result from the fact that when I look at the p-values, I see that the effect of habitat is significant ( p values lower than o.o5) in many species while other two hypotheses are much less supported. So, the p-values are not distributed randomly among these three hypotheses (with hypothesis one much more often supported). My second doubt is due to obvious observation that each species has separate life-history. Thus, there is no reason to assume that they behave as proverbial black and white balls in a basket generating significant results at chance (I agree with Frederico here).
Yes, I agree that if I tested the effect of habitat in general sense then I would use species as a random effect. However, this is paper on applied science, thus it is important from practical reasons to identify these species that respond positively to urban habitat and these responding negatively (in terms of abundance). Thus, we did separate GLM analyses for each species (actually one of referees insisted on this). We also performed ordination analysis, which partially solves the multiple testing.
My general observation is that whichever correction I use it corrects p-values to so low values that I am barely ever to able find significant results. So, by using FDR or Bonferroni correction I probably increase second type error.
Sven - Yes, I agree... but referee in a journal does not understand this :)
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics