I have 2 groups, Group A and B. Group A is experimental, B is control. I measured performance on a skills test for both groups, which is my core interest. However, I want to look at how they performed on other written tests, (same cohort of participants) and see if the intervention affected these written tests. By doing so, do i need to correct for multiple testing using Bonferroni correction, alpha/number of performance tests as my criterion? Thanks!
Hello Poh,
Yes, it would be a good idea to impose some sort of aggregate risk cap on the set of tests you intend to run. R offers a very simple command (p.adjust) for doing this using methods that are superior to the Bonferroni, such as the Benjamini-Hochberg.
See this link: https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/p.adjust
Good luck with your work.
In addition to Prof Morse suggestion you might want to download this attached reprint from ETH. All methods are available in R package bioconductor. Best wishes David Booth
My view is that whether or not you apply a correction for multiple tests depends on your objectives. To some extent, it depends on whether you want to prioritize finding potential "significant" findings, or being conservative against identifying false "significant" findings. ... Often we apply corrections for multiple tests in cases where we have a "family" of hypotheses. The most common case is when there are pairwise comparisons of e.g. group means. Do you consider your hypotheses a "family" ? ... For me, if I had multiple dependent variables (if that's what you're asking about), I wouldn't correct for tests across multiple DV's. ... To some extent, you have to consider what it is you're trying to accomplish by making the correction. Let's say I conducted an experiment where I was testing a dietary supplement and measuring three potential outcomes: sleep, happiness, and exercise. If I found ---- for sleep ---- a significant effect, with a reasonable effect size, that had practical importance, what do I gain by adjusting the p-value for the fact that I had three DV's ? Maybe that is the right path, but it's worth asking yourself what difference that makes in your conclusions. ... It's important to not take the correction-for-multiple-tests idea too far. If we conduct 1000 agronomy experiments in the U.S. per year do we have to have to adjust the p-values for all of them ? No. But but we also know that there's some chance of significant results from any of them coming from chance. It could be the case that birds happen to poop on some plots and not others, and that made the corn grow better coincidentally with some treatment we were testing. That's just life. Would adjusting the p-values for these experiments have helped us in this case ? ... Overall, don't place too much importance on arbitrary p-value cut-offs (e.g. 0.05). Take into consideration effect sizes and practical implications of your results. With this perspective, it may be that adjusting p-values for multiple tests takes on less importance. ...
- Badges
- Science topic
- Similar topics
- Correction