I found some authors using Bonferroni approach while deciding the significance level of Pearson's correlation in which the relationship between two variables is said to be significant when the p value is less than .05 divided by the number of pair of variables. For example, if we are examining the Pearson correlations between five variables (10 pairs), the corrected p value is .001.
Hi Arjun. The Bonferroni correction is one simple, widely used solution for correcting issues related to multiple comparisons. The webcomic XKCD illustrates the real world issue when significance is not corrected when multiple tests are conducted.
One issue with Bonferroni correction is that it can lead to Type II errors. This is because the power to detect an hypothesis while controlling the familywise error-rate is greatly reduced when the number of tests increases. E.g., instead of running 5 tests, you ran 1000, the minimum p-value become < .00005. You will now likely fail to reject the null hypothesis H0, even if there is a difference H1=true.
An alternate form of correction has been proposed by Benjamini & Hochberg, 1995, which addresses this issue, has become a very popular alternative to Bonferonni correction.
http://xkcd.com/882/
Article Controlling The False Discovery Rate - A Practical And Power...
Article The Control of the False Discovery Rate in Multiple Testing ...
Hi Arjun. The Bonferroni correction is one simple, widely used solution for correcting issues related to multiple comparisons. The webcomic XKCD illustrates the real world issue when significance is not corrected when multiple tests are conducted.
One issue with Bonferroni correction is that it can lead to Type II errors. This is because the power to detect an hypothesis while controlling the familywise error-rate is greatly reduced when the number of tests increases. E.g., instead of running 5 tests, you ran 1000, the minimum p-value become < .00005. You will now likely fail to reject the null hypothesis H0, even if there is a difference H1=true.
An alternate form of correction has been proposed by Benjamini & Hochberg, 1995, which addresses this issue, has become a very popular alternative to Bonferonni correction.
http://xkcd.com/882/
Article Controlling The False Discovery Rate - A Practical And Power...
Article The Control of the False Discovery Rate in Multiple Testing ...
Interesting answers here. I am wondering why nobody brings up Tukey's HSDs as an alternative to Bonferroni. https://en.wikipedia.org/wiki/Tukey's_range_test
In particular, Paul, you don't seem to use it. Why not?
You've already got some good answers here. My personal feeling has always been that the Bonferroni correction is far too stringent unless you are dealing with a small number of significance tests. As Dr. Yarnold says, you'll get lots of Type II errors - missing stuff that's really there.
If you are running ANOVAs, then you have lots of other options. Tukey tests (mentioned by the last commenter) control for Type I errors across all possible pairwise comparisons. Scheffe tests are even more conservative, controlling for all possible between-group comparisons. Other procedures tend to be a bit looser. But in most cases, they'll yield similar results.
If you are working from a correlation matrix, I'm not sure what options you have. In my own work, I've gotten away with a straightforward discussion of the issue. If, for instance, I am presenting a table with 20 correlations that are relevant to the study, then I need to acknowledge that there is a very good chance that random data would have yielded at least one "significant" result (p
Hi Steven and Arjun,
I am curious, Arjun, in what you have decided to use. I am in a similar situation where I am running multiple Pearson r's (>700)/groups. I have FDR-corrected the p-values which statistically is the valid thing to do. However, I also have two control groups in addition to an experimental group and when I run the same analysis no uncorrected significant correlations are found, suggesting Type I error with this type of data/comparisons is not an issue. The correlations found are specific in the experimental group only. This to me means that corrections are not needed. Can anyone chime in if this reasoning is fair for the data at hand?
Hi, Eric,
This is an interesting situation. The correlation matrices seem to be different for experiment and control. You could test that with Mantels test. However, the problem of multiple comparison exists irrespective of the data you use. It is a problem of the statistical methodology.
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis