Hi Klaus, my personal opinion is that the exact p-values should be disclosed until the p-value is below .001, where little software reports a p-value. Further, authors should refrain from conducting unnecessary tests, such as the common practice of displaying a correlation matrix loaded with p-values. Furthermore, they should certainly avoid analysis that is guided by cherry picking significant results (such as from a big correlation matrix) to move on to a next phase, but be guided by theory and conducting a limited number of tests. Worst, of course, is something like stepwise regression that might involved an untold number of tests even though they are somewhat hidden from the analyst. The authors should disclose all the tests they have conducted. Personally, I prefer to just lay out the exact p-values and leave it to the readers to decide how the multiple tests affect decisions about the findings, because I don't believe there is any magic number for probability adjusted or unadjusted. As a reviewer, I did just point out the problem of multiple tests to authors who had to test a slew of hypothesis by swapping variables out of a model one at a time. Bob

Hi Robert, thank you for your comprehensive answer. I agree totally. The only problem is whether the "normal" reader is able to decide what the set of p-values in a multiple-test-problem means.

For those people who don't know how to decide precisely, I recommend the review "Multiple Hypotheses Testing" by J. P. Shaffer (nnu. Rev. Psychol. 1995. 46:561-84). From the tests describe in this article I prefer Holm's Sequentially-Rejective Bonferroni Method.

Hi Klaus, I do tend to agree with you that little attention is given to that in training, which is also why there is so much testing of things that are of no theoretical importance, such as correlation matrices (often Table 2 or Table 1) and crazy stuff like stepwise regression. Even if all readers don't know what to make of actual p-values and seeing al the relevant tests, it would be nice to see better practice in the presentation of results and the reduction in unnecessary testing. I am actually not a very strong advocate for strict correction, because I sort of believe in the counter argument of "lifetime Bonferroni" in that one could argue, the correction might apply to all the tests I do as an analyst or all the tests done on a given data set not just done in a specific set of analyses by a specific researcher working on that data set, so at this point maybe one would have get p-value of something like .0000001 to claim anything from the Framingham Heart Study or for a specific type of association, such as diet an arterial blockage in the Framingham heart study, which has probably been looked at thousands of times. Given all the analyses on that data set, thousands of them should be significant by chance alone, but there have been hundreds of researchers working independently over decades on those data. Same applies to me. How many thousands of statistical tests have I run in my life? Admittedly they are on many data sets, many of them because they were null, left aside while we moved on to look for something more interesting and publishable. That's actually the origin of my lifetime Bonferroni idea, but I think it applies just the same, maybe more so, to data sets. Bob

Hi Bob, your argument with the "lifetime Bonferroni" is a nice idea. But I believe (like you) that this requirement is too strong. It would mean that the probability for a specific scientist to make one (or more) false rejection in his whole life is only “alpha” (e.g, 0.05) on average. Or, in other words, only every 20th scientist would make a false rejection in all his life. This would almost convert the stochastic behaviour of statistics to determinism. (-:

More realistic is the "study-wide Bonferroni". But, as you said, the overall rejection rate depends on many things (on statistical dependence of the data and tests an so on).

Personally, I would be satisfied if I could trust that the false rejection probability (of the H0) would be less than 0.05 (or 0.01) PER TABLE (i.e., less than every 20th table would show wrong significances). But I often see tables with 18 (and much more!) test results and with alpha=0.05 (or even 0.10) without specification of any p-value. That is what I deprecate. In this case, the probability to see one or more significant tests (i.e., stars = *) is more than 60% PER TABLE (=1-(1-0.05)^18). Then on average more than every second table shows significant “differences” even if there are no differences at all! Klaus