In the article “Determination of the relationship between water use efficiency, carbon isotope discrimination and proline in sunflower genotypes under drought stress“ by Önar Canavar et al. (Australian Journal of Crop Science, 02/2014; 8(2):232-242) we have the problem of neglecting the multiplicity again (see my remarks on the former article of I. Blinova and F.-M. Chmielewski).
If I have understood the very short description of the statistical analysis on page 240 correctly, the authors used “Fisher’s least SIGNIFICANT difference (LSD) method” (they call this test “Fisher’s least SQUARES differences method, but I guess that they used the normal LSD) for detecting significant differences between the means. In contrary to the “least significant difference-Bonferroni test”, this test does not take into account the multiplicity (see, e.g., http://de.wikipedia.org/wiki/Post-hoc-Test or http://www.graphpad.com/support/faqid/189/). Probably, this problem is not too serious because the degrees of freedom (df=k-1) between the k groups are small (e.g., df=3 for the four C=Cultivars). However the number of comparisons (=k(k-1)/2=6) exceeds df=3, and in this case the Bonferroni-modification is strongly recommended (see https://umdrive.memphis.edu/yxu/public/SPSS%20ANOVA.pdf).
More serious is that they made 18 „independent“ test (for 18 different dependent variables) in their Table 2 and did not reduce the critical individual p-values. Hence the probability of a Type I Error increases considerably. Because they did not list the actual p-values, there is no chance to determine the true adjusted p-values.
Certainly, there are any significant differences in Table 2 because many of the factors have “significant” influence (because of the non-adjusted p-values, possibly not “true significant”). However one does not know (not even probabilistic) which of these factors are “significant by accident only”.
Similar considerations apply to Table 3 (here we have multiple tests for the significance of correlation coefficients).
A very simple remedy for such a multiple testing problem would be to report “exact “ p-values rather than inequalities (p
Hi Klaus, my personal opinion is that the exact p-values should be disclosed until the p-value is below .001, where little software reports a p-value. Further, authors should refrain from conducting unnecessary tests, such as the common practice of displaying a correlation matrix loaded with p-values. Furthermore, they should certainly avoid analysis that is guided by cherry picking significant results (such as from a big correlation matrix) to move on to a next phase, but be guided by theory and conducting a limited number of tests. Worst, of course, is something like stepwise regression that might involved an untold number of tests even though they are somewhat hidden from the analyst. The authors should disclose all the tests they have conducted. Personally, I prefer to just lay out the exact p-values and leave it to the readers to decide how the multiple tests affect decisions about the findings, because I don't believe there is any magic number for probability adjusted or unadjusted. As a reviewer, I did just point out the problem of multiple tests to authors who had to test a slew of hypothesis by swapping variables out of a model one at a time. Bob
Hi Robert, thank you for your comprehensive answer. I agree totally. The only problem is whether the "normal" reader is able to decide what the set of p-values in a multiple-test-problem means.
For those people who don't know how to decide precisely, I recommend the review "Multiple Hypotheses Testing" by J. P. Shaffer (nnu. Rev. Psychol. 1995. 46:561-84). From the tests describe in this article I prefer Holm's Sequentially-Rejective Bonferroni Method.
Hi Klaus, I do tend to agree with you that little attention is given to that in training, which is also why there is so much testing of things that are of no theoretical importance, such as correlation matrices (often Table 2 or Table 1) and crazy stuff like stepwise regression. Even if all readers don't know what to make of actual p-values and seeing al the relevant tests, it would be nice to see better practice in the presentation of results and the reduction in unnecessary testing. I am actually not a very strong advocate for strict correction, because I sort of believe in the counter argument of "lifetime Bonferroni" in that one could argue, the correction might apply to all the tests I do as an analyst or all the tests done on a given data set not just done in a specific set of analyses by a specific researcher working on that data set, so at this point maybe one would have get p-value of something like .0000001 to claim anything from the Framingham Heart Study or for a specific type of association, such as diet an arterial blockage in the Framingham heart study, which has probably been looked at thousands of times. Given all the analyses on that data set, thousands of them should be significant by chance alone, but there have been hundreds of researchers working independently over decades on those data. Same applies to me. How many thousands of statistical tests have I run in my life? Admittedly they are on many data sets, many of them because they were null, left aside while we moved on to look for something more interesting and publishable. That's actually the origin of my lifetime Bonferroni idea, but I think it applies just the same, maybe more so, to data sets. Bob
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics