Hello,
When running multiple analysis it is common to adjust the p value to reduce the chance of a type 1 error. However, I am unsure whether it is also appropriate to do this when running numerous normality tests prior to running the analysis itself. For example, I want to run 20 independent t-tests in a study (two groups). Prior to running these tests I want to check for normality using Kolmogorov–Smirnov tests for each dataset so that I know if I need to log transform the data (20 x 2 groups = 40 KS tests). It seems to me that running 40 x KS tests will massively increase the chance of a type one error, showing that some data is non-normal, when it is in fact normal. Is it appropriate to adjust the p-value (e.g bonferonni) for the normality test itself when I am running so many?
Many thanks,
Rob
Hi, Robert,
In such circumstances one may, a priori, partition the experimentwise p in accordance with the perceived importance of the tests in the context of the study. This Bonferroni methodology is known by various names including "a priori alpha splitting", "ensemble-adjusted" and "ordered". A discussion and example are provided in pp. 83-86 of Yarnold and Soltysik (2005), Optimal Data Analysis: A Guidebook with Software for Windows, APA Books.
No reason really, unless your normality test is theoretically driven (substantive Hypothesis).
Hope this helps
Matt
I side with Prof. Yarnold. A good introductory reference is attached. This is extremely common in Gene expression data. Take a look at the reference. HTH David Booth
Robert Browne
I would avoid comparing P-values to a pre-specified subjective threshold like 0.05.
Treat them as a tail probability when the null is true.
Statistical tests will never tell you if your data is normal: they just can't do that.
Unless you have skewed data, t-test will be applicable under general conditions.
In case your data is far from being normal, then use an appropriate model instead
******************************************************
Kolmogorov–Smirnov tests for normality are of little use when you estimate the parameters from the data. The P-values will always be very large: you can run a simulation and check for yourself.
Article A Cautionary Note on the Use of the Kolmogorov–Smirnov Test ...
Personally I would still take all those tests into account, although what specific correction to apply is less clear: testing normality prior to applying a t-test will result in an increase of type I error eventually
Article Wilcoxon–Mann–Whitney or t-test? On assumptions for hypothes...
Thanks all for your helpful responses. As always, it seems it was not as simple a question as I first thought!
Robert Browne technically using your original data you could set up a simulation in order to see the expected type I error and adjust your alpha accordingly
but again the KS test is of little use if you estimate the parameters from the data
Stefano Nembrini You may find my favorite linear models reference of use as well as the math stat book. Best wishes, D. Booth
https://www.amazon.com/Applied-Linear-Statistical-Models-2013/dp/1259064743/ref=sr_1_1?keywords=applied+linear+statistical+models&qid=1561563770&s=books&sr=1-1
and
https://www.amazon.com/Mathematical-Statistics-Applications-Dennis-Wackerly/dp/0495110817/ref=sr_1_3?keywords=mathematical+statistics&qid=1561564004&s=books&sr=1-3
- Badges
- Science topic
- Similar topics
- Citations