One of the assumptions in linear regression is that the data should be normally distributed. My data is skewed. I tried log, square root, and several other transformations which I could find still is not entirely normally distributed and the KS normality test rejects the null hypothesis.
Finally, with the help of INT in the Rank Normal Omnibus Association Test(RNOMNI) R package, I am able to get a proper bell-shaped curve. Can this normalization be used because finally, I am shifting the data to a non-parametric form?
Wow you have a problem and your approach is very unlikely to be correct. Use a Box-Cox transformation package (a nice one is in R) and that should solve your normality problem and the rest should be routine. Best wishes, David Booth
PS I forgot to mention that it's possible that you misunderstood your Kolmogorov Smirnoff test results. As the attachment shows large p-values favor the hypothesis that your distribution is normal NOT small p-values. This error is often made. Check this out. If the test indeed doesn't favor normality then follow the steps in my previous post. Best wishes, David Booth
PP plot (QQ plot) and the histogram of the transformed data are good criteria to check the appropriateness of the transformation as well.
To get an appropriate transformation, it is extremely helpful if you find the distribution of the original data. After that, you can find the rank or the order of the data (the distribution) of them. Since both the obtained distribution function and the distribution function of a normal variable are uniformly distributed over the interval [0,1], the desired transformation can be computed through the equality of the two mentioned random variables.
Thanks David Eugene Booth and Babak Jamshidi for the answers. The INT transformation I used is transforming my data to a normal distribution. I am not preferring it because I am not able to get the unit change of my variable after my association tests are done. Like u see if I am doing with Haemoglobin levels I can finally relate to the associated SNP is having an effect of 3 unit change in the original variable level. This statement can be said only if I use a simple transformation such as log or sqrt etc. Thanks for suggesting Box-Cox transformation. But I have read from a blog(https://towardsdatascience.com/box-cox-transformation-explained-51d745e34203) that one of the limitations of this method is the same. " If interpretation is your goal, then the Box-Cox transformation may be a poor choice. If lambda is some non-zero number, then the transformed target variable may be more difficult to interpret than if we simply applied a log transform." Is there a way that we can finally relate to the target variable after the regression or association if we use @box-cox?
@Alagu please don't forget that it possible to back transform your fitted model to the original units and should be done in your case. See the attached file for details. BTW this should be done for ANY transformation for your application. Best wishes. David Booth
One piece of advice is to not use a test to determine if your data are normal or not. If you have a large sample size, such a test is likely to identify the non-normality in a distribution that is close-enough to normal.... If your data are really troublesome, you might consider a form of robust regression, or a non-parametric approach like Kendall–Theil Sen Siegel linear regression.
- Badges
- Science method
- Similar topics
- Biological Science
- Ecology