Regarding either a linear or a machine learning based regression analysis, how should we perform the normality test of the model? should we consider all data or just the training or testing dataset? I would be grateful if anyone could describe it with more details!
Make residual diagnostic plots (normal-QQ, amongst others), ideally on the test set.
In addition, you can use histogram of residuals to check if it is bell-shaped. You can use statistical tests like Kolmogrov-Smirnov or Shapiro-Wilks.
Since the model is based on training data, they must be checked for normality. I must add that the separation to test and training has to be completely random. You can not intentionally select a proportion of data so that they be distributed normally.
addint to the post of Kelvyn Jones : having outliers often is a sign that the assumption of a normal distribution is not reasonable.
("I am on an outlier! - I just haven't found my distribution yet.")
Jochen Wilhelm ; I agree that outliers can be very informative as to an improved model; as well as potentially signalling the need for a different distribution for the stochastic part of the model, they may also indicate other issues such as omitted variables, and non-linearity. It is just that I would not lose a lot of sleep over mild non-normality.
Kelvyn Jones , yes, fully agreed :)
From my perspective (research in life science), it's very common that people try to analyze concentration- and intensity-like measures, and they often find groups with scaringly high positive outliers. By a bad reflex, such values are removed. But in fact, such measures are more like log-normal or gamma distributed, and when looking at the log values there are usually no outliers anymore.
The problem is slightly more complicated in practice because people (biologists, physicians etc.) love to manually apply all kinds of (more or less strange) normalizations and standardizations to get measurements from different experiments to a similar level, thereby complicating the distributional properties of these normalized values.
Funny side note: exacly whene they would not have to worry about such things - e.g. in qPCR, where the measurements are altedy taken "at the log scale" - they do all kinds of exponential transformations to get their beloved problems back ;)
- Badges
- Science topic