I am conducting a panel data regression for a research on economic growth of few countries. In real life, it is hard to find data that are normally distributed and most of the control variables are correlated with each other in one country or another.
However, the regression test results are satisfactory and all show that the residuals are normally distributed, there exists no serial correlation and heteroscedasticity. Even the CUSUM and CUSUMSQ tests show that the model is stable.
In such a case, are the diagnostic tests enough to justify that the results of the regression model are reliable and valid even when data are not normally distributed and there exists correlation among them?
Thank you in advance for your responses.
There is no assumption about normality of predictor or outcome variables in linear regression (only about the normality of residuals/errors). Also, linear regression is designed to handle correlated predictors, so there should be no problem unless the correlations are extremely high.
A "statistically significant" model should never be equated with "true or valid." There are Type I and Type II errors. There are also other errors, such as Simpson's paradox. Finally, one study may find a significant relationship that is only significant under the experimental conditions at that one location. There are significant issues of reproducibility: can someone redo your experiment and get the same result? In some cases, scientists have tried to replicate their own experiments and were unable to get the same result. If there is sufficient replication and the experimental design and the subsequent analyses conform to the norms in your field of research, then it is likely that your work will be suitable for publication. That does not make the results true or valid. Because it is in print and reviewed by three other researchers does not make it true. However, without other evidence, we often decide to behave as if the results are true.
Strictly speaking most of the above answers are correct for explanatory models. However predictive models are a bit different. In that case cross validation is definitely necessary
A good explanation is given in steyerberg's book referenced in the attached paper which also gives an example of these methods. Best wishes David Booth
« the regression test [...] all show that the residuals are normally distributed, there exists no serial correlation and heteroscedasticity. Even the CUSUM and CUSUMSQ tests show that the model is stable. » : all of this is wrong.
Regression tests did not found evidence against a Normal distribution, against serial correlation and so on. This is very different. They did not found either because data are Normal etc. (optimistic point of view, almost always false), because of bad luck (importance of reproductibility), or because of lack of power (most frequent situation). Lack of power may be because the sample size is too small (quite frequent), because the departure is of a different kind than the one the test is designed to find, or (inclusive) because the departure is "small".
The fact that the departure is "small", so cannot be found by the regression tests, does not mean it has no consequences on the results of the model, that may be sensitive to that departure (this is especially true in small sample sizes). However, the situation is expected to be not as bad as when the test manages to detect something.
Last, there are other assumptions that are not always checked by these tests, including influential points, quasi-separation, overfitting, misspecified model and so on.
So, in brief, passing all tests is a good starting point, but does not replace a careful examination of the model and its results, and the coherence of its results.
If the tests say the model is not overfitted, but there are several panel members with high correlation of explanatory variables, that might suggest splitting the panel on other attributes and deriving sparser models for each.
- Badges
- Science topic