I am trying to model population growth rate for three different species in response to multiple climatic variables using linear regression. In this situation, is there a need for a correction procedure that controls for family-wise error rate? It is my impression that FWER corrections primarily apply to multiple comparisons of means tests (for example, mean A differs from mean B, and so on). Any input is appreciated!
Using multiple regression is somehow complex. Population growth can have nonlinear relation with any of your climatic variables. To test this nonlinearity you should check the residual plot against each independent variable.
Ase you have three population, type I error do not inflate significantly in comparisons. But if you insist on it you can use this formula to calculate actual alpha level.
actual alpha=1-(1-alpha(commonly 0.05))^k
K is the number of comparisons. If you reach an actual alpha more than 0.05, lower the alpha value that you enter to this formula.
Comparison of regression between populations, specially multiple regression, needs some considerations:
1- Climatic variables shouldn't have any correlation. This can cause collinearity.
2- The model should be additive. This means your climatic variables should not have interaction effect on growth.
3- All model selection criteria should be at good condition (R2, RMSE, AIC, BIC, Durbin-Watson, etc)
In short: the FWER is not controlled by the p-values given. If the FWER should be controlled, then the p-values (or alpha) must be appropriately adjusted.
But, Thomas, why will you "test" anything at all? It seems that your experiment is exploratory, not confirmatory. But if you really want to test, can you specify a reasonable alternative (minimum relevant effects)? Can you quantify the costs of wrong decisions and the benefit of a correct decision (to sensibly select alpha and beta?). And if so, do you want to control the FWER? Or rather the FDR? Or yet the TWER? ... And if you have to do the tests because a boss or a reviewer wants them, then simply do what they suggest. If they want a FWER control (at what level?) than do it. Otherwise leave it.
I think testing is a stupid idea here at all (I am not talking about calculating p-values; they may still be of some use to rank predictors - testing means to base a definite decision on the p-values, be it only to reject H0). It would be more useful to interpret the estimated coefficients (and their estimated precisions) w.r.t to their relevance.
I agree with much of Ehsan said, except his points 1-3:
1 - Only very tight correlation is a problem. However, it does not change the model predictions. The model can just not decide which of the highly correlating variables provides information that is not provided by the other. Therefore the variances (i.e. the model uncertainty) about the respective coefficients are inflated. If these coefs should be estimated with high precision, one has 2 options: either to decide to omit one ore more predictor(s) or using a "compound predictor" in the model (e.g. the loading vectors of a PCA). (the problem is simple for pairwise correlations that are easy to detect and easy to resolve [omit one], but multicollinearity could be a problem; here, a resonable solution is only the use of a compound predictor).
2 - Additive is fine, but interactions can explicitely modelled! They are often the biologically most interesting thing. Additivity is important only for the coefficients (otherwise it yould not be a linear model) and the errors (otherwise the least-squares estimation won't work [remedy: use generalized linear models]).
3 - Model selection based on statistics is never sound (it should be guided by expertise, not by data!). Further, all mentioned statistics can be quite "bad" and still the model can be "the best possible" and even useful. Also, the meaning of "good" or "bad" for these statistics is always relative and there is no objective way to classify them into these categories.
Thank you for your precise explanations Jochen. Besides, your recommendations are very useful. I mentioned these three points as if an moderate statistician do the job (such as me). As you mentioned, tight correlations can cause multicollinearity problems. In such situations, even one predictor can be a proxy of the others. I think in this way we can solve the problem while the interpretation is still simple using proxies. PCA can do the job too, but interpretation of the results can be a little messy.
Additionally, as you said almost all criteria I've mentioned are relative and one absolute value doesn't make any sense.
Thank you for your feedback Ehsan! Just a note, to clarify one thing: the fact that several variables are all very highly correlated is just a special case of multicolinearity, and as you said, in such a case any of these variables may serve as proxy. However, multicollinearity also arises when one of the variables tightly correlated to a linear combination of a group of other variables. Or when the linear combination of one group correlates to a linear combination of another group of variables. These cases can, to my knowledege, only be identified by the anaysis of variance inflation. In the first case again one variable can serve as proxy again, but in the latter case only some compound variable can cover the whloe information of the group of variables.
Dear Jochen,
If second situation exist, in my opinion, confirmatory factor analysis may lead to a more interpretable results than exploratory Factor or PCA (if all assumptions met), am I right??
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics