Hello Valentin,

What you are suggesting above is questioning the principles of science.

Statistically non-significant results are often of the utmost importance and they are extremely significant in terms of their practical application.  

For example, the pharmaceutical company hires a researcher if a new drug would have a significant impact on the therapy. This new drug is very expensive and is going to replace the other drug currently in use for a decade being three times cheaper.

The study results indicate that in terms of treatment the difference between the types of drugs is NON-SIGNIFICANT. This is extremely important information as there is no excuse now to implement a new, very costly drug. 

Hence as researchers we are not on a "fishing expedition in search of significant p values in an ocean of data" (excuse my perhaps a bit dumb metaphor here). 

Also, you should apply a number of fit indices, not exclusively AIC (you could add chi-squares, BIC, WRSMR, etc) there is at least one hundred of them, Just look into literature in your research area to find out about other fit indices applied. 

What I suppose would be the best way to approach your problem is to prepare a couple or several THEORY driven models to test rather than at all cost search for a best fitting model. 

Best wishes

Thank you Witold for your answer.

What is confusing for me is that adding all X (explanatory) variables in one complex model, yields the smallest AIC from my list of models; yet, in terms of predicting how an observed Y value fits in one class or another (considering the multinomial aspect of Y) is not the best model.

Simpler models, but with 20 to 50 AIC units bigger, can predict as good or even better for some classes of Y (I refer here to the percent of correctly predicted observations).

This suggests me that I should consider simpler models and not use all X variables in a 'soup' and hope to taste good. However, I would be interested in following a sort of path that perhaps others experimented or proposed when it comes to model selection (in the case of  multinomial logistic regression).

Hope I made my question a bit more specific.

Hello Valentin,

I am not sure if I understood everything you wrote correctly. What I am sure though is it would be better if you used multiple fit indices rather than relied exclusively on AIC.

Also, path analysis based on regression could be examined with Structural Equasion Modelling techniques. 

Finally, perhaps some of the predictors within your models explain common variance (partially) hence it is possible that increased number of predictors is not adding much and at the same time being penalized (by AIC) due to its complexity.

Regards

Witold

My dependent (explained) variable Y has 3 levels (e.g. class1, class2, class3). I use several X independent (explanatory) variables to predict Y.

Actually the most complex models don't get penalized by AIC that much. However, even if these complex models are the best in terms of AIC, they are not necessarily the best in correctly predicting the classes of Y.

How should I deal with model selection in such cases? Is there any technique?

I upload many papers about model selection of seven LDFS such as Fisher's LDF, logistic regression, SVMs and my methods.

If all your explanatory variables are independent then all model selection procedures give the same results. But that is not the case in most practical situations. Usually explanatory variables are highly correlated and your model suffer from the multi-collinearity problem. Therefore, it is a good strategy for obtaining the most parsimonious model by selecting the useful set of independent (uncorrelated) explanatory variables and build your model then drop variables in the order dictated by their significance.

As for part b) from your original question - what to do IV's that are not "significant" by p-value but you believe are clinically important  to take into account - the answer is that it depends on your principal research question.

If you'r principal research goal is to determine whether a specific IV is related to your outcome, the selection of which "other" variables should go into the final model is not driven by the p-values of those variables, but rather their influence on the relationship between the primary IV of interest and the outcome.

For example, suppose that my research hypothesis is that "Patients that receive drug A will experience lower post-operative mortality than patients that receive Drug B."  My co-authors raise concern that perhaps elderly patients are more likely to receive Drug A vs. Drug B, so they ask me to "adjust" for age in the model.  I create a multivariate model with two IV's: Drug and Age.  I notice that the p-value for Age is not significant; however, I also notice that the beta coefficient for Drug in the multivariate model has changed quite significantly from the model with Drug alone.  In this case, even though Age is not significantly related to the OUTCOME by a strict statistical definition, it is important to retain Age in the model because it has a strong effect on the estimate for the effect of Drug.

In some cases, the most "parsimonious" model is not actually the one that answers your research question.  It depends what you're trying to determine,  The flip side of this is that you can have a variable that is "significantly" related to your outcome which does NOT modify the primary relationship of interest (i.e. Drug - Mortality in this case), but some reviewers will say that "I noticed you didn't adjust for (confounder), you have to redo your models while taking that into account" - and sometimes will persist even after you explain that this variable has no effect on the principal effect you are studying.

This principle can be quite aggravating to explain to reviewers, who sometimes believe that you have to adjust only for variables that they think are significantly related to the outcome, when in reality that is not the case.