Dear Ronny,

I am afraid you might be misinterpreting the principle of parsimony, as outlined in your question. I see you asked this two years ago, but I'll explain my view anyway :-)

The most parsimonious model

The information theoretic criterion chosen (in your example AICc) is as such an estimate of model parsimony. It tries to resolve the trade-off between model fit and complexity, to achieve the best predictive ability. AICc already includes a penalty for model complexity, so no need for a subsequent round of ranking according to the number of estimated parameters.

If you trust in AICc, the model with lowest AICc is the most parsimonious, the one with 2nd lowest AICc is the 2nd most parsimonious, and so on. Any measure derived from AICc, such as ΔAICc, Akaike weights or evidence ratios are just other representations of essentially the same thing.

If you choose a cut-off point at say ΔAICc < 2 and subsequently favor the model with fewest estimated parameters, you are applying an additional round of penalty for complexity. Doing so feels like you are not trusting AICc does the job properly. You may get rid of a couple of weaker effects with say p < 0.12 and perhaps end up with a larger proportion of "significant" effects according to the conventional p < 0.05 applied in null hypothesis testing. But why use information theoretic criteria, if you are looking for the answer most consistent with null hypothesis testing?

Uninformative parameters

Another, related question is that of uninformative parameters (see Arnold 2010: J. Wildl. Manage. 74: 1175-1178). Many authors have suggested that one should disregard models which contain such uninformative parameters, if they appear in the set of considered models (may it be ΔAICc < 2; or something else). "Disregarding" can mean e.g. exclusion from model averaging.

Imagine that the most parsimonious model (with ΔAICc = 0) is a regression with:

Y ~ a + bX

There is another lower ranked model (e.g. ΔAICc = 1.5), which is effectively the best model plus one parameter more (parameter c):

Y ~ a + bX + cZ ,

which actually makes the model less parsimonious in terms of AICc. The model performs fairly well among the competing models only because the best model is a special case of it. One could argue that c is uninformative, as it only harms predictive ability in relation to another competing model. Why consider such a model?

Using regular AIC, the largest possible penalty for one added parameter is 2, so the largest possible ΔAIC is also 2. When considering all combinations of nested models using AIC or AICc, one will typically end up in a set of competing models, many of which contain uninformative parameters. It is a very common and fairly often neglected problem.

Notice that occurrence of uninformative parameters in a model does not mean it is less parsimonious than a model with larger AIC. It just means that there are obviously better alternatives, and that inclusion of some parameters is poorly justified.

Best wishes, Andreas

Ronny,

A common practice is to rank the models based on the Akaike weight. For this purpose first take the differences between the smallest AIC and all the others. Then calculate the likelihood and then the weights. The model with the highest support is the one with the highest AIC weight (AICw closest to 1).

If none of the models is overwhelmingly supported or two or more models have AICw close to each other, model averaging is recommended. 

However these practice must be supported by theory or biological reality. Any model can fit data well, and have seemingly low AIC. There is a problem called Freeman paradox where use of AIC may be misleading.

All the best

Thanks for answer. I use library(AICcmodavg) in R and I see it calculates the weights:

From one of my tests, see attached file.

If I interpret it correctly, from this model 3 has most support. Model 7 the second best, followed by model 4 (due to K=7 vs 8). 

What is regarded as close AICw values?

I see there is a option called compute evidence ratio:

> evidence(aictab(cand.set = Cand.models, modnames = Modnames))

Evidence ratio between models ' mod 3 ' and ' mod 4 ':

1.56

Ronny, your interpretation is correct. The evidence ratio shows that model 3 has 1.56 times more support than 4. However, the AIC weights are all very small and none of the models are overwhelmingly supported or there is substantial model uncertainty.

What is your goal? Predictive or explanatory modelling? If it is predictive modelling, the simpler model 3 (with fewer parameters) is better. If it is  explanatory modelling you may wish to consider  even the second best model because you do not want to leave out an important explanatory variable. To be on the safe side look at the beta weights of the variables in the saturated model (all 9 parameters) and see if the variables left out of model 3 have the lowest explanatory value. Check also if all parameters in your saturated model have been correctly estimated. For some of these concepts please see my article on the "Relative standard error ..." on my research Gate profile.

Since the AIC is often liberal you may also wish to compare your results with the Bayesian alternative using the BIC weights.

In my papers, I examin AIC, BIC, Cp and other staics ets. by model selection. See my papers on RG.

Hi, Ronny

Before today, I am busy the gene analysis and finish the surprising research yesterday.

After my desatation in 2000, I compute all posible models with AIC, Cp, BIC etc.

I follow themodeks selected by the forward and backward models with AIC etc.

I checked the minimum AIC and Akaike adviced me to check the models within "Min AIC+2". And I propose the results for specialist to discuss about results.

You check my uploading papers except for recent gene analysis. You can see many examples of your questions. 

AIC is just one of multiple goodness-of-fit measures. However, goodness-of-fit is not the only consideration for model selection. In general, modelling seeks to find a solution to some problem for which the most general approach is the inverse problem solution. Inverse problems are not solved by goodness-of-fit, but by minimizing error propagated of the parameter of interest.

It is all too common to just use ordinary least squares and assume that one has finished with the numerical methods needed for finding a model. In that case, one can indeed use AIC, or BIC or Chi-squared, or t-testing, or adjusted R2, or partial probabilities during ANOVA.

However, typically that will produce inferior models, which inferior models will then be compared with AIC or other goodness-of-fit measure to sub-select the least inferior of the inferior quality models, under those information content assumptions only. Consider for example that if the data is proportional type such that the errors are more homoscedastic after taking logarithms of the data and model, then that should be done first, as the results will be power function treatment and better, see link. Moreover, sometime the problem is ill-posed and applying ordinary least squares models to it will have instability and poor quality, for which Tikhonov regularization will improve the stability. However, in that case, goodness of fit becomes irrelevant, indeed, counterproductive as a model selection criterion.

Here is another example, suppose we have data with some really wild x-axis outliers. If we then use ordinary least squares fitting we will be duped into thinking we have wonderful goodness-of-fit. However, what we really have is garbage in/garbage out. If we really want to see what the tendency is in the data, we need to use some other method like Theil regression that is less affected by outliers. Now since the information obtained is reduced using nonparametric methods, e.g., including median slope, then we should analyze the fit quality nonparametrically as well, and however that is done, it would not be properly done using AIC, as the information types do not agree. Moreover, AIC tells us nothing about which regression methods to use and how to compare the same model with different regression methods which makes it a really bad thing to use, as it is then smoke and mirrors used by people who do not want to pose the "What the heck am I doing?" question.

The moral of the story is that all information is context dependent. Goodness-of-fit is often selected as a default context for information, but usually without doing the homework needed, for example, heteroscedasticity testing and many other critical considerations needed to ensure that the information sought by performing regression is at all related to the information reduction performed by the selected regression method.

https://en.wikipedia.org/wiki/Goodness_of_fit

https://en.wikipedia.org/wiki/Inverse_problem

http://www.itl.nist.gov/div898/handbook/mpc/section5/mpc55.htm

http://journals.lww.com/nuclearmedicinecomm/Abstract/2006/12000/An_improved_method_for_determining_renal.7.aspx

https://en.wikipedia.org/wiki/Tikhonov_regularization

https://en.wikipedia.org/wiki/Heteroscedasticity

A discussion of what AIC is and is not is presented on the Cross-validated site. http://stats.stackexchange.com/questions/215654/what-does-the-akaike-information-criterion-aic-score-of-a-model-mean/234662#234662

Dear Ronny,

I am afraid you might be misinterpreting the principle of parsimony, as outlined in your question. I see you asked this two years ago, but I'll explain my view anyway :-)

The most parsimonious model

The information theoretic criterion chosen (in your example AICc) is as such an estimate of model parsimony. It tries to resolve the trade-off between model fit and complexity, to achieve the best predictive ability. AICc already includes a penalty for model complexity, so no need for a subsequent round of ranking according to the number of estimated parameters.

If you trust in AICc, the model with lowest AICc is the most parsimonious, the one with 2nd lowest AICc is the 2nd most parsimonious, and so on. Any measure derived from AICc, such as ΔAICc, Akaike weights or evidence ratios are just other representations of essentially the same thing.

If you choose a cut-off point at say ΔAICc < 2 and subsequently favor the model with fewest estimated parameters, you are applying an additional round of penalty for complexity. Doing so feels like you are not trusting AICc does the job properly. You may get rid of a couple of weaker effects with say p < 0.12 and perhaps end up with a larger proportion of "significant" effects according to the conventional p < 0.05 applied in null hypothesis testing. But why use information theoretic criteria, if you are looking for the answer most consistent with null hypothesis testing?

Uninformative parameters

Another, related question is that of uninformative parameters (see Arnold 2010: J. Wildl. Manage. 74: 1175-1178). Many authors have suggested that one should disregard models which contain such uninformative parameters, if they appear in the set of considered models (may it be ΔAICc < 2; or something else). "Disregarding" can mean e.g. exclusion from model averaging.

Imagine that the most parsimonious model (with ΔAICc = 0) is a regression with:

Y ~ a + bX

There is another lower ranked model (e.g. ΔAICc = 1.5), which is effectively the best model plus one parameter more (parameter c):

Y ~ a + bX + cZ ,

which actually makes the model less parsimonious in terms of AICc. The model performs fairly well among the competing models only because the best model is a special case of it. One could argue that c is uninformative, as it only harms predictive ability in relation to another competing model. Why consider such a model?

Using regular AIC, the largest possible penalty for one added parameter is 2, so the largest possible ΔAIC is also 2. When considering all combinations of nested models using AIC or AICc, one will typically end up in a set of competing models, many of which contain uninformative parameters. It is a very common and fairly often neglected problem.

Notice that occurrence of uninformative parameters in a model does not mean it is less parsimonious than a model with larger AIC. It just means that there are obviously better alternatives, and that inclusion of some parameters is poorly justified.

Best wishes, Andreas

Thanks for reply and valuable information.

Few comments, on top many other good hints: It makes little sense to add more and more models and let only AIC (or BIC) decide. Also a "bad", "unphysical" model can get a non-zero weight, and thus can make e.g. the averaged model worse. AIC may also exclude the fully correct model, just if it is complex and if you do not have enough data, then aim to get more data, instead of rejecting a good model. Be also careful if you calculate a weighted average on something highly nonlinear, better try in such case a linearizing transformation upfront. A good practise is to use bootstrapping to check your estimations and to make experiments.

In "New Theory of Discriminant Analysis After R.Fisher", the Min - values of AIC, BIC (Cp) are indicated by six types of data. You can feel relieved if they match, because there are cautions when they are different. In addition, we selected a model with minimum mean error rate (M2) in the validation sample with "100 fold cross-validation for small sample". You should compare them using all available.