you might want to consider

Burnham, K.P. & Anderson, D.R. (2002). Model Selection and Multimodel Inference. A Practical Information-Theoretic Approach. (2nd ed.). NY: Springer.

They vote against the "true model approach"; moreover, they consider the question as such misleading (referring to Taub, 1993: a model is not truth by definition; Box 1976: all models are wrong, but some are useful, etc). Rather, they focus on numerous other model selecting procedures.

You are so right, linear AIC is just another proxy to the truth, and there are 'a few' other proxys to that

George Box: all models are wrong but some are useful. AIC is a way of measuring usefulness of a model.

There's a great explanation here (below). It's more clear-cut when considering BIC (cf. quasi-true models).

http://www.sortie-nd.org/lme/Statistical%20Papers/Burnham_and_Anderson_2004_Multimodel_Inference.pdf

The truth is out there...

Seems to me, I prefer BIC & CAIC not AICc!

The AIC is a modification of the loglikelihood and penalizes the likelihood of the data given the model by taking model complexity into account. This implies that the credibility of your model will drastically decrease when its likelihood was poor and the model involves many parameters. The BIC further extends the AIC by adopting a Bayesian perspective. The BIC can be approximated as a reformulation of the AIC that also takes sample size into account. This implies that the likelihood of complicated models is less penalized when the models were derived from large samples.

In essence, the LogLikelihood, BIC and AIC are alternative approaches to quantify discrepancies between a model and data. If the data at hand is the total population, the LogLIkelihood tells you what you need to know and there is no need to penalize since the model can only be applied to existing instances (i.e. generalizability is not an issue). However, if your data represents a sample, it is less clear what the value of your model is. Particularly, the fit of your model in the derivation data will be an overoptimistic estimate of its fit in new samples. This overoptimism will increase as you allow more flexibility (i.e. allow the model to focus on individual instances from your data). Because simpler models are smoother, their predictions are less likely to strongly deviate from new instances. This is why simpler models are usually preferred, unless you have much data at hand. The BIC and AIC attempt to integrate this philosophy, but do not necessarily provide the right answer. After all, model complexity and model fit represent mathematical concepts and have no direct relation to each other, or to the so-called truth (e.g. samples and populations change over time, or may be influenced by extraneous factors that cannot be measured. Also, variable definitions may change or not well or no longer represent tangible entities)

The following article presents a very insightful discussion: Ye, M., P.D. Meyer, and S.P. Neuman (2008), On model selection criteria in multimodel analysis, Water Resources Research, 44, W03428, doi:10.1029/2008WR006803.

True model need not be a member of the family of models considered in order for AIC, BIC or any model selection criteriion. However, it is important to understand what the properties of these model selection criteria are under different conditions: (i) true model contained or not, (ii) asymptotic versus finite sample, and (iii) prediction versus model consistency. The results that we know about for AIC and BIC are all symptotic results. We know that BIC is a consistent model selection criterion, i.e. it will select the correct model as sample size goes to infinity, when the true model is in the family of models considered. AIC does not have consistency property, but it gives the optimal prediction model asymptotically.

I thought that Multi-model inference protocols were designed with the assumption that there isn't a true model, or even a best model, but a set of models that are likely to be approaching the truth. Model averaging then allows you to make best use of the identified set of models. From a frequentist viewpoint, p-values become less relevant and models containing non-significant parameters may be the best you have. Working with ecological data means that you know a priori that variance will be the bane of your life, you deal with it in a damage limitation way because you cannot collect all values of all variables affecting a natural system. So your random terms don't 'behave' as well as you'd like. Lets not forget, most models wear hats! Their parameters are estimates for a population that we will never know the truth about. Keeps us all busy though!

In response to Mark Warren: model averaging, in fact, does assume that one of the included models is 'correct'.

Monteith, K., Carroll, J.L, SEppi, K Martinez, T (2011) Turning Bayesian model averaging into Bayesian model combination . In: Proceedings of the 2011 International Joint Conference on Neural Networks, Edition (, ed), p 2657--2663.

Domingos, P. (2000) Bayesian Averaging of Classifiers and the Overfitting Problem. In: In Proceedings of the 17th international conference on machine learning, Edition (, ed), p 223-230. Morgan Kaufmann

Minka, T. (2000) Bayesian model averaging is not model combination. MIT Media Lab note.

Thanks for the correction. I suppose that sort of answers the original question 'does AIC require the truth to be in the set of candidate models?' Although it doesn't require it, methods that use AIC assume that 'truth' is represented somewhere in the models presented. From an ecological viewpoint, I believe this is an unreal assumption and modellers working with such data should assume they don't have the truth but are trying to move closer towards it. Being new to Bayesian methods I better read those papers!

It seems to be a common misconception that model averaging is a method of model combination; I was not aware of it myself until recently. Although model averaging technically does combine a set of models, the averaging procedure usually results in skewed weights because it assumes that one of the models is correct (and small deviations in log-likelihood then lead to strong differences in model probabilities).

Im not sure in what context the original question "Does AIC require the truth to be in the set of candidate models?" was posted; direct comparison of AIC or BIC may for instance be performed to select a 'better' (but not necessarily correct) model. This is also one of the reasons why several authors pose that performance statistics should never be interpreted on its own, but rather be compared to other models that were fitted on the same data.

The situations where the true model is or is not contained in the class of models are sometimes known as M-closed and M-open, respectively in the Bayesian model averaging (BMA) literature. BMA is applicable under both M-open and M-closed situations. Only the motivation and interpretation change. What loss function are you trying to minimize? See Bernardo and Smith "Bayesian Theory" (1994) or Clyde and George article titled "Statistical Uncertainty" (Statistical Science, 2004).

you might want to consider

Burnham, K.P. & Anderson, D.R. (2002). Model Selection and Multimodel Inference. A Practical Information-Theoretic Approach. (2nd ed.). NY: Springer.

They vote against the "true model approach"; moreover, they consider the question as such misleading (referring to Taub, 1993: a model is not truth by definition; Box 1976: all models are wrong, but some are useful, etc). Rather, they focus on numerous other model selecting procedures.

.

"As far as I understand the derivation, the truth is not required and, in fact, we aim at the KL-optimal model as proxy to truth and the nature of truth is not relevant to the derivation at any point."

Correct from my understanding,mainly built from the Burnham and Anderson book (Rainer's comment above)

This used to be a pretty explosive topic ! Just for fun, from Burnham and Anderson :

AIC Myths and Misunderstandings

http://www.colostate.edu/Dept/coopunit/pdfs/AIC%20Myths%20and%20Misunderstandings.pdf

"AIC assumes a true model.

This is not correct; see Chapter 7 of Burnham and

Anderson (2000). This error may come from the fact that there are several

derivations from K-L information to AIC. One such derivation notes that the

deviance (the first term in AIC) has a relationship to the chi-squared distribution

(and is therefore the basis for likelihood ratio tests). The chi-squared distribution

of the test statistic comes about only if the models are nested. We are guessing,

but this might be the source of the confusion. Our clear position on this issue is

that nothing need be assumed about a true model when justifying or using AIC (or

AICc or QAICc); see details in Sections 7.2 and 7.2 of our 2002 book."

.

.

for a nice discussion of BIC and AIC, see

AIC and BIC : Comparisons of Assumptions and Performance

Jouni Kuha

Sociological Methods & Research, 2004 33: 188

the online version of this article can be found at:

http://smr.sagepub.com/content/33/2/188

this discussion (see particularly pp 204-208) allows to reconcile the apparently conflicting views of Burnham/Anderson and Ye et al.

.

I am work with asymmetric models (see http://www.sciencedirect.com/science/article/pii/S1572312710000948 and http://www.tandfonline.com/doi/abs/10.1080/00949655.2013.828057#.UkwbkNJwr7Q). If you generate of a skew-normal model with strong asymmetry may be that a skew-t-normal model have better AIC and BIC than skew-normal, for example.

This SAS simulation of several model-selection methods, for a known ("true") model with up to 10 predictor variables, showed that several methods--including Mallows' Cp and stepwise regression--beat AIC in choosing the right model. However, the Schwarz Bayesian Criterion was the clear winner among 9 methods:

http://analytics.ncsu.edu/sesug/2007/SA05.pdf