Hello,
We are trying to find the best model (in R) for a language acquisition experiment. We have looked at Age (5, 6, 7, ... adults) versus Group (child versus adult) as a fixed factor mediating results. Both have moderate effects, but when comparing models, we find that AIC is lower in one model while BIC is so in the other. We have noticed that there is a lot of debate online as to which criterion is better to use, but no clear answer. Any thoughts?
I am really new to this business of logistic regressions. It is not specified if Age was treated as a continuous variable or as grouping labels. I would guess that de d.f. would be the same in both cases, but then I do not see how AIC and BIC got to change in opposite directions. I know that BIC corrects more agressively for the d.f. used by the model,but there would need to be something else if the défé are the same in the two models. The opposite effects on AIC and BIC likely indicate that both models are relatively as good.
If you want to see if indivisual ages, rather than making all children the same age and all adults the same age (in the Group effect), it seems that it should be possible to include both Age and Group in the same analysis, by scoring Age within group as deviation from the group mean. Then there could also be a term for Age:Group that would document a different slope in the two groups, since presumably an asymptote might have been reached in adulthood. If neither Age nor Age:Group are significant, age would not seem to count.
Merci André. The models are different. In the Age model, age is treated as a continouous variable but is grouped (5, 6 7 etc). While the second model has only two groups (child and adult). The AIC says that the Age model is better, while the BIC says the opposite. Thank you for your suggestions.
Not sure if I understand your problem correctly but never interpret absolute values of the many information criteria out there (AIC, BIC, CIC, DIC,...WAIC). You need to look at the difference in values for the same IC when comparing models. If differences for different ICs go in opposite directions check if the differences are not just "anecdotal" (>3). Most likely the differences are not substantial.
Yes Martin, we are seeing that in comparing models the BIC comparison promotes one while the AIC comparison promotes the other. The differences are significant (and tested with an anova).
Given that the model with several age categories consumed more degrees of freedon (which was not clear in the original description) and that BIC corrects more severely for the number of parameters fitted, it could be predicted that BIC would degrade more or improve less than AIC. Since the point seems to be to decide if age adds to simple category, the informal answer is that it does not, since BIC degraded. The suggestion to code age deviation within category would show this more formally, since the models compared would be in a hyerarchical relationship.
There is no clear answer because they are of similar likelihood function just that BIC introduces a larger penalty for the model with the most sophistication or parameters to address issues of overfitting. Hence AIC is supposed to be higher than BIC although the results will be close. So for example, if model 1 AIC=10 and BIC=8 and model 2 AIC =12 and BIC=7, which model will have the best likelihood of fit? I will go a step further and calculate their MAE and RMSE to further evaluate the models before making such a decision.
- Badges
- Science topic
- Similar topics
- Statistics
- Time Series