I want to compare the evidence for three models using AIC criteria. I have three models that I want to compare. Two models were computed using non-linear least squares. One model was computed using log-likelihood. I want to calculated the AIC value for these three models. AIC values for two models will be calculated using MSE. The AIC value for one model will be calculated using log-likelihood. Are these three AIC values comparable?
Hi John,
What do you mean exactly by "calculated using MSE" ? Can't you just compute log-likelihood for all of your three models and then compute AIC ? (or is the question was : "I compute GLM estimators using MSE, is that okay to plug them into log-likelihood to compute AIC value and then compares them with AIC values that are derived from max-likelihood estimators ?)
Anyways, please find some explanations about AIC by someone who is far better than me, you'll find interesting facts about your question there http://robjhyndman.com/hyndsight/aic/ (point 1 includes, I believe, the answer to your question ! )
Best regards,
MSE and log-likelihood are different statistic, you can't compare them directly. -2Log L between two different models for the same data is subject Chi-square distribution. MSE/sigman2 follows chi square distribution, hence you can see they are different.
AIC=-2log L+ 2*p, p is the number or unknown parameters, hence it is based on likelihood rather than least square.
I would recommend an out-of-sample evaluation as it is more objective:
Section 4 of our paper explains how to do that:
https://www.researchgate.net/publication/284947381_Forecast_Error_Measures_Critical_Review_and_Practical_Recommendations
Chapter Forecast Error Measures: Critical Review and Practical Recommendations
There are two ways to calculate AIC. One is with the Residual Sum of Squares: AIC = n*ln(RSS/n) + 2*K and the other is with log likelihood: AIC = -2*ln(likelihood) + 2*K. If I calculate two AIC values, but each AIC value is computed using one method and the other using the other method, are they comparable? Can you get the same value for a model using both methods? That is my main question.
It seems to be that AIC should be better since it penalizes over-fitting, however picking the smallest MSE seems to do the same thing, we should make some simulation to explore this.
There was a thread on StackExchange that asked essentially the same thing. The key is to use the same AIC calculation for the the different fitting methods, otherwise the value of the AIC comparison is meaningless. Check out the link below and scroll to the answer by user "Scortchi".
https://stats.stackexchange.com/questions/87345/calculating-aic-by-hand-in-r
Below article fit the MAS into AIC to get the best fit model, while the attached picture (Decoding Renaissance Technology Hidden Markov Model) states mostly using BIC to compare weighted, Bayesian or Markov models provided low number of parameter. 🙂
http://kourentzes.com/forecasting/2016/06/17/how-to-choose-a-forecast-for-your-time-series/
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling