I recorded data from 17 subjects 600 trials each. I fitted the data for a linear model(2 parameter) and exponential (3 parameter) model. Overall r square for the exponential model was better and SSE was less. I calculated AIC, BIC, HQC for both model. Average of 17 subjects are
Exponential: 17.700531(AIC) 14.388660(BIC) 11.232778(HQC) 0.829579 (r sqaure)
Linear: 19.355811(AIC) 16.375185(BIC) 14.153079(HQC) 0.723486 (r sqaure)
is it valid to select the exponential model?
One model has better AIC, BIC, HQC, and r-square than the other. That sounds like good reason to choose one model over the other. ... However, if you have only 17 data points in the model, AICc may be a better statistic, to avoid over-fitting. Also, you might check to see if the r-squares are truly comparable. The r-squares for the exponential model is probably a pseudo r-square and not comparable to a true r-square.
This answer comes from the website Crossvalidated:
The rule of thumb, outlined in Burnham & Anderson 2004, is if Δi is AICi - AICmin, then the best model has Δi=0, and all other models have Δi>0. Therefore, the larger the AIC for any model, the less plausible it is.
@Adrian Esterman, what would be AICi and AICmin in this case? What you posted makes no sense to me without additional context.
In my point of view, model comparison and selection will make sense if you have used the same number of predictors. As you can see, the R square value is high for the exponential model because of the 3 variables used (R square increase with the number of independent variables). Try to harmonisze this and use many statistics (RSE, RMSE, PRES, %Error or MAD etc...) to select your models.
First you have to understand one thing that AIC or BIC is an estimator of the relative quality of statistical models for a given set of data. Given a collection of models for the data, AIC estimates the quality of each model, relative to each of the other models. Thus, AIC provides a means for model selection. When a statistical model is used to represent the process that generated the data, the model will almost never be exact; so some information will be lost by using the model to represent that process. AIC estimates the relative information lost by a given model: the less information a model loses, the higher the quality of that model. In making an estimate of the information lost, AIC deals with the trade-off between the goodness of fit of the model and the simplicity of the model. There is no value absolute good or bad, it just reflects the relative quality of a model to the other.
Balima Larba Hubert , statistics like AIC, AICc, and BIC are valid for different models with different numbers of parameters, as long the same data is used for each, the dependent variable is the same, the dependent variable isn't transformed, and some other conditions.
Big Thanks to all. Dear Salvatore S. Mangiafico sorry for confusion about number of data points. We recorded 17 subjects' data. Y axis was reaction time and x axis was processing time. we binned x axis(processing time) in 7 bins. We then took average RT in that bin for y axis. So technically fitting was done on 7 data points. Moreover, few subjects did not have data in few bins. so few subjects produced n-k-1 as zero. It leads AICc to Inf. How to deal with it? To avoid Inf values , I used AIC. The fitting was done with max 7 data pints for all 17 subjects individually , then AIC was averaged.
The problem is that binning is dependent on the values of processing time that is 0-350 ms and we cant take less than 50 ms bins. It makes few few data cases in a bin. how to increase the no of data points? is number of subjects no of data points?
Please suggest practical solutions?
I always look at the AIC difference on an odds or probability scale. If the difference in AIC is 1.6 that is e^0.8 on an odds scale or about 2.226. That is about 2.226/3.226 = .69 on a probability scale. This isn't a convincing evidence that one model is better than another (69% vs 31%). The data only slightly favour one of the two models.
Thanks Thom S Baguley . But this is the data and maths. Can I conclude that the process is both a linear and exponential? In our previous experiments, the diff of AIC corrected was >10 supporting the exponential. Can it be taken as a extra evidence to conclude that the exponential model is the model?
There are several issues here. Having a good fit to a linear or exponential model doesn't imply the process is linear or exponential (there are always many other possibilities). If those are a priori the most reasonable candidate models then the data provide some support for the exponential. It is also quite reasonable to infer the process from fits across multiple experiments (and one could formally combine them). Certainly differences in AIC of > 10 do provide strong evidence favouring one model over another.
- Badges
- Science topic