I'd like to compare more than two models (in my case the're six models, all binomial with two continuous predictors each), in order to assess which model provides the best fit. I suppose I have to use the Deviance Information Criterion (DIC) and I am using the rjags package. However, it seems that you cannot compare more than two models simultaneously in the function -- we have to compare the difference between two models and choose the one with best fit. I thought I could perform a pair-wise comparison (with only six models it would not be that time-spending), but I don't know if it really makes sense. I also thought I could simply rank all models by their DIC values and assume that the model with lower DIC is preferred, does that make sense?
Could anyone give some guidance on that? I would greatly appreciate any suggestions.
Hi Rafael, problem with DIC is there does not appear to be consensus as to which values are different. Difference in AIC values have general rules of thumb so that it is possible to identify which models are most/least informative, but DIC does not have any such consensus. Could be a good question for stackexchange:
https://stats.stackexchange.com/search?q=DIC
Thank you all!
And thank you, Darren, for the tip. In fact, I'm not really sure if DIC is appropriate for this case. I've already asked a question on stackexchange.
Hi Rafael,
I would look in to posterior predictive checks of the model.
If you can port your model to Stan - http://mc-stan.org/
Then there is a lot of options such as WAIC or LOO
http://www.stat.columbia.edu/~gelman/research/unpublished/loo_stan.pdf
Also, if you are using straight forward simple models there are lots of packages which can easily fit the model with little or no need to write the model. See brms and rstanarm packages in R.
I would like to calculate DIC and WAIC in Stan. Please , anyone, help me writing stan codes for DIC and WAIC.
I second Sreenath Arekunnath Madathil 's suggestion of posterior predictive checks. You could do it for each model you have. It would be a good first start. There is good information here:
https://stats.stackexchange.com/questions/174280/what-is-posterior-predictive-check-and-how-i-can-do-that-in-r
- Badges
- Science topic
- Similar topics
- Psychology
- Social Psychology