Hey everybody!
I'm implementing a Bayesian Negative Binomial using STATA 17. Because of some colinearity or convergence issues, I needed to put my variables in different blocks in the modeling process. Yet, it is a bit confusing to choose the most optimum number of blocks (and also their exact set of variables) for a model. Do you have any idea about it?
Apart from that, what criteria do you suggest (DIC, Acceptance rate, Efficiency, variables significance, etc.) for comparing models developed using various number of blocks?
I appreciate any help you can provide in advance.
It depends on how many cases are available.
Generally, DIC is a good criteria. It, however, requires that you know the true data generation process including the exact distributions.
In practice this is not always a realistic assumption.
This is why out-of-sample evaluation may be needed.
I recommend the FEW-1 workflow for forecast evaluation described in this paper:
Article Assessing Point Forecast Bias Across Multiple Time Series: M...
As a quick check if model is adequate I recommend the Overestimation Percentage corrected (OPc) metric applied to your point predictions. The Relative out-of-sample Mean Absolute Error (RelMAE) is also applicable. But please see the notes in Tables 10 and 11 in the above paper.
Dear Andrey Davydenko ,
I appreciate the time you put into answering my question as well as suggesting me a valuable reference.
In the last paragraph, you said if the model is "adequate." What do you exactly mean by "adequate"?
Bests.
I don't quite understand the "blocks" portion, but if you are trying to split variables it may be the case that the pathologies you have observed (poor convergence) that you should Google more about "identifiability" in the context of modelling.
I may be too far removed from academia since leaving for industry, but quite frankly all the criteria you mentioned are garbage. What is important is what the posterior distribution looks like (which you already have as a precursor to those criteria) and how it behaves. So take the samples and do that fun part of science where you "play" and look into what the predictions do and change when you fiddle with things and try to gain insight
- Badges
- Science topic