Do you have a large Poisson mean or very small Poisson mean? In my understanding, overdispersion is a phenomenon when variance is larger than the Poisson mean. Please correct me if I am wrong.

Mohammad Razaur Rahman Shaon,

Thanks for your reply. No, the Poisson mean is fine. The issue is the difference between the Poisson mean and the variance. As you said, when the variance is larger than the mean in a Poisson distribution, the data is over-dispersed and is better represented by a negative binomial distribution with dispersion parameter θ such that:

Var(y) = E(y)*[1 + 1/θ E(y)]

Now, what if the variance is larger than the mean, but the difference is extremely small, and yet significant. Say, θ in the above equation is 1000. My question is would you simply use the math and use a Poisson distribution, or would you keep the negative binomial distribution because of that dispersion, albeit it is extremely small?

Thanks for the clarification. The way I understand it, the Gamma distribution in NB model used to account for extra noise in the data. If not account for this extra noise, the model can result in biased parameter estimates. In case of very small dispersion parameter, there shouldn't be any significant difference in the parameter estimates as well as residuals from the fitted models. For example, in our study where we used NB-Lindley distribution, we observed smaller dispersion parameter compared with NB distribution because a part of extra noise is already accounted for due the Lindley distribution.

Mohammad Razaur Rahman Shaon,

You're right. The difference in the parameter estimates is very small too. Again, I want to explore this from the theoretical point of view, not from the statistical point of view.

So, my understanding is that if we believe (from the theory) that the distribution is Negative Binomial, then we should keep it. The implication of doing so, though, depends on the context. If the context is epidemiology, even a small difference between the expected value and the variance is important. By definition, the expected value of a disease in a stochastic negative binomial mechanism is going to be different after 1000 (very large) repetitions. So, I would keep the negative binomial distribution even if the difference is very small because even 1 different record of such disease in 1000, matters.

If you are running a regression model assuming a Negative Binomial distribution and you get a large overdispersion parameter (>> 1), this says that the observed variance is larger than that expected under the Neg Bin distribution. Hence, this likely is due to one of the following:

1) you are mssing important variables (fixed or random) that explain that extra variability, or

2) the distribution is not negative binomial but another prob distribution where the variance is even larger than that expected under the negative binomial. This usually means you have a very skewed distribution in your data, more skesed than expected under the Neg Bin distribution.

Good luck with your analysis.

Thank you Mary C. Christman for your response. I think I was trying to ask about the inverse dispersion parameter in the question. My apologies for this confusion (I have now edited the question). I actually get extremely large "inverse dispersion parameter" which means that the observed variance is much smaller than that expected under the Neg Bin distribution; yet, the variance is significant.

So, my question would be whether you forget about the hypothesized theory and switch to a Poisson distribution or you maintain the hypothesized theory and you keep the Neg Bin distribution?

This is an interesting topic. Mathematical wise if the dispersion parameter is close to 0 then NB is same as Poisson.

I had the same issue before; the value of the dispersion parameter was significant but slightly greater than 0 (something like 0.002). This implies that we need to use NB.

In my opinion, we need an additional test to decide between the models. In my case, I decided to run a Lagrange Multiplier to learn the truth (!!). The test indicated that the over dispersion was not an issue in my case.

Amin Mohamadi Hezaveh

Thanks Amin for your reply.

When the over-dispersion is extremely small, the difference in the "truth" between two distributions is maybe 1 in 1 million. So, I argue that the "truth" you are talking about is a context-specific term. Even if another metric (e.g. Lagrange Multiplier) shows that the over-dispersion is mathematically negligible, the context may require keeping the Neg Bin distribution.

If we are talking about crashes, 1 in 1 million is "not an issue". But, if we are talking about cancer, or Ebola for example, then yes, I argue that even one person difference in one million is an issue.

Amir Pooyan Afghari , why 1 in 1 million is "not an issue" when "we are talking about crashes"?

That one crash might be a fatal crash.

Seyedehsan Dadvar you are absolutely right. That's exactly why crash severity must be considered in addition to the total counts of crashes.