Forgive the lack of a reproducible example in this question, as my problem stems from analysing a large (>50000 rows) dataset.
I want to ask a question about generalised linear mixed effects model diagnostics, I'm less familiar with handling GLMMs over GLMs. I am analysing a dataset where the response has a ‘fat tailed’ distribution. Its characteristics are: continuous values, all non-negative and greater than 0, with a strong positive skew & a maximum value of 1.
I have used R package lme4 and glmmTMB for the models themselves, and packages DHARMa and MuMIn (& base R) for my diagnostics.
I have fitted models with the following link functions: Gamma(inverse), Gamma(log), Beta(logit) and Gaussian(log). From some reading around I’m using simulateResiduals() in DHARMa because a normal QQ plot isn’t appropriate for most of these distributions.
My issue is I’ve fitted a selection of models to try to settle on the most appropriate and get conflicting results from different diagnostics, so I’m not sure what to do next.
In order of best to worst looking at the DHARMa QQ plot & residuals vs predicted plots is:
When using AIC (or AICc or BIC) the order is:
When I fit the mean estimate to the response data and eyeball it, the order is:
When I look at the prediction intervals, the order is:
And if I look just at fixed effects for confidence intervals, the order is:
At the moment, I am thinking the model with a beta family is the one to go with, even if the mean estimate is ‘worst’ (it’s still quite a good fit from eyeballing, it’s just the logit link flattens the estimate vs others), the prediction intervals and QQ plot are best and the AIC is OK. Also it's the best one on paper in terms of how it matches the characteristics of the response data. Is that a reasonable assessment of things? Does anybody have other ideas either about what I’ve done to check these models, or other things I could do that I haven’t thought of?
I believe that you will end up with a better decision if you think about the nature of your data and the processes that generated it before choosing a probability model instead of just picking the one that fits best.
We already know your response is highly skewed and strictly positive, so the normal distribution likely won't make much sense. In addition, we know that your response is strictly bound by 1, which may cause trouble with the Gamma as it does not rule out values above 1 and can have very long tails.
I therefore assume that unless all your data are at a safe distance from the upper and lower bound, you are best off with bounded distributions defined on the support of [0, 1], like the Beta distribution.
However, the Beta may cause trouble if you have true 0 and/or 1 values because those values are not defined for shape parameters below 1. If you find this to cause you trouble (it did in all my attempts to do Beta regression), you might have to play around with zero-/one-inflated Beta models, or if possible try different bounded distributions like the Kumaraswamy distribution.
Best,
Roman
I believe that you will end up with a better decision if you think about the nature of your data and the processes that generated it before choosing a probability model instead of just picking the one that fits best.
We already know your response is highly skewed and strictly positive, so the normal distribution likely won't make much sense. In addition, we know that your response is strictly bound by 1, which may cause trouble with the Gamma as it does not rule out values above 1 and can have very long tails.
I therefore assume that unless all your data are at a safe distance from the upper and lower bound, you are best off with bounded distributions defined on the support of [0, 1], like the Beta distribution.
However, the Beta may cause trouble if you have true 0 and/or 1 values because those values are not defined for shape parameters below 1. If you find this to cause you trouble (it did in all my attempts to do Beta regression), you might have to play around with zero-/one-inflated Beta models, or if possible try different bounded distributions like the Kumaraswamy distribution.
Best,
Roman
Thank you Roman.
I arrived at a Beta distribution in a roundabout way (initially was running a Beta GLM, decided there were aspects of the data that needed handling with a GLMM, no implementation of Beta in lme4, discovered an implementation in glmmTMB) so I was a little surprised when I did compare it to my earlier models, things like AIC came out worse.
It sounds like I can ignore 'better' performance in diagnostics as these come from badly specified models.
Malcolm
I am not sure if I would recommend 'ignoring' these. But the reason why people use GLM(M) instead of just making simple transformations that get things approximately normal and homoschedastic is usually because they want to use a probability model that more closely reflects the nature of their data. This may come at a high cost, because you often pay with all sorts of convergence issues, and especially for GLMM your inference is based on approximations that may not always work as intended. I therefore do not see much sense in replacing one probability model that does not quite reflect the nature of your data by another one that has just the same problem - plus a load of additional pitfalls.
For example, if your data are bound by 0 and 1 and you nonetheless use a gamma to account for the skew while ignoring the difference in support, I am not sure why this should be preferable to simply log-transforming the response and fit a regular LMM. In my (limited) experience, the latter leads to models that are much more well-behaved than Gamma GLMMs, and both have just the same mismatch of support.
Btw. I am not sure if you can compare AIC between GLM and glmmTMB (if that's what you did) because different packages often have slightly different expressions for log likelihood functions. If you are not absolutely sure identical model specifications result in the same log likelihood, you probably better shouldn't.
Hope that's all useful for you...
Best,
Roman
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Software
- R Statistical Package