I'm having trouble interpreting the results of some recent bayesian mixed effects models I ran. I am looking to predict a continuous outcome variable ('accuracy') from a binary, dummy coded, within subjects factor ('intervention').
Model 1 is defined as accuracy ~ intervention + (1 | items) + (1 + intervention | subjects), and here intervention negatively affects accuracy.
Model 2 is defined as accuracy ~ intervention + control variable + (1 | items) + (1 + intervention | subjects) ('control variable' is continuous and between-subjects).
Model 2 results in no effects whatsoever, so the previous effect of intervention is lost but also control variable doesn't affect accuracy.
Model 2 has a barely higher R2 (36 vs 37) and a model comparison using lOOIC indicates that model 1 is the better model.
I have looked into suppression effects, but as far as I understand it, none of the conventional definitions of suppressor effects apply to my results because control variable (A) doesn't increase the prediction strength of my model, and (B) isn't correlated with accuracy.
To better understand what is going on I performed a median split on control variable and ran two independent mixed models for low and high control variable values.
Here I see the same drug effects in both low and high models (albeit with slightly higher uncertainty surrounding the effect, presumably due to lower df). This is confirmed when plotting the data, I see the same decrease in accuracy for both low and high control variable groups.
From my understanding all of the above indicates there is an effect of intervention and I should trust model 1 - I would just like to understand what happens in model 2.
I would be highly grateful if anyone could direct me to some relevant literature or maybe provide an explanation as to what might be going on here!
Could you please add the following information:
- are your control and intervention variable (cor)related?
- how big is the difference in beta values for your intervention variable?
BTW this sounds like a stepwise approach. If so don't use it. stepwise is well known to be a bad choice. Simply google for details or Austin and Tu's paper in J. Clin. Epidemiology available on researchgate I believe. Best, D. Booth
Thanks both for your answers.
Sven Greving Control and intervention variable are not at all related - because control variable is between subs and intervention within, the same value in control exists for both levels of intervention.
As for the difference in beta values between the two models: the beta value of model 2 becomes more negative (from -0.56 in model1 to -1.92 in model 2), but the credible interval is now almost centered around zero, whereas before there was a clear distance to zero.
David Eugene Booth I know of the debate around stepwise model building, however I'm not sure how else I would find out about scenarios like this - where I have a potential confounder (possibly) concealing a real effect - without the stepwise approach.
This is guessing on my part, but I think the control variable and intervention status are related as otherwise why would you be using a control variable? In model 2 the effect on accuracy is then being 'shared' between the effect of the control variable and that part of the intervention variable that is between subjects ( See Article Understanding and misunderstanding Group mean centering: a c...
)and therefore the intervention effect is reduced as the control is includedSo fit a new model to find out - that is put the 'within' observed intervention as your response and include the control variable as a fixed effect (albeit measured at subject level ) with just random intercepts.
I would appear that you are also fitting in both models random slopes for intervention - is that what you intend? (Sorry, but I much prefer equations and not syntax from software that I do not use.) I do not know what you level 1 unit is but with repeated measures within subjects, time is usually included in the fixed part as well as time with random slopes between subjects: see for a worked example.
http://www.bristol.ac.uk/media-library/sites/cmm/migrated/documents/repeated-measures.pdf.
I too think you are not adopting a stepwise approach but rather examining two theoretically interesting models in which one is a subset of the other.
Kelvyn Jones thanks very much for your answer.
Control variable and intervention are not at all related.
Maybe it helps to explain my variables a bit better: intervention = drug vs placebo, and control variable = a baseline (i.e., placebo) working memory score (which I use as a proxy for individual drug susceptibility). So for a participant, for both levels of intervention I have the same score of control variable, because I use only the placebo scores. This is why they are not related, and I suppose why the shared variance explanation isn't applicable here.
I was expecting either an interaction of drug and control variable, where only individuals with high drug susceptibility show effects on accuracy, or no interaction with a persisting effect of drug - but the current results with the drug effect disappearing when control variable is entered, in absence of a relation of control variable to either the outcome or my main predictor, confuses me.
I have tested a model with time (i.e., day 1 vs day 2) added as a fixed effect, and adding time also increases the uncertainty around my main effect of drug, but not in the same extreme way as adding control variable.
If the variables are not related Prof Jones' approach will show it. You are making an unsubstantiated claim. You should show that you claim is true. Best wishes, David Booth
I apologise if I wasn't clear - I thought it would go without saying that I have of course double checked my claim - both with a simple correlation and also with Prof Jones' approach. No (cor)relation whatsoever. But also from my understanding of correlational analyses a correlation here wouldn't make any sense.
OK.
First, when you say uncorrelated - do you mean zero correlation or just a low non-significant correlation. If the correlation is non-zero then it will decrease statistical power in model 2 to test the intervention effect relative to model 1 (even if the reduction in power is small it would matter in some cases).
Second, convergence issues could arise - worth checking for warnings and look at the trace plots etc.
Third, are there missing data for the control variable?
Thanks very much Thom Baguley for your answer!
I wasn't aware that even small, non-significant correlations between two predictors can influence the power of a model. Do you by any chance have any literature at hand that explains this in more depth?
The correlation between my control & intervention variables is r = .005, p = .82. (the coefficient of control variable predicting intervention variable in a mixed model is ß = -0.23, CrI = [-5.04, 4.61])
As far as I am aware my models did converge, as diagnosed based on Rhat values and trace plots.
And yes, my control variable does have about 10% missing values. I did already run my model 1 with a reduced dataset equivalent to the size of the dataset used when control variable is entered and got very similar results. So it seems it's not the missing data in model 2 that is leading to the lack of effect here either.
I wrote this for students a while back (loosely based on my book).
http://psychologicalstatistics.blogspot.com/2013/11/multicollinearity-and-collinearity-in.html
See also: https://janhove.github.io/analysis/2019/09/11/collinearity
Also: https://davegiles.blogspot.com/2011/09/micronumerosity.html
If r = .005 then collinearity will be negligible (unless there are other predictors that the intervention correlates with).
The most likely culprit is missing data - if the 10% cases are not missing completely at random then this could bias the estimates (it isn't just about sample size). I'd consider comparing the results with multiple imputation. That's relatively easy if you are using brms.
https://cran.r-project.org/web/packages/brms/vignettes/brms_missings.html
Thank you very much for your last answer Thom Baguley and for directing me to multiple imputation - I've experienced some trouble implementing this with my data, so haven't got an answer as to whether it would help me better interpret my models. Will update here as soon as I get the functions to work.
Many thanks again to everyone who has offered their advice so far.
- Badges
- Science topic