1) Because I am a novice when it comes to reporting the results of a linear mixed models analysis, how do I report the fixed effect, including including the estimate, confidence interval, and p-value in addition to the size of the random effects. I am not sure how to report these in writing. For example, how do I report the confidence interval in APA format and how do I report the size of the random effects?
2) How do you determine the significance of the size of the random effects (i.e. how do you determine if the size of the random effects is too large and how do you determine the implications of that size)?
3) Our study consisted of 16 participants, 8 of which were assigned a technology with a privacy setting and 8 of which were not assigned a technology with a privacy setting. Survey data was collected weekly. Our fixed effect was whether or not participants were assigned the technology. Our random effects were week (for the 8-week study) and participant. How do I justify using a linear mixed model for this study design? Is it accurate to say that we used a linear mixed model to account for missing data (i.e. non-response; technology issues) and participant-level effects (i.e. how frequently each participant used the technology; differences in technology experience; high variability in each individual participant's responses to survey questions across the 8-week period). Is this a sufficient justification?
I am very new to mixed models analyses, and I would appreciate some guidance.
It is not complicated at all:
1. Don't report p-values. They are crap!
2. Report the fixed effects estimates. These represent the best-guess average effects in the population. Try making quantitative statements:
"Overall the effect is strong/weak/negligible". That depends on your domain of research, of course.
3. Report the confidence limits. Make statements on uncertainty:
"The effect apprears to be is strong, but CIs are wide. We cannot exclude that it is much weaker or stronger"
"The effect is practically zero and we can say so with good certainty"
4. Report how variable the effect is between individuals by the random effects standard deviations:
"On average the effect is strong, but there sis considerable variation between subjects"
What works well for me is the following table structure:
parameter name | beta | lower-95 | upper-95 | random effect (SD)
If you are using an R implementation (lme4, rstanarm, brms), you can also extract the participant-level estimates and create spaghetti plots for 4.
https://www.researchgate.net/project/Book-New-statistics-for-the-design-researcher
Answers to your questions will take time and space. Best would be to talk to someone in your Uni's Stats dept. See if the links below are of any help
http://arxiv.org/ftp/arxiv/papers/1308/1308.5499.pdf
http://wiki.bcs.rochester.edu/HlpLab/StatsCourses?action=AttachFile&do=get&target=Groningen11.pdf
http://www.stat.cmu.edu/~hseltman/309/Book/chapter15.pdf
http://www.bristol.ac.uk/cmm/software/mlwin/
Look at this site they have a libary of publications showing how other people do it.
Hi Sabina,
See Table 2 of this article (http://ursulakhess.de/resources/HDH11.pdf) for an example of a mixed model reported in APA format. Although this table simply reports the estimated effect and its standard error, you could substitute the standard error for the 95% confidence interval).
To get a sense for the size of the random effects, partition the variance in your dependent variable. That is, calculate the proportion of variance in your dependent variable attributable to between-cluster differences and the proportion attributable to within-cluster variability. To do this, run a null model (i.e., model your dependent variable with no independent variables). Then, add the two residual/random variance estimates together to get a total estimate of the residual/random variance. Then divide the between-cluster residual variance by the total residual variance. If you 30% of your dependent variable's variance is attributable to between-cluster differences, then 70% must be attributable to within-cluster differences. Of course, the meaningfulness of that balance will depend on your dependent variable.
That the random effects were weak in your study reflects the fact that individuals' performances across the 8-week period remained relatively uniform. Were the within-person random effects non-significant? Even if that is the case, you can still justify the use of mixed modeling for some of the reasons you stated, primary amongst them the missing-data issue.
In the case of the LMM, you should report on biological rationale and level of significance of explanatory variables, taking also into consideration the estimates of the parameters in G and R matrices. The covariance structure should be selected among several candidates, i.e. autoregressive, compound symmetry, unstructured, etc., to estimate the fixed and random effects and selecting and their Akaike’s Information Criterion (AIC) and Bayesian’s Information Criterion (BIC) should be reported. Also if the models were fitted and compared using an specific method such as restricted maximum likelihood, should be stated. Please check a publication by Bueno and Bevilacqua in the following link for further reading.
http://www.sisef.it/iforest/contents/?id=ifor0843-006.
Hi Subina. It very much depends on why you have chosen a mixed linear model (based on the objetives and hypothesis of your study). Random effects, in your case, seem to simply control for random factors that may affect your response variable. Usually you report the proportion of the total variatiion they account for (as a ratio or percentage). But if it is not a large proportion it does not mean you can neglect them. Som times you have to consider them regardless of its magnitud (like in the case you measure the same experimental unit several times accross the study). A way to check for their statistical significante is to perform a Likelihood Ratio Test (i.e, you compare a model with the random effect vs. a model without it, using a Chi square with the d.f. based on the difference on the number of parameters in the complete vs. the reduced model (variances and covariances) or to use the Akaike Information Criterion to compare alternative models). Most of the statistical programs give you this information as well as the BLUEs (fixed) and BLUPs (mixed) effects and their confidence intervals. Hope this helps.
If I understand you well you have the participants as experimental units, so participants are no factor. You correcrly use a fiv´xed factor for the technoklogies and a random one for the weeks. The effects of a random factor can not be estimated, because they are random. You only can (and should) estimate the variance component of the random factor by one of the four methods offered by SPSS. All you need to know you find under two-way cross classification, mixed model in one of my books.
As mentioned by others, more time and space than available here are needed to introduce and explain mixed model methodology. I would suggest you to have a look at the SAS manual entitled SAS for Mixed Models by Littell and al, 2nd edition which gives you both basics in linear mixe models and many examples of applications as well as the text books by Geert and Molenberghs (Linear models for longitudinal data) and Variance components by Searle Casella and McCulloch. I also strongly recommend use to test and compare several models using information criteria eg AIC or BIC (depending on your osbjectives) before drawing any conclusion.
I recomend this paper of Plos One: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0112653
It isn't exactly a LMM but this paper explain how you can report a GLMM.
Best regards,
Martí Casals.
I am analyzing data of a randomized clinical trial that randomly enrolled 60 patients to two groups (treatment vs control). Patients were assessed at baseline, two and four weeks after randomization. I have chosen to perform LMM to account for the dependency between repeated measurements collected for each participant across time. I'm using SPSS 20.0 to run all analysis.
I am having some trouble in finding out how to report the results of my analyses. I've read several textbooks (Andy FIelds'; Landau & Everitt's;Twisk's) and none of them helped me to find out how to properly report not just the model fit (RLM or RL; AUC, etc), but the estimates for each group at each time point, the between-group mean differences and the 95% confidence intervals of the differences.
In my mixed model, each subject is the experimental unit. "Group" and "Time" are fixed factors; "Time" is the random effect (no random intercept or slope could be added as this resulted in convergence problems, possibly due to overfitting [?]); variance components were used as covariance structure (convergence problems appeared when using AR; Compound symmetry and unstructured covariance). The syntax can be found as it follows:
MIXED Dor_Perna BY Group Time
/CRITERIA=CIN(95) MXITER(100) MXSTEP(10) SCORING(1) SINGULAR(0.000000000001) HCONVERGE(0, ABSOLUTE) LCONVERGE(0, ABSOLUTE) PCONVERGE(0.000001, ABSOLUTE)
/FIXED=Group Time Group*Time | SSTYPE(3)
/METHOD=REML
/PRINT=SOLUTION TESTCOV
/RANDOM=Time | COVTYPE(VC)
/EMMEANS=TABLES(Group) COMPARE ADJ(BONFERRONI)
/EMMEANS=TABLES(Time) COMPARE ADJ(BONFERRONI)
/EMMEANS=TABLES(Group*Time)
/EMMEANS=TABLES(Group*Time) COMPARE(Group)
When looking at the "Fixed Effects", no Group by time interaction was found (p = .065); however, in the Group*Time table a significant difference was found (mean difference 2.1; 95% CI [-3.47 to - .859] p = .001)
Are the results given by the "Pairwise comparisons" table the ones that I will present? How to explain the differences between the "Fixed effects" table and the "Pairwise comparisons" table?
I thank you all in advance
Especially for whom is using R and lme4
I recommend this two links taken from Cross Validated, where several number of examples and papers are cited and reported.
https://cogsci.stackexchange.com/questions/9765/how-should-results-from-linear-mixed-models-lmms-be-reported
https://stats.stackexchange.com/questions/26855/example-reports-for-mixed-model-analysis-using-lmer-in-biology-psychology-and-m
Best, Marianna.
A colleague of mine says "You should drink from the sources, not from the puddles" Check out Bolker et al. 2009 http://dx.doi.org/10.1016/j.tree.2008.10.008
You can use Wald Xi square tests in order to estimate the explained deviance, however is hard to measure the residuals, you can try the R package "car" Anova function and the "phia" package for post-hoc test. Both work for GLMMs and LMMs
1) How do I report the fixed effect, including the estimate, confidence interval, (...) For example, how do I report the confidence interval in APA format and how do I report the size of the random effects?
-> There are various ways to report. One is: report the fixed effect with CI/p-values in a table. My personal favourite to include variation of the random effect is to show the random effect for each of the 16 participants in a graph with CI in descending order. Also possible is to display the observed means and the estimates from the regression against time for each group (setting/no setting). There are programs like R where you can easily create the plot for random effect (here an example with poisson regression: https://stats.idre.ucla.edu/r/faq/random-coefficient-poisson-models/).
2) How do you determine the significance of the size of the random effects (i.e. how do you determine if the size of the random effects is too large and how do you determine the implications of that size)?
-> That's you and your knowledge who decides what size is relevant for the topic (not significance, that's a statistical term).
3) Our study consisted of 16 participants, 8 of which were assigned a technology (...) How do I justify using a linear mixed model for this study design?
-> The design is a repeated measurment design. This can be analyzed with ANOVA or Mixed Model (see advantage of MM against ANOVA: http://www.theanalysisfactor.com/advantages-of-repeated-measures-anova-as-a-mixed-model/). Sincerly, I wouldn't justify. Just mention that you performed a linear regression, respectively, poisson regression with patient as random variable.
I found also this!
https://web.stanford.edu/class/psych253/section/section_8/lmer_examples.html
Cheers!
There are many pieces of the linear mixed models output that are identical to those of any linear model–regression coefficients, F tests, means.
But there is also a lot that is new, like intraclass correlations and information criteria. And a lot of output we’re used to seeing, like R squared, isn’t there anymore.
And because the model is more complicated, you may need to include in your paper more information about how you set up the model. For example, you usually need to say whether you included a random intercept or slope (and at which level) and which covariance structure you chose for the residuals.
It is not complicated at all:
1. Don't report p-values. They are crap!
2. Report the fixed effects estimates. These represent the best-guess average effects in the population. Try making quantitative statements:
"Overall the effect is strong/weak/negligible". That depends on your domain of research, of course.
3. Report the confidence limits. Make statements on uncertainty:
"The effect apprears to be is strong, but CIs are wide. We cannot exclude that it is much weaker or stronger"
"The effect is practically zero and we can say so with good certainty"
4. Report how variable the effect is between individuals by the random effects standard deviations:
"On average the effect is strong, but there sis considerable variation between subjects"
What works well for me is the following table structure:
parameter name | beta | lower-95 | upper-95 | random effect (SD)
If you are using an R implementation (lme4, rstanarm, brms), you can also extract the participant-level estimates and create spaghetti plots for 4.
https://www.researchgate.net/project/Book-New-statistics-for-the-design-researcher
Assalam alaikum Mr. Subina Saini
I think you could use SPSS program to analyze your mixed linear model from analyze tab.........> mixed model.........>linear , also the following reference my help you (Simultaneous optimal estimation in linear mixed models)
Mi-Xia Wu · Kai-Fun Yu · Aiyi Liu · Tie-Feng Ma
Metrika (2012) 75:471–489
Reporting Linear Mixed models can be tediously difficult if you do Not have basic foundation of statistics and in particular the random and fixed effects as basic requirement. dismantling the estimate outputs from those models depends on what kind of model you have run, what type of data, covariates and repeating and how those co-variates and predictors vary across the levels of other predictors. I have Not seen two statisticians follow the same method of interpreting and reporting the findings from the linear Mixed methods in the same way, why? because each analyses and models are unique, each model tells a different story and you should begin first by writing and understanding your own model story via literature review and doing exploratory data analysis, i.e., do Not rush to mixed models interpreting if you do NOT have those foundations.
Please read the pdf on this link, https://arxiv.org/ftp/arxiv/papers/1308/1308.5499.pdf
very simple to learn how to report GLMM
Can I ask a question? I am midway through a statistics assignment and looking at a 2 x2 mixed ANOVA. I really struggle with the concept of a one-tailed and two-tailed test. I believe the one I am exploring is a one-tailed because:
The researcher predicts that intelligence will be greater for a 2004 year group over a 1991 year group.
That is one-tailed because they predict the direction of the hypothesis?
Am I right or wrong?
Many thanks,
Ruth
Ruth Bennetts - Yes, you are correct. The exact same data analysis will be run, however, any significant findings can only be considered significant if they fall within the scope of the hypotheses. Additionally, the p-value must be halved, for any significance that is observed, when it comes to writing up your findings. The primary draw to a one-tailed hypothesis is that you massively increase your statistical power by effectively making more 'targeted'(/directional) observations.
Perhaps this may be helpful for you.
Article The anova to mixed model transition
- Badges
- Science topic