'lme4' is taking over development from nlme. Douglas Bates is the primary developer. Both have their pros and cons depending on what you want to do -- see http://glmm.wikidot.com/faq for a viewpoint. And while I recommend purchasing Julian Faraway's book "Extending the Linear Model with R" the updated chapters on mixed models are available as free downloads at http://www.maths.bath.ac.uk/~jjf23/ELM/

'lme4' is taking over development from nlme. Douglas Bates is the primary developer. Both have their pros and cons depending on what you want to do -- see http://glmm.wikidot.com/faq for a viewpoint. And while I recommend purchasing Julian Faraway's book "Extending the Linear Model with R" the updated chapters on mixed models are available as free downloads at http://www.maths.bath.ac.uk/~jjf23/ELM/

Please check chapter 5 Models for Repeated-Measures Data: The Rat Brain Example

in the following book, there are good examples using R, SPSS, SAS, HLM, and Stata to do repeated measure study.

(Brady T. West, Kathleen B. Welch, Andrzej T. Galecki. (2007). Linear Mixed Models: A Practical Guide Using Statistical Software. Chapman & Hall/CRC, Taylor & Francis Group)

See http://www.uvm.edu/~dhowell/StatPages/More_Stuff/Mixed-Models-Repeated/Mixed-Models-for-Repeated-Measures1.html I don't know if you will have problems with few subjects, or with fewer subjects than variables, but it is worth a try. There is R code at the end of the document.

Dave Howell

The book Mixed-Effects Models in S and S-PLUS by Pinheiro and Bates (http://www.springer.com/statistics/statistical+theory+and+methods/book/978-1-4419-0317-4) is an excellent reference if you really want to learn about the theory and practice of working with these models. It's directly related to the nlme package with many detailed examples. While I agree that the lme4 package has some nice added features, I've worked with both of them and it's not hard to transition between the two.

lme or lme4 package and the book mentionned above are the way to go.

If the linear mixed model is with a simple random effect with standard correlation matrix and validation of assumptions, the following code is all that is needed:

lme(response~factor1+factor2,random=~1|subjects,data=data)->results

anova(results)

For multiple comparisons, it is easier to use the package multcomp in my experience:

library(multcomp)

summary(glht(results,linfct=mcp(factor1=”Tukey”)))

Good luck!

Thanks everyone for your thoughtful suggestions! I'll take a look at your suggestions over the next few days and run some code on some pilot data. (The data I mention above is still being coded, so I want to improve the analysis.)

Thanks again....I'll likely have additional questions as I step through some of these examples in the next few days.

If the data are normal, then the lme model given by Hugues Beaufrere is common to model repeated measures - in lme4 you'd run

m1=lme(response~factor1+factor2,random=~1|subjects,data=data)

anova(m1)

summary(m1)

m2=lmer(response~factor1+factor2 + (1|subjects),data=data)

anova(m2)

summary(m2)

There are a few things you should know however. The anova command that Hugues shared yields Type I sums of squares and F stats with the associated p-values. Many statisticians (although not developers of nlme like Doug Bates) recommend that you use Type III sums of squares and the associated p-values, implement that nlme by

anova(m1,type="marginal")

And multcomp does not (as far as I know) appropriately deal with tests of fixed effects when an interaction is in the model. That is to say, for a model such as:

m1.int=lme(response~factor1*factor2,random=~1|subjects,data=data)

So, to test whether your fixed effects of interest have a significant effect on the response, an alternative approach is to perform a likelihood ratio test of your fixed effects of interest. For example, to test the sigfnificance of factor1:

m3=lme(response~factor2,random=~1|subjects,data=data)

anova(update(m3,method="ML),update(m1,method="ML"))

or to test the significance of the interaction

anova(update(m1,method="ML),update(m1.int,method="ML"))

Note that lme4 does not even provide p-values, which is why Hugues suggested the package multcomp in the first place, specifically Tukeys multiple comparisons, which is absolutely appropriate for fixed effects models. The problem with using multcomp or anova( ) F tests of models which include random effects is that they use degrees of freedom which are incorrect (google "Doug Bates p-values and all that" for an interesting discussion). The issue is that when the data are normal, then models with only fixed effects have been worked out precisely, and so the (denominator) degrees of freedom for the associated F tests are well known and calculable. But once random effects (to model repeated measures) are introduced, no one can tell you what degrees of freedom to use for F tests, t-tests, or Tukey tests inc. multcomp.

Al Parker

Yes thanks for pointing out that type III SS are more appropriate in this circumstance.

I think p-values can be obtained using lmer objects using the Kenward-Roger method of approximation of denominator degrees of freedom. The full model is compared with the reduced model obtained through lmer and then:

library(pbkrtest)

KRmodcomp(full.model,reduced.model)

The good thing with lmer is that you can model more complicated random effects such as nested and factorial random effect which are more cumbersome to do with lme.

Finally, contrast statements can be used in these models I believe:

library(gmodels)

fit.contrast(results,"trt",c(0,1,-1))

It might also be interesting to take a look at Harald Baayen's book "Analyzing Linguistic Data: A practical introduction to Statistics using R". Although mainly aimed at psycholinguists, I have found his introduction to the often complex materials at the right level of complexity - discussing conceptual issues in depth but leaving the low level mathematics to the experts.

This book also introduces the package languageR, which provides the function pvals.fnc. Since it is impossible (or difficult?) to calculate df when random effects are introduced (see the above discussion), this function uses Markov chain Monte Carlo simulations to get confidence intervals (and p-values). This package is often used in psychological research, and looking at the papers that reference the above mentioned book (or the paper by Baayen, Davidson & Bates, 2008) might give some interesting pointers.

Statistical inference with asymptotic tests is tricky in mixed-effects models, as the degrees of freedom are unknown. If that is your concern, you may want to try MCMCglmm. There's a great tutorial written by the author of the package. It's Bayesian, which means you get true credibility intervals.

In response to Hugues suggestion to use a Kenward-Roger df for lme models, I am as skeptical as Doug Bates when he says that there is no reason to think that we can find the right "fudge factor" to adjust the degrees of freedom to get a valid test (in http://rwiki.sciviews.org/doku.php?id=guides:lmer-tests) .

Here is the link to Doug Bate's reasoning for not providing p-values in lme4 at https://stat.ethz.ch/pipermail/r-help/2006-May/094765.html.

Martin recently suggested using MCMC to estimate p-values, Doug Bates shows how to use MCMC for this purpose in http://rwiki.sciviews.org/doku.php?id=guides:lmer-tests, but I am afraid it is not very straightforward to follow. Perhaps Martin can share a link to the great tutorial written by MCMCglmm's author?

And a very interesting new contribution, advocated by Andrew Gelman, is the use of 'Bayesian lme modeling' in the recent R package, blme.

Al

The tutorial is delivered as a vignette to the package, see below. I am excited to see that Gelman finally made his own "mer" implementation. I can also highly recommend his book Data Analysis using regression and multilevel/hierarchical models. It features numerous examples on using lme4.

http://cran.r-project.org/web/packages/MCMCglmm/

Oh, I misread "Gelman advocates" as "Gelman introduced". Just tried the new package. Works like a charm.

Thanks everyone! I'll actually be working with the real data soon in the next week or two, so all of this has been helpful. I'm sure this is going to be an ongoing thread useful to many people....I'll post questions and updates here too as I work through any issues. Such a knowledgeable community!

First of all, we have to clarify the scale of your dependent variable. Them, we can define a specific probability density function. Maybe the normal distribution is not the correct density function to represent your dependent variable measured.

So, can you provide more information of your research? Like:

a) dependent variable measured (discrete or continuous);

b) fixed effects investigated;

c) The objective of the experiment;

d) the frequency of measurements on the experimental unit (Alumni).

I do not know how to use the R software but I can design a SAS runtime to deal with the correlation between measurements on the same student.

Best regards.

http://www.bodo-winter.net/tutorial/bw_LME_tutorial.pdf

I recommand this article including easy example.

Michael Crawley's "The R Book" and Zuur's "Mixed Effects models..." texts have some good examples for ways to specify your repeated-measures model. Both texts are likely available for free download in your university library has an agreement with Springer publishing.

One thing that people sometimes have a difficult time getting their heads around is the specification of the random effect structure in the lmer function (package "lme4"). The time variable should most definitely be specified appropriately in the random effects structure. On top of this, if you're also wanting to estimate a parameter for the time variable, you should also include it in the fixed effect component of your model (complete with any necessary interactions).

In my opinion, I would worry less about p-values and just present some sort of model selection approach. They will likely allow the same inference in the end, but you can go both routes and see for yourself.

I do, however, agree that Bayesian methods will likely give between parameter estimates. That said, I've personally seen few cases in which the parameter estimates or error estimates are drastically different, especially if the response data can't be approximated by some simple distribution (e.g. Poisson). Others, of course, may have different perspectives. Generalized Estimating Equations also may be good for such longitudinal data, but I'm not sure if your data will be entirely amenable to this. They tend to be more robust that GLMM to deviations from the assumptions surrounding the specified error structure (Poisson, etc.) and can be much more friendly, in general, for longitudinal sampled data.

Sounds like a great data set you have. Keep us informed.

Here is a simple tutorial of SPSS on the topic

Hi Krista, we applied linear mixed model to repeated measures in this paper, http://link.springer.com/content/pdf/10.1007%2Fs00223-011-9518-9.pdf; have a look to the Statistical Methods and might be helpful!

BW

Michael

From your description it sounds like you want to do multiple comparisons among 10 different conditions that you're treating as levels of one variable (virtual world). If that's all you really want to do then taking a preliminary step of doing a MLM or ANOVA does nothing but obfuscate that this is just multiple comparisons (unless you have an extraordinarily complex model). It sounds like pairwise.t.test will serve just fine.

For some reason psych students come out of stats classes believing that you do an ANOVA and then you do all of the t-tests to understand the ANOVA. You don't. Now, in your case it sounds like you really wanted to do all of the t-tests right from the start. If that's the case then the ANOVA or MLM is pointless additional work. (FYI, if one really had a pattern or expectation from the ANOVA and didn't need all the t-test then it was a good idea to do first, but never do all of the t-tests after.)

As an aside, one thing that's been missed in the advice so far, given that you're coming from ANOVAs, is data aggregation. In repeated measures you often have multiple measures per condition per subject and you get the averages of those to do an ANOVA so that you have one measure per condition per subject. With MLM you do it on the raw data. You do not aggregate because you're modelling each data point (sort of), not each condition.

I am not sure that I understand: How manny groups are the students in? 10?.

What you must first do is simply subtract the baseline score from the endpoint-and test the difference, with t-tests or anova-that would give you an idea whether differences are far from giving significant results. But a simpel Kolmogorov-Smirnof

test should be preformed-to verify whether data are normali distributed. (this test is found in SPSS under nonparametric test( one sample test)). If not the linearmixed model should be dropped as it assumes interval level data.

Why is N so low? Could you not include more students?

You should never do tests to verify normality because they don't verify normality. They can only verify if things are not normal and even then only in a way that's not terribly relevant for assumptions of normality. They test against exactly normal and can fail even if the data are quite normal. Try the following in R to see my point...

y

Re: John Christie's first post above, please define MLM, googling it does not help very much, it is not 'standard' statistical parlance. Maybe you mean Maximum Likelihood Method.

More importantly, I think it should be pointed out to anyone who is interested in implementing an lme model that many statisticians would disagree with two of John's broad assertions:

1) using multiple t-tests (implemented via R's pairwise.t.tests) "will work serve just fine" for repeated measures data.

If there are repeated measures from the same subject, multiple t-tests (even after correcting to maintain small Type I error rates or a small FDR) do not account for the correlations amongst these repeated measures and so are an incorrect approach, that is why Kristen had the foresight to use a mixed effects model to begin with. If the modeler feels that the fit of the lme model is inappropriate (by looking at the residuals as John correctly points out), then one may want to fit multiple lme models (and correct for Type I error/FDR).

2) "In repeated measures you often have multiple measures per condition per subject and you get the averages of those to do an ANOVA so that you have one measure per condition per subject."

Averaging the data per subject is feasible (and I agree with John is a good approach) only if the repeated measures are from the same level of the fixed effect(s) (e.g., multiple measurements from the same virtual world). However, if each subject tested each level of the a fixed effect (i.e., each virtual world) once, and if you want to compare mean responses across the levels (i.e., virtual worlds), you can not average each subjects responses.

And how long does Research Gate keep a thread open for? Leaving them open for so long, in my opinion, obfuscates the discussion to something that does not resemble very much the question of the original poster.

Cheers,

Al

Al, I, of course meant paired tests for the t-tests. And my point wasn't that they're an adequate substitute for an overarching model. It was that if you're going to do them anyway, as the questioner says they will, an overarching model is pointless.

I'm not sure how the averaging in your point 2 disagrees with me. I just didn't flesh out the details and I was only talking about multiple measures per condition per subject.

And you're right, the threads should eventually close.

John,

I tried to find a way to send you a personal message off of this thread, but I don't know how, so I am adding one here. Your paired t-test approach would work fine, after aggregating the data for each subject per test condition, and then pairing these aggregates by subject. Of course this only works when there is only a single random effect (subject). But I don't understand why you say 'if you're going to do them anyway, as the questioner says they will, an overarching model is pointless.' If the assumption of equal variance amongst groups is appropriate, the 'overarching model' has more power since it provides a better estimate of the (within subject) common variance and higher df for this variance. Multiple comparisons via t-tests estimate the variance for each subset, which is sweet when the between group variances are heterogeneous, but lose you power otherwise. Re: 'point 2', I read that you were trying to make a distinction between ANOVA and "MLM," with only "MLM" being able to use 'raw data.' There is no requirement to aggregate the data for each subject when applying ANOVA with mixed effects (i.e., lme).

My concern is whether multiple comparisons will be useful with 10 virtual worlds to compare. Even if true differences may not exist, you could find differences by chance alone. I think comparing the 95% confidence intervals will be more straightforward.

I don't know if my partial answer get sent, so here it is in abbreviated form.

For an example using mixed models with repeated measures, see http://www.uvm.edu/~dhowell/StatPages/More_Stuff/Mixed-Models-Repeated/Mixed-Models-for-Repeated-Measures1.html. (Don't worry about part 2.) This gives code for lme and lmer at the bottom. You just translate "time" to "world" and drop reference to groups.

BUT, I agree with John Christie that the overall anova will only tell you that times (worlds) are different. Presumably you already know that. In addition, I would hope that you don't really want to compare every world with every other world. Presumably there are theoretical reasons to look at groups of worlds. If so, do that instead.

Gudeta's suggestion about confidence limits is a good one, and I recommend that you calculate them.

Attached below is a very nice paper wrtten by Dr. Singer. I think this is one of the best papers for anyone who would like to learn repeated measure modeling. It doesn't matter if you use SAS or not.

http://isites.harvard.edu/fs/docs/icb.topic1140782.files/S98.pdf

Hi, we used a linear mixed model in the following paper: http://www.plosone.org/article/authors/info%3Adoi%2F10.1371%2Fjournal.pone.0067344;jsessionid=3AABFB2A9D2158563FE69F74C71F1158

This can be downloaded free of charge.

Have fun!

You only need to use a linear mixed model if you have correlated observations to the extent that the matrix will not invert. However, your number of subjects sounds too low. I recommend getting more subjects then creating a priori hypotheses.

packages ‘rptR’(Schielzeth and Nakagawa 2013) of R will be very helpful.

This book is great to show you how to do it in SPSS. Great screenshots.

I found this tutorial very helpful (in addition to Bodo Winter's recommended above). https://m-clark.github.io/mixed-growth-comparison/#add_a_cluster_level_covariate