David

Thanks for coming back and providing useful further detail - many people do not.

I asked about the distribution of the dependent variable in case it was discrete; I would not worry about mild non -normality of the response. 

I know much more how the IGLS/RIGLS estimator works so I will say something about that. IGLS is a Full information max likelihood estimator and the random part is estimated as if the fixed part is known ( ie assumed not to have been estimated) and the fixed part is estimated as if the random part. is known. The RIGLS estimator is restricted max likelihood and estimates the random part given that the fixed part has been estimated, ( a bit like dividing by n-1 when estimating a variance around a mean).

 The correlation is derived not as a directly  estimated parameter but as the co-variance divided by the product of the square root of the associated variances. All three of these parameters can have uncertainty around them ( especially given 'only' 53 effective units at the high level). The problem can be much worse and in my MLwin manual there is an estimated correlation coefficient of +2.126!

https://www.researchgate.net/publication/260771330_Developing_multilevel_models_for_analysing_contextuality_heterogeneity_and_change_using_MLwiN_3_Volume_1_updated_March_2017_Volume_2_is_also_on_RGate

I improve on this by switching to MCMC estimation ( see the appendix to chapter 10) which has full uncertainty modelling (both the random and fixed part as seen as estimates) and the correlation of 2.126 becomes a more plausible 0.627 for the same data and the same model. STAN is a R package that  estimates multilevel models via MCMC  (http://mc-stan.org/ )

More generally, there is a recent trend to go maximal and fit very complex models. Thus, Barr et al. (2013) argue that not to include random slopes is anti-conservative. On the other hand, Bates et al. (2015) counter that analytical models should also be parsimonious, and fitting models with many random effects quickly multiplies the number of parameters to be estimated, particularly since random slopes are generally given co-variances as well as variances. Sometimes the data available will not be sufficient to estimate well such a model.

Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 10.1016/j.jml.2012.11.001. http://doi.org/10.1016/j.jml.2012.11.001Bates, Douglas, Martin Mächler, Ben Bolker, and Steve Walker. 2015. “Fitting Linear Mixed-Effects Models Using lme4.” Journal of Statistical Software 67(1): 1–48.

So back to your question - I suggest that you use a more appropriate MCMC estimation due to your relative small number of units at level 2 coupled with your small number of units at level 1 ( less than 2 on average per higher level unit if I have got it correct). At the very least you well see how sensitive your results are due to different estimation procedures. It would be useful to know what happens. The estimation of these complex models is in its infancy and we need all the experience we can get.

Best

Kelvyn

Book Developing multilevel models for analysing contextuality, he...

By what method have you estimated this (FIML versus REML versus MCMC ...) and by what software and how many higher level random terms are there and how many level 2 units are there and what is the assumed distribution for the response?

Thanks for the reply Kelvyn, here are the details you asked for:

I'm using the lmer function in the lme4 package in R. It uses REML.

There are only three random terms in the model: a random subject intercept, a random subject slope, and a random item intercept.

There are 53 subjects and 77 items.

The response variable has a slight positive skew.

I'm not an expert on this, but here are some vague intuitions:

Perfect correlations don't sound ecologically plausible for most kinds of data, so I doubt these are really reflecting something useful about the data; they're more likely to be reflecting the model's inability to estimate meaningful correlation parameters (perhaps because there's not enough data to model this).

w.r.t. what to do about that problem: I honestly don't have much idea. I am skeptical of suppressing random correlation components, because in my experience it sometimes has had weird side effects (for an example, see fig. 7 in http://www.mypolyuweb.hk/~sjpolit/pubs/Politzer-AhlesPiccinini_jphonms.pdf, the corresponding discussion is in section 5 of that ms). On the other hand, if you include the random correlation components, then it seems like the model tries to force the random effects to be highly correlated even if they aren't in reality (at least with the default settings I have been using), as shown in this example:

library(lme4)

library(languageR)

# model with random correlations

summary( m1

David

Thanks for coming back and providing useful further detail - many people do not.

I asked about the distribution of the dependent variable in case it was discrete; I would not worry about mild non -normality of the response. 

I know much more how the IGLS/RIGLS estimator works so I will say something about that. IGLS is a Full information max likelihood estimator and the random part is estimated as if the fixed part is known ( ie assumed not to have been estimated) and the fixed part is estimated as if the random part. is known. The RIGLS estimator is restricted max likelihood and estimates the random part given that the fixed part has been estimated, ( a bit like dividing by n-1 when estimating a variance around a mean).

 The correlation is derived not as a directly  estimated parameter but as the co-variance divided by the product of the square root of the associated variances. All three of these parameters can have uncertainty around them ( especially given 'only' 53 effective units at the high level). The problem can be much worse and in my MLwin manual there is an estimated correlation coefficient of +2.126!

https://www.researchgate.net/publication/260771330_Developing_multilevel_models_for_analysing_contextuality_heterogeneity_and_change_using_MLwiN_3_Volume_1_updated_March_2017_Volume_2_is_also_on_RGate

I improve on this by switching to MCMC estimation ( see the appendix to chapter 10) which has full uncertainty modelling (both the random and fixed part as seen as estimates) and the correlation of 2.126 becomes a more plausible 0.627 for the same data and the same model. STAN is a R package that  estimates multilevel models via MCMC  (http://mc-stan.org/ )

More generally, there is a recent trend to go maximal and fit very complex models. Thus, Barr et al. (2013) argue that not to include random slopes is anti-conservative. On the other hand, Bates et al. (2015) counter that analytical models should also be parsimonious, and fitting models with many random effects quickly multiplies the number of parameters to be estimated, particularly since random slopes are generally given co-variances as well as variances. Sometimes the data available will not be sufficient to estimate well such a model.

Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 10.1016/j.jml.2012.11.001. http://doi.org/10.1016/j.jml.2012.11.001Bates, Douglas, Martin Mächler, Ben Bolker, and Steve Walker. 2015. “Fitting Linear Mixed-Effects Models Using lme4.” Journal of Statistical Software 67(1): 1–48.

So back to your question - I suggest that you use a more appropriate MCMC estimation due to your relative small number of units at level 2 coupled with your small number of units at level 1 ( less than 2 on average per higher level unit if I have got it correct). At the very least you well see how sensitive your results are due to different estimation procedures. It would be useful to know what happens. The estimation of these complex models is in its infancy and we need all the experience we can get.

Best

Kelvyn

Book Developing multilevel models for analysing contextuality, he...

Kelvyn and Stephen, thank you both for the very helpful replies!

Hi David,

to check whether the model is overparametrised I would use the rePCA function of the RePsychLing package that is mentioned in the Bates et al. (2015) paper (page 5 middle): https://arxiv.org/abs/1506.04967v1

If it tells you that you have too many random components remove the random effects correlations by using || instead of | in the random part specification and see if this helps (check again with rePCA). If yes, you can include correlations one by one again and specify the model's random components as maximal as the data allows.

A practical example is discussed in Reinhold Kliegl's presentation notes from last year (http://www.uni-potsdam.de/attlis2016/assets/Kliegl_AttLis2016.pdf), for the rePCA function see p. 13.

HTH, Tom

Thanks Tom!