When the number of free parameters in your regression starts to get close to the number of data points then your equation can predict anything perfectly. This applies to both random and fixed effects, so I think part of the answer to your question is that you can have too many intercepts. However, I do not think that is the case here. I think what’s going on here is that when you add a random effect you are making the model ask the question: does the variation due to my fixed effect (the independent variables) exceed the variation due to the random effects. In your case, it sounds like you may have an effect that is not significant by word (even though it is significant by subjects). That means that while your effect generally holds up in all of your talkers, but it may not hold up in all of your words. To think about this in an ANOVA framework you should read Herb Clark’s classic 1973 paper on item analyses – the logic still applies even though you are doing simultaneous subject and item analyses. Clark argues that both types of statistics are equally important in psycholinguistics and I agree. One caveat though, when you say a word only occurs once, do you mean it only occurs once per speaker? If so that’s fine. But if it literally only occurs only once (only one speaker said it), it may be difficult to estimate subject and item random effects simultaneously because the effects are highly correlated (at least for that word). In this case you might consider dropping any items that only appear for a small number of subjects and repeating your analysis, or you might consider doing separate models with subject and item random effects (but with all the data).

I think you can have several intercept random effects. I suggest you read this paper: Baayen, R. H., Davidson, D. J. & Bates, D.M. (2008) Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language 59, 390-412.

I don’t know in detail your data but a first point of view I think these variables not significant. I would like to encourage to ask this question to R-sig-mixed-models --. Its a forum about mixed models.

Hope it helps.

Martí Casals.

It is possible, as Marti said, but the question is how this affects your results. Usually not much attention is paid to the intercepts, but including them does not hurt. You will have non-significant values for them.

In general, the fact that after a traditional regression analysis you have statistically significant independent variables does not ensure independence: when dependence is non monotonous. A trivial example: let X be a random variable with pmf or pdf symmetric around zero, and let Y=|X|. Using standard regression analysis you will get that (X,Y) are statistically significant independent variables!

When the number of free parameters in your regression starts to get close to the number of data points then your equation can predict anything perfectly. This applies to both random and fixed effects, so I think part of the answer to your question is that you can have too many intercepts. However, I do not think that is the case here. I think what’s going on here is that when you add a random effect you are making the model ask the question: does the variation due to my fixed effect (the independent variables) exceed the variation due to the random effects. In your case, it sounds like you may have an effect that is not significant by word (even though it is significant by subjects). That means that while your effect generally holds up in all of your talkers, but it may not hold up in all of your words. To think about this in an ANOVA framework you should read Herb Clark’s classic 1973 paper on item analyses – the logic still applies even though you are doing simultaneous subject and item analyses. Clark argues that both types of statistics are equally important in psycholinguistics and I agree. One caveat though, when you say a word only occurs once, do you mean it only occurs once per speaker? If so that’s fine. But if it literally only occurs only once (only one speaker said it), it may be difficult to estimate subject and item random effects simultaneously because the effects are highly correlated (at least for that word). In this case you might consider dropping any items that only appear for a small number of subjects and repeating your analysis, or you might consider doing separate models with subject and item random effects (but with all the data).

As Bob already indicated, it would be good to have a more precise description of your data. Generally, random effects modelling splits up the residual error term in several components. But, this is only possible if you have repeated measure on the random factor, which implies in your case that items must be measured repeatedly, in the sense of every item must be encountered by multiple participants.

Furthermore, I wonder how you have created your random effects structure. Did you use any model tests to verify that a random effect is valid.

A special situation where a fixed effect can disappear if you add the random effect, is when your explanatory variable is associated to the random effect. I am not a linguist, but the situation could for example happen, if your experimental treatment is the frequency of words, such that you would have low frequency condition and high frequency condition. What you actually do, is use the natural variation in your material sample as sort of a pseudo-treatment. Adding a material level random effect would consume exactly this variation and make your fixed effect disappear.

As final thought, but I am not sure about this, in mixed effects models so-called shrinkage happens. This basically means that group means are pulled closer towards each other (the grand mean, respectively). Gelman, Hill Yajima (2012) even argue that because of shrinkage we do not even have to adjust the family-wise error when doing multiple comparisons. As I said, I am not sure about this, but perhaps adding more random effects also increases shrinkage. Overall, shrinkage is a desirable effect as it prevents over-fitting.

Gelman, A., Hill, J., & Yajima, M. (2012). Why We (Usually) Don’t Have to Worry About Multiple Comparisons. Journal of Research on Educational Effectiveness, 5(2), 189–211. doi:10.1080/19345747.2011.618213

When I say "some of which occur only once". I mean that the word only occurs one time in the entire corpus, that is, one time in one speaker.

I am using an R script created by Daniel Ezra Johnson called Rbrul (http://www.danielezrajohnson.com/Rbrul_manual.html). It performs step up/down regression analyses. This from the manual: "The deviance is -2 times the log-likelihood. If two logistic models are nested -- fit to the same data, and one model is a subset or 'special case' of the other -- then the difference in deviance can be tested against a chi-squared distribution with degrees of freedom equal to the difference in the number of parameters of the two models. This is called a likelihood-ratio test. For example, if we try collapsing two factors into one within a factor group, this represents one degree of freedom. If the difference in deviance is greater than 3.84, then the simpler model can be rejected with p < 0.05. The R command to obtain a p-value where x is the value of chi-squared (difference in deviance between two models) and df is the degrees of freedom (difference in number of parameters between two models) is:"

Jeroen -- given that some words were only produced once by one subject, I don't think you can estimate subject and item effects simultaneously (though perhaps if only a few do you could get away with it -- you'd want to look at the distribution of counts by item and word). If I were in your shoes, I would first drop the item effects and analyze by subject, and then drop the subject effects and analyze by item. Both are necessary to establish that your effects generalize (to new subjects and new items) so don't feel you can get away with just the subject analyses because they "work" (significance does not equal a good model). The other option is to restrict your analysis to a subset of the items that do appear in multiple speakers.

More broadly though, it sounds like you don't have a lot of experience or training with regression or mixed models (please pardon me if that is not the case and I am jumping to conclusions, I hope I do not sound insulting or condescending -- this is meant in a very constructive way -- the way you described the X2 test made it feel like you may not have had training in this). Mixed models and logistic regression can be very tricky to use right and to evaluate, and there are often no way for external evaluators (e.g., reviewers, editors, readers of your paper) if they were done properly. For that reason it is VERY important to get it right as you can't count on reviewers or editors to catch any mistakes. In general, I don't think you can "hack" your way to a good mixed model, cobbling together scripts from the internet and suggestions on discussion boards. I would strongly suggest that if you don't have a strong background in this approach that you enlist the help of an expert to work with you and your data more directly. You'll learn a lot and you can be more confident in your findings (which of course is the goal).

The general problem with LMM is that the degrees of freedom cannot be determined right away. Bolker et al.(2009): "the effective number of parameters used by a random effect lies somewhere between 1 (i.e. a single standard deviation parameter) and N - 1 (i.e. one parameter for each additional level of the random effect". They suggest using MCMC instead of asymptotic tests to determine the model fit.

Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., & White, J.-S. S. (2009). Generalized linear mixed models: a practical guide for ecology and evolution. Trends in ecology & evolution, 24(3), 127–35. doi:10.1016/j.tree.2008.10.008

Bob - You are right, I haven't enjoyed any form of training on this. The script is not just "from the internet" though, it was designed by Dan Johnson to make R more widely usable to linguists. It mimicks the functionality of David Sankoff's VARBRUL and extends and improves it in various ways. But this does not change the fact, as you point out, that I have no idea what I am doing :).

Thanks for being good natured about it. I think you're on the right track, and the question of random intercepts is a really important one for linguistic (and psycholingusitic research) because we are sampling both from the pool of possible subjects and from the pool of possible words, and of course we want to be able to generalize both across subjects and items (e.g., words), so figuring out how to deal with them is crucial. I hope that you are able to find some help in Antwerp.

BTW: Martin's answer about d.f.'s is right; however, it may not apply to binomial linking functions (logistic regression) -- I've seen it more as an issue with lienar models. Not sure which you are doing. Best of luck.