Before you do anything else, your approach to model building is flawed. Your model is much too complex, and you shouldn’t have used stepwise regression to build it. I also believe that your random effect structure is too simple.

Thanks for your response. Okay, yes, I realize it's a very complex model at the moment and I think there are too many interactions to interpret. What would be the good way to build it then - how can I find/select the most appropriate one?

Christina,

First and foremost, you should focus on only including interactions that are theoretically motivated or predicted by past research. I don't know how much data you have nor which predictors are between- vs. within-subject. For an EEG experiment, I'm guessing that your number of subjects is small which would severely limit the number of between-subject predictors/dfs you should consider. Age, for example, would be first on my list of variables to drop unless the study was designed to produce a large range of ages.

Second, you should only consider a very small number of alternative models. For example, if one theory hypothesizes that Group won't matter whereas another one does, then test two models - one with Group and another without. Or, if one says that Group will interact and another says not, then test two models - one with the key interaction and another without.

To interpret your "== 0" statements, you have to know how your categorical variables were coded and whether continuous predictors were centered. A test against 0 is often uninformative unless you use effect coding of categorical variables and centered continuous predictors.

I think you need a stat consultant or a lot of reading. Your questions and analysis are too involved to easily address via research gate.

Dear Michael,

the variables that I included are motivated by literature and some (such as Age) are covariates - the groups that I want to test differed somewhat in age, for example, but some literature says that age might have an influence on the ERP component on its own. But I believe Age is indeed a variable that I could drop, it doesn't seem to have too much influence. For differences between the groups themselves, there's not much literature yet, that's why I tried models both with or without group.

As for the data, there are 23 participants for each group, with different numbers of trials that are left after exclusion of false responses and artifact correction (but usually around 50-60 for each translation direction)

Okay, I guess I should center my continuous variables.

Understood... unfortunately, I don't have access to statistical consultation. Do you perhaps know of some good (and understandable) literature or links you could point me to that would be relevant for this case?

Thanks a lot, I really appreciate your help.

Also, regarding the random effect structure, I did want to add more to it (put in random slopes and also put in a random intercept for the Domain/the electrodes), but I always get errors that the model then fails to converge, so I assumed it was too complex for the number of observations I have or so

Christina, you will get grief from multilevel modelers if you only include intercepts. First, don't consider interactions as random effects - see https://arxiv.org/pdf/1506.04967.pdf. You likely lack the power to detect these variations. Second, focus on those effects that are most likely to vary across subjects. Hopefully you have prior research to guide that decision. I'm guessing that it's more likely that subjects will respond differently to your within-subject variables than your words will (although certainly include the item intercept as you've done).

Do you get errors about convergence or warnings? The first is a show stopper, the second may still be okay depending on the situation.

Dear Michael, thanks for the link, I am reading it now. I get warning messages such as this one (and it usually takes very long for the model to load/for this message to appear):

For the model: fm1 500.00ms)`~ Group+Direction+DomainElRow+LateralityLCR+scale(LexTALE, center=TRUE, scale=FALSE)+FamStimMeanAccGroup+ImStimMeanAccGroup+(1|DomainElRow)+(1|Subject)+(1|UnifiedWordNo)+(Age|Subject)+(FamStimMeanAccGroup|Subject)+(ImStimMeanAccGroup|Subject)+(FamStimMeanAccGroup|UnifiedWordNo)+(ImStimMeanAccGroup|UnifiedWordNo), data=GrandERPtable, REML=FALSE)

Warning messages:

1: In optwrap(optimizer, devfun, getStart(start, rho$lower, rho$pp), :

convergence code 1 from bobyqa: bobyqa -- maximum number of function evaluations exceeded

2: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, :

unable to evaluate scaled gradient

3: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, :

Model failed to converge: degenerate Hessian with 3 negative eigenvalues

How can I know whether it's still okay or not?

Okay, so I try to start out with a 'maximal' model that makes sense according to my predictions and that would still be interpretable (so including the factors I think are important and perhaps some pairwise, but not three-way interactions) and a richer random effect structure at least for subject. I use scaling for the continuous variables for which a "zero intercept" doesn't make sense. (Do I have to do effect coding of categorical variables even if there are just two levels?) And I also define a zero-correlation model (i.e. no interactions, just the main effects). Then I use PCA to check the dimensionality and use model comparison.

Did I understand correctly?

First, age|subject makes no sense - only within subject variables can play that role. Without knowing your full design, I can’t tell if your very complex random effect structure makes sense.

Okay, makes sense, I excluded the between-variables from the subject random slopes. I looked into the paper you suggested and into effect coding and I am currently working on the (weighted) effect coding as found here (since my data is not balanced)Article When size matters: advantages of weighted effect coding in o...

(As for the interactions, it is possible to do weighted interactions as well - see http://www.readcube.com/articles/10.1007/s00038-016-0902-0?author_access_token=gaEOw1aGoukpJEmm_3nHpPe4RwlQNchNByi7wbcMAY6a0AXYucd-I5s1D1ZtLilB_SuZNmkuGXfc1g_Qli45XpET8PwA0HF66o2nM1SFmF7nPxblY6g9wSAdE4jp77_8C3vv-kxsuScmDsxZeQ7qZw%3D%3D and

#https://www.youtube.com/watch?v=Fp4MV9ltfEI - but in my case I think I will stick to the unweighted interactions since the number of observations are balanced across my interacting factors.)

(I'm adding some details here in case it might be helpful for others)

I also have reaction time data for the same experiment with the same factors (except for the EEG specific ones) and so I decided to do this first since it is simpler (there is no Domain and Laterality Factor). I found this presentation, which exemplifies the paper on parsimonious models/rePCA and am following this procedure to select the best model.

I still get warning messages about convergence, though.

For example:

I started out with this:

contrasts(RTtable$Group)

Sorry for the lengthy answer

But I think I'm making some progress now :)

For RT, consider Gamma with a log link.

i recommend considering simpler re models and doing model comparison to simplify, if possible. Be systematic in your backward approach to re structure.

are the errors generated by rePCA? I wasn’t familiar with that command, so thanks for the tip.

I think I'd like to stick with the identity link, based on this paper: Article To transform or not to transform: Using Generalized Linear M...

Okay, sounds good. The warnings are generated both by the lmer function, the summary function, but not the rePCA. Sure, happy if it's helpful.

How many RE components did the rePCA suggest that you use? If fewer are suggested, then you should drop more random effects.

Regarding the identity link, it depends on what you assume about how RTs are combined, but it's your call.

Good luck!

The thing is, rePCA suggests dropping one random slope for Item, but then if I take it out and compare the models, the more complicated one is significantly better (in AIC, Log-likelihood, and Chi-squared, although BIC is better for the smaller one). Also, if I compare the correlation model with the zero-correlation model, the former is not significantly better, so I'm what this means, but I think it's probably because the model didn't converge correctly. So I'll try to simplify it more and see what happens.

It sounds like you're on the right track, Christina. Given that you're exploring a random effect structure, the more conservative BIC might be a better guide. Whatever you choose, do it consistently. :)