Hi, I am currently researching visual word processing in second language acquisition. I have four independent variables: participant's native language background (3 levels), participant's proficiency level (2 levels), SOA condition (3 levels), and word priming type (5 levels). The independent variable is the reaction time. The formula for the Full model is like: log(RT)~prime type*SOA*Native*Proficiency+(1|Participant)+(1|Item). For each participant, they were assigned to one SOA condition, and had 15 items for each prime type (75 items in total). However, my sample size seems relatively small, with about 15 participants for each Native*Proficiency group(e.g., 15 Intermediate Japanese learners).
Can anyone give me some advice on at least how many participants should be included in each group of a linear mixed effects model? Attached reference papers or books would be greatly appreciated.
Dear Mingyang Yu,
this is a classic question in statistics in SLA (and other fields). The answer is that the more participants you have, the more likely you are to find significant results (and vice versa). How many participants you would (quote-unquote) need as a minimum to find a significant result depends on the effect size of the relation you are investigating (say: effect of condition on word recognition). Larger effects can be detected with less participants (and vice versa). Since the effect sizes are usually unknown before a study, the advice is the one above: The more participants you have, the more likely you are to find significant results.
This paper is probably also helfpul: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4148275/
The question is, what (which effects) you would like to model with the linear mixed effects (LME) model? And - as the response above states - what size of the effects do you expect (or are likely)? You say your sample size is "relatively small" - what does that mean precisely?
It seems to me that your factors SOA condition and and priming type are within-subject. So the groups size is made up of the two predictors native language and proficiency. While the first might be a factor, the second could be a continuous predictor. Someone told me once that it is a good rule of thumb to have at least 15 observations for each design cell (I can't give any references). So it would be good to have at least 15 participants in each of your native language groups. With regard to the proficiency level, here it depends if it is continuous or categorical. In the first case, the 15 participants in each L1 group might be enough - as long as the observed values represent a large enough range of the possible values (or the range of interest) and if the effect is really linear (probably not). If proficiency is a factor, then it depends on how many levels it has. If it has three (high, mid, low) then - according to the advice above, you should have 15 participants for each level (amounting to 45 participants in each language group!). If proficiency in the LME model is modelled independently (i.e. not nested under L1), one could have half the number because the 15 participants of each proficiency level could be distributed across the different L1 groups.
I don't have any references with regard to sample size but these papers might be useful in general:
- Meteyard, L., & Davies, R. A. I. (2020). Best practice guidance for linear mixed-effects models in psychological science. Journal of Memory and Language, 112, 104092. https://doi.org/10.1016/j.jml.2020.104092
- Linck, J. A., & Cunnings, I. (2015). The utility and application of mixed-effects models in second language research. Language Learning, 65(S1), 185-207. https://doi.org/10.1111/lang.12117
- Kumle, L., Võ, M. L.-H., & Draschkow, D. (2021). Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R. Behavior Research Methods. https://doi.org/10.3758/s13428-021-01546-0
- Bates, D. M., Kliegl, R., Vasishth, S., & Baayen, H. (2018). Parsimonious mixed models. arXiv, 1506.04967v2. https://arxiv.org/abs/1506.04967
PS: If you expect that you can't get data from as many participants as you would like, you might compensate a little by getting more observations (test more trials) with each one of them.
Andreas Wirag Thank you so much for your response. The link is very useful.
Tom Fritzsche Thank you so much for your response. I have added more details to my question. The papers you shared are very useful.
Hi. Mingyang. Happy to reach out. You may kindly check the citations below for a more concrete estimation of the power analysis and sample size. For example, at least 1,600 observations for each condition in the second reference.
Westfall, J, Kenny, D A, & Judd, C M (2014) Statistical power and optimal design in experiments in
which samples of participants respond to samples of stimuli. JEP: General, 143(5), 2020–2045.
Brysbaert, M, & Stevens, M (2018) Power analysis and effect size in mixed effects models: A tutorial. Journal of Cognition, 1(1): 9, 1–20.
Yushu Wu Hi! Yushu! Thank you so much! The references are very helpful.
hi, I would like to thank you very much for your distinguished contributions, which helped me to progress well in my research axes, in this case language sciences. Sincerely yours.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Applied Statistics
- Survey Design
- Questionnaire