I am in the middle of designing a study with three/four levels of analysis and I would appreciate if you could give me an advice concerning the sufficient sample size at each given level. Are there any recommendations in literature?
Hi, I think there are theories on this problem, and one creteria for this is the ratio of item to case, which should be at least 1 to 3, and ldeally 1 to 5 or more, and there should be at least 100 cases when the model contains five or less constructs, or 150 -300 with seven or fewer, or 500 with more than seven constructs, according to Hair, J. F., Black, W. C., Babin, B. J., & Abderson, R. E. (2011). Multivariate data analysis (Seventh ed.). My suggestion is 500 is better than fewer cases. Please refer to Hair et. al's book or other references.
I don't know of any small sample studies that have gone beyond two-levels. In two-level models, without using any small sample correction (e.g., Kenward-Roger), with continuous outcomes, about 20 units are needed at the highest level to obtain unbiased estimates (power will be quite low though). With discrete outcomes, about 50 units are needed at the highest level with at least 5 observations per cluster.
A relevant paper might be McNeish & Stapleton (2014, Educational Psychology Review) which is a review paper and contains several references that may be helpful for the specifics of your situation. A full-text is available on my page should this be of interest.
Hi here are some references:
Scherbaum, C. A., & Ferreter, J. M. (2009). Estimating statistical power and required sample sizes for organizational research using multilevel modeling. Organizational Research Methods, 12(2), 347–367.
Maas, C. J. M., & Hox, J. J. (2004). Robustness issues in multilevel regression analysis. Statistica Neerlandica, 58(2), 127–137. doi:10.1046/j.0039-0402.2003.00252.x
Best,
Holger
This is a widely discussed statistical question. The links below may provide some more information
http://www.stats.ox.ac.uk/~snijders/PowerSampleSizeMultilevel.pdf
http://joophox.net/publist/methodology05.pdf
https://www.amstat.org/sections/srms/proceedings/y2010/Files/308112_60089.pdf
As already pointed out by one colleague, the key, I guess, is to have sufficient sample size at the top level of the hierarchy. A related issue is the optimality criterion to use to derive an optimal sample size - a decision that needs to be made considering the end use of the results from multilevel modelling.
A lot depends on your target of inference. Taking the example of pupils (level 1) in schools (level 2) - if you want to infer about the between -school variance than you need plenty of schools. If you want to also infer about a particular school you also need plenty of pupils in that school. This issue most obviously comes up with people in households - you can do a multilevel analysis on this if you have plenty of households but you are unlikely to be be able to say anything about any specific household as the number of members is likely to be too low for precise inference. In a similar manner multilevel models can be used to analyze data for twins where there can only be 2 people (level 1) in the twin pair (level 2).
At the other end of the scale where you have thousands of people (at level 1) in a country (at level 2)- a simulation study has shown ( and using MCMC Bayesian methods) that the method works quite well with only a dozen or so countries even when there is a discrete outcome and you are estimating cross-level interactions
How Many Countries for Multilevel Modeling? A Comparison of Frequentist and Bayesian Approaches http://onlinelibrary.wiley.com/doi/10.1111/ajps.12001/abstract
For general orientation on this - see
http://www.bristol.ac.uk/cmm/software/mlwin/mlwin-resources.html#datadesign
You can use the following software to do the power calculations
http://www.bristol.ac.uk/cmm/software/mlpowsim/
and
Optimal Design for Longitudinal and Multilevel Research Documentation for the “Optimal Design” Software
http://www.stat.cmu.edu/~brian/463/week14/od/od-manual-20111016-v300.pdf
http://sitemaker.umich.edu/group-based/optimal_design_software
Not the most recent paper, but interesting.
Article Sufficient Sample Sizes for Multilevel Modeling
- Badges
- Science topic
- Similar topics
- Psychology