That's a bit of a debated point. While you can calculate sample sizes for univariable logistic regression, most studies will require a multivariable analysis.
There was a very influential paper published in the 1990s by Peduzzi et al (1996) based on simulation studies which concluded that for logistic regression you needed ten events (not patients) per predictor variable if you were calculating a multivariate model.
More recently, bigger and more comprehensive simulation studies have cast doubt on this hard-and-fast rule. Vittinghoff and McCulloch (2007), in a very widely-cited paper, concluded that “problems are fairly frequent with 2–4 events per predictor variable, uncommon with 5–9 events per predictor variable, and still observed with 10–16 events per predictor variable. Cox models appear to be slightly more susceptible than logistic. The worst instances of each problem were not severe with 5–9 events per predictor variable and usually comparable to those with 10–16 events per predictor variable.”
In other words, with between 5 and 9 events per predictor variable, their models performed more or less as well as models with 10-16 events per variable
However, since then further simulation studies where prediction models are validated against new datasets tend to confirm that 10 events per variable is a minimum requirement (see Wynants 2015) for logistic regression. These studies are important because they are concerned with the generalisability of findings.
Based on current research, the sample should have at least 5 events per predictor variable ideally 10. Sample sizes will need to be larger than this if you are performing a multivariate analysis with predictor variables that have low prevalences. In this case, you may require up to 20 events per variable, and should probably read the paper by Ogundimu et al.
A reading list
- Courvoisier, D.S. et al., 2011. Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure. Journal of Clinical Epidemiology, 64(9), pp.993–1000.
- Kocak M, Onar-Thomas A. A Simulation-Based Evaluation of the Asymptotic Power Formulas for Cox Models in Small Sample Cases. The American Statistician. 2012 Aug 1;66(3):173-9.
- Ogundimu EO, Altman DG, Collins GS. Adequate sample size for developing prediction models is not simply related to events per variable. Journal of Clinical Epidemiology. Elsevier Inc; 2016 Aug 1;76(C):175–82.
- Peduzzi, P. et al., 1996. A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 49(12), pp.1373–1379.
- Vittinghoff, E. & McCulloch, C.E., 2007. Relaxing the rule of ten events per variable in logistic and Cox regression. American Journal of Epidemiology, 165(6), pp.710–718.
- Wynants L, Bouwmeester W, Moons KGM, Moerbeek M, Timmerman D, Van Huffel S, et al. A simulation study of sample size demonstrated the importance of the number of events per variable to develop prediction models in clustered data. Journal of Clinical Epidemiology. Elsevier Inc; 2015 Dec 1;68(12):1406–14.
http://www.intellectusstatistics.com/sample-size-write-up/sample-size-multinomial-logistic-regression/
Article Sample size determination for logistic regression revisited
That's a bit of a debated point. While you can calculate sample sizes for univariable logistic regression, most studies will require a multivariable analysis.
There was a very influential paper published in the 1990s by Peduzzi et al (1996) based on simulation studies which concluded that for logistic regression you needed ten events (not patients) per predictor variable if you were calculating a multivariate model.
More recently, bigger and more comprehensive simulation studies have cast doubt on this hard-and-fast rule. Vittinghoff and McCulloch (2007), in a very widely-cited paper, concluded that “problems are fairly frequent with 2–4 events per predictor variable, uncommon with 5–9 events per predictor variable, and still observed with 10–16 events per predictor variable. Cox models appear to be slightly more susceptible than logistic. The worst instances of each problem were not severe with 5–9 events per predictor variable and usually comparable to those with 10–16 events per predictor variable.”
In other words, with between 5 and 9 events per predictor variable, their models performed more or less as well as models with 10-16 events per variable
However, since then further simulation studies where prediction models are validated against new datasets tend to confirm that 10 events per variable is a minimum requirement (see Wynants 2015) for logistic regression. These studies are important because they are concerned with the generalisability of findings.
Based on current research, the sample should have at least 5 events per predictor variable ideally 10. Sample sizes will need to be larger than this if you are performing a multivariate analysis with predictor variables that have low prevalences. In this case, you may require up to 20 events per variable, and should probably read the paper by Ogundimu et al.
A reading list
- Courvoisier, D.S. et al., 2011. Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure. Journal of Clinical Epidemiology, 64(9), pp.993–1000.
- Kocak M, Onar-Thomas A. A Simulation-Based Evaluation of the Asymptotic Power Formulas for Cox Models in Small Sample Cases. The American Statistician. 2012 Aug 1;66(3):173-9.
- Ogundimu EO, Altman DG, Collins GS. Adequate sample size for developing prediction models is not simply related to events per variable. Journal of Clinical Epidemiology. Elsevier Inc; 2016 Aug 1;76(C):175–82.
- Peduzzi, P. et al., 1996. A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 49(12), pp.1373–1379.
- Vittinghoff, E. & McCulloch, C.E., 2007. Relaxing the rule of ten events per variable in logistic and Cox regression. American Journal of Epidemiology, 165(6), pp.710–718.
- Wynants L, Bouwmeester W, Moons KGM, Moerbeek M, Timmerman D, Van Huffel S, et al. A simulation study of sample size demonstrated the importance of the number of events per variable to develop prediction models in clustered data. Journal of Clinical Epidemiology. Elsevier Inc; 2015 Dec 1;68(12):1406–14.
Great post, Ronán. But if I am not mistaken, the rules of thumb you discuss are focused on avoidance of over-fitting. Whether or not one will have sufficient power to detect some minimally important effect size is a separate question. HTH.
p.s. - I'd add this nice article by Mike Babyak to your reading list.
https://people.duke.edu/~mababyak/papers/babyakregression.pdf
An important distinction, Bruce – thank you for pointing it out. It makes me realise that sample sizes are determined by two not-necessarily-convergent constraints : statistical power and generalisability. As you say, my comment focusses on the generalisability of the model.
And thanks – I collect ‘the only reference you need to read on…’ references, and I suspect that the Bayak paper is the one stop shop here.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size