Seeking to clarify my understanding of determinants of minimal sample size for logistic regression.
I am looking at predictors of worry after injury. There are n=120 people in the ‘not worried’ group, and n=33 in the ‘worried’ group (I am aware some may take issue with categorising a continuous outcome!).
My understanding is that the sample size is largely dependent on the number of people in the smaller grp (so in this case, the ‘worried’ group. I also understand that the 10 events per predictor rule of thumb guide for logistic regression tends to underestimate the sample required.
Given this, do I have any grounds to run a logistic regression with even just 2 predictors (eg Age and Sex)? Or am I limited to t-tests/chi-squared as a first step / means of preliminary exploration of associations?
You can check the assumptions of model fitting in logistic regression. Indeed, a limited sample size can give a large confidence interval estimate. Also, have to think, about whether Age and sex are actually predictors of dependent variable worrying.
Hello Prudence Butler. Nowadays, Frank Harrell recommends 20 events per parameter, not 10 or 15.
- https://discourse.datamethods.org/t/author-checklist/3407#arrow_rightoverfitting-and-lack-of-model-validation-26
But see also the "much better way" he links to:
- https://www.bmj.com/content/368/bmj.m441
If you use either Stata or R, you could try the pmsampsize package to implement the "better" procedure described in that article.
Notice that I said events per parameter rather than events per variable. This excerpt from the BMJ article linked to explains why I did that (boldface added for emphasis):
The word “variable” is, however, misleading as some predictors actually require multiple β terms in the model equation—for example, two β terms are needed for a categorical predictor with three categories (eg, tumour grades I, II, and III), and two or more β terms are needed to model any non-linear effects of a continuous predictor, such as age or blood pressure. The inclusion of interactions between two or more predictors also increases the number of model parameters. Hence, as prediction models usually have more parameters than actual predictors, it is preferable to refer to events per candidate predictor parameter (EPP). The word candidate is important, as the amount of model overfitting is dictated by the total number of predictor parameters considered, not just those included in the final model equation.
Note too that rules of thumb about events-per-parameter are concerned with reducing the likelihood of overfitting. Conventional sample size estimation (power analysis), OTOH, is about ensuring sufficient power to detect the smallest effect size of interest.
I hope this helps.
Determinants Sample Size Number of Events: In logistic regression, the number of events—that is, the total number of people in the "worried" group—is sometimes regarded as the effective sample size. The effective sample size is quite small, consisting of only 33 incidents. Events per Variable (EPV): To prevent overfitting and guarantee accurate estimates, it is commonly recommended to have a minimum of 10 events for each predictor variable. Ideally, you would require at least 20 occurrences for 2 predictors. Recent research, however, indicates that depending on the model's complexity and the distribution of the data, even this criterion may underestimate the necessary sample size.
About Your Research Small Sample Size: Using more than a few predictors in a logistic regression may result in inaccurate estimates and overfitting, as there are only 33 occurrences in the "worried" group. Two Predictors: It should be possible to meet the 10 events per predictor criteria with just two predictors, such as age and sex, albeit very narrowly. This means that the model does not need to account for many additional factors because it is assumed that age and sex will have an influence. Preliminary Exploration: To gain insight into the correlations between predictors and the outcome variable, preliminary exploration can be conducted using chi-squared tests for categorical variables and t-tests for continuous variables.
Bootstrap Validation: To validate your logistic regression model, using bootstrap techniques. In order to evaluate the stability and dependability of your model estimations, this method resamples your data. Other Approaches: As an alternative, think about utilizing Bayesian techniques or penalized regression models, which have the potential to be more reliable when dealing with small sample numbers. In brief Given your current sample size, you can move forward with logistic regression utilising two predictors. But be mindful of the constraints, and think about using bootstrap techniques to validate your results. Commencing with exploratory analyses (t-tests, chi-squared) can assist you in determining the most significant predictors. Regularisation techniques and other robust methods might be necessary if you intend to add more predictors in the future.
Reference
- Vittinghoff, E., & McCulloch, C. E. (2007). Relaxing the rule of ten events per variable in logistic and Cox regression. American Journal of Epidemiology, 165(6), 710-718. DOI:10.1093/aje/kwk052
- Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996). A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 49(12), 1373-1379. DOI:10.1016/S0895-4356(96)00236-3
Re the Vittinghoff & McCulloch (2007) article that Mahamudul Hasan mentioned, I think it is important to note that the simulations in that article addressed the very specific situation where there is one primary explanatory variable with all other variables being included for adjustment only. As V&M put it:
"We conducted a large factorial simulation study with binary as well as failure time endpoints, focusing on a primary predictor, either binary or continuous, and regarding the covariates as adjustment variables."
IMO, therefore, people who plan on interpreting the coefficients of all variables that happen to be significant must not use this article to justify having fewer than 10 events per predictor variable parameter.