Recently a collaborator came with a project which aims to validate at his setting some prediction models for digestive bleeding. They are the rockall score, glsagow blatchford score, AIMS65 score, Charlson morbidity index, Child-Pugh and MELD classification. His question was how to estimate a reasonable sample size. I took a look at the original papers and their validation sets ranged from 197 subjects with 41% outcome to 32504 subjects with 2% outcome. An absence of a refreshing happiness occurred when I saw that in the largest set of validation there were significant coefficients as low as 0.3 with SE of 0.18. Also, I did Iook around for some guidance and found the following comments in Steyerberg's book where there is a sample size for validation studies part: "modest sample sizes are required for model validation"; "to have reasonable power, we need at least 100 events and at least 100 non-events in external validation studies, but preferably more (>250 events). With lower numbers the uncertainty in performance measures is large." But in the text there are several simulations results showing that it depends a lot of the coefficients and SE, which could lead, even with these amounts of outcomes to power as low as 50%. Taking these rule of thumb, and expecting a 4% outcome in a validation cohort, it would be necessary to include 2500 to 6250 subjects. Pretty scary and with very wide range, which does not help much in the planning time. I found a logistic regression sample size formula, but it did not help much as it allows only two predictors at a time and the permutation predictors coeff and Se in the formula, the N ranged from a few hundred to dozens of thousands. http://www.dartmouth.edu/~eugened/power-samplesize.php
I would like very much to be comfortable in recommending a sample size of 2500 taking the Styerberg's rule of thumb as the basis of it. I would like to hear from those who have some experience in this issue.
For validation studies, there is little empirical evidence to suggest what sample size (and more important the number of events), except Vergouwe et al, J Clin Epidemiol 2005; 475-483 and another paper by Peek et al J Clin Epidemiol 2007:491-501. Where the former recommends a minimum of 100 events and 100 non-events (also mentioned in Ewout's book, as he co-authors the Vergouwe paper)- I would try and go much higher.
Also when you are comparing models (see the Peek paper) as well and sample size requirements will be much higher (I'm just currently writing something up on this, based on some simulation studies we've carried out, for cox regression models, though the results should carry over to logistic I guess). The problem comes when you are comparing models with insufficient numbers of events. It's quite common to see inflated c-statistics (for example) when the number of events are low. Picking the 'winner' by choosing the model with the highest c-statistic can also be problematic when numbers of events are low.
Just some thoughts...
Suppose I stop recruitment when all predictors have at least 10 events, however there is less then 100 outcomes. Is this reasonable for a model validation?
In my haste to leave the office to avoid the traffic, I forgot to add...the number of predictors is irrelevant in (external) validation studies. This is only relevant for when you are developing a model. At present the only guidance is the number of events. So don't multiply the number of events by the number of predictors in the model. All you want is a minimum of 100 events/100 non-events as a bare minimum, but if you are comparing multiple models, you may want many more.
For classification tasks, use a multinomial distribtuion to estimate the sample size. It is, statistically spoken, on the safe side. See Tortora (1978): http://www.jstor.org/stable/2683352
Pedro - I just ran across this old question on ResearchGate, and thought I would put in a comment: For the case of linear regression through the origin, I have developed a simple way to estimate sample size needs. For multiple regression, to obtain this estimate, it would be easiest to use a linear combination of regressors in place of the one regressor. If interested, you can find this paper on my ResearchGate page now: "Projected Variance for the Model-Based Classical Ratio Estimator: Estimating Sample Size Requirements." - Jim
Yikes. Sorry. In my excitement at seeing a topic related to something about which I have such an interest, I misread that your question was for validation. This is related, but certainly my suggestion for collapsing regressors would not be helpful. Mea culpa.
Parallelization of a Modified Firefly Algorithm using GPU for Variable Selection in a Multivariate Calibration Problem
Hello, Pedro! I think that perhaps one of my papers called
may help you with this issue. If you wish to read it completely, please let me know and I will send it to you.
Upload it in researchgateand post a link to it.
Olá, Pedro! Obrigado por vote up minha resposta. Por favor, me passe seu email inbox para eu lhe enviar o pdf do artigo. Devido a revista não permitir, não posso divulgá-lo abertamente aqui. Desculpe por não responder antes. Infelizmente, o researchgate ainda não envia alerta sobre respostas em perguntas que você respondeu anteriormente.
You could read the paper:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC28769
Sample size calculators:
http://www.raosoft.com/samplesize.html
http://www.calculator.net/sample-size-calculator.html
Another calculators:
http://www.dartmouth.edu/~eugened/power-samplesize.php
http://www.surveysystem.com/sscalc.htm
http://powerandsamplesize.com/Calculators/
Would you please check the PMC ID (28769) cited above? It shows no item found. Thanks.
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis