Validating logistic regression models.
I derived a logistic regression model for predicting clinical response to a drug. I have two groups: responders and non-responders, classified according to a clinical scale. The predictors of the model were two polymorphisms and three environment variables such as smoking. I generated the predicted probabilities using logistic model for all patients. Then using predicted probabilities i constructed a roc curve for responders vs non-responders.
No. Validation neccessarily requires new data. The predictive performance is far overestimated when the the data is predicted that was already used for fitting
In the foggy in-between there are methods like cross-validation (CV), for instance leave-one-out CV.
No. Validation neccessarily requires new data. The predictive performance is far overestimated when the the data is predicted that was already used for fitting
In the foggy in-between there are methods like cross-validation (CV), for instance leave-one-out CV.
Hello, Veeramanikandan.
As Jochen has pointed out, you may develop a cross-validation in order to validate your logistic regression model. I like the split sample techniques, but they are suboptimal.
I posed this question last year in RG. Have a look at:
https://www.researchgate.net/post/Internal_validation_of_a_regression_cross-validation2
There are very good comments.
Regards.
There are some other things you can do.
You can select several different groups of subjects and create a logistic regression for that. Then use the model you just made on the other subjects. This is not a true validation step. If there is an issue with your model, this might make it more apparent.
To expand a bit on what Jochen said, you can do more than leave one out. Suppose that you have 1000 subjects. Create different models where you choose multiple groups of 400, 600 and 800 of the subjects and see what those models look like.
Again this is not a true validation. But, it does give you an idea of what your model might look like.
Something that can, and probably will come up, is when you run a validation study, the model will look different. Jochen idea or mine will help you prepare for this idea.
You might even be able to get away with adding a factor or two to your model. If you had 1000 subjects collected at 4 different sites, add site into your model. If you have an "early" and "late" group, you can add that too. This too will give you an idea about how your models will change with site and time.
No, it is not.
You can not validate your estimated parameters with the same data you used to estimate them in the first place. And there is no way arround this.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling