Dear All,
As stated in the heading: I need to understand when any of two model fit stats:Hosmer-Lemeshow Goodness of Fit & Kolmogorov-Smirnov Test should be used.
As per the reserach Hosmer-Lemeshow Goodness of Fit is to be used when there are one or more continuous predictors in the model,however for Kolmogorov-Smirnov Test it can be used to analyze maximum difference between the cumulative true positive rate and the cumulative false positive rate.
Do advice as which measure should be used & how. In my case for logistic regression most of my independent variables are categorical hence H-L can be ruled out.
The Hosmer-Lemeshow test is a measure of how well your model fits the data. It is has nothing to due with the structure of the variables though it will effect the m-asymptotic possible values if you had five, two class categorical variables versus just one continuous (the reason for this is that the asymptotic structure in logistic regression is genereally based off of the number of covariate patterns that can be made and not the number of individuals. Regardless, what is done (off the top of my head) is use the predicted probabilities and separate them into decile groups. The for each of the ten different deciles you will have a actual observed value for each of the predicted values . The closer the observed match the predicted in dependent variable probability the lower the hosmer-lemeshow test and the less significant it becomes suggesting that the model does not differ drastically then would be theoretically expected.
It is better to use a receiver operator curve (ROC) than mess around with the best cut-off points as this choice is selfish based off of desired cutoff where as the ROC curve is always the same for a given model and is essentially interpreted as the percentage of times that the outcome variable was accurately predicted (this is sort of off but I more or less am on point).
Wishing you the best,
Logan Netzer
The Hosmer-Lemeshow test is a measure of how well your model fits the data. It is has nothing to due with the structure of the variables though it will effect the m-asymptotic possible values if you had five, two class categorical variables versus just one continuous (the reason for this is that the asymptotic structure in logistic regression is genereally based off of the number of covariate patterns that can be made and not the number of individuals. Regardless, what is done (off the top of my head) is use the predicted probabilities and separate them into decile groups. The for each of the ten different deciles you will have a actual observed value for each of the predicted values . The closer the observed match the predicted in dependent variable probability the lower the hosmer-lemeshow test and the less significant it becomes suggesting that the model does not differ drastically then would be theoretically expected.
It is better to use a receiver operator curve (ROC) than mess around with the best cut-off points as this choice is selfish based off of desired cutoff where as the ROC curve is always the same for a given model and is essentially interpreted as the percentage of times that the outcome variable was accurately predicted (this is sort of off but I more or less am on point).
Wishing you the best,
Logan Netzer
Thanks Logan,
so for my model my ROC curve is way above the line "hump shape" and also AUC is more than 92%so i am pretty good here.
That is a huge c-statistic, my friend. Very impressive but let us hope your logistic regression model is not in violation of any assumptions, no warnings of a quasi- or complete separation, and the Hosmer-Lemeshow test was non-significant. Very rare to get a number that high.... rare but not impossible. That is as good as you can hope for any higher and you most likely would definiately run into model problems. Just make sure everything is in check because it would be a shame to let a c-statistic (AUC) that high go to waste.
Wishing you the best of luck,
Logan Netzer
Thanks Logan,
I did had issues of quasi- or complete separation. To take care of this i have used the weight of evidence & information value techniques .
Initial model had close to 22 predictor variables which came down to 9-10 predictors.
Also along with these i also ran the anova test to check the deviance. With all the remaining variables there was significant change in deviance and null deviance,
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis