I have a dataset where I want to assess and compare the performance of different mortality scores in burned pediatric population. For certain score, there is a probability of mortality. I want to test how accurate is the estimated probability (score calculated) with the actual observated event. For this Im using Stata where first I calculate the mortality score and then the probability (from 0 to 1) for each subject.
Q1. For the fitted model, Y= observed mortality, and X1=mortality probability estimated from score. Is this correct? or shoud I transform the probability into log(odds)?
Q2. To assess the utility of the score, I use the AUC ROC from the previous model with just one X variable. Is this correct? (c-statistic)
Q3. Does it make any sense tu try to estimate the calibration of the model, given I only have a summary of it (just the probability, not the model itself)?
Q4. If Q3 is affirmative, shoud I use AIC, BIC (other measure of fit) for the comparisson of the different scores or shoud I just compare the AUC ROC (roccomp)?
Hello, Emiliana
Your proyect is interesting
To perform external validation of the original models , you have to evaluate their calibration and discrimination hability in your population
For discrimination you measure AUC ROC in your population
For calibration you measure Hosmer Lemeshow test and SMR
You can read material and methods section from this paper, in which i am coauthor
I think it can be useful
http://www.scielo.org.ar/scielo.php?script=sci_arttext&pid=S0325-00752015000300006
For question 1 you use the predicted probability, not the score
Q2- OK
Q3 Yes
Q4. To validate the performance of each model in your population , you use both AUCROC and HL
Pilar
Muchas gracias, proximamente haremos mas estudios de validación. Estare contactandola encaso de mayores dudas.
Muchos saludos desde Perú
- Badges
- Science topic
- Similar topics
- Papers