Hello,
My aim is to study how the inclusion of several risk factors improve a clinical model predicting the incidence of stroke in a survival analysis. I want to use the NRI to compare two models (1 baseline model vs baseline model+new risk factor). However, I would prefer to stratify the probabilities of these models in categorical variables (i.e., risk 0-3%, risk 4-6%, risk >7%...). For that purpose I have prepared the following code with R, but I am not sure whether it is correct or not and I would prefer to clarify it =).
I will use the open survival data “lung” in order to show a reproducible example. IMPORTANT: I will not consider censored data in this analysis, but it should be analyzed as appropriate in further analyses:
library(survival)
library(pec)##To calculate probabilities of event at a point of time.
library(Hmisc)##To calculate NRI
data(lung)
lung
read
Title NRI for Risk Prediction Models with Time to Event and Binary
The NRI for time to event data can be calculated in different ways. A popular group of the the NRI family is the so-called "prospective NRI". See this article and the references in it: https://annals.org/aim/fullarticle/1814428/net-reclassification-improvement-computation-interpretation-controversies-literature-review-clinician-s
Also notice that there is a lot of controversy around the use of NRIs. See for instance this article: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3918180/
I suggest you reconsider the use of NRI. I have read that Pencina regrets introducing it (https://www.fharrell.com/post/addvalue/). I've written myself about it and that the IDI is a much better measure. Essentially the NRI considers a change of risk of 0.1 the same as a change of risk of 0.00001, whereas the IDI takes into account the magnitude of change. Also, please make sure you report the IDI separately for those with and those without the event of interest. It makes no sense to add IDIevent and IDInon-event together (https://cjasn.asnjournals.org/content/7/8/1355). Hope this helps.
Thanks for your answers which have been very helpful.
Certainly NRI is not the best approach to compare two models due to it doesn't consider the magnitude of the effect (e.g., 0.1 ~ 0.0001) and it has several source of bias that may inflate this index. This has been empirically proven ( Article A note on the evaluation of novel biomarkers: Do not rely on...
) .Perhaps, a solution is not trust just an individual index and compare both models by means of different tests depending on your question (brier score, time-dependent roc curve, IDI...). I have been using the score() function from riskregression package ( https://rdrr.io/cran/riskRegression/man/Score.list.html ).This package includes several utilities to compare survival models with censored data (calibration plots, time-dependent roc curves and brier score). This can be complemented with improveprob() or survIDINRI() functions.
Thanks for your information!
- Badges
- Science topic