Hello,
I was wondering if anyone could shed light on an issue I am experiencing with the package cvAUC (link provided below).
I wish to get the function cvAUC to work, yet I am not getting the expected results. Maybe the issue is caused by the format of my data. What I have is a dataframe with a column for the Dependent Variable, one column for the Predicted Probabilities (predicted on the basis of the model I have previously fitted).
From the help documentation, the function's arguments are:
cvAUC(predictions, labels, label.ordering = NULL, folds = NULL)
Assuming that my dataframe is named 'mydata', the way I am using the function is:
cvAUC(mydata$pred.prob, mydata$DV)
I seem to get the AUC for just one fold. Am I missing anything here? Is the issue caused by the way in which the DV is coded (0=absence, 1=presence)?
Thank you for any guidance on the subject.
GmA
https://cran.r-project.org/web/packages/cvAUC/cvAUC.pdf
Hello!
Thanks for your reply. I am quite puzzled. I will look further into the issue, since I would very like to use the function. Unless another similar package/function exists. Do you know of any other alternative?
Best
Gm
Hello! Yes, I happened to come across those two, but I am interested in ROC curve in the context of cross-validation. But maybe I could gave another look at those two.
Thanks
Gm
UPDATE:
after some time, I have put together an R function to calculate a optimism-adjusted AUC value via bootstrap resampling. The method is nicely (yet succinctly) described in:
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. ., Eijkemans, M. J. ., Vergouwe, Y., & Habbema, J. D. F. (2001). Internal validation of predictive models. Journal of Clinical Epidemiology, 54(8), 774–781. http://doi.org/10.1016/S0895-4356(01)00341-9
The function (available from the link added below) returns a chart showing:
-the distribution AUC value in the bootstrap sample (auc.boot), which represents "an estimation of the apparent performance" (according to the aforementioned reference);
-the distribution of the AUC value deriving from the model fitted to the bootstrap samples and evaluated on the original sample (auc.orig), which represents test performance (i.e., the model performance on independent data).
At the bottom of the chart, the apparent AUC (i.e., the value deriving from the model fitted to the original dataset) and the AUC adjusted for optimism are reported.
Hope this helps.
http://cainarchaeology.weebly.com/r-function-for-optimism-adjusted-auc.html
UPDATE 2:
I have put together an new R function to perform internal validation by repeated cross-validation, implementing part of the procedure described by
Arboretti Giancristofaro R, Salmaso L. Model performance analysis and model validation in logistic regression. Statistica 2003(63): 375–396
The procedure consists of the following steps:
1) the whole dataset is split into two random parts, a fitting (75%) and a validation (25%) portion;
2) the model is fitted on the fitting portion (i.e., its coefficients are computed considering only the observations in that portion) and its performance is evaluated on both the fitting and the validation portion, using AUC as performance measure ;
3) steps 1-2 are repeated B times, eventually getting a fitting and validation distribution of the AUC values. The former provides an estimate of the performance of the model in the population of all the theoretical training samples; the latter represents an estimate of the model’s performance on new and independent data.
The function is available here on ResearchGate (DOI: 10.13140/RG.2.1.1925.6086) and is fully described here:
http://cainarchaeology.weebly.com/r-function-for-binary-logistic-regression-internal-validation.html
Hope this helps.
http://cainarchaeology.weebly.com/r-function-for-binary-logistic-regression-internal-validation.html
Code 'model.valid': R function for binary Logistic Regression int...
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Software
- R Statistical Package