I don't see a CI in the SPSS output, even though I selected that option, and want to test the significance of the difference between two AUC's.
The video linked below does a pretty good job of explaining how to use the ROC command in SPSS (including how to get the AUC with CI in the output). But note that SPSS does not have a built in method for testing the difference between two AUCs (see second link below).
If the two AUCs are correlated, you can use the method described by DeLong & co-authors (3rd link below)--you'll have to code it yourself.
If they are two independent AUCs, then you can use a simple z-test, as follows:
z = (AUC1-AUC2) / SEdiff
where SEdiff = SQRT(SE12 + SE22)
Alternatively, if you have access to another statistical software besides SPSS, it may have a built in procedure. (E.g., Stata and R have methods to do this, AFAIK.)
HTH.
https://www.youtube.com/watch?v=_2zN2a3MgmU
http://www-01.ibm.com/support/docview.wss?uid=swg21480005
http://www.ncbi.nlm.nih.gov/pubmed/3203132
Thanks Bruce this is helpful. Not sure what it means, though, that two AUCs would be correlated? Do you mean that the predictor measures are correlated?
(Follow up: now I see what that means - "correlated" means the AUCs were computed on the same sample).
Whenever you lack a formal standard error (which would result in a confidence interval) you can "build" yourself a robust version of it using the bootstrap (which has the added advantage that it does not rely on assumptions such as normality!)
http://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf
Regarding correlated AUCs: this usually refers to the question "Which of these two classifiers works better for my data?" In this case, the AUC curves would necessarily be correlated. It's even more clear when you think of any given point on those curves -- c.f., McNemar's test for correlated binary data.
Donald Bamber (1975) was the first to provide confidence intervals for the AUC. This method is extended to factorial designs in Kaufmann et al. (2005). Also disgnostic trials involving clustered data can be handled by this approach, see, eg., the brief review paper by Brunner and Zapf (2013). Range preserving confidence intervals are discussed in Lange and Brunner (2012, 2013). The method is described in detail in Brunner and Munzel (2013, Sec. 2.1.5).
See also the excellent textbook by Zhou, McClish, and Obuchowski (2009): Statistical Methods in Diagnostic Medicine, Wiley.
Please find below the cited references.
Bamber, D. (1975). The Area above the Ordinal Dominance Graph and the Area below the
Receiver Operating Characteristic Graph. Journal of Mathematical Psychology 12,
387-415.
Kaufmann, J., Werner, C., and Brunner, E. (2005). Nonparametric methods for
analyzing the accuracy of diagnostic tests with multiple readers. Statistical
Methods in Medical Research 14, 129-146.
Lange, K. and Brunner, E. (2012). Sensitivity, specificity and ROC-curves in
multiple reader diagnostic trials - A unified, nonparametric approach.
Statistical Methodology 9, 490-500.
Lange, K. and Brunner, E. (2013). Analysis of Predictive Values Based on
Individual Risk Factors in Multi-Modality Trials. Diagnostics 3, 192-209.
Brunner, E. and Zapf, A. (2013). Nonparametric ROC Analysis for Diagnostic
Trials, in Handbook of Methods and Applications of Statistics in Clinical
Trials, Vol. 2: Planning, Analysis, and Inferential Methods (N. Balakrishnan,
Ed.), Wiley, 471-483.
Brunner, E. und Munzel, U. (2013). Nichtparametrische Datenanalyse. Springer,
Heidelberg. / 2. Überarbeitete Auflage: Springer, Heidelberg.
Hi Warren,
there is also some R-psackage available:
https://github.com/KatharinaLange/facROC
In case of any problems with this R-package, please contact Katharina Lange (now she is married and has changed her name to Katharina Kramer)
We habe also provided some SAS-macros for AUC-ROC analysis. In case of need, just send me an e-mail: [email protected]
Best regards
Edgar
Use Hanley & McNeil 1982 formula to calculate standard error first (in terms of AUC). If the sample size of both diseased and non-diseased class are large enough, then AUC can be assumed as normally distributed. The final estimate of AUC with the confidence limits should equal to (AUC-Z(alpha/2)*SE), (AUC+Z(alpha/2)*SE). Z(alpha/2) = 1.645 for 0.05 significance level for example.
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software
- SPSS