The ROC curve, and the area under the curve (AUC) parameter derived from it, are an important tool for optimising classifications.
As I understand it the ROC is derived by varying the decision line, recording how the true positive and false positive fractions as the line is adjusted and a curve is plotted from the resulting data. I have often heard the AUC claimed to measure the probability of correctly classifying a random sample.
Surely it is in fact measuring the probability of correctly classifying a sample when independently and randomly defining an arbitrary decision line, i.e. two simultaneous uncertainties. When you have finalised your test protocol based on the calibration/training set you no longer choose an arbitrary decision line, you use the one selected in the calibration phase. To me this means you can no longer calculate the ROC as you can no longer vary the decision line. this would make the ROC/AUC fundamentally unsuitable for validation/test sets.
In Hanley and McNeil (Radiology, 1982, vol 143, pg 29-36) it is defined subtly but significantly different and more accurately, in that it is the probability of correctly ranking a pair of one true positive and one true negative in their true order (not necessarily getting both right). However, since we ultimately want classification models to make a hard decision on what class a sample belongs to, the relative rank of a pair is not important in a test set, just what the probability of being correct?
I would be inclined to use measures such as Odds ratios, from which significance can be calculated in both calibration and validation sets.
I would be interested if anyone has insights or can recommend some useful background reading on the issue.
Use the expected performance curve. The ROC is inherently biased because of the following problem: in the real world, you choose the operating point _before_ you see the test data.
Reference: bengio.abracadoudou.com/publications/pdf/bengio_2005_icml.pdf
Thanks Christos for the paper, it was an interesting read and makes sense using the desired performance parameter directly. I am generally wary of the ROC, but can see the point of it in terms of choosing a subset of tests from a larger bank of possible tests. I don't feel it is a terribly useful statistic myself, beyond guiding development, it will define the relative probility of being able to make a useful classifier but will provide little useful information on how two tests actually compare in action. However, it is firmly embedded in the psyche of the medical field, so it does need tackled.
I would tend to follow the approach in that paper myself less formally. I optimised for a standardised specificity (true negative rate) across all my tests because that was the most important trade off for my test. The issue is that most people in the biomedical field will still want the AUC presented alongside everything else and I do not feel, based on basic principles, that it is applicable to validation/test sets. Is there any work that specifically looks at this issue?
As usual the answer is it all depends on what you are trying to do.
In the development of a clinical diagnostic, at some stage you will need to do a pivotal trial and test the performance of the diagnostic (sensitivity, specificity, PPV, NPV) at a specified cut point in a specified clinical population. The ROC curve will not help you there.
But in early stage development, especially when one is developing a multivariate index assay (a machine learning generated diagnostic index combining several features or analytes) you may well still not be able to specify the clinical cut points, but you need to protect against the over optimism that can arise from a tuned machine learning exercise. Computing the ROC area in a validation set is essential in that context.
It is important to remember that all of the standard measures of diagnostic performance **are critically dependent on the population** as well as the test. For example, I worked on optimisation of a sepsis diagnostic. We had clinical sites in public hospitals and in private hospitals. The public hospital cases were mostly severe sepsis with multi organ failure. The private hospital cases were middle class sepsis - diagnosed and treated early, much less severe. The ROC area, sensitivity, specificity, NPV and PPV were quite different in the public and private hospital. It was the same disease, the same test, the same inclusion criteria - but a subtly different demographic.
In general, in early stage development your groups will be well defined and ROC areas will be high. As research continues, groups will become closer to the actual target population and diagnostic measures will usually look as though they are degrading.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics