Use the expected performance curve. The ROC is inherently biased because of the following problem: in the real world, you choose the operating point _before_ you see the test data.

Reference: bengio.abracadoudou.com/publications/pdf/bengio_2005_icml.pdf

Thanks Christos for the paper, it was an interesting read and makes sense using the desired performance parameter directly. I am generally wary of the ROC, but can see the point of it in terms of choosing a subset of tests from a larger bank of possible tests. I don't feel it is a terribly useful statistic myself, beyond guiding development, it will define the relative probility of being able to make a useful classifier but will provide little useful information on how two tests actually compare in action. However, it is firmly embedded in the psyche of the medical field, so it does need tackled.

I would tend to follow the approach in that paper myself less formally. I optimised for a standardised specificity (true negative rate) across all my tests because that was the most important trade off for my test. The issue is that most people in the biomedical field will still want the AUC presented alongside everything else and I do not feel, based on basic principles, that it is applicable to validation/test sets. Is there any work that specifically looks at this issue?

As usual the answer is it all depends on what you are trying to do.

In the development of a clinical diagnostic, at some stage you will need to do a pivotal trial and test the performance of the diagnostic (sensitivity, specificity, PPV, NPV) at a specified cut point in a specified clinical population. The ROC curve will not help you there.

But in early stage development, especially when one is developing a multivariate index assay (a machine learning generated diagnostic index combining several features or analytes) you may well still not be able to specify the clinical cut points, but you need to protect against the over optimism that can arise from a tuned machine learning exercise. Computing the ROC area in a validation set is essential in that context.

It is important to remember that all of the standard measures of diagnostic performance **are critically dependent on the population** as well as the test. For example, I worked on optimisation of a sepsis diagnostic. We had clinical sites in public hospitals and in private hospitals. The public hospital cases were mostly severe sepsis with multi organ failure. The private hospital cases were middle class sepsis - diagnosed and treated early, much less severe. The ROC area, sensitivity, specificity, NPV and PPV were quite different in the public and private hospital. It was the same disease, the same test, the same inclusion criteria - but a subtly different demographic.

In general, in early stage development your groups will be well defined and ROC areas will be high. As research continues, groups will become closer to the actual target population and diagnostic measures will usually look as though they are degrading.

Thanks Mervyn, that was very useful. I can see that the ROC is useful for testing the potential for test will have an optimal solution within the new population too. I can see the value now, I was concerned that I would be using an inappropriate statistic.