The ROC curves measures the accuracy of a diagnostic test for the entire range of values and allows the identification of the cut-off that best discriminate between false positives and true positives. This statistical tool provides numerous sensitivities and specificites for each cut-off value and the researcher decides which one is ideal through a balance of the two. Different from other fields of medicine in oncologic research greater sensitivities are preferable at the expense of lower specificities due to the nature of the diseases treated. Does anybody has experience on what are the acceptable sensitivity values in oncology and especially can you provide a useful reference to cite?
Remember that the ROC area and the sensitivity and specificity at a given threshold depend on the clinical population. This is very sensitive to the case mix. In early stage diagnostic development is is quite common to work with samples that are much more polarised than genuine clinical populations: that is the positives have cancers that are later staged or more aggressive, and the negatives are often healthier. As the R&D programme development progresses, the sample populations become closer to the clinical reality and ROC areas and sensitivities and specificities degrade. Here is an example outside the oncology area. I was involved in development of a diagnostic for an infectious disease. We had trials running at a public hospital and at a private hospital. The diagnostic performed much better in the public hospital than in the private hospital - because in the public hospital the cases were predominantly late stage with organ failure, and in the public hospital they were predominantly early stage ( middle class people consult their medical advisor earlier). The trial has the same inclusion criteria, and was run in the same city. But the socio economic catchments of the hospitals were different and that affected the performance estimates. The diagnostic was validated in both trials, but there was a difference in performance.
Measures of diagostic performance are critically dependent on the specification of the clinical population.
Thanks for the answer and I found the reference suggested very useful.
Remember that the ROC area and the sensitivity and specificity at a given threshold depend on the clinical population. This is very sensitive to the case mix. In early stage diagnostic development is is quite common to work with samples that are much more polarised than genuine clinical populations: that is the positives have cancers that are later staged or more aggressive, and the negatives are often healthier. As the R&D programme development progresses, the sample populations become closer to the clinical reality and ROC areas and sensitivities and specificities degrade. Here is an example outside the oncology area. I was involved in development of a diagnostic for an infectious disease. We had trials running at a public hospital and at a private hospital. The diagnostic performed much better in the public hospital than in the private hospital - because in the public hospital the cases were predominantly late stage with organ failure, and in the public hospital they were predominantly early stage ( middle class people consult their medical advisor earlier). The trial has the same inclusion criteria, and was run in the same city. But the socio economic catchments of the hospitals were different and that affected the performance estimates. The diagnostic was validated in both trials, but there was a difference in performance.
Measures of diagostic performance are critically dependent on the specification of the clinical population.
Any ranking can be partitioned (everything above the cut-off, test "positive", everything below the cutoff test "negative") to generate different trade-offs of sensitivity vs. specificity. An ROC "curve" is actually not a curve but a set of possible partitions for a given test modality against a given population with constant condition incidence. Assuming your future population has the same incidence and disease characteristics as the population used to develop the test, the estimated real-world cost per false negative divided by estimated real world cost per false positive (the "cost function ratio") will be closest to one particular point on the ROC partition. This is the point that should be chosen for that context.
A good rule of thumb: if the real-world cost function ratio is as yet unknown, or is at best an estimated range, chose the point on the ROC curve, or within the cost function ratio range, that maximizes the mutual information. This is the point on the partition for which knowledge of whether an individual test outcome is "positive" or "negative" maximizes the average reduction in uncertainty as the to the actual condition of the individual.
Note that knowing population incidence is not needed to calculate the FP and FN rates associated with any given point on the ROC "Curve" - but you do need true incidence to calculate the actual costs for any point, and to find the point that maximizes mutual information.
- Badges
- Science topic