Various issues come up in this discussion. With a single cut-off: max(Youden) is max(sens+spec); the upper left point in ROC space. Accuracy relates to the absolute sum of true-positive (TP) and true-negative (TN) classification: max(TP+TN) / N. This measure has often been used, but arguably suboptimal as a performance measure in medical applications; for example, if prevalence of disease is 1%, accuracy should be at least 99%, since 99% accuracy is achieved by labeling all patients as negative.

The more interesting perspective for medicine is the cost-based (or utility-based) perspective. The NCSS and other software manuals rightly refer to this approach.

I would like to bring up the link with the choice of the decision threshold for classification of patients as positive vs negative, with an action such as "treat" connected to positive classification. The decision threshold is in terms of a probability of the endpoint, say probability that disease is present in a diagnostic problem. Prediction models are useful to estimate individualized probabilities, and ROC analysis is frequently applied to quantify the quality of a prediction model. The upper left point in ROC space corresponds to using the mean prevalence of disease as the threshold. This makes sense since the likelihood ratio of such as prediction is 1: the prediction after using the prediction model is equal to the prior probability of the disease without using the model, and the likelihood ratio is the tangent of the ROC curve. The utility-based approach leads to a decision threshold that reflects the ratio of harms (overdiagnosis, FP classifications) to benefit (gain for those correctly diagnosed, TP classifications). The classic decision-analytic formula is that the odds of the threshold is equal to this ratio, for example 1:1 implies a threshold of 50%; 1:4 implies a threshold of 20%. This decision-analytically motivated threshold can then be used in ROC space to see what sensitivity and specificity relate to this cutoff.

So, note that the optimal combination of sensitivity and specificity follows from the utility-based choice of the threshold, rather than that we start with discussions on the balance of sensitivity vs specificity. This balance can only be addressed in combination with the mean prevalence of the disease, and may be more complex for medical applications than the direct question: how many false-positive classifications are you willing to accept for 1 true-positive classification?

Article Performance Measures for Prediction Models and Markers: Eval...

Article Sensitivity and Specificity Can Change in Opposite Direction...

Hi, Good Morning

The methods to determine cut-off points in an ROC Analysis, Kindly see the link.

http://ncss.wpengine.netdna-cdn.com/wp-content/themes/ncss/pdf/Procedures/NCSS/ROC_Curves.pdf

OptimalCutpoints package of R has various methods to compute optimal cutpoints in diagnostic tests:

http://cran.r-project.org/web/packages/OptimalCutpoints/OptimalCutpoints.pdf

Various issues come up in this discussion. With a single cut-off: max(Youden) is max(sens+spec); the upper left point in ROC space. Accuracy relates to the absolute sum of true-positive (TP) and true-negative (TN) classification: max(TP+TN) / N. This measure has often been used, but arguably suboptimal as a performance measure in medical applications; for example, if prevalence of disease is 1%, accuracy should be at least 99%, since 99% accuracy is achieved by labeling all patients as negative.

The more interesting perspective for medicine is the cost-based (or utility-based) perspective. The NCSS and other software manuals rightly refer to this approach.

I would like to bring up the link with the choice of the decision threshold for classification of patients as positive vs negative, with an action such as "treat" connected to positive classification. The decision threshold is in terms of a probability of the endpoint, say probability that disease is present in a diagnostic problem. Prediction models are useful to estimate individualized probabilities, and ROC analysis is frequently applied to quantify the quality of a prediction model. The upper left point in ROC space corresponds to using the mean prevalence of disease as the threshold. This makes sense since the likelihood ratio of such as prediction is 1: the prediction after using the prediction model is equal to the prior probability of the disease without using the model, and the likelihood ratio is the tangent of the ROC curve. The utility-based approach leads to a decision threshold that reflects the ratio of harms (overdiagnosis, FP classifications) to benefit (gain for those correctly diagnosed, TP classifications). The classic decision-analytic formula is that the odds of the threshold is equal to this ratio, for example 1:1 implies a threshold of 50%; 1:4 implies a threshold of 20%. This decision-analytically motivated threshold can then be used in ROC space to see what sensitivity and specificity relate to this cutoff.

So, note that the optimal combination of sensitivity and specificity follows from the utility-based choice of the threshold, rather than that we start with discussions on the balance of sensitivity vs specificity. This balance can only be addressed in combination with the mean prevalence of the disease, and may be more complex for medical applications than the direct question: how many false-positive classifications are you willing to accept for 1 true-positive classification?

Article Performance Measures for Prediction Models and Markers: Eval...

Article Sensitivity and Specificity Can Change in Opposite Direction...

Hi Everyone, 

Thanks for your insightful contributions. All comments and references provided have given more in-depth understanding of the concept.

Hi Paul your last response - tabulating the cut-points, Youden index and ESS helped more to deal with the puzzle. More so, how can one have access to the UniODA...I guess its a stat software?

Given that cost-based (or utility-based) perspective is perhaps one of the best approaches to deal with the cut-off thing, can any one provide references on methods to determine cost of FP/FN of a given diagnostic test?

Many thanks