Hi Miss Leclair, the "best" cut-off actually depends on what you want to have - you can either maximize sensitivity at the expense of specificity in order to create a test, that is maximal sensitive (gets everyone that is affected, but produces a lot of false-positives). Likewise, you can increase specificity (minimize the axis 1-specificity) and you will get a test, that is better in "ruling-out" the disease.

If you want to maximize both, sensitivity and specificity, you can apply the Youden's index. For this, you aim to maximize the Youden's index, which is Maximum=Sensitivity + Specificity - 1. So you choose those value of the ROC-curve as a cut-off, where the term "Sensitivity + Specificity - 1" (parameters taken from the output in the same line as the observed value, see attachments) is maximal. Software specialised in medical statistics (f.e. MedCalc) provides the Youden's index for each ROC-analysis.

Hi M. Haslacher,

Thank you very much for providing this detailed explanation. However, I am still puzzled about the following:

My test has values 0 to 10, but the output is not giving me sensitivity for each of these values, it seems to be calculating averages, according to the footnote «All the other cutoff values are the averages of two consecutive ordered observed test values». (see output). I would like to know the sensitivity and specificity for each of my 11 values, such as what has been previously done in the literature (see output). 

Before you start wondering about what a software does, invest some time and do the calculations "manually". That's not too difficult and as a side-effect you will better understand how things work.

How to do:

For each cuf-of value you like to consider (like 0, 1, 2, ... 10):

calculate the proportion of correctly classified cases based on that cut-off:

- correctly classified as "positive" = true-positive-rate = sensitivity

- correctly classified as "negative" = true-negative-rate = specificity

Plot sensitivity against (1-specificity) to get a ROC diargam.

Youden-Index: Detemine the point for which (sensitifity + specificity) is maximal. 

Another "optimal cut-off" is the value for which the point on the ROC curve has the minimum distance to the upper left corner (where sensitivity=1 and specificity=1). By Pathagoras' theorem this distance is sqrt( (1-sensitivity)²+(1-specificity)² ).

Note that both ways do NOT consider any costs of wrong classifications or benefits of correct classifications. For some purpose a higher sensitivity may be more important than a higher specificity (or vice versa). Hence, there exists no "generally optimal cut-off", and "optimal" cut-off is optimal only w.r.t. specific conditions and contraints.

http://www.medicalbiostatistics.com/roccurve.pdf

Here is some syntax to help you generate your own set of 2x2 tables, as suggested by Jochen. The syntax assumes variables called TestResult and GoldStd. Note that it has not been tested, and may require some tweaking. But it should give you the general idea.

VECTOR T = Test0 to Test10.

LOOP # = 0 to 10.

- COMPUTE T(#) = TestResult GE #.

END LOOP.

RECODE Test0 to Test10 (0=2).

FORMATS Test0 to Test10 (F1).

VALUE LABELS Test0 to Test10

1 "Test GE cut-point"

2 "Test LT cut-point"

.

CROSSTABS Test0 to Test10 BY GoldStd / CELLS = COUNT COL.

HTH.

Further to Jochen's excellent reference, here's a good discussion of the various indices that are used to determine optimal cutoff points:

Liu, X., 2012. Classification accuracy and cut point selection. Statistics in medicine, 31(23), pp.2676–2686.

https://www.researchgate.net/publication/221808472_Classification_accuracy_and_cut_pointselection

http://onlinelibrary.wiley.com/doi/10.1002/sim.4509/abstract

See also

Fluss, R., Faraggi, D. & Reiser, B., 2005. Estimation of the Youden Index and its Associated Cutoff Point. Biom J, 47(4), pp.458–472.

http://www.researchgate.net/profile/Benjamin_Reiser2/publication/7600740_Fluss_R_Faraggi_D_Reiser_BEstimation_of_the_Youden_Index_and_its_associated_cutoff_point._Biom_J_47_458-472/links/0912f50f46c4a42455000000.pdf

Article Fluss R, Faraggi D, Reiser BEstimation of the Youden Index a...

Article Classification accuracy and cut point?selection

Dr. Wilhem, Weaver and Conroy, thank you all. I agree that understanding the meaning behind formulas is important and gives you more satisfaction as to better understanding your research.

It is great to have the input of the research community for daunting issues. 

I'd also add that in R there's an extremely simple and well documented package called pROC. 

I am also ask ROC-analysis.

Youden index is the way; Also you may use weights for sensitivity and specificity, so it will be modified Youden index.

Hi Emir. I guess this is the sort of thing you have in mind, is it?

Article Weighted Youden index and its two-independent-sample Compari...

How nice to see true nice scholars spare some times for others!

DIFF = abs(sensitivity– specificity);

and

D Value = Sqrt((1-Sensitivity)²+(1-Specificity)²)

Reading various articles on ROC, I feel like youden index might work for me. Can anyone suggest how it can be calculated on training, validation set on fastai. As of now only way I can see is to calculate it on my test set.