Dear Safi

You can use Genetic Algorithm to find the optimal point on the ROC. It is available on Matlab

If disagree with Osama. ROC curves show the trade-off between true positives and false positives. If you go farther to the right on the curve, you can increase the true positives a little bit, at the cost of a lot more false positives. It could be that that little bit at high cost is important, but also that it is not.

If your data are about e.g. pancreas-cancer detection, it is a clinical decision what is acceptable. That clinical decision is not easily captured in some 'objective' calculations, no matter what algorithm you use.

If agree with Osama and Lambert. To find the trade-off between true positives and false positives on the ROC, you can use any one of evolutionary search algorithms like GA, SA etc. and you can make the goal function is f-measure.

http://upload.wikimedia.org/math/9/9/1/991d55cc29b4867c88c6c22d438265f9.png

http://en.wikipedia.org/wiki/F1_score

The f -measure factor involves the false positives and the true positives factors. Also I am agree with Lambert the optimal point is relative to our application.

Cheers

Just a short comment. It depents on the application, mostly regading the False Positives/Negatives. Tey are your optimisation variables! Thing about, e.g. banknote authentication, what would you like to have, a counterfeit or a real one?. The f-measure is a nice possibility for optimisation.

To my experience the choice of cut-off due to max accuracy is not the optimal approach. The cut-off should be chosen according to the application of the test and the "costs" of false positives and false negatives, respectively. E.g. if false positives should be avoided, one has to choose a cut-off at a high specificity level, let's say 95%.

This is depending on the optimal definition first. 

F1 is an OK definition for optimal thresholds. We consider geometric mean: Sqrt(A1*A2) is better than F1, please see why it is better from

Hu, B.-G., and Dong, W.-M., “A study on cost behaviors of binary classification measures in class-imbalanced problems”, March 26, 2014. (http://arxiv.org/abs/1403.7100). 

All conventional definitions, say F1 or  geometric mean, are not applicable for a reject option, but only mutual information does. 

You can see the optimal thresholds on mutual information from

X. Zhang, and Hu, B.-G., “A New Strategy of Cost-Free Learning in the Class Imbalance Problem”, July 22, 2013. (http://arxiv.org/abs/1307.5730). （Accepted by IEEE Transactions on Knowledge and Data Engineering)

This is a well known problem and the only sensible solution involves accounting for costs, or utilities, of FPs, TPs, FNs and FPs and underlying probability P(D) of the disease.  The optimal cutoff  of X corresponds to the  value of  X where the ROC gradient is  O *(U_FP-U_TP)/(U_FN- U_TP) and O=(1-P(D)/P(D). I think it is due to Hanley and McNeil.

I do not know why hardly anyone working on ROCs seems aware of it.

Roger.

@Roger, maybe it is because not everything is readily expressed as 'costs'. In medicine, the cost of a false positive (e.g. diagnosing a cancer while the patient does not have that particular disease), can more or less be calculated, although the psychological damage is hard to express in money.

But the cost of a false negative, which could lead to death of an untreated or too late treated patient, is very hard to calculate. What is the value of a saved life or a less troublesome life?

I do not recommend using ROC for a final decision about a threshold.

Instead, I suggest to consider to use two thresholds - say c1 and c2 - instead of one and simultaneously to optimize true positive, sensitivity, specificity, etc... instead.

So some cases will be unclassified however the performance of you classifier is likely to increase.

ROC curve, in general, indicates the efficiency of a model by analyzing all possible cutoff values. It is better to use as model performance comparison rather than using it for choosing optimal values. Instead, the cost approach can be applied to examine optimal cutoff values.