Most research papers usually mention 'Sensitivity for a fixed specificity (mostly 95%)' or 'Specificity for a fixed sensitivity (mostly 95%)' . I do know that for single feature classifiers, these sensitivity/specificity objective can be achieved by controlling the threshold. But when I'm using a classifier like Random Forest ( that too with 15-20 features) , how do I tune my classifier to achieve a certain sensitivity? Another related doubt is that when papers mention classification accuracy etc. can they report any specificity-sensitivity pair on the ROC or the one to which the classifier has been tuned?
It probably depends on the software you use. In case of R and package randomForest, you could use function predict with argument type = "prob" to get predicted probabilities. Then, you could use the predicted probabilities to find a cut-off that leads to a given sensitivity.
Do you mean accuracy = probability of correct classification (PCC) or accuracy = AUC? The AUC is fixed, but PCC = prevalence*sensitivity + (1-prevalence)*specificity.
Best
Matthias
The software i'm using is WEKA. As of now, I can click a point and observe the SE/SP and threshold value at that point. But I can't understand how to train my classifier to that precise point. I have found out that the Threshold Selector can be used, but can't find any examples of how.
I had meant Acc=PCC. When I mention the Se/Sp, can I report any point on the ROC? My basic aim is to fix one of them and show that the other is better than previous results (something which I can already see in the ROC).
On further playing around with the ThresholdSelector in WEKA, I've managed to achieve my objective. By using Cost/Benefit analysis curve, I could find out the threshold value to suit my needs. Setting appropriate parameters in the ThresholdSelector then gave me the tuned classifier.
I understood choosing a particular Se/Sp will change my accuracy. (Acc,Se,Sp) is a triplet which will change if one of them if changed. Thus, in my understanding, when Se for a fixed Sp or vice-versa is mentioned, Acc is not mentioned and the figure just gives an idea about how finely the classifier can be tuned to suit the needs. Is this correct?
ACC has several drawbacks eg. you should be careful in case of class imbalance. Anyway, the figure (ROC curve) in my view is the better alternative to present the performance instead of giving just one number.
Best
Matthias
The dataset i'm using contains only around 7.5% positive samples (sample size ~130,000). I agree that ACC will not give a good representation of the result in this case.
Then, ROC curve comparison and AUC are the two parameters I can use to compare my results with previous research works. However, WEKA also gives me 'Kappa statistic' and 'F-Measure' as parameters to estimate the performance of my classifier. But I don't think there is any point in presenting them since not many research works in the past have used them, rendering them useless for any comparison.
Are you balancing the class distribution before training the classifiier?
Regarding performance measures under class imbalance see www.pitt.edu/~jeffcohn/biblio/Jeni_Metrics.pdf