It depends on problem domain. There exists several off-the-shelf metrics, e.g. accuracy, precision, recall and so on. Each of these metrics indicates different aspects. So, the best metric cannot be generalized. It depends what the point of interest is based on problem domain. However, the mostly used metric is the accuracy for evaluating a supervised binary classifier with balanced classes.

Hi Sami,

Both MCC and F-Measure can be used. I recommend also exploring the "Kappa statistic".

HTH.

Samer

If the classes are balanced, accuracy or AUC_ROC is good, otherwise, you should look at kappa coefficient or AUC precision-recall.

If your classes are balanced, why not use the most intuitive measure, Accuracy? If you are not sure, that your classifier chose the best decision boundary you should go for AUC, which is equivalent to the probability that your classifier assigns the higher score to  the relevant tweets compared to the irrelevant tweets. Be careful with any other metrics, because they usually don't have a clear meaning and as soon as you get into the regime of unbalanced classes, most metrics are misleading, especially the F-Measure should not be used at all (proof in the attached paper).

https://www.frontiersin.org/articles/10.3389/fncom.2014.00043/full

in the case of the balanced dataset, precision, recall, and F1 are the good measurement. If your dataset is unbalanced, you would better use ROC curve or macro/micro precision

Thanks to all of you for your answers.

Classification Assessment Methods

This paper introduces a detailed explanation with numerical examples many classification assessment methods or classification measures such as:

Accuracy, sensitivity, specificity, ROC curve, Precision-Recall curve, AUC score and many other metrics. In this paper, many details about the ROC curve, PR curve, and Detection Error Trade-off (DET) curve. Moreover, many details about some measures which are suitable for imbalanced data are explained.

Your comments are highly appreciated.

https://ac.els-cdn.com/S2210832718301546/1-s2.0-S2210832718301546-main.pdf?_tid=20353999-8c84-422a-8191-fd1059469272&acdnat=1536185805_03b19e159f0675a01f35512961973eb5

Presentation of the paper is here

https://www.researchgate.net/publication/327403649_Classification_assessment_methods_a_detailed_tutorial

Precision recall

ROC-AUC

Accuracy

Log-Loss

Both Accuracy and ROC are appropriate.

It depends on problem domain. There exists several off-the-shelf metrics, e.g. accuracy, precision, recall and so on. Each of these metrics indicates different aspects. So, the best metric cannot be generalized. It depends what the point of interest is based on problem domain. However, the mostly used metric is the accuracy for evaluating a supervised binary classifier with balanced classes.

In balanced classes, AUC-ROC is good, otherwise, you should look at precision, recall.

Precision, recall, and F1 are good besides AUC-ROC.