Assuming you are working on a binary classification task, I recommend using the F1 score.

You are correct that you should not use accuracy here since the algorithm could default to always predicting the majority class and achieve 90% accuracy. This result might look good, but the underlying model will be useless since it will never predict a negative result.

The F1 score metric is the harmonic mean of the precision (ratio of true positives to the sum of true and false positives) and recall (ratio of true positives to the sum of true positives and false negatives) metrics and thus balances for the effect of the imbalance within the dataset. This metric is commonly used for binary classification tasks on imbalanced datasets. The F1 score is between 0 and 1, and a higher F1 score generally indicates a better classifier.

Thank you so much for the detailed response, Matthew. Your are correct, I am working on a binary classification task for a dataset of chemical compounds (class labels 'toxic' and 'non-toxic', toxic compounds being the majority class).

It is preferable that in case of inaccuracies, such models have false positives rather than false negatives (a false positive would be incorrectly predicting a safe molecule as toxic, which is to err on the side of caution, however a false negative would be incorrectly classifying a toxic compound as safe, which would be disastrous.

Would you still recommend the F1 score for such cases?

Anish Gomatam These are some of the scenarios that may arise while assessing multi-label classifiers.

1. Duplicate data in your test data.

2. Only a subset of the expected labels is predicted by your model.

3. Your model correctly predicts more labels than predicted.

4. High recall — High precision

5. High Precision — Low Recall

6. Precision is low, but recall is high.