Dear Hatem Ali :

I would say "it depens on the relative cost". I took this sentence from Witten & Frank, Data Mining, Practical Machine Learning Tools and Techniques with Java Implementations, 2000, Morgan Kaufmann.

Yours faithfully.

Monica Perez-Priede

Is your training dataset heavily unbalanced in the sense of consisting of many negative and only a few positive instances? That could explain your observation because in such a situation a classifier predicting all or almost all instances to be negative would achieve high accuracy and specificity.

Hence, it might be worthwhile to repeat your classification exercise with alternative classification methods that take into account training dataset imbalances.

Dear Hatem Ali ,

as Dirk Tasche earlier mentioned, your dataset is probably highly imbalanced and as you may already know, accuracy is not a convenient metric for evaluating imbalanced datasets (suppose your dataset contains 99K images of non-infected COVID-19 and 1K of infected ones. If your model classifies every sample into the healthy class, you will achieve accuracy of 99%, while your model is performing terribly discouraging!)

Additionally, individually considering sensitivity (also known as Recall) and specificity is still not a good practice. In the above example, while your model achieves sensitivity of 1 (99K/99K), its specificity would be equal to 0 (0/1k)! This inconsistency will be revealed while computing AUC, f1-score, and so on.

In conclusion, as you already doubted, the obtained results are not reliable and your model is perhaps suffering from dealing with imbalanced datasets (with dominating negative class).

I hope it helps you!

Thank you very much Behzad and Dirk

You are absolutely right.

The data is heavily imbalanced.

Now, I repeated it again and giving a bit better results:

In the training dataset: AUC 0.73, Accuracy 0.99, specificity 0.99, sensitivity 0.93, PPV 0.9, NPV 0.9.

In the test dataset, everything is almost the same (AUC, Specificity, NPV)

However , the sensitivity and PPV are terribly bad and almost zero.

Is there a reason for that?

I am getting these results using a decision tree.

I would prefer very much to try to improve the decision tree rather than trying another model

I would appreciate your valuable opinion

Dear Hatem,

I'm not an expert for classification trees. In general, some adaptation of 'weighting' is a possible approach for dealing with class imbalance. Perhaps you'll find this tutorial helpful: https://machinelearningmastery.com/cost-sensitive-decision-trees-for-imbalanced-classification/

Best regards, Dirk

Dear Hatem Ali :

First of all, I would try to look for another feature to expand your dataset.

Yet, there is another book that could be interesting to you: The Elements of Statistical Learning. Data Mining, Inference and Prediction. Trevor Hastie, Robert Tibshirani and Jerome Friedman. Second Edition. Springer Series in Statistics.

Yours faithfully.

Monica Perez-Priede