For a balanced dataset, we use a confusion matrix and the derived performance metrics; accuracy, precision, recall, F1-score. But what if the dataset is unbalanced, I think these metrics aren't enough, is it right, if yes share with me extra details please.
I recommend Brier score based on leave-one-out cross-validation. But more details about the dataset are needed in order to better understand your concerns.
Andrey Davydenko Thanks alot
I am talking about UNSW intrusion dataset which have millions of records [Normal] and a hundred thouthands of records [anomal]
You can check this article - https://machinelearningmastery.com/tour-of-evaluation-metrics-for-imbalanced-classification/
If there are millions of records, the leave-one-out approach may not be feasible, but it depends on the settings you have.
I prefer Brier score since it lets you evaluate the quality of uncertainty estimates. It should be suitable for both types of the records you mentioned. And the link provided by Ferdib Al Islam is great, I think it describes the Brier score very well.
Let me also recommend this work:
Chapter Forecast Evaluation Techniques for I4.0 Systems
It describes relevant data formats to store your predictions and also presents error metrics for numeric predictions.
There are many performance metrics that can be used to evaluate unbalanced datasets, such as sensitivity, specificity, false-positive rate, false-negative rate, geometrical mean, positive likelihood, Diagnostic Odds Ratio (DOR), Discriminant Power (DP), and YI.
It's better to solve the imbalance issue before evaluating to create more reasonable results. Try to use SMOTE techniques.
Try sensitivity, specificity, false positive, and negative rate.
And the best solution is to solve the imbalance between classes for reliable results.
Please check the following
The F1_score represents a more balanced view compared to the ACC, Prec., and Recall metrics but could give a biased result because it doesn’t include True Negatives (TN). Recently, several scientists highlighted drawbacks of the F1_score measure [1-3], and in fact, Hand and Peter [4] claim that alternative measures should be used instead of F1_score.
On the other hand, MCC considers all four entries of the confusion matrix (FP,FN,TP,TN). MCC is not affected by the unbalanced data, and MCC is preferred over F1_score as it is a more ‘balanced’ assessment of classifiers, no matter which class is positive. MCC produces a more informative and truthful score in evaluation than accuracy and F1_score.
[1] Chicco, D., Totsch, N., & Jurman, G. (2021). The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData mining, 14(1), 1-22.
[2]Naulaerts, S., Dang, C. C., & Ballester, P. J. (2017). Precision and recall oncology: combining multiple gene mutations for improved identification of drug-sensitive tumours. Oncotarget, 8(57), 97025.
[3] Chicco, D., & Jurman, G. (2020). The Advantages Of The Matthews Correlation Coefficient (Mcc) Over F1 Score And Accuracy In Binary Classification Evaluation. Bmc Genomics, 21(1), 1-13.
[4] Hand D, Christen P. A note on using the F-measure for evaluating record linkage algorithms. Stat Comput. 2018; 28:539–47.
Accuracy won't hold well for unbalanced data. The metrics like Sensitivity-Specificity Metrics and precision-recall metrics can however give a good overview of performance. Similarly, True Positive rates and False Positive Rates along with ROC AUC or G-mean can also provide a good overview of performance.
Try to handle the class imbalance problem first. You may use synthetic minority over-sampling technique on the training sample. You may get useful information regarding the model's performance evaluation for imbalance data by visiting the following link: https://machinelearningmastery.com/tour-of-evaluation-metrics-for-imbalanced-classification/
Omar Elzeki, It depends on the LEVEL of your imbalance dataset. For a modest imbalance (say 4:7) dataset, the Fawlker-Mallows index could be performed better. And for the drastic imbalance dataset (say 1: 900), the Jacob index could be better. Sometimes, Jacob performs less compare to the Fawlker in drastic imbalance dataset.
I think that you should apply both evaluation metrics and see the performance according to your dataset.
Omar Elzeki in addition to everything already recommended, there is also balanced accuracy
https://statisticaloddsandends.wordpress.com/2020/01/23/what-is-balanced-accuracy/#:~:text=Balanced%20accuracy%20is%20a%20metric,the%20presence%20of%20a%20disease.
For unbalanced data, try Matthews correlation coefficient (MCC).
Check https://en.wikipedia.org/wiki/Matthews_correlation_coefficient.
A better metric for unbalanced data would be K-S statistics that tells you the separation power of the model.
- Badges
- Science topic
- Similar topics
- Filing