Hello, my fellow researchers,
I am dealing with purely unbalanced data for a supervised classification Deep Learning (DL) problem. The model that I am currently running is dealing with acceleration signals collected from sensors with the aim of assessing damage into civil engineering structural systems.
As it is well known, most DL algorithms work best when the number of sample in each class are about equal. In this case, the metric to assess the performance of the DL model is accuracy. However, in the case of unbalanced data, another metric called f1-score (The weighted average of precision and recall) is the performance metric to assess how well the model is classifying the data points.
I am having some results now, but my f1-scores are a bit low! (35%), however, the accuracy of the model is very good (more than 90%). These values can be accepted for thesis submission, however, I would like to have an idea of how scientific journals deal with this kind of situation.
Generally, for the case of balanced data, the reported accuracy is more than 90%. Yet, for f1-score, is there any minimum value that should be reported or what?
I look forward to hearing from you.
Thanks for your time and consideration.
Majdi
Is it a binary or multiclass problem? Are true negatives and true positives equally important? F1 doesn't account for true negatives so that's something that should be considered.
For unbalanced problems, I'd suggest AUC or Matthew's Correlation Coefficient if it's a binary classifier, although there is a multiclass extension of MCC!
Here's a paper discussing the advantages of MCC over F1/Accuracy:
Article The advantages of the Matthews correlation coefficient (MCC)...
Multiclass extension of MCC can be found on the Wikipedia page:
https://en.wikipedia.org/wiki/Matthews_correlation_coefficient#Multiclass_case
Let me know what you think!
For unbalanced or non uniform data, the appropriate models are non-linear probability distribution models, see e.g., Free-Downloade https://epdf.pub/aspects-of-multivariate-statistical-theory.html
See also
Chapter In Directional Statistics
Ethan G. Armstrong It's a multiclass problem. Besides, the TN and TP aren't equally important.
I have been recommended to read this paper. Article Classification Assessment Methods: a detailed tutorial
Where the author insisted that the F1 score is sensitive to class imbalance. For this reason, I am thinking of AUC for both binary and multi-class.
But, I didn't know about MCC before, I will try to implement too and see!
Thanks so much.
Ethan G. Armstrong I am wondering if there are any built-in functions (ready to use) to compute the MCC in my DL model?
Hey Majdi Flah ,
For built-in functions to compute MCC:
R: mltools
- Link: https://cran.r-project.org/web/packages/mltools/mltools.pdf
Python: scikit-learn
- Link: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html
Hope that helps!
- Badges
- Science topic