Technically, you can just look at formulas for that.

https://en.wikipedia.org/wiki/F-score

It is nicely explained by Ma et al. in 2019 [Analyzing the Leading Causes of Traffic Fatalities Using XGBoost and Grid-Based Analysis: A City Management Perspective] in figure:

https://www.researchgate.net/publication/336402347/figure/fig3/AS:812472659349505@1570719985505/Calculation-of-Precision-Recall-and-Accuracy-in-the-confusion-matrix_W640.jpg

This is the reason in image classification you should always make a confusion matrix, as it will show what your model does well and wrong.

In addition, you should always save your datasets, compare results and print out images which were classified incorrectly, this will help you understand what your model is lacking… maybe it was a noisy data, maybe label was wrong, maybe it should have been different class, you never know unless you check it.

These results are clearly not acceptable and appear to be incorrect, and this incorrectness could logically have several reasons. According to the obtained results, when the recall is 100% and F1-score is 100%, the accuracy must be 100%. If you feel visually that the accuracy is not 100%, you probably did the calculations wrong.

Md Foysal Ahmed

No, even if you have a perfect F1-measure, precision, and recall (all at 100%), it doesn't necessarily guarantee a perfect accuracy (100%). This is particularly important to consider in imbalanced datasets. In such cases, a model can achieve high accuracy by simply predicting the majority class all the time, even though it might miss important information about the minority class. Consequently, make sure to understand your data and evaluate your predictive model via techniques like cross-validation to ensure model validity and generalizability.

HTH.

Dr. Samer Sarsam

Samer Sarsam I think you are correct, such as imbalanced or small dataset such as only two image dataset or two types of class only. But when it's about big data then it's quite impossible to check the data manually. because, the model is blind only know what we are giving during the training. But in my model, small and big all dataset have showing the same results, in custom dataset and some public dataset that i modify according to the project theme. But if i want to publish this then there should be a lot of question, where i manually checked the testing is working correctly. So what logic i should reply to their question where they will say if the precision, recall and f1-score is 100% then the accuracy also should be 100%. It will be great, if you could suggest me some logics on this situations.

Md Foysal Ahmed

Before reporting the results of your study, it's imperative to conduct a comprehensive evaluation of the algorithm's quality and suitability for the problem domain. This involves rigorous validation techniques such as 10-fold cross-validation to ensure robustness. In selecting evaluation metrics, it's essential to prioritize relevance to the problem at hand. While accuracy is commonly used, it may not always reflect the nuances of model performance. Precision, recall, and F1-score offer more meaningful insights in many cases. Factors such as data size, quality, and class imbalance can influence the metric selection and interpretation of results. By focusing on metrics that align closely with the problem domain and considering potential dataset limitations, you can better assess the model's effectiveness and ability to generalize to real-world scenarios.

NB: I partially agree with Hadi's answer, Hadi Orak. This is because Hadi's statement is based on one assumption that may not always hold true in practice. While it's logical to expect that if precision, recall, and F1-score are all 100%, accuracy should also be 100%, this is not always the case.

Kind regards,

Dr. Samer Sarsam

With respect to the opinion of Dr. Samer Sarsam

When the value of FP and FN is zero, there is no chance to decrease the accuracy value. Anyway, if you give an example to prove this, we would be very grateful.

achieving 100% precision, recall, and F1-score indicates outstanding results in classification tasks, showing an occasional occurrence of false positives and false negatives. Although a 99.9% accuracy rate implies some infrequent misclassifications, it can be considered acceptable based on the specific context and the level of tolerance for errors, as long as it satisfies the necessary criteria for precision, recall, and F1-score.

Hello all,

Md Foysal Ahmed and Hadi Orak, in addition to my previous reply, here is further explanation:

Accuracy considers both true positives and true negatives, while precision, recall, and F1-score primarily focus on true positives and may not fully represent the model's performance across all classes. Therefore, while high precision, recall, and F1-score indicate strong performance, accuracy alone may not fully capture the model's effectiveness, especially in complex scenarios like class imbalance. In scenarios of class imbalance, where there's a significant disparity in class distribution, achieving high precision, recall, and F1-score for the underrepresented class is essential. However, if the model is biased towards the majority class due to this imbalance, it may prioritize predicting the majority class more frequently, potentially leading to inflated accuracy while neglecting the minority class.

Here is the answer to Hadi's question Hadi Orak:

While a scenario with zero FP and FN values may contribute to high accuracy, it's essential to recognize that accuracy can decrease in other situations due to various factors beyond FP and FN values.

I hope that helps.

Kind regards,

Dr. Samer Sarsam

Hello

With utmost respect to the Dr. Samer opinion. It should be mentioned that the explanation given by Dr. Ahmed shows that the data is balanced and even with the change of the data, an unacceptable result (accuracy less than 100%) has been obtained. Also, in the confusion matrix, factors beyond (FP and FN values) are not checked. It is expected that calculations, labeling, etc. are the cause of this error.

Best regards,

Hadi Orak

Hadi Orak

Even with balanced classes and no false positives (FP) or false negatives (FN), accuracy can still be influenced by factors like mislabeling or data ambiguities. Additionally, the model's performance near decision boundaries or with noisy data can independently impact accuracy, irrespective of FP and FN values. Hence, while low FP and FN values play a role in accuracy, they may not be the sole determinants in every case. It's crucial to validate the algorithm's code for quality assurance. Furthermore, as mentioned in my previous second response, various other factors can contribute to such outcomes.

Kind regards,

Dr. Samer Sarsam

Just show us the confusion matrix and all will be clear.

We can argue all day but numbers don't lie.

I feel that if the dataset is for binary classification, Hadi Orak is right as accuracy will be directly calculated from the confusion matrix and isn't affected by any factors beyond FP and FN.

But in the case of multiclass classification, it depends if the reported precision, recall and F1-score are average metrics over all classes or for any one of them. If they're average, still the results are unacceptable but if they're for some particular class, these might be acceptable and we'll have to check the average metrics or confusion matrix for more clarity

Results after some modifications in the code where I made mistakes before. Now, the accuracy, precision, recall, and F1-score remain stable and consistent after half of the epochs during training.

From what I see, all is fine. Your validation dataset might have 1-2 odd images reducing accuracy, you might want to check it as maybe some label is wrong, or maybe images have a lot of noise. Your test set is perfectly predicted. Congratulations and I wish you a nice article. You can check it by computing confusion matrix of validation dataset, or training set if you used percentage of that data for validation. Or you can just search by index which label in list is wrong.

Hello

excellent

The results are very pleasing. I recommend checking the images that caused the model to degrade, as they may be abnormal images or outliers.

The analysis still needs to be improved. However, the dataset seems unbalanced, and you don't have a binary classification. I agree with Dr. Samer Sarsam

Fernando Lavin Thank you for your advice. Could you please suggest some of the most advanced techniques suitable for hybrid model analysis, as well as the best techniques for handling imbalanced datasets? Even if I evaluate on a public dataset (because the custom dataset i won't be able to be made public, since it belongs to XYZ company). I will definitely investigate according to your suggestions.

No problem, Md Foysal Ahmed. In your case, I recommend reviewing the literature related to your field to explore potential hybrid model analysis techniques. Additionally, I would like to request Dr. Samer Sarsam to address your second question regarding the best techniques for handling imbalanced datasets.