In terms of evaluating how well a neural network performs in a classification task with the number of classes greater than 2 (for example, classifying an observation into one of the 4 classes), which would be a better measure to use: (i) error-based measures such as the cross entropy loss, or (ii) strict accuracy rate?
And what would be the advantage(s) of one measure over the other?
Thank you.
The more important question is usually:
is your data balanced, i.e. do you have roughly the same amount of data for each class in your data set?
If that's the case, multi-class accuracy is a good and intuitive metric. Usually that's not the case and you are dealing with class imbalance. Here Cohens Kappa would be a good option:
https://towardsdatascience.com/multi-class-metrics-made-simple-the-kappa-score-aka-cohens-kappa-coefficient-bdea137af09c
Referring from the ResNet literature ( "Deep Residual Learning for Image Recognition", He et al. Using this due to its high number of cites ), we see that he compares his model on various fronts with respect to the size, type etc. While error-based measures are the most applicable, any one test wouldn't be enough to "validate" that model A is better than B. The efficient net paper ("EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks", Tan & Le) focuses on accuracy rather than the error rate. Both these paper are top-notch hence to answer your question directly, both are "viable" options.
Dataset is also to be taken into account as it isn't necessary model A works well on both Dataset A and B. Many papers also add the number of parameters generated.
My suggestion would be to make a study of the metrics of the literature you are trying to better and add the top 1% and 5% error/accuracy rates always while keeping common data. It would be an added bonus if you could add the same for the previous architecture attempts which have been made.
Soumya Snigdha Kundu: You should be careful, though. Top-k accuracy is only meaningful if the number of class instances in the validation and/or test data is known. This is the case for common benchmarks as ImageNet, but not necessarily for your custom data set.
An example: We have a small data set with 3000 samples for class A and only 10 samples for classes B,C and D each. We would get 99% correct predictions by writing a program that simply outputs "A" for any given input. Without knowing the class instance counts, accuracy is pretty much useless if you aim to convey the actual quality of your model.
Edit: It is still usefull to compare model performance, though. It's just not very expressive about the overall prediction quality of your model if your classes are not balanced.
Benjamin Staar: Thank you very much for pointing that out, I'll definitely keep that in mind!
But I said that in the research perspective because of this forum. I seldom find self-made datasets proposed in high quality research hence I went ahead with that notion. Will make sure to point it out next time!
The instance of imbalanced data I feel is a conditional constraint here. If we were to compare 5 techniques in the same imbalanced data, the top-k accuracy would be helpful right? (asking you for clarification)
Yes, for comparison there should be no problem.
In applied research, e.g. industrial or medical, applying deep neural networks to a novel, custom data set is pretty common. Not everybody is concerned with improving scores on ImageNet, MS COCO, CIFAR et al. ;)
Let's say I develop a new image-based system for classifying different types of mushrooms collected from the forest. Top-k accuracy would allow me to compare e.g. the effects of different tweaks and hyperparameters. However, it would not necessarily tell me if my system is good, i.e. if I can trust it with my life ;)
Benjamin Staar: Thank you for the clarification once again! Ill make sure to include this in my answers from now on!
- Badges
- Science topic