I am training a deep learning model toward a image classification task. The VGG-16 model is trained individually on two different training sets with varying degrees of data imbalance. Set-1 had only 25% positive samples (class-1 (positive, label:1): 100; Class-0 (negative, label:0): 400). Set-2 had only 80% positive samples (class-1 (positive, label:1): 320; Class-0 (negative, label:0): 400). Set-1 is highly imbalanced compared to Set-2. The individual trained models were evaluated on a common test with equal numbers of positive and negative (n=100 each) samples. Since the dataset is imbalanced in both Set-1 and Set-2, the output probabilities would not be calibrated, so, I applied temperature scaling to rescale the probabilities. Figure (a) and (b) shows the results before and after calibration for the model trained on Set-1 and (c) and (d) for the model trained on Set-2. I observe that the expected calibration error and maximum calibration error reduce after temperature scaling and the probabilities are scaled to closely follow the y=x diagonal. However, it is not clear how to interpret the pre-calibration and post-calibration curves in terms of data imbalance. How the fraction of positives in each confidence bin, pre- and post-calibration, can be explained in terms of data imbalance?
Dear Sivaramakrishnan Rajaraman,
1. Use the augmentation of those classes for which you have an imbalance; 2.if you use the metric for evaluating the quality of the model as "Accuracy", then there should be no big problems, since this metric takes into account the imbalance of samples by classes.
Shahboz Sharifovich Qodirov Thanks. The question is not about the augmentation performance. Provided I get a reliability diagram for a model trained on an imbalanced dataset, I have to interpret how this data imbalance is reflected in the diagram. For instance, trained on a minimal number of positives, the model may overlearn the majority class (negatives) and may misclassify all instances as belonging to negative class. In that case, the probabilities of all classified instances may lie between [0 0.5]. But how this is reflected in the reliability diagram?
Sivaramakrishnan Rajaraman ,
Refer to the work in [1] p.6 for explanation of the temperature scaling with respect to the softmax.
Regards
[1]Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017, July). On calibration of modern neural networks. In International Conference on Machine Learning (pp. 1321-1330). PMLR.
Available at: proceedings.mlr.press/v70/guo17a/guo17a.pdf
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training