I have trained a VGG-16 model toward a binary classification task. The model was trained on equal numbers of abnormal and normal images (n=2000). Literature studies demonstrate that model calibration is performed, when the model is trained on an imbalanced dataset, to rescale the probabilities to reflect the true likelihood of occurrence of the samples of a given class. However, I experimented to see if calibration impacts performance on a model trained on balanced dataset. The model, though, trained on a balanced dataset, underpredicted the positive class as observed from the below figure as the uncalibrated outputs were lying above the y=x diagonal. I observed that on applying various calibration methods, the expected calibration error (ECE) decreased compared to that obtained with the non-calibrated output (Non-calibrated ECE:0.039 vs. Platt calibrated output: 0.02). Also, the calibrated outputs closely followed the y=x diagonal. After calibration, the precision, Kappa, F-score, and MCC metrics increased compared to that obtained by the uncalibrated outputs.
I have the following questions:
Dear Sivaramakrishnan,
Whether or not the classifier resulting from the training exercise is a calibrated probabilistic classifier depends on the training criterion (optimisation objective) you apply. Neural networks typically deliver uncalibrated soft classifiers as their output (like also SVMs do; that's what motivated Platt to proposing his calibration method). The paper on beta calibration ("Beyond sigmoids: How to obtain
well-calibrated probabilities from binary classifiers with beta calibration") by Kull, Silva Filho and Flach explains this in more detail. Again depending on the training criterion, when compared according to different criteria the calibrated version of the classifier may perform than the uncalibrated original classifier.
Best regards, Dirk
Why not just cut to the chase and use logistic regression? See the attached search for references. Especially the second one..
David Booth
Yes, it can as the CNN/NN training always gives an uncalibrated classifier for the generic applications considered at the time of training. However, once calibrated the CNN/NN can lose the genric nature of the application.
I think calibration error does not depends on the capacity of model learning towards positive and negative samples because the model capacity depends on how good is the database and it is independent of calibration error
- Badges
- Science topic