Hi, I am working on deep learning for object detection (binary classification). I read from various sources that it is preferable to have a balanced training data. I was hoping if someone could link me to some papers that substantiate this claim ?
I ask because I am performing hard negative mining to augment my training set of negative samples. This however, will lead to an imbalanced training set as it will result in more negative than positive samples. Is there any way I can alleviate the problem of imbalanced training data ? And is hard negative mining a good idea in CNNs ?
Dear Haziq, your notion is correct. Imbalanced classes lead to substantial problems in binary classification and the interpretation of results. One way to address this is to employ a different performance measure than accuracy (e.g. AUC-values) in conjunction with permutation tests. Although from another domain, in my article I try to shed some light on how to tackle class imbalance in applied machine learning.
Article EEG Responses to Auditory Stimuli for Automatic Affect Recognition
Hi,
I am no expert on CNN object detection, but in "Taking a deeper look at pedestrians" the training is also performed using an imbalance. They state that the influence of this imbalance is minimal on the resulting accuracy. Note however that the samples they use are generated using another pedestrian detection algorithm.
I think it depends on the application if the balance matters. In the case of object detection, the amount of samples that should be classified as negatives is in most cases a lot larger as the ones that are positive. This is also reflected in the training procedure when using non-CNN approaches (ACF, ICF, ...), where the amount of negatives is in a lot of cases 5-10 times larger as the amount of positives. I would not see a reason this should be different for CNN.
Imbalanced class sizes aren't a problem for maximum-accuracy methods. The following paper presents a brief description and comparison of various legacy methods versus enumerated-optimal classification tree analysis.
- https://www.researchgate.net/publication/291947229_Using_data_mining_techniques_to_characterize_participation_in_observational_studies
Article Using data mining techniques to characterize participation i...
There are many ways to solve your class imbalance, the most used approach that I'm aware of, and which is actually easy to do and also implemented on almost every Deep Learning framework, is to use penalized models. On these models, they impose an additional cost on the model for making classification mistakes on the minority class during training. These penalties can help the model to pay more attention to the minority class, and in summary you're forcing the model to pay more attention to the classes where you have less data.
Dear Haziq,
you might find this paper useful. It suggests an interleaving learning method to balance between hard and easy samples and improved the results of a CNN generating descriptors for optical flow.
A correlation between the difficulty of samples and their subcategory should be considered since some classes might be neglected or because the optimization of the harder class might disrupt the easier one. On that case, a learning method that uses all samples but account their difficulty could be preferred.
Conference Paper Optical Flow Requires Multiple Strategies (but Only One Network)
with HDLtex you can create several layer to make you data set balanced hierarchically
Please read this paper,
Conference Paper HDLTex: Hierarchical Deep Learning for Text Classification
Please take a look at Conference Paper Class Rectification Hard Mining for Imbalanced Deep Learning
- Badges
- Science topic