Dataset imbalance is a major problem usually encounter in machine learning practice. Some technique such as SMOTE or ADASYN generate addition data (synthetic data) to avoid imbalance problem. However I was wondering about the quality and correctness of these synthetic data and their corresponding label.
Should we place our trust on the new data label ?
Actually machine learning models need lots of data for training but collection of real field data in various circumstances is very difficult and impossible. In this point of view techniques like SMOTE and ADASYN helps to increase the available data. The proposed model which is trained on synthetic data when we test on images in the same data may give good results but when we test our model on the real field, accuracy may be less.
How well your synthetic data corresponds to real data depends on your dataset that you are synthesizing from. Just like in classification, generalizability is determined by how well your data represents the real world problem. Methods like SMOTE and ADASYN try to learn patterns from your data and generate observations based on that, this systematic generation can be seen as patterns in the data (Look at the image, provided by Machine Learning Mastery).
How are going to use synthetic data is also important. Adding it to training data and then testing with real data only is the right way, so that you don't affect the performance evaluation with your synthetic data.
There are several synthetic data generation methods and each method has it's underlying assumptions. If the assumptions are correct, then the synthetic data will be correct/representative of the real system. Therefore the quality of the synthetic data depends on the assumptions of the generating method. However, there is no straight forward/easy ways to evaluate these assumptions, domain knowledge / experience might be of help.
- Badges
- Science topic