I want to know if have some way to combine datasets with a different number of instances into one so that the final dataset is not biased to larger datasets.
For instance: we have the dataset A, B, and C with 2000, 500 and 50 instances, respectively. If we combine into a D dataset, with 2550 instances, the result should be biased to A, with 2000 instances.
Since I would like to use statistical tests such as Wilcoxon signed-rank test, my idea is to replicate, sampled, the smaller datasets until they have the same number of instances as the largest.
Following the example, we would have the dataset D with 6000 instances, being 2000 instances of A, 4 * 500 instances of B, and 40 * 50 instances of C.
Does this make sense?
https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test
Hi Vitor,
From the information you provide, you can model this problem as an imbalanced dataset problem. In this case, it makes sense to oversample your "smaller" datasets and there are several ways of doing it. A very famous approach is by using SMOTE:
https://www.jair.org/media/953/live-953-2037-jair.pdf
It will create synthetic instances based on the existing ones (thus it is not simply repeating instances).
There is a vast amount of oversampling (like SMOTE) and undersampling techniques. A practical way of applying them is by using this python library:
https://github.com/scikit-learn-contrib/imbalanced-learn
WEKA and other data mining tools also have scattered implementations of sampling techniques.
Best Wishes,
Heitor
Thank you, Heitor,
but, is SMITE used to learn a model in imbalanced-class, right? My problem is to evaluate/measure the result. Something like "how to normalize the F1-score of this dataset built combining 3 different datasets?" (and use a statistical test to confirm it).
Can I use SMOTE to balance the number of instances (not class) in the dataset?
Hi Vitor,
SMOTE will create new samples based on the class label, what you can do is "trick" it by assigning a dummy class label for each dataset, e.g., instances from dataset A will be assigned class A. You just need to be sure if it is ok that SMOTE generate synthetic instances for your problem. Notice that these instances will be at least slightly different from the original instances (so be aware of that).
Best Regards,
Heitor
Vitor,
If you were working from INESC today chances are that we were sitting right next to each other as I am a visiting researcher at INESC and I was there at LIAAD today :)
- Badges
- Science topic