I have a huge set of training data devided into two class. The class no. 1 contains over 300,000,000 cases while class no. 2 has only about 2000 cases. I want to use machine learning technics to create classification model according to the data (I started working with neural networks). The question is, should I select approximately the same number of cases from class 1 as in class no. 2, so the training set would have 2000+2000=4000 cases? What will be with the rest unused data? I assume that it depends on the data but I ask for the general convention. Maybe some anomaly analysis would be appropriate in that problem?
Sincerely,
Mateusz Soliński
Yes! It does matter. See if the following technique could help:
https://www.cs.cmu.edu/afs/cs/project/jair/pub/volume16/chawla02a-html/chawla2002.html
Hello. The number of samples matters very much in any statistical estimation technique, and machine learning (aka classification and regression theory) is no different. It's useful to start with simple least-squares regression to get a feel for how much data you need to get a given accuracy. For 1-D, the variance of the estimation error typical goes down inversely with the number of samples. In any case, a good rule of thumb is to have roughly equal number of cases for each class in supervised learning. Since you have a lot more cases in one class, you could forgo the resampling for that case when you do cross-validation and use independent samples for your case 1.
I assume you are trying to classify your observations as coming from one of your classes or the other. Depending on your method, it will be hard to get your model to predict that an observation comes from the smaller class. If one group has three times the observations as the other, then you can lean toward the small group whenever you get a predicted probability greater than 25%. Your case is far more extreme. Everything gets easier if your classes are balanced.
Yes, it does matter. Weighted SVMs can be used to account for the unequal class numbers. However, yours is an extreme case. I would go with 2000 samples from each class. Use the rest of cases from class one as an independent test set and compute how good my model performs.
Yes, an imbalance of the sort you are describing will lead to something called classifier bias.
Whether this is a good or bad thing depends on the nature of the data you are working with. Is the division of samples due to the rarity of one class in the population, or due to sampling bias? If your data set does not reflect the makeup of the population than you have a case of sampling bias and should take steps to correct it. If your data reflects the actual population and the minority class is just very rare then changing the proportion of samples from each class may lead to more false positives.
The attached paper details some methods for up-sampling minority classes or down-sampling majorities that you might find useful.
http://www.eecs.wsu.edu/~cook/pubs/icdm13.2.pdf
Conference Paper Handling Class Overlap and Imbalance to Detect Prompt Situat...
- Badges
- Science topic