Suppose that we have a unbalanced data-set for a binary classification problem and we want use 10-fold cross validation for training and testing fitted model.
* Is this correct that we only use sampling methods (under-sampling, over-sampling or SMOTE) in training data?
* If yes how we can implement this sampling methods for 10 fold cross-validation? We should re-sample minority class before cross-validation or we should use a new structure of k fold cross-validation?
* Anyway to implement sampling methods with sliding validation? (I'm working on a binary time series perdition - one step ahead, up-turn and down-tern of output of t+1 comparing to t - t is time).
* Is it not more appropriate to use these sampling methods separately for every year of data-set?
Number of up-turn and down-turn samples in every year:
Total Up Down
_____ ___ ____
2009 234 135 99
2010 243 153 90
2011 241 132 109
2012 240 133 107
2013 240 155 85
2014 241 110 131
2015 243 126 117
2016 29 24 5
All data 1711 968 743
I must admit I don't follow what you are doing. Nevertheless in validating logistic regressions i have found that bootstrap cross -validation was often preferable to K-fold cross- validation. I have attached 2 talks that I found on the internet. that give the basic idea. The links are also included. I hope this helps a little. Best wishes.
https://lagunita.stanford.edu/c4x/HumanitiesScience/StatLearning/asset/cv_boot.pdf
http://stat.cmu.edu/~brian/724/week11/lec27-bootstrap.pdf
@Juan Thank you for answer. So your proposed solution for this problem is only using stratified cross-validation not sampling methods? I think using this method does not solve the main problem (unbalanced data set) because finally we are training model with all samples for predicting future. Is this correct?
@David . Thank you for answer David. Based on cited papers bootstrap cross -validation has some problems (bias and variance) and i think it is appropriate for high unbalanced data sets not my data set. What do you think?
Honestly, I don't know. Prof. Steyerberg, who is an expert, answered a sort of similar question. His answer there might be helpful. Link attached. Hope some of this helps a little. Good luck.
https://www.researchgate.net/post/How_to_interpret_the_results_of_5-fold_cross_validation
- Badges
- Science topic