Dear all, In machine learning, the most common way of splitting a dataset to obtain the training and test datasets is to randomly allocate the cases into one or the other taking into consideration the predefined p% of cases to be included in the test set. This approach will lead in some cases, particularly when using small datasets, to unbalanced training and test sets, that is, the training set may not contain extreme values of the predicted variable and/or combinations of the input variables. After training, the "whichever" learning algorithm is being used, will have, most likely, poor prediction ability and as a consequence a smaller applicability domain.
In what circumstances you decide not to randomly assign the cases but to force some of them to be present in the training set?
Do you, for instance, stratify your dataset into classes/bins reflecting the value of the variable(s) to be predicted and allocate a certain p% of cases of each class into the test set and (1-p%) onto the Test set, instead of randomly selecting the p% cases from the whole dataset? Do you stratify according to the predicted value or according to the input variables values?
Regards, Luis
Sir, there is a nice solution regarding your training-testing data splitting problem, that is k-fold training and testing. You may try 5 fold or 10 fold approach to take the decision.
Regards!
https://stackoverflow.com/questions/17200114/how-to-split-data-into-training-testing-sets-using-sample-function
https://cran.r-project.org/web/packages/dataPreparation/vignettes/train_test_prep.html
Is it helpful ?
Sir, there is a nice solution regarding your training-testing data splitting problem, that is k-fold training and testing. You may try 5 fold or 10 fold approach to take the decision.
Regards!
Luis F. Gouveia try this link
https://towardsdatascience.com/train-validation-and-test-sets-72cb40cba9e7
Thank you all for your contributions. However, I asked your help regarding a particular issue "when not to randomly divide the whole dataset into train and test sets" and I still can´t find the rational for it, namely by using stratifying or k-folding strategies.
Regards, Luis
Me and Andrzej Foremny
have classified construction contracts to three sets. One of them was "collusion highly expected". In our 250 cases dataset there were only 9 cases matching the set "collusion highly expected". We trained the classifier with 5 of them and controled the result with 4 cases. The total number of sets in vaidating data was 70, but finding the cases of "collusion highly expected" was a core of the research. Despite the low number of training cases of this kind, we've got 3 (out of 4) true-positive. It was our decision not to train fully random.Regards
Hubert
Dear Hubert Anysz , (long time no see..) I trust you're doing well and I appreciate your contribution. In fact that's the kind of feedback I was expecting. Although the k-fold cross-validation approach suggested by Md. Asadur Rahman works well in many cases, I would expect it to fail in the example you shared. Now, the question is: how did you picked the 5 cases to train? Was there any formal rational (based on the predictor variables, for instance) ou you just randomly pick 5 out ou those 9 "collusion highly expected" available cases?
Thanks again and best regards, Luis
you can use randomization before splitting train and test datasets except you have time series data or order is important. In that case there are ways to do it.
If you have imbalanced dataset then you can do oversampling or undersampling or SMOTE before randomly splitting it.If you do undersampling, you can try k-fold cross validation method.
I'm glad to meet you Luis F. Gouveia here again and answer sth (not anly asking) :-)
Yes, it was a random choice 5 out 9 to training dataset. Once the cases were classified to the "c. h. expected" we didn't make any additional assesment among them. Just random choice.
Take a look at the two papers about creating the split considering distribution shift:
Preprint High Dimensional Restrictive Federated Model Selection with ...
and
Preprint Variational Resampling Based Assessment of Deep Neural Netwo...
- Badges
- Science topic