I have a data set which is already divided into 10 folds with each fold having training,validation and test sets. I'm not able to understand how to apply 10 fold cross validation on this data set.
In general, if we want to apply k-fold cross validation on a data set, the procedure is as follows
- shuffle the data set
- split the data set into k-fold
- for each unique fold, take it as test set and the rest of the folds as train set. Fit the model and then evaluate the model. This procedure is repeated for all the folds.
- Take the average of all the models.
In my case, each fold is internally divided into train, test and validation sets. How to apply 10-fold cross validation on this kind of data set?
Hi Kalyan,
Cross validation is much popular technique in DS/ML communities. There are multiple objectives behind using this technique. Few of them are: to get best hyper-parameters set using grid search(e.g. in case of Xgboost), to get generalized model (having trade-off between bias and variance), etc.
As I can see in your case, you have already divided data in training, validation & test set. Please note that CV is always performed on training set. In case of k-fold cross validation, say number of records in training set is 100 and you have taken k = 5, then train set is equally divided in 5 equal parts (say: t1, t2, t3, t4 & t5).
Now, in first iteration t1 will be left out and remaining 4 parts (t2, t3, t4 & t5) will be used for training the algorithm and t1 to validate it. Once algorithm is trained this trained model will be validated on t1 from this you will get error/accuracy metric. Now in 2nd iteration t2 will be left out and again algorithm will be trained on remaining 4 parts( t1, t3, t4 & t5) and once algorithm is trained it gets validated on t2. It continues till all 5 iterations are over. At last you will get average train and validated error/accuracy metrics from cross validation exercise. And this is how CV works and is used.
Here I would recommend to just divide data set into train and test set and then use CV on train set. Internally CV will divide train set into k equal parts which in parallel both train and validate the algorithm. Once algorithm is trained you can check the error/accuracy metrics on test set.
Hope I am able to answer your queries. Thanks!
Hello,
You're not supposed to have training, validation and test sets from each fold. The steps is as follows:
(1) the data set must be randomly divided into k parts (in your case k = 10, and you have already done so);
(2) one of the parts is taken away from the original data set, resulting in a training set consisting of k-1 parts, and a validation set of 1 part;
(3) you train your model based on the training set (k-1 parts) and predict the validation set (part kept out). Its performance is evaluated by comparing the prediction for the part kept out and its real values from part kept out (e.g. MAE, RMSE, MSE, MBE).
(4) you repeat step 2 and 3, considering at each time a different fold as validation set and k-1 as training set. Then you may average the performance from all the 10 folds procedures.
You may also perform a repeated k-fold cv, where you'll repeat step 1-4 by using random 10-fold splits from original dataset, and averaging all performance values stored.
-> n = number of times you re-split original dataset, example repeated k-fold cv (n=10, k=10).
Regards,
Hello Kalyan Katikapalli,
In this case, there is three options, to put it short, group all the data in just 3 groups, train, validation and test, using a "for" loop will be enough. In the other hand, you should apply k-fold cross validation to each different set (in this case 10 times). The last option, is to group the data in train (train + validation) and test. Example:
1. Use a for loop to create 2 groups, if the label is train or validation ---> train, is the label is test ----> test,
2. Use a function to create the folds, either, train_test_split in scikitlearn in python or a function made by yourself,
3. Your datasets are created
Regards,
- Badges
- Science topic