Hello,

To the best of our knowledge, usually K-fold cross validation is used when no test and validation data are available or when only small samples are available.

K-fold cross validation method allows to use different training and testing data which will avoid the overfitting and give better generalization ability.

It also prouves that the classifier performance are independent from the training data.

Best regards.

Hi Wassila,

K-folds cross-validation is to divide your data into for example 3 parts ( part1, part2 and part 3). Then making fold1 contains part1 as a training set, part2 as a validation set and part3 as testing and fold2 contains part 2 as a training, part 1 as a validation and part 3 as a testing. Also, fold 3 contains part 3 as a training, part 2 as a validation as part 1 as a training. This will form 3-folds cross validation. Thus, you can generalise your results on this dataset.

OR at least 2- folds for training and testing only without validation parts if you can guarantee that your approach has not overfitting or underfitting issues.

Best regards,

 Osman

Hi,

My understanding of the role of CV in machine learning is to estimate the tuning parameter. For example, decision tree has a complexity parameter which effects on how bushy of a tree grows. Since CV data are inherent in the training data, the prediction accuracy can be estimated for CV data to guide the selection of the turning parameter to have minimal CV error (actually 1-SE rule can be applied to be on the conservative side of overfitting). With the calibrated tuning parameter, you can prune the tree to have somehow objective criterion for model selection.

The external validate data are not part of algorithms but can serve as an independent testing data for verifying the model.

Best,

Weidong

I do not think so. If you having k-folds and each fold contains training, validation and testing and then you change training to become validation set in one other fold and testing set in another fold and similar for other parts. This will mean that you recover the whole dataset to become as test set (Unseen data) for the training models by your algorithm. Other things, your algorithm can not generalized as learning model on this whole dataset as unseen test data  with validate the result in every part of the dataset, while overfitting is a different issue.

Wassila and Osman, 

From the name of cross-validation and validation set, one can understand that both help in validating learning process and avoiding overfitting, though. Cross-validation is very useful because one round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (a.k.a. training set), and validating the analysis on the other subset (called the validation set or testing set). Thus, if you are using cross-validation, then no need to have a validation set (and vice versa). In sum, having both approaches in one process is not quite sensible.

Note that, in order generalize the model, cross-validation needs to perform additional run (11 th) over all the training set in order to build the ultimate predictive model.

HTH.

Samer

Additional run with different dataset parts because you just train on one part and test in different because the performance may be varied as happened in some LETOR datasets with me. This will require from you to take the average predictive values. Also, every run on the same fold will varied by small variation that will need to take the average of number of runs in each fold. Furthermore statistical test p-value should validate you significance in results

Well, in details:

The standard way of predicting the error rate of a learning technique given a single, fixed sample of data is to use stratified tenfold cross-validation. The data is divided randomly into 10 parts in which the class is represented in approximately the same proportions as in the full dataset. Each part is held out in turn and the learning scheme trained on the remaining nine-tenths; then its error rate is calculated on the holdout set. Thus, the learning procedure is executed a total of 10 times on different training sets (each set has a lot in common with the others). Finally, the 10 error estimates are averaged to yield an overall error estimate. Running cross-validation for the 11 th time aims at building the final model from all the loaded data where this extremely useful for model generalization. 

Sometimes people tend to divide the original data into 3 parts (training, validation, and test) with a specific percentage value for each. This case can be used when cross-validation is not applied. 

In sum, as I mentioned before, both cross-validation and validation test are used for validating the predictive model (i.e., provide a general idea of the performance of the algorithm) in a fairly way, although they might not exactly produce same results.  

Cheers, 

Samer

The better option to provide closed-box (.exe, .jar) with test benchmarks if you want to document it as an academic article for researchers to other researchers compare with it and test it. But if this belongs to the company, you do not need to document it as research outcomes in competition with other researchers because it is private research belong to companies and achievement is the achievement for that company not for research. No one can compare with research from academia except the same company that have the data and packages.

You are going far from your main question "Why there is a need for K-fold cross validation regardless if you have validation set in the dataset?", so your reply makes no sense. 

However, to correct your statement "statistical test p-value should validate you significance in results", generally, we do rely on p-value when comparing the performance of several learning algorithms (examining what algorithms perform better than others) in terms of, e.g., F-measure, accuracy, etc., but not to validate the algorithm that we built.

Cheers, 

Samer