Hello Luis ..  Here is the practical DM answer: If you have a really large dataset; that is one that you are certain provides more than needed range and resolution of the input and output variable values, then you could chance using only one training and one test set.  However, in general, it is always a good idea to perform cross-validation when building statistical models.   Perhaps most importantly, you can use the performance statistic of k-cross-validation model builds to estimate the confidence of the average model coming up with an accurate prediction.  If you like, you can actually use all cross validation models; asking each to vote on the predicted class (or compute the average real output) - but that gets us into ensemble methods (another story) . 

Here is a slightly more mathematical ML answer:  Every model can suffer from the bias of the ML algorithm, or the variance of the chosen training data.   Ideally, a model has been developed with low bias and low variance, however this is difficult to achieve with a single ML algorithm and finite training set.  All algorithms have their inherent bias, and each training set will tend to have those examples it favours and therefore different variances.  Therefore, the best you can do to test a chosen ML algorithm and its configuration is to build multiple models using different training and test sets and compute the mean performance accuracy and standard deviation (or confidence interval).  Note that you then still have that problem of choosing the best model for use in production.   If I am going to pick one, I choose the one closest to the mean performance calculated and not necessarily the most accurate model.

.. Danny 

It is recommended to use cross-validation everytime because test error of a ML method will never be the same as trainning error. Generally, test error is greater than training error and cross-validation helps you to choose among several ML methods. The size of the test set depends on the size of the entire data set. You can start for example with 50%-50% if you have a very large data set but you can also use something between 70-80% for training dataset.

regards,

Bogdan

First of all you should have a large dataset (which might introduce bias in the study) and at the same time a sufficient number of events (in order to create the k-fold). Both conditions are essential for the use of cross validation. If the dataset is not quite large you don't need of cross validation, if it doesn't includes a sufficient number of events you can proceed with bootstrap method.

Remember that overfitting depends on the number of parameter you put in your model.

The relationship between test and validation set is guided by the number of events too. Anyway in some field the inner validation is less powerful than the external one.

Ale

With 70% training dataset and 30% test dataset, it seems you are applying a holdout method. The holdout method is the simplest kind of cross validation. The data set is separated into two sets, called the training set and the testing set. The function approximator fits a function using the training set only. Then the function approximator is asked to predict the output values for the data in the testing set (it has never seen these output values before). The errors it makes are accumulated as before to give the mean absolute test set error, which is used to evaluate the model. The advantage of this method is that it is usually preferable to the residual method and takes no longer to compute. However, its evaluation can have a high variance. The evaluation may depend heavily on which data points end up in the training set and which end up in the test set, and thus the evaluation may be significantly different depending on how the division is made. Though the method is good, I always advocate for K-fold cross validation.

K-fold cross validation is one way to improve over the holdout method. The data set is divided into k subsets, and the holdout method is repeated k times. Each time, one of the k subsets is used as the test set and the other k-1 subsets are put together to form a training set. Then the average error across all k trials is computed. The advantage of this method is that it matters less how the data gets divided. Every data point gets to be in a test set exactly once, and gets to be in a training set k-1 times. The variance of the resulting estimate is reduced as k is increased. The disadvantage of this method is that the training algorithm has to be rerun from scratch k times, which means it takes k times as much computation to make an evaluation. A variant of this method is to randomly divide the data into a test and training set k different times. The advantage of doing this is that you can independently choose how large each test set is and how many trials you average over. For more information browse https://www.cs.cmu.edu/~schneide/tut5/node42.html

Hi Luis,

The standard way of predicting error rate of a classifier method is to utilize stratified tenfold cross-validation. However, you could use holdout method when your aim is doing fast learning process.

HTH.

Samer

You may need to simply apply a single hold-out (eg 70-30% split or similar), a cross validation, or a nested cross validation, or the Tibshirani and Tibshirani correction. Please, read our paper "Performance estimation properties of cross-validation based protocols with simultaneous hyper-parameter optimization" which contains the details: http://mensxmachina.org/en/publication-details/?docid=f542c003-8f6b-30fd-b5c0-da5981a35285

regards, hope it helps IT

You may try different proportion in cross-validation using Weka and test and analyze the result sets....

K-fold cross-validation is a good choice, although my experience is that it tends to show higher accuracy than when the same method is used on production (previously unseen) data.  Still, it gives a good sense of which algorithms perform best and the results will be in the ballpark.  As noted by another poster, be sure to use stratified cross-validation to ensure that the classes are proportionally represented in each fold.