If you are using the same partition every time, then it does not matter as most machine learning algorithms are deterministic and same training data would produce same classifier. If you partition randomly at every iteration, then it is one of the many ways of doing 10 fold cross validation. 10-fold cross validation can be done in following ways:

1. Create 10 instances of 90-10 split randomly. In this scenario, some of the data may always only occur in train/test set. This is called 10-fold random cross validation

2. Create 10 equal sized partitions of data randomly and at 10 instances of learning, use 9 of them for training and 1 for testing. Make sure every partition is used only once of testing. This is called 10-fold cross validation.

There might be other ways but these are the most commonly used techniques. If you are using any standard classification packages such as LIBSVM, then there are options to do cross validation without writing a code for it. Otherwise, you could use MATLAB to generate partitions randomly using function: cvpartition

Good Luck!

http://www.mathworks.com/help/stats/cvpartition.html?refresh=true

If you are using the same partition every time, then it does not matter as most machine learning algorithms are deterministic and same training data would produce same classifier. If you partition randomly at every iteration, then it is one of the many ways of doing 10 fold cross validation. 10-fold cross validation can be done in following ways:

1. Create 10 instances of 90-10 split randomly. In this scenario, some of the data may always only occur in train/test set. This is called 10-fold random cross validation

2. Create 10 equal sized partitions of data randomly and at 10 instances of learning, use 9 of them for training and 1 for testing. Make sure every partition is used only once of testing. This is called 10-fold cross validation.

There might be other ways but these are the most commonly used techniques. If you are using any standard classification packages such as LIBSVM, then there are options to do cross validation without writing a code for it. Otherwise, you could use MATLAB to generate partitions randomly using function: cvpartition

Good Luck!

http://www.mathworks.com/help/stats/cvpartition.html?refresh=true

Hi, 

You can use Weka tools, I think it do what you're looking for.

Also, for the accuracy, you can save your learning hypothesis parameters that gives the maximal accuracy.

Best regards

Two comments:

1) It is recommended to repeat cross-validation several times to get an even better idea of the variability of the results.

2) A single performance measure such as accuracy can be misleading (eg. for imbalanced data). You should look at the confusion matrices resp. consider performance measures that reflect the results of all classes you consider.

hth

Matthias

As previously mentioned WEKA can do it for you so does Matlab. There are built in functions that partition the data for you.

However note that the accuracy of the classifier is the average of the different runs not the sum !

You can also consider a leave one out (LOO) approach for testing the accuracy of your model, though you have to keep in mind that LOO is the worst case scenario and might behave strangely.sometimes.

Best of luck

This is cross validation. 

First randomly shuffle youeyour data. Then remove the first 10% of samples into a test set. The remainjng is the training set. Find the rate for this iteration and save it. Repeat by removing the samples 11% to 20% into the test set the rest are  used as the training set. Find the rate of this iteration and save it. Repeat this 10 times to cover all samples. Find the average of the rates as a final error rate.

1) 10-fold crossover validation is one way to test a learning method to guarantee that the testing results of the method do not strong depend of the way you split your dataset into 10 equal size subdatasets.

2) It is observed that testing results strongly depend on how you partition your dataset and varies from one testing fold to another. Therefore, to make a comparison of the performances of the two (or many) considered methods to be sound it is required that

2,1) The partition of your dataset is to be considered as "equalitarian", i.e. your partition must be considered as quite objective. To obtain such a partition of your dataset you should split your dataset into 10 folds quite randomly.

2.2) As the testing results of 10 folds are largely varied, you must take the averaged sum of these testing results of each folds.

2.3) Viewing testing results as random quantities, a statistical analysis with respect to method testing results should be done. In the recent literature, you can find out, for instance, Wilcoxon statistical test  requires to analyze about 30 testing results on the same 10-folds and, hence, the considered methods should be run on the same partition repeating three times. If you apply 5-fold crossover validation, they should run on the same partition repeating 6 times. By this, you can find in internet available 10-folds or 5-folds of benchmark datasets (of classification or regression problems). Note that the comparative study between two or more learning methods makes sense only when these methods are run on the same dataset partitions.

Hopefully, these are useful for you.

I think it is easy to write a small code to create 10 pairs of training/test dataset from one as you want. If you would like to find some tools to implement this, probably WEKA is helpful.

10-fold method is essentially different from your split method: for 10-folds the 10 testing datasets are disjoint, whist it is not for your case. The question is why one needs 10-folds. The purpose of this is that in order to evaluate soundly the prediction ability of a learning method, one wants to test the method on a sufficient number of real datasets. Assuming one has only one, but one want to make 10 independent experiments. To design such experiments, one splits the datasets into 10-folds. So, each fold can be used to evaluate the prediction ability of the method when it does its learning on the remaining 9 folds. Not that it is required that 10-folds should be done based on a quite random procedure to ensure that the obtained 10-folds are created quite objectively.

Hello everyone,

Joining this nice discussion, I have one query for my work.

I am using time series data for hydrologic predictions. I am using M5 model trees in WEKA software. The concern from my end is that while supplying the original data set and then creation of training, validation and testing data sets into proportions, let us say 60%, 20% and 20%, whether the chronology of time series data is preserved or the splitting is totally random?

In hydrologic forecasts, the general assumption is that the past gives an important cue about the future. So the training of past data can enable us to find robust parameters governing a particular process which can be used for future predictions like rainfall or stream flows.

Kindly please give your valuable responses.

Thanks!

Priyank Sharma

Hello everyone,

I have a query. I could find some papers that randomly divide the data set into train/test with 10-15 different seeds. It means for a given seed, they have a randomized selection of train/test data. The results are then averaged over the different seed points to get a robust estimate of accuracy. Is this an accepted way of doing cross-validation? (though technically it is not a 10-fold cross-validation :))

Thanks!

SR

Yes you should do that and its called 10 fold cross validation. However before you start partitioning the data shuffle the samples randomly. Then create the ten partitions.