This looks like n-fold cross validation. n-fold cross validation should be used when there is insufficient data to both train a model and test it. So the data is randomly split inot n equal size partitions (folds). One fold is held out for testing, whilst the rest are used to train a classifier (I'll use classifier as an example, also works with regfression). The classifier is tested with the held out fold and the result recorded. The held out fold is returned and a different one removed. The process is repeated until all of the folds have been used for testing. In 10-fold cross validation you will now have 10 results (e.g. classification accuracy). Average these for your overall result.

The fundamental purpose of n-fold cross validation is to provide enough data for the model, but not to generate too many folds, which can result in computationally infeasible designs. So don't just accept "everybody uses 10 so it must be right" So how to select n for your experiment?

In my study of decision tree classifiers for dialogue acts, I used a rule of thumb I fond tucked away in Quinlan's work - you need about 1,000 cases for a "moderately complex" problem. I was able to generate 1,200 cases on my dataset (600 in each class). I used 60-fold cross-validation which gave me 1180 cases for training and 20 for testing in each classifier.

If you have very little data (as in the experiments for our Silent Talker Lie Detector), you can use leave-one-out cross validation, where you train on all but one case in each iteration. This is commonly used in psychology studies where you might have 20 participants in a trial.

A good method used by an ANN researcher at an IEEE conference i attended was to do an initial experiment with different numbers of fold and find the lowes n that actually resulted in mse reducing during training.

Now it's possible I've misunderstood you.

Another reason for using smaller quantities of data is "windowing" - it's part of the ID3 approach from times when you could have more data than it was computationally feasible to process, so you *reduced* the size of the training set. I'm not as familiar with this so I invite other researchrers to explain why it is still used today (apparently produces better results).

You can find some of my papers using n-fold cross-validation either through my researchgate entry or my blog, http://www.semanticsimilarity.net 

Dear James, I am using training patterns from 1 to 1000 and then 1001 to 2000 because my ANN model is not converging if i give it all 2000 patterns at once.

Interesting, Aamir,

have you investigated the effect of randomising and splitting the data several times, does this always happen? (That's what I'd look at next).

Dear Aamir,

It is strange that your data are converging very quickly. One reason could be that your data does not need to be solved by an ANN method or another reason could be the architecture of your ANN. Can you please indicate the percentage of data that you are using for testing?

I agree with Dr. O'Shea it is better if you select set of data randomly instead from pre-specified starting point and check the difference.

Best regards