The question of finding good training set and its size depends on the problem at hand and domain specific . Good training set should help to generalize without over fitting.

Leave one out method is most suitable for verifying training set.

To my humble knowledge, leave one out method works for splitting the dataset into training and testing but does not measure how good your training set is

Good question.

In my view the problem is very closely related to that of determining the sample in statistical studies. What we want, in the end, is a training set which is a good sampling of the population we are about to submit to the classification process. It is a problem of making assertions about a population based on the sample (training set).

I found an interesting discussion here:

http://www.bmj.com/about-bmj/resources-readers/publications/statistics-square-one/3-populations-and-samples

Thank you Francesco for the useful link. Selecting good samples of the training set is, at least to my humble understanding, challenging task due to the diversity nature of the selection criteria. And to your wide knowledge, how can we determine that the samples we are selecting adequately sufficient and reliable to reach our judgment on the problem at hand? Is it wise to select good samples and trash weak ones? or is it good to include both samples? Is the selected samples reliable representations (good samples)?

I think that the samples should provide a good representation of the classes involved in the classification process. Personally, I would stick to those samples for which there is a strong agreement about the class each of them belong to. But also practical matters may come into play, such as the availability of samples itself. That may depend a great deal on the application you are working on. It would be interesting to know a bit more about your specific application.

Try cross-validation to test training set

The best training set is a training set of one. Only then can you be sure of not overtraining.

Thank you Franscesco for your concise comment n feedback, I am working on pattern recognition.

Fred, thank you for the comment, It would be highly appreciated if you can elaborate a few more lines on "a training set of one"...? thank you

A single representative sample means that class membership can be determined by measuring the extent of commonality with the sample. The similarity with the training sample may be based upon different structure for each pairwise comparison. It is quite possible therefore for patterns to be members of the same class that at have nothing in common. This sort of approach avoids the commitment to intuitively selected features that are expected to be present to some extent throughout all patterns in each class. This is rarely the case except in artificial problems and certain problems that are understood e.g. Iris recognition. It also avoids the problems above associated with training itself.

The question of finding good training set and its size depends on the problem at hand and domain specific . Good training set should help to generalize without over fitting.

Very sensible remark, Baskar, I couldn't agree more

One typical scenario is when data variation in two classes heavily overlap.

Then, one might want to determine what data in overlapping area is representative of what class. There is no any honest way to exclude subjectivity here, unfortunately. The best solution here would be not making this call and keep all the data -- regardless of their 'representativity'. Surely, this way we add to ambiguity in discrimination; but this seems a lesser evil. So, the common wisdom is avoiding pruning the train data: it narrows the data variance and thus the reach of the classifier constructed.

as rightly pointed out by Nikita Orlov, determining what data in overlapping area might help . the subjectivity part can be minimized by taking in larger training set.

I found Fred's comment is valid. Regardless of size of the training set, the most importance is the representative of training set to each class member. Rather than focus on quantity aspect of the training set and testing set, I would focus on the quality of the training set and testing set.

Apart from the type and size of available data, I would like to go with Dorota in supporting the cross validation techniques. This method takes into account the complete set of training data while taking care of overfitting as at a particular instant only a chunk of data is used for training and testing.

A good training set is one that minimises training time of a classifier

Yiu can use, double cross validation for extracting the best (more reliable) training set.

See Kecman paper:

Testing adaptive local hyperplane for multi-class classification by double cross-validationT Yang, K Vojislav, C Longbing… - Neural Networks (IJCNN …, 2010 - ieeexplore.ieee.org

Cheers

WitekD

The key idea is how to preprocess the data before using the training test. Once you prepare your data it will be easy to select and apply the training test. Also it depends on your data and on the problem itself. The size of sample will definitely push you to use a powerful technique. What I mean that sometime you need to apply a special technique such as cross-validation, bootstrap or even both together. Here you need to reform the full sample before you train your models. Simply a good starting point will give you the chance to train the models in more professional way. So I see the process as an interrelated and no part could be ignored.