Strictly speaking you are right: conclusions derived from the test set are applicable only to that set.

However, there is a probability that those conclusions are also valid for the rest of the population. That probability becomes higher if your test set is representative of the population.

How does one know whether the test set is representative? Well, you don't!

Again a probability: take RANDOM samples of the population, and take ENOUGH random samples. Even if you do that, you can be unlucky.

First test it against another representative set: the validation. You can have bad luck here too; the validation set confirms the conclusions based on a non-representative test set.

If your conclusions are very interesting (but based on an unlucky test set), other institutes will repeat your experiment, and (again) with high probability can not confirm those conclusions.

Good LUCK !

Strictly speaking you are right: conclusions derived from the test set are applicable only to that set.

However, there is a probability that those conclusions are also valid for the rest of the population. That probability becomes higher if your test set is representative of the population.

How does one know whether the test set is representative? Well, you don't!

Again a probability: take RANDOM samples of the population, and take ENOUGH random samples. Even if you do that, you can be unlucky.

First test it against another representative set: the validation. You can have bad luck here too; the validation set confirms the conclusions based on a non-representative test set.

If your conclusions are very interesting (but based on an unlucky test set), other institutes will repeat your experiment, and (again) with high probability can not confirm those conclusions.

Good LUCK !

.

see "structural risk minimization" for an introduction to bounding the generalization error from the knowledge obtained from a finite dataset (empirical error) and the "capacity" of the model

as a starter :

http://www.svms.org/srm/

or V. Vapnik's book (a milestone !)

"The nature of statistical learning theory" (1995, Springer)

.

On the practical level, you can "overcome" it if you correctly use statistics, as Lambert or Fabrice suggest. And if you're interested in more fundamental and philosophical aspects, then Karl R. Popper's classics "The Logic of Scientific Discovery" is a good place to start. Because what you mention, is actually one of the essential problems of the methodology of science.

Thanks you all

but i concluded from your answers that my real problem is in validation method?

is this what you mean?

Are you asking as an engineer, or as an empirical scientist interested in "how the world works"? If the former, than I'd suggest you have for your classification at least three disjoint (and possibly large) sets of data (apart from the training data): held-out data, testing data, and validation data. First of all, lock the validation data somewhere safe and forget them. Use held-out data as a model of "the world outside the training data" during the iterative training process of the classifier. Once you're happy with the the classifier, test its performance on the testing data and you'll see how it will most likely perform in the "real world". However, since you'll most likely repeat this testing iteration several times in order to improve the classifier, the testing data become somehow "spoiled" by being implicitly present in the training "meta-loop". So just before you go to present the classifier to you boss or customer or whoever, unlock the validation data set and you'll see how you're really doing with the classifier (but then again safely forget the validation data for another year or so).

And if you're asking as an empirical scientist (with some interests in philosophy and methodology of science), then it's quite impossible to answer in this little textbox - but the aforementioned Popper's book tries to address very well what you're asking.

Thanks sir Jan Romporti, really i'm interested in the rough sets but i faced this problem that i mentioned

but I'd like to ask is there any standard way to choose the test data set or i should choose it randomly so it may be normally distributed (describing the whole population) or skewed in some way?

i.e. is there any constraint i should follow when choosing the test data set to be sure it is well describing the whole problem? or it depends on my luck.

.

"Thanks you all

but i concluded from your answers that my real problem is in validation method?

is this what you mean?"

no, at least for the pointers to the litterature i suggested !

structural risk minimization does not rely on validation (simple or cross)

i do not know much about the links between rough set theory and Vapnik's approach ; one pointer however :

PAC Learnability of Rough Hypercuboid Classifier

Tian Yu, Jin-Mao Wei, and Jian Li

Abstract. Probably approximately correct (PAC) learnability of classification models is crucial in machine learning. Many classification algorithms were introduced and simply validated on benchmark data. And they were not further discussed on under what condition they are assured to be learned successfully, because it is commonly hard to address such PAC learning issues. As one may accept, it would be even crucial to investigate the PAC learnability of the classification models if they are exploited to deal with some special data, such as gene microarray data. Rough hypercuboid classifier (RHC) is a novel classifier introduced for classification based on gene microarray data. After analyzing the VC-dimension and the time complexity of RHC, this paper proved that RHC is a PAC-learning model. The proof gives support to the further RHC applications in classifying cancers based on gene microarray data.

http://link.springer.com/chapter/10.1007%2F978-3-642-31576-3_82#page-1

.

Hi Ahmed,

Regarding your concerns, there is a paper published in Bioinformatics (freely available from our lab's website): http://gsp.tamu.edu/Publications/journal-publications/performance-reproducibility-index-for-classification

This is a paper talking about an index for reproducing conclusions one obtains from a small-sample setting, and the way one can get the best out of these settings.

Hope it would be helpful.

Regarding the generalization outside of the training set, the books and references that several authors have suggested are a very good start. A few more insights on why you need a lot of random samples, as Lambert has said:

What you are looking for is to be Probabably Approximately Correct (PAC). This means that you want the probability of the error you are observing and the error in the general population being different by more than a maximum acceptable value epsilon (ε) to be bounded by some value of delta (δ). It is a direct implication from the law of large numbers (especially a form known as Hoeffding's inequality) that if you have a lot of samples your control over the bound is going to be better:

http://en.wikipedia.org/wiki/Hoeffding's_inequality

Adding on top of that structural risk minimization (as Fabrice suggested) effectively tells you this:

Between all the models that have the same empirical error in the training set, we cannot intuitievely assume that all will be equally good for generalization. For example in the simplest of all cases that you have a two class classification problem in a two dimensional feature space and your training data are linearly seperatable, there will be possibly infinite lines perfectly separating the two classes. Some of these lines will be almost touching one class or the other, some will be a bit further away and so on. There will be however a line that has the maximal margin between these classes and this line Vapnik (the guy who introduced SVMs) argues that will have the best generalization potential. This is intuitive also if you think that some of the samples could have been affected by random noise, but as long as the separating line (or hyperplane in a higher dimensional feature space) is far enough, the samples will still be classified correctly. Therefore, SVMs optimize for a cost function that maximizes the margin (Hinge loss)

With small data set, I use bootstrapping, feature selection methods, and then I analyze the population of models in order to be confident on their stability. See my abstracts.

Thanks to you all for your helpful answers

I agree with the Lambert Zip answer, but I would also add some consideration on the possibility of over-sampling the time independent variable if and only if (iff) you are working with signals time behaviours. If you aren't working with signals time evolution, oversampling has no sense!