Does it make sense to validate a NN after training?
Let A, B be training sets of n-vectors. Vectors in A are examples, or vectors with expected output 1. Vectors in B are counterexamples, with expected output 0. After training, a perceptron NN
P:R^n \rightarrow \{1,0\}
has outputs 1 on A, and 0 on B. So, it is 100\% accurate on the training data.
Recall that training is performed, in most cases, taking into account only the numerical values of the data vectors. Usually the training algorithms have no place for the 'meaning', 'interpretation', or 'empirical content', of the data.
Next, in order to 'validate' P, two sets of test data are given, A', B' and A'', B''.
Evaluating P on A', B' turns out to be 0\% accurate, that is, P(A')=0 and P(B')=1. The NN completely fails the first validating data set.
Finally, evaluating P on A'', B'' is 100\% accurate. Validation is completely successful in this case.
Many intermediate percentages of accuracy occur in practice.
All taken into account, the question arises: Was P well trained?
The criteria the neuroscientist used to choose both the training and the test vector sets ---in principle a subjective judgment--- must be taken into account. But the literature on NNs seldom carries an explicit discussion of these criteria.
Another way to put it: How to validate validation? This can be referred to as the problem of metavalidation.
References and opinions on this topic are welcomed.
Hello Daniel,
It is possible that you are overfitting the model...100 % accuracy during training is hard to get, and you could be fitting to noise. Also is possible that your training set doesn't have enough cases, so you could try changing your training and validation percentages to better use the data. (see Bishop's book for a complete description)
P may or may not be well trained. The only certainty is that A', B' and A'', B'' are poorly chosen or have too few patterns. Your description does not preclude putting all misclassified validation patterns in A', B' and all correctly classified patterns in A'', B'' .
After training, network validation errors can be reduced by
(1) pruning the hidden units such that validation error is minimized, or
(2) varying the regularization parameter lambda, for output weights, until validation error is minimized.
Hi,
I agree with everything Raul and Daniel said. The short answer is: test it with data you havent used during the training and you'll see if its overfitted.
hello !
as far as i am getting your question..you are getting 100% accuracy on training set which is pretty abnormal according to my view..i think you are overfitting the neural network model...overfitting is the stage where your model is over trained and give correct results only on the data it has been trained but does not give correct or valuable results on unseen data sets...so it directly effect generalization which is most valuable feature of neural network for its usage.
use 70% data for training 15% for testing and 15% for validation..and use different
data set for testing from the set which you used for training.
I hope this will helps, and I apologize in advance if I didn't understand your question and just told you stuff you already knew ;)
good day.
Daniel,
You don't want to train your network to perfectly fit the training data. Instead, divide your sample into training and test sets, and use the test set to determine when to stop training (min error of the test set - training beyond the min, error on the test will rise as the network overfits the training set). Given sufficient samples, you can use a third unseen data set to validate the model and give an unbiased estimate of error, as the validation set was not used to train the model or determine the stopping point.
See Neural Smithing by Reed and Marks, Chapter 15.
Agree with Carlos & Raul above. 100% accuracy on training is bad news, in the sense of too good to be true.
Usually, you define 3 different sets (training, test and validation). The test set is used to stop training before memorization kicks in. As such, it cannot be considered an independent set and that is why you need a validation set. The standard approach (before you embark into boosting/bagging, which involve multiple classifiers), is to perform k-fold cross-validation with randomly assigned training, test and validation sets. That way you'll get an average and standard deviation on the performance. Criteria for defining the relative sizes of training-test-validation vary, but a typical rule of thumb is to use 60-20-20 or similar.
Inconsistent validation results indicate either poorly trained P, or badly chosen validation subsets A', B' and A'', B''. To obtain meaningful validation, A, B, A', B', and A'', B'' should be combined into new sets; call them AA and BB. Next, AA' and BB' -> _representative_ subsets of AA and BB should be selected and separated out for validation - either randomly, or by some good clustering algorithm (we use Kohonen maps). The remaining parts of AA and BB are then used for training. If error values on training and validation sets are similar, then the model is considered valid _within_ (!) its applicability domain.
I agree with James Pokorski's approach, though "a third unseen data set" use is easier said than done ; otherwise obtaining unbiased error estimates would had been a triviality which is not ususally the case.
Agree with Michael Manry. At least, it is better to assure that validation set (sets) came from the same probabilistic distribution (of joint (inputs,class), provided that inputs are random). In high dimensions, this testing could be cumbersome, but at least the use of Naive Bayes density estimation or data support estimation with 1-class SVM should be carried out, if the origin of validation set is unknown. Both training and validation sets should have "same" support (at least) and "same" likelihoods against Naive Bayes - estimated probability density.
First, Depends on your data (quantity and quality to represent the function to learn), your NN (dynamic or not, enough non linear, ...) and your training algorithm (BP, spatial, temporal, spatio-temporal, ????).
Second, never validate a learned NN after training sorryt to shock everyone but it is better to insert validation procedure in the training algorithm !!!!!!!!!!!!!! you avoid to loose your time after and you can never be sure of your cross-validation so it means it is faster to create a lookup table than a NN. LOL :)))
Third, never give up, the problem in this equation is you.
Hello everyone.
I believe everybody understood very well the question. And I am aware of the overfitting problem, which I describe as follows.
If P is overfitted then a small perturbation of vectors a or b will often result in vectors a', b' that are misclassified by P. This means that the decision boundary for P is too close to A, or too close to B, or both.
Let d be the distance from set A to set B. That is, any pair (a,b) of n-vectors with a in A and b in B satisfies |a-b| (distance from a to b in euclidean space) is no smaller than d, and for at least one such pair the distance is actually equal to d.
The algorithm I apply has an unusual property. The trained multilayer perceptron neural network P is such that the following two conditions hold simultaneously:
1.- If a' is any n-vector in Euclidean space at distance less than d/2 from any of the elements of the training examples set A, then the NN evaluates to 1 on a', P(a')=1
2.- If b' is any n-vector in Euclidean space at distance less than d/2 from any of the elements of the training counterexamples set B, then the NN evaluates to 0 on b', P(b')=0
This means that the decision boundary for P is located at least at distance d/2 from A and, simultaneously, at least at distance d/2 from B. This is a kind of 'best fitting' situation. Under these conditions it can be said that P is "well adjusted" to the training data and therefore there is no overfitting.
Let us look back at the original question assuming additionally that, in the above sense, there is no overfitting and ask again: Is there a reasonable way to solve the metavalidation problem? What are good criteria to select the training/test/validation vectors.
Since the algorithm I use is so fast, and P is well fitted (in the technical sense above), I tend to go along with Sébastien Dourlens: Assuming the vectors are 'well' selected (good representatives of the empirical problem at hand), use them all and obtain P that is a) well fitted to the data and b) 100\% accurate on all vectors involved. But it is not clear that this solves the metavalidation problem, unless you are willing to bypass, or ignore, both testing and validation.
Let me repeat. P is well fitted, 100\% accurate and can be obtained very quickly. So, it is not unreasonable to dismiss validation. In this new context the problem then reduces to: What are good criteria to choose A and B?
Thanks for your time, reflected in the many interesting postings. With all this help I am sure each of us ---and especially myself--- will enrich his understanding of NNs.
Most cordially,
Daniel Crespin
Hi Daniel,
When you have a 100% function fitted for all given inputs, it does not mean that for any other inputs the function is learnt but just almost sure. That's why cross-validation is necessary. To avoid the validation, you must be sure that stability or any other constraints/limits of a NN are respected for all possible new sets of input/output not learned. There is a solution: modify the structure of NN composing thepolynomial equation based on weigths to fit the stability constraints before the training step. you may download a paper at http://s.dourlens.free.fr/these/articles/1.pdf written for the Neurocomputing'98 conference.
Another solution is ANFIS (Yang 1993) where the NN is ensured to be trained on the complete I/O dataspace of a fuzzy logic system.
Best regards
Daniel, thanks for clarification. You are testing the fact that none of b-points lie within d/2 support of a, and similar for a-points. (Rest of the space is not relevant and is not interesting at all). In this kind of hyphothesis testing, validation data should come strictly from within both estimated supports (random sampling of arbitrary points for validation is not relevant - these points will provide 0's and 1's from areas which are not the subject of the test).
Daniel, the fact that your class boundary is well-behaving does not necessarily exclude overtraining. Overtrained NN models also exhibit this behavior, although their boundary's shape is quite complicated. Let me use simple analogy from 2D: having N pairs of (x,y) points you can create two models:
1) y = f1(x), where f1 in the N-th degree polynomial fitted by interpolation
2) y = f2(x), where f2 is simpler function than f1 fitted by least squares
Model 1) will appear to be well-behaving (f1 is a smooth function that goes through all the points, with perfect "prediction" of this "training set"). Model 2) will have higher "prediction error". However, I will pick model 2) over model 1) any time, because 1) is varying wildly between points and it is not robust. Model 1) fits both signal and noise in the data, while 2) has a tendency to filter noise out to some extent.
Hello to all.
The "good fitting" property, mentioned in my previous posting, and to be now called "weak fitting" , is implied by a "strong fitting". The weak fitting says that vectors d/2-close to A are classified as examples and vectors d/2-close to B are classified as counterexamples.
Let D be the decision boundary of P. Let S_1(d/2) be the open strip around D having amplitude d/2 and lying on the examples side of D; similarly, S_0(d/2) denotes the strip around D having amplitude d/2 and lying on the counterexample side of D.
The algorithm mentioned before produces P such that no point of A belongs to the strip S_1(d/2) and no point of B belongs to S_0(d/2). This I call "strong fitting". It says that, uniformly around the decision boundary D, there are (d/2)-strips where all the training vectors are excluded. This means that A is located uniformly at depth (d/2) inside the example region of P, and B is located uniformly at depth (d/2) inside the counterexample region.
More generally, if s, t are numbers with s+t = 1 then P can be obtained such that no point of A belongs to the strip S_1(s) and no point of B belongs to S_0(t).
These facts provide the metavalidation problem a somewhat different perspective.
With best regards,
Daniel Crespin
CORRECTION: The strips at the end of my previous postings should be the following:
S_1(sd) S_0(td)
So, s and t are positive proportionality factors that add up to 1, and are to be multiplied by d. In particular one can take
s = t = 1/2
and obtain S_1(d/2) and S_0(d/2)
Hello Freddy, hello all.
The problem does not go away. Here is the situation:
The NN training software I use is extremely fast and always succeeds 100% with the training data. The test/validation data can be added to the training data without much extra time or memory cost. There is no overtraining because the decision boundary is as far away from the training data as can be expected. Does it make sense to validate? The software is freely available (Windows XP, with a full GUI) at the following download link
http://www.matematica.ciens.ucv.ve/dcrespin/Pub/Crespin.zip
Unzip and install running the executable. It comes with full documentation and with many numerical training/test data files.
Best regards
Daniel Crespin
Hello Raul, hello all.
ABOUT OVERTRAINING/OVERFITTING:
As I understand it, overfitting happens when the decision boundary is very close to one of the data sets: to the set A of examples (training vectors with desired output 1), to the set B of counterexamples (training vectors with desired output 0), or to both.
For NNs built with radial basis functions, overfitting could occur if balls with very small radii around each n-vector in A are selected for a NN, say R. The region where R outputs the value 1 would then be a small neighborhood U of A. For all vectors outside U, R would output 0. In this case for a given vector x, the value R(x) just provides a rough idea of x being, or not being, close to A. In general this is a systemic difficulty with radial basis NNs.
In the case of classical perceptron neural networks [PNNs] you can choose very small simplexes, one around each example, and obtain a PNN with similar limitations as above. Even narrow strips, instead of small simplexes, will do.
Assume a PNN, say P, has 100\% accuracy on the pair of training data sets A, B, with A being examples and B counterexamples. It may happen that P if overfitted to the sets A, B. But this is not necessarily the case.
Note first, that a perceptron neural network P has a 1-valued maximal open region R_1, and a 0-valued maximal open region R_0. These regions are disjoint, and are separated by a common frontier D, the decision boundary. Suppose now that all the vectors belonging to A are located deep inside R_1, and all vectors of B are deep inside R_0. Why should P be considered overfitted to the data A, B?
The software I use trains a PNN in a rather sophisticated way ---still very fast--- to avoid the overfitting described above. If there are other reasonable definitions of overfitting, references would be appreciated.
The idea seems to be around that, if you train too much your NN (say, train with too many iterations of backpropagation) you are then at risk of ending up with an overfitted PNN. But overfitting, 100\% accuracy, and overtraining, are very different concepts. No one of them implies in general the other. Let us see why. Consider, data sets A, B and a PNN P. The following are some relevant definitions.
P is *overfitted* to A, B, if the decision boundary is too close to A, or to B, or to both.
P is *subfitted* to A and B if A is way too deep inside R_1 and B is way too deep inside R_0.
P is *well fitted* to A and B, if the A is deep enough, but not too deep, inside R_1 and B is deep enough, but not too deep, inside R_0.
[What *deep enough* exactly means depends of course on the empirical origin of the data sets A, B. A good answer to *deep enough* could provide a solution for the metavalidation problem].
P is *overtrained* if it was calculated using too many iterations of an algorithm ---say of backpropagation--- thus reaching a stage where weights are stuck and behave as if a local minimum of the quadratic error was reached.
P is *100\% accurate* if P(a)=1 for all a in A, and P(b)=0 for all b in B.
By the way, I have allways been discussing plain vanilla *supervised learning* of perceptron NNs.
ABOUT DATA FILES:
The installation software unpacks its included numerical data and calls a subprogram that creates data files in a DATA directory. All numerical data are somehow included in the executable, but I cannot tell if it is easy to extract them without installing. It depends on the dark, unfathomable ways Microsoft Visual Studio packs and installs programs. I told the "Package and Deployment Wizard" the programs, documents and data files I wanted, and PDW somehow produced the executable.
Some of the files are created "as the original" benchmark problems, for example the Sonar, Mines vs Rocks. This is a set of more than 200 vectors of dimension 60. Additional data files are then created by adding random perturbations to the originals. Small, medium and large perturbations provide corresponding files with little perturbed, moderately perturbed and highly perturbed vectors. The idea is to check the trained neural networks on these perturbed data. Beware that perturbation sizes are maintained, but the actual random values of perturbations vary each time the program is installed.
Similarly for the "original" XOR problem in dimensions 2 to 13.
There are inbuilt limitations for the training program:
1.- Vectors cannot be more than 100-dimensional.
2.- At most 350 vectors from a data file are processed.
This certainly allows for the full Sonar data set, and for plenty of testing on your own favorite data files.
The 13 dimensional XOR problem has 8192 binary vectors of dimension 13. I once ran the program [with limitations removed] on this problem and it took, if I remember correctly, nearly 30 minutes. For the full Sonar data set it takes less than one second to calculate architecture and weights of a 100\% accurate PNN.
It turns out that, the number of vectors being the same, a higher the dimension implies it is easier for the software to calculate the PNN.
Vectors in general position, and in high dimensions, are best for the training. Noisy signals typically provide vectors satisfying these conditions.
I am aware of the Irvine repository. Thanks anyway. The installation software already creates 10 MB of data files. Perhaps some day, patience willing, I will add more data from Irvine.
ORIGINAL METAVALIDATION PROBLEM:
The metavalidation problem arises when you look at training from a practical viewpoint.
Assume what you want is to obtain a PNN that performs reliable pattern recognition. For a dramatic effect, suppose you want to rate mortgage loans, or to use electrocardiograms to predict 1-year survival of cardiac patients, or detect possible cancer from mammograms, or determine the worthiness of a jet engine from its noise spectrum.
Training usually means:
1.- Collect historical data.
2.- Preprocess collected data.
3.- Create training data files and validation data files.
4.- Figure out an initial architecture and weights.
5.- Train with backpropagation.
6.- Check the scores for the PNN obtained and somehow decide if that is enough for the purpose at hand.
But with this software you can ignore 4. and 6. Furthermore, 5. is replaced by the new, very fast software. What does this means in practice? Perhaps training should now be redefined as follows.
Let P be a PNN built with given data vectors A, B in a way that P is 100\% accurate, data are deep enough inside regions, etc. Consider additional sets A', B' of data vectors.
To *train* P with A' B' means to evaluate P on these vectors and each time a' in A' or b' in B' are misclassified, modify P to obtain P' that
1.- Is still 100\% accurate on A, B.
2.- Correctly classifies a' (or b').
Once all vectors a', b' have been evaluated and the required modifications of the successive PNNs done, a final PNN that is 100\% accurate on A, A', and B, B' results.
As Raul accurately stresses in his post, "What you score on the test data is the final score. Not the training." I agree fully. From this perspective scores here are perfect because all test data have been assimilated into training data.
Raul, I regret you have no Windows XP platform available to install the software. Just in case, a couple of data files are attached.
I appreciate the interaction. It forces me to explain ideas and this helps into the next stage of the present work, which is to define trainable, 100\% accurate NNs of a new kind that are better models of biological brains than PNNs.
Cordial regards to Raul and all others,
Daniel Crespin
Correction: here and in next posting, the actual training data files. Previous one is mistaken.
ADDITION:
Sets of data vectors A'', B'' where P has 0\% accuracy can always be defined. For example, if P is 100\% accurate on A', B', just take A''=B' and B''=A'. This kind of "forced failure" is the main motivation to study metavalidation.
- Badges
- Science topic
- Similar topics
- Computer Science