Hi everyone,
I am trying to build a Neural Network to study one problem with a continuous output variable.
I am trying to understand the attached learning (error vs. number of training samples) and validation (error vs. regularization parameter lambda) curves.
[Figure 1: Learning curves and validation curve.]
I am relatively new to machine learning and I was wondering if someone could give me some advice on the analysis of these results.
Do these curves look ok for you? I can see that both training and validation error do not improve with increasing the number of samples (something characteristic of high bias situations) but the errors are relatively small, right?
I have also tried to include an additional hidden layer but the results are very similar.
Any comments or suggestions are more than welcome.
Thanks in advance,
David
Hi David,
I don't see any issue with your result (first plot). The model generalized nicely with reasonable performance using about 500-700 training instances, and adding extra instances didn't show an improvement in its performance.
HTH.
Samer
I find it very strange that the "validation error" is twice smaller than the training error. I do not believe in a model whose generalization performance is better than the training performance. This may happen due to small sample effects, but not with 5000 samples (unless the number of variables is very large). Compute the variance of the training error and of the validation error, and perform a t-test in order to assess whether the differences in the training and validation errors are statistically meaningful.
I do not understand the "error is relatively small" statement. The real question is: how is your validation error with respect to your experimental error, i.e. the uncertainty on your data? If the uncertainty on your data is 10-6 and the validation error is 10-3, your model is very poor; If the uncertainty on your data is on the order of the validation error, the model is OK. If the uncertainty on your data is 1 and the validation error is 10-3, there is tsomething very wrong with the methodology or with the code.
I do not know what lambda is, so I cannot comment on the effect of lambda, except that increasing lambda increases the error, so that you might as well set lambda to zero. If lambda is the parameter of weight decay, the curve should be U-shaped.
There are many other possible sources of problems, but you do not provide the relevant information. What is the number of variables of your network ? How many hidden neurons in the hidden layer (forget about using more hidden layers) ? What training algorithm do you use ? How many different weight initializations do you perform ? What do you mean by cross-validation (how many folds) ? And, most importantly, what results do you get on a test set, i.e. a data set that is neither used for training nor for choosing the number of hidden neurons ?
- Badges
- Science topic