I trained a set of data and the first performance plot in the attached document was obtained. I then retrained the network and the second performance plot was obtained. The mse value decreased after retraining but I am not sure if a performance plot with the best validation performance at 0 epochs indicates that the data was overfitted.
When the Neural Network is overfitted... it learns the data "by heart" so is able to remember the data used to train it, but the overfitted Neural Network looses the capacity of generalization, so if you show the neural network a new sample he may not be able to give a correct answer...
Hi Beverly, you need to compare training error and testing/validation errors. The hallmark of overfitting is that you get good training error but much worse test/validation error. That means your network learned the wrong structure in the training data. That could have several reasons: (1) too few data points (2) bad/noisy data (3) data from temporal process which is nonstationary (4) the network is too big for the available data so it can memorize everything instead of learning and underlying structure.
.
overfitting is not the only possible reason
when the "best" performance is obtained very early in the training, it is often due to a too large value of the learning step (or too high momentum, if you use momentum) : try with a much smaller learning step, it will sure take a longer time to reach the best performance but the NN might behave much better !
.
Fabrice is making a good point. Specially when you see the training error going up the learning rate might be too high.
- Badges
- Science topic