I am currently using an online update method to update the weights of a neural network, but the results are not satisfactory. I want try switching to a batch update method now. Firstly, I will tell you what I have understood about batch update:
We calculate the mean square error for all input patterns and the update weights. Now I don't understand how to calculate delta of error(delE). In online update method we use formula: error*derivative of o/p (i.e assuming sigmoid activation we get y(i)( 1-y(i) ) ). But o/p to take in batch update(last one or ?). Similarly, we calculate weight change (wtC) usign the formula
for hidden to o/p layer:
wtC=learning rate*delE(delta of error)*Hidden o/p;
and for input to hidden layer:
wtC=learning rate*delE(delta of error)*input;
What values should I take for hidden o/p and input in batch update for calculating wtC?
.
.
Suppose you are at the beginning of an epoch :
- show the first sample, calculate the updates to the weights (as you would do for the on line version of the BP algorithm), do NOT update the weights but keep the updates in memory
- show the second sample, calculate the updates to the weights (as you would do for the on line version ...), do NOT update the weights but add the updates to the updates in memory
- show the third sample, calculate the updates to the weights , do NOT update the weights but add the updates to the cumulated updates in memory
- and so on until the last sample of the training set
-THEN update the weights by the sum of the updates you have built during the whole epoch
and start again for a new epoch ....
obviously the results are independent of the order in which you show the samples (a difference with the on line version of the BP algorithm)
batch training may seem faster (much less updates, of course) but it is more prone to get stuck in local minima and requires actually many more training epochs than the on line training
you will find a detailed analysis of both versions of BP in the following paper :
http://axon.cs.byu.edu/papers/Wilson.nn03.batch.pdf
The general inefficiency of batch training for gradient descent learning
D. Randall Wilson, Tony R. Martinez
Abstract
Gradient descent training of neural networks can be done in either a batch or on-line manner. A widely held myth in the neural network community is that batch training is as fast or faster and/or more ‘correct’ than on-line training because it supposedly uses a better approximation of the true gradient for its weight updates. This paper explains why batch training is almost always slower than on-line training—often orders of magnitude slower—especially on large training sets. The main reason is due to the ability of on-line training to follow curves in the error surface throughout each epoch, which allows it to safely use a larger learning rate and thus converge with less iterations through the training data. Empirical results on a large (20,000-instance) speech recognition task and on 26 other learning tasks demonstrate that convergence can be reached significantly faster using on-line training than batch training, with no apparent difference in accuracy.
.
Hi, Prashant.
The gradient error is basically calculated using derivative of error with respect to the network weights. The batch calculation of error gradient is quite similar to recursive type BP except that the calculation is done on the whole data sample, N.
You can refer to this book for mathematical formula of the gradient error.
"Neural networks for modelling and control of dynamic systems" by Magnus Nørgaard, Ole Ravn, Niels K. Poulsen and Lars K. Hansen .
The book also comes with Matlab codes for several type of NN trainings including the one you looking for. The Matlab toolbox can be download from:
https://www.mathworks.com/matlabcentral/fileexchange/authors/1119
or
http://www.iau.dtu.dk/research/control/nnsysid.html
.
.
Suppose you are at the beginning of an epoch :
- show the first sample, calculate the updates to the weights (as you would do for the on line version of the BP algorithm), do NOT update the weights but keep the updates in memory
- show the second sample, calculate the updates to the weights (as you would do for the on line version ...), do NOT update the weights but add the updates to the updates in memory
- show the third sample, calculate the updates to the weights , do NOT update the weights but add the updates to the cumulated updates in memory
- and so on until the last sample of the training set
-THEN update the weights by the sum of the updates you have built during the whole epoch
and start again for a new epoch ....
obviously the results are independent of the order in which you show the samples (a difference with the on line version of the BP algorithm)
batch training may seem faster (much less updates, of course) but it is more prone to get stuck in local minima and requires actually many more training epochs than the on line training
you will find a detailed analysis of both versions of BP in the following paper :
http://axon.cs.byu.edu/papers/Wilson.nn03.batch.pdf
The general inefficiency of batch training for gradient descent learning
D. Randall Wilson, Tony R. Martinez
Abstract
Gradient descent training of neural networks can be done in either a batch or on-line manner. A widely held myth in the neural network community is that batch training is as fast or faster and/or more ‘correct’ than on-line training because it supposedly uses a better approximation of the true gradient for its weight updates. This paper explains why batch training is almost always slower than on-line training—often orders of magnitude slower—especially on large training sets. The main reason is due to the ability of on-line training to follow curves in the error surface throughout each epoch, which allows it to safely use a larger learning rate and thus converge with less iterations through the training data. Empirical results on a large (20,000-instance) speech recognition task and on 26 other learning tasks demonstrate that convergence can be reached significantly faster using on-line training than batch training, with no apparent difference in accuracy.
.
- Badges
- Science topic