In one of my publications, I have mentioned that the learning rate balances the intensity of downing the error after each epoch. The learning rate affects a larger or smaller portion of the respective adjustment to the preceding weight. If the factor is set to a large value, then the neural network may learn more quickly, however, if there is a large variableness in the input set then the network may not learn very well or at all. In real terms, setting the learning rate to a large value is improper and counter productive to learning. Characteristically, it is better to set the factor to a small value and edge it upward if the learning rate appears slow. Momentum runs as a low-pass filter to settle sudden changes in the progress. Momentum basically permits a change to the weights to persist for a number of adjustment cycles. The degree of the persistence is controlled by the momentum factor. If the momentum factor is set to nonzero value, then increasingly greater persistence of former adjustments is permitted in modifying the current modification. This can progress the learning rate in some situations, by helping to smooth out atypical conditions in the training set. The Question is that what are new methods to adjust these parameters to achieve the optimal solution?
Right, but why not to use Metaheuristic approaches and adopt GA rather than PSO or Ant Colony? Is effect of optimizing neuron weights more beneficial or that of Momentum-Learning rate?
The most recent work in this area that I know of is from 2008:
A Fast and Scalable Recurrent Neural Network Based on Stochastic Meta Descent
Zhenzhen Liu and Itamar Elhanany
You can check the references to Nicol N. Schraudolph which are an excellent resource on stochastic meta descent.
.
you may find the following interesting :
ADADELTA: AN ADAPTIVE LEARNING RATE METHOD
Matthew D. Zeiler
ABSTRACT : We present a novel per-dimension learning rate method for gradient descent called ADADELTA. The method dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent. The method requires no manual tuning of a learning rate and appears robust to noisy gradient information, different model architecture choices, various data modalities and selection of hyperparameters. We show promising results compared to other methods on the MNIST digit classification task using a single machine and on a large scale voice dataset in a distributed cluster environment.
http://www.matthewzeiler.com/pubs/googleTR2012/googleTR2012.pdf
.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence