I am trying to train a complex system by neural net with Levenberg-Marquardt algorithm, it is faster than stochastic gradient descent, bot it is not fast enough. which training algorithm is the fastest one for complex systems?is Bayesian regularization faster than Levenberg-Marquardt?
I think that Levenberg-Marquardt algorithm is the fastest, but it usually requires a lot of memory.
Several algorithms can be used for training a neural network; such as Levenberg-Marquardt, BFGS Quasi-Newton, Resilient Backpropagation, etc. The gradient method is distinguished by its simplicity but is slow in nature.
The conjugate gradient method improves the convergence speed through its principle and its choice of the convergence step. In addition, it is inspired by the gradient method. This guarantees its simplicity. But, she uses the exact linear search method to calculate the convergence step. This method increases the cost of calculation that can be minimized with one of the inexact linear search methods. Newton's method improves the speed of convergence. But, it requires the calculation of the Hessian matrix which requires a fairly high calculation cost. Moreover, this matrix can be non-invertible. A solution to this problem is developed by the Levenberg-Marquardt method of adding a term in the Hessian matrix to ensure its invertibility. On the other hand, quasi-Newton methods make it possible to reduce the cost of calculating the Hessian matrix.
Levenberg-Marquardt algorithm is the best one. If you want to proceed with some other optimized technique you ll get fastest training
so we can conclude that the best choice is Levenberg-Marquardt training algorithm , and if the performance of the training is not good for my network with LM, no other training algorithm will help to get better results?
is my conclusion true?
It depends on your problem.but, In my opinion is better you use meta-heuristic algorithms for training neural network
is it possible that for some application, the stochastic gradient descent outperforms the Levenberg-Marquardt training algorithm?
- Badges
- Science topic