vSGD-l uses local gradient variance terms and the
local diagonal Hessian estimates, leading to
i =
(gi)2=(hi vi),
vSGD-g uses a global gradient variance term and an
Pupper bound on diagonal Hessian terms: =
(gi)2=(h+ l),
vSGD-b operates like vSGD-g, but being only global
across multiple (architecture-specic) blocks of
parameters, with a dierent learning rate
per block. Similar ideas are adopted in
TONGA (Le Roux et al., 2008). In the experiments,
the parameters connecting every two layers
of the network are regard as a block, with the
corresponding bias parameters in separate blocks
Use Lyapunov stability method to make the learning rate adaptive. Following are the papers:
1. https://link.springer.com/article/10.1007/s00500-017-2500-3
2. Article Comparative Study of Neural Networks for Control of Nonlinea...
- Badges
- Science topic