vSGD-l uses local gradient variance terms and the

local diagonal Hessian estimates, leading to

i =

(gi)2=(hi vi),

vSGD-g uses a global gradient variance term and an

Pupper bound on diagonal Hessian terms: =

(gi)2=(h+ l),

vSGD-b operates like vSGD-g, but being only global

across multiple (architecture-specic) blocks of

parameters, with a dierent learning rate

per block. Similar ideas are adopted in

TONGA (Le Roux et al., 2008). In the experiments,

the parameters connecting every two layers

of the network are regard as a block, with the

corresponding bias parameters in separate blocks

Use Lyapunov stability method to make the learning rate adaptive. Following are the papers:

1. https://link.springer.com/article/10.1007/s00500-017-2500-3

2. Article Comparative Study of Neural Networks for Control of Nonlinea...