The Xavier Glorot weight initialization is a widely used method, which keeps the variances of hidden unit activations stable, independent from the network size, because the formula depends on the input unit count and the output unit count. This is a good initialization if you experiment with different network sizes.

The paper:

http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf

A further explanation:

http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization

A code example:

http://deeplearning.net/tutorial/mlp.html

Best - Stefan

Hi Alexandra,

I guess you are using the neural network toolbox of Matlab as your analytical tool. If so, for a standard network you can change the initial values of weights and biases for each layer separately using:

net.layers{i}.initFcn = 'initwb'

I don't know how many layers you use in your networks but in the standard ones (not deep learning) one or two would be sufficient.

If you use a custom network,

Set net.initFcn to 'initlay'. This sets net.initParam to the empty matrix, because initlay has no initialization parameters.

Set net.layers{i}.initFcn to 'initwb'.

Set each net.inputWeights{i,j}.initFcn to a weight initialization function. Set each net.layerWeights{i,j}.initFcn to a weight initialization function. Set each net.biases{i}.initFcn to a bias initialization function. (e.g., rands and midpoint.)

In order to make that efficient for larger networks you have to use a nested loop for your initialization.

Best regards,

Felix

Have you considered having a new variable that set the maximum number of nodes forward that an edge can point? This can be done for the backward connection too.