I will use Backpropagation as an example of  why the learning rate must be small(it is usually very small), since you are trying to minimize the error using the learning rate as steps. If for example you are trying to minimize the error in a parabola, the use of a large learning rate will accelerate convergence but at the same time you may overshoot your goal.

That is the direct answer to your question, what I think your real question was is the weight initialization assignment of neural networks. Taking again Backpropagation neural networks, it was a topic of research a long time ago. To see some results of the learning parameters such as weight initialization take a look at some of the foundational papers such as:

1) Thimm, G., & Fiesler, E. (1997). High-order and multilayer perceptron initialization. Neural Networks, IEEE Transactions on, 8(2), 349-359.

2) Fahlman, S. E. (1988). An empirical study of learning speed in back-propagation networks.

If you have other types of neural networks in mind let me know and we can discuss them

Dear Arturo,

I have received a notification that you've added an answer to this question, but there is no answer here. I've followed the answer in your profile. What's the problem that the answer does not show here?

I would be thankful if you could check it in order to resolve this issue.

Anyway, thank you so much for your comment. I totally agree with you by the back propagation ANN example. But the issue is that there's no proof that shows the exact domain for the learning rate existing. I know the bigger learning rate can cause instability on the system and it is an important factor to use small rates.

Personally I am looking for the purpose of that domain. Maybe on some topology of ANN we need to use bigger rate (for example greater than one). So the ambiguous point lies here. If the system needs to update weights in each step in order to achieve a goal at a bigger difference therefore maybe it needs to use a big learning rate to reach its desired weights. Although we should consider the updating process does not cause "instability". The converging rate, instability and other factors on ANN are very important to be matched with each other.

By the way, this is not an assignment. I've been confronted with this issue and I'm looking for an acceptable answer. I've searched a lot and read articles in order to convince myself why we should use that domain and if we use the bigger rate for some type of problems, would that not be a violation of basic concepts?

Regards,

Amirmohammad

Learning value usually between 0 till 1, my experience the best between 0.4 till 0.75

How do we compute learning rate anyway?

Amirmohammad,

Thanks for the notification on the posts, this has previously happended due to hiccups in RG and are usually resolved by the programming team in RG.

I think that you might benefit from two particular proposed solutions to the problem that I think you are after:

1) Adaptive learning rate- You can take a look at stochastic meta descent. Take a look at the work of Nic Schraudolph http://nic.schraudolph.org/bib2html/sort_date.html .

2) Stochastic algorithms such as simulated annealing on NNs.

I have done some previous work on combining both algorithms(annealing for wide search and stochastic meta descent when the algorithm is close to a solution) but there is still much work to do to couple them in an optimal manner.

Hope this helps

Dear Andrea,

By choosing a small number between 0 to 1. We usually use it to control the rate of updating weights on a neural network in order to accelerate convergence. I, personally, believe it depends on the topology of ANN. You might use the larger value in order to reach your goal faster. But you have to be aware of the instability on the system which is one of the side effects of using a larger rate. Larger rates can cause instability and the system cannot reach its desired goal.

It's something like "trial and error" so you have to check every value to find the best matching one.

Regards,

Amirmohammad