Normalization is necessary because in input  layer the multiplied value of weight and input variable should activate to very small less than 3 so it is necessary to get better result it should be normalized. 

Normalisation is required so that all the inputs are at a comparable range.

Say there are two inputs to your ann, x1 and x2. x1 varies from to 0 to 0.5 and x2 varies from 0 to 1000. A change of x1 of 0.5  is 100 percent change where as a change of x2 by 0.5 is only a change of 0.05%. Hence normalization helps

Hello Bikash,

Please download and read my thesis which is found on this link.

https://www.doria.fi/bitstream/handle/10024/98414/Rotich%20%28M.%20Sc%20Thesis%29.pdf?sequence=2

and check particularly pg. 18 the second paragraph.

If you find it useful, do not forget to cite it and of course vote this comment up :)

Regards,

NR

Hi Bikash,

Please see Matlab NN Toolbox Manual. It is quite useful in explaining the reasons of normalization. Good luck! Luiza

To smooth the imperfection of ANN programs using normalization.

In real biological neural networks normalization missing. Improve program.

Normalization is needed because it removes geometrical biases towards some of the dimensions of the data vectors. In this way every bit of data gets treated in a "fair" manner. Another way of posing this is to realize that all learning algorithms depend on numerical properties so one should try to avoid small numbers, large numbers, and large differences.

Normalization (or scaling) is  one of  the main parts of ANN learning process. If you do not normalize your inputs between (0,1) or (-1,1) you could not equally distribute importance of each input,  thus naturally  large values become dominant according to less values during ANN training.  

Consider a neuron with (excitatory) input x_1..n and the corresponding synaptic weights w_1..n.

Without normalization: activity = w.x, "more is better" (more inputs leads to more activity in the postsynaptic neuron). So selectivity to a given input pattern (10010...) is impossible.

With normalization: activity = w.x / |x|. With the euclidian norm, this is proportional to the cosine of the angle between w and x, and it is maximal for colinear vectors. Arguably, this kind of selectivity is more interesting.

Normalization is important in ANNs because real data obtained from experiments and analysis most times are distant from each other. The effect is great because the common activation functions such as sigmoid, hyperbolic tangent and gaussian produce result that ranges between [0,1] or [-1,1]. It is important to normalise the values to be in that range.

The common normalization approach includes Statistical normalization (using mean and standard deviation) and Min-Max Normalization

It is needed to ad as comparable range more information in tools of MATLAB it will be easier to explain

Thank you very much for nice explanation...

Hi Bikash. 

Normalisation or scaling is not really a functional requirement for the NNs to learn, but it significantly helps as it transposes the input variables into the data range that the sigmoid activation functions lie in (i.e. for lofistic [0, 1]  and tanh [-1, 1]. 

Note that instead of normalising (i.e. using mean and standard deviation) you can also linearly scale the data into an interval suitable to your activation function. This often works better as normalisation can distort non-stationary data.

Instead, you can also have a fixed connection weight to a single neuron with linear activation function and a 1:1 connection to the input layer which does this for you (as it calculates a regression and can hence scale any input into any output range). I believe this was recommended by Zimmerman and Neuneier in a book called Neural Networks Tricks of the trade and gives you a purely connectionist approach to data preprocessing. 

Hope this helps

Sven

imagine that you are six feet tall but your brother is 4 feet. Now your brother is asked to stand on a bench which is 3 feet in height . For a person entering the room for a fraction of a second, he will see that your brother is 7 feet tall and you are shorter than him. But had he seen the bench , he would realize otherwise. Similarly for any training phase the data needs to be be normalized such that the 'benching effect' is avoided between the data columns which represent data in different units. So if u have a variable for which .01 mm (suppose) has a significant effect compared to another input data for which 100 gms make a similar effect on the the output. For the ANN identification , the effect variation needs to be appropriated such that all involved input parameters contribute to the model. This is absolutely a basic way i learnt and want to tell you but please cross check it as per the ANN concept.this helps. 

Instead of having another layer for scaling the inputs, it is always better to preprocess the data to normalize it and then feed it to a neural network.

Thank you sir for those nice suggestion 

As you can see from the link, there is no need to normalize data if you use standard programs e.g., FITNET, PATTERNNET, TIMEDELAYNET, NARNET & NARXNET,

All of the normalization and de-normalization is done automatically.

http://www.mathworks.com/matlabcentral/answers/25759-normalizing-data-for-neural-networks

Normalization is not really a need for properly codes ANNs.

Use the codes for training ANN without normalizing inputs and targets;

https://www.researchgate.net/publication/305325563_Codes_in_MATLAB_for_Training_Artificial_Neural_Network_using_Particle_Swarm_Optimization

Code Codes in MATLAB for Training Artificial Neural Network using...

we should normalize the data because sometimes the input and desired variables have very different ranges so one should always normalize both the desired and input data files between 0 (or -1) and 1.

In the application of neural networks to real-world problems, it is very important to have a criterion for accepting the solution. Only then can we successfully act to overcome any potential difficulties.

The learning curve is a valuable indicator for observing the progression of learning, but the MSE in the training or test sets is only an indirect measure of classification performance. The MSE depends on the normalization and characteristics of the input data and desired response. We should normalize the total error by the variance of the desired response to having an idea of how much of the desired variance was captured by the neural model. This is reminiscent of the correlation coefficient for linear regression, but there is no precise relationship between classification accuracy and MSE.

Normalization of the data allows us to get experience with step sizes and use systematic weight initializations. Since the data is all positive, we normalize it between [0, 1]. We divide the data in training, test, and validation sets. The training set is used to arrive at optimal weights, the test data is used to gauge the performance of the classifier, and the validation set is used to help us stop the training at the point of best generalization

please look at.:

- Neural and Adaptive Systems: Fundamentals through Simulations

- Bioinformatics: Concepts, Methodologies, Tools, and Applications

Normalisation becomes even more important when activation function is Sigmoid. If you look at it's graph, you'll see it saturates (gradient close to zero) at very high values. This means that at those high values, the gradient/slope will have very small values. If the network needs to learn, ie., update weights during backpropagation, it solely depends on the gradients of output wrt the weights. If that gradient is very small, the update in weights is small and thus learning becomes slow. 

Şenol hocam, normalizasyon yapmadan bunu çözmen zor. ancak matlab programında uygulamalar var. bundan yararlanabilirsin.

Teşekkürler hocam. İyi çalışmalar dilerim

Latest research helps in demo that normalization also help in dealing with vanishing gradient and hence train deeper networks

With a few exceptions, it helps the gradient descent converge faster since it make more uniform steps through the feasibility space of the error function .

Also, in some cases, it helps zero-center your data, avoiding that the gradient descent algorithm optimize in a zigzag behavior. When your data is not zero centered the gradient descent can only optimize weights of a same node in zigzag [1].

[1] Y. LeCun, L. Bottou, G. B. Orr and K. R. Muller, “Efficient backprop,” In Neural networks, tricks of the trade, Lecture Notes in Computer Science LNCS 1524,1998.

to have good convergence

While I've have heard many of these arguments before, this need for normalization smacks of poor convergence techniques. Perhaps, the Adam optimizer is limited in scope. Has anyone experimented using a logarithmic activation function for the 1st hidden layer?