Let's assume we have a standard feedforward ANN with just a single hidden layer. It is standard practice to normalize the input data ,usually in [0,1] or [-1,1]. Let's assume min-max normalization. If we have sigmoid activation function, wouldn't it be more sensible to normalize in a range like [-4,4] or [-5,5] ? The sigmoid function is in essence linear in [-2,2] so if we normalize in [-1,1] the approximated function is linear for the most part. It might be argued that for certain weights the input for the sigmoid activation function can be outside the normalized range but still that's generally few cases (depending also on what values the weights might take).
As for how to initialize the weights: A common formula is in the range [-b,b] where b = 1 / sqrt(Ninput+Nhidden) (assuming sigmoid function).
It is usually said that small weights are more preferable (also if you like at regularized error functions that penalize large weights ) since large weights have higher chances of leading to overfitting.
Any thoughts?
Hey, I am not talking about the normalization formula but whether this affects the output of the network for sigmoid activation function
Dear Nikolaos Bismpikos,
I think both are different things. Weight depends on the learning algorithm. So we can't find the range of weight, it depends on used training algo. If you use GA, then u can fix the range of weight. Because in GA, u defined your own equation to compute the value of weight.
Hi Nikolaos,
In my experience, normalization procedure is used in NN input if the different features that are being used are in a different numeric value/ranges. The normalization is done feature-wise usually, so the impact on the network would be similar.
The sigmoid function is not applied directly to the input features, I mean, if we have X = [x_1, ... , x_n] as input an W_i = [w_i1, ..., w_in ] the synaptic weights corresponding to the i'th neuron of the following hidden layer, the activation of that neuron w.r.t. the sigmoid function would be:
Y=sigmoid(X · transpose(W_i)) that it's always between the range (0,1).
If you start with very large values in the input and weights, the sigmoid signal would be 'crushed' to 1.
There are several ways to initialize the initial weights, but small random numbers is usually fine. At the end, these weights will be updated according to an optimization function to complete the designed task.
Hope that some of this information helps you.
Cheers.
Hey, yes I already know this infromation. But if the weights are small and the input normalization is at [-1,1] most of the time we'll be on the linear part of the sigmoid. That was my main concern asking if it makes any impact ( in terms of including non-linearity) to use a wider range like [-4,4]
Please see the attached link for similar thread and interesting answers:
https://www.researchgate.net/post/How_to_determine_the_appropriate_pre-processing_technique_for_artificial_neural_networks_ANNs?_tpcectx=profile_questions
Hello, normalizing on the [-1,1] range is a good decision, because you are interested in the threshold behavior of the Sigmoid function and not its value range per se.
If you use standard backpropagation algorithm than you should initialize the weights randomly (within a certain value range, maybe between [-1, 1] and not equally.
I hope that helps a bit.
Regards, Wolfram
Hello, using large valued weights (e.g. in the range [-4, 4] as stated by Nikolaos) could have two major effects somewhat irrespective of input value ranges; one has already been stated by Menex. On the other extreme, the pre-activation value (or Total Potential: T.P) of a hypothetical neuron could be driven to a very small value, such that the activation value of the Sigmoid function is almost zero.
Let, T.P = (Wx+b).........total potential or pre-activation value for a hypothetical neuron.
then, y= Sigm(Wx+b)..........activation value for a hypothetical neuron y.
y→ 1, for large values of T.P (i.e. T.P→ +∞)........extreme condition 1.
y→ 0, for extremely small values of T.P (i.e. T.P → -∞).....extreme condition 2.
Note that neither of the two extreme cases is desirable in neural networks, as it removes us from the linear region of the Sigmoid curve; we then are in the extremely upper or lower saturation domain of the Sigmoid curve. Hence, during backpropagation of error signals for weights update, derivatives of activation values within the saturation regions yields zeros which are of little or no use for learning. i.e erases backpropagated information on learning from the output layer!!!
I hope this helps.
Oyebade
Hey Oyebade, thanks for your reply. Are the above cases problematic even when no back-propagation / iterative algorithm is used for learning the weights? For example if we use a meta-heuristic algorithm/evolutionary algorithm to learn the weights are these two extreme cases also undesirable?
Hello Nikolaos, I think for meta-heuristic algorithms, the extreme cases shouldn't be a problem once such algorithms do not rely on error gradients for weights update. In fact, haven worked with particle swarm optimization (PSO) for neural network, it is not uncommon to find that network weights are well outside the [-1, 1] range after training. Although one could intentionally clamp the network weights to the range [-1, 1] in PSO, I have little doubt that this significantly affects the performance of the network.
Oyebade
Nick,
I've checked that standarization method influence the prediction accuracy of ANN if sigmoidal function is applied.
"The influence of input data standardization method on prediction accuracy of artificial neural networks" will be published in two weeks time. There is another issue: what is an activation function. We should fit the range of data value for for the activation function e.g. we can not calculate ln(x) for x
- Badges
- Science topic
- Similar topics
- Medicine
- Disease
- Infectious Diseases
- Immunization
- Vaccination