I have a recurrent ANN model with one recurrent neuron for time series forecasting and I use a sigmoid activation function for that which restrict the outputs between the range 0,1. Because of using sigmoid activation function I normalize my time series data(both input and target) to the interval 0.1,0.9, but all outputs of my model are bounded in 0.5. When I replace the sigmoid with the hyperbolic tangent (tanh) function, it solve the bound problem and it put all outputs of my model in the interval 0.1,0.9 (like sigmoid activation function interval). I wanna to know:
1-The reason which makes the bounded problem with sigmoid function
and 2-Can I use hyperbolic tangent (tanh) with output interval -1,+1, for min-max normalization between the interval 0.1,0.9?
Any suggestion or ideas to aid me is greatly appreciated.
Thanks in advance for your help.
.
basic checks (we all do silly mistakes sometimes ...)
- does your sigmoid indeed map to [0, 1] ? There could be a stupid (1/2) factor hidden somewhere
- do you have a bias input for your neuron ?
otherwise, there is no reason for the sigmoid activation function to behave poorly ; it has been reported that the sigmoid vs tanh choice is relevant to the learning speed but this is not what you are concerned with :
//begin quote
In Symon Haykin's "Neural Networks: A Comprehensive Foundation" book there is the following explanation from which I quote:
"For the learning time to be minimized, the use of non-zero mean inputs should be avoided. Now, insofar as the signal vector x applied to a neuron in the first hidden layer of a multilayer perceptron is concerned, it is easy to remove the mean from each element of x before its application to the network. But what about the signals applied to the neurons in the remaining hidden and output layers of the network? The answer to this question lies in the type of activation function used in the network. If the activation function is non-symmetric, as in the case of the sigmoid function, the output of of each neuron is restricted to the interval [0,1]. Such a choice introduces a source of {\em systematic bias} for those neurons located beyond the first layer of the network. To overcome this problem we need to use an antisymmetric activation function such as the hyperbolic tangent function. With this latter choice, the output of each neuron is permitted to assume both positive and negative values in the interval [−1,1], in which case it is likely for its mean to be zero. If the network connectivity is large, back-propagation learning with antisymmetric activation functions can yield faster convergence than a similar process with non-symmetric activation functions, for which there is also empirical evidence (LeCun et al. 1991)."
The cited reference is:
Y. LeCun, I. Kanter, and S.A.Solla: "Second-order properties of error surfaces: learning time and generalization", Advances in Neural Information Processing Systems, vol. 3, pp. 918-924, 1991.
//end quote
(from http://stats.stackexchange.com/questions/142348/tanh-vs-sigmoid-in-neural-net )
.
yes you can use the tanh with a [.1, .9] normalization, but you could prefer a [-.9, .9] normalization !
.
Many thanks @Fabrice Clerot
yes my sigmoid map to [0, 1]
what is your mean about stupid (1/2) factor?
I just used recurrent neuron with AR and MA inputs
.
By "stupid 1/2 factor" i mean that there might be a silly bug such as having (1/2)sigm as a transfer function instead of sigm ; the same could happen for the normalization but this is unlikely as it would with tanh
.
Dear @K.-L. Du
yes I used [0.1,0.9] range for normalization, but the sigmoid bounded every output and all of my outputs with sigmoid will be given equal to 0.5, when I replace Tanh all outputs are given in the range [0.1,0.9].
Dear @Fabrice Clerot
I check normalization and transfer function code, every things is okey.
to point out that, I used a M-stimator function inested of least square as a cost function in ANN model,
will cost function be related to this problem?
my means is choosing the transfer function can be related to selected cost function?
If yes, why? and how?
Thanks in advance for your answers.
.
M-estimate should not make a difference : as long as it works with tanh, it should work with sigm ...
Just be careful to the difference in span of the two functions : assuming a bisqare Tuckey M-estimator ( https://en.wikipedia.org/wiki/Redescending_M-estimator ), the gradients are zero for errors larger than the parameter k ; say that i consider values away from [.05, .95] as outliers (adapted to sigm), i would set k = 0.45 ; if i were to consider values away from [-.95, .95] a outliers (adapted to tanh), i would set k = 0.95
(the values above are just given as examples ... depending on your application, you may choose other values to define the "outliers")
.
@Fabrice Clerot
Dear Prof.Clerot, thank you very much. Your last guide solved my problem thoroughly.
- Badges
- Science topic