I would like to ask a question from Neural nets community, that there exist many activation functions such as Sigmoid, Tanh, ReLu etc., to fire a neuron. To insure non-linearity and better update weights in neural nets layers, which between Tanh and ReLu functions perform better in text classification tasks and why.?
In deep learning the ReLU has become the activation function of choice because the math is much simpler from sigmoid activation functions such as tanh or logit, especially if you have many layers. To assign weights using backpropagation, you normally calculate the gradient of the loss function and apply the chain rule for hidden layers, meaning you need the derivative of the activation functions. ReLU is a ramp function where you have a flat part where the derivative is 0, and a skewed part where the derivative is 1. This makes the math really easy. If you use the hyperbolic tangent you might run into the fading gradient problem, meaning if x is smaller than -2 or bigger than 2, the derivative gets really small and your network might not converge, or you might end up having a dead neuron that does not fire anymore.
It depends upon the model’s architecture, the hyperparameters and the features that we are attempting to capture. Ideally, we utilize the ReLU function on our base models but we can always try out others if we are not able to reach an optimal result.
Mohamad Taheri you are right, most of researchers recommend ReLU activation function and it works fine in most cases. However, it depends on model's architecture as well as the requirements of our output, as many researchers make use of Softmax in attention-based bidirectional LSTM.
It depends on the need and the architecture. Generally ReLU is a better choice in deep learning. I would try both for the case in question before making the choice.
tanh is like logistic sigmoid but better. The range of the tanh function is from (-1 to 1).
Advantage:
==> Negative inputs will be mapped strongly negative and the zero inputs will be mapped near zero.
Drawback:
==> Activation is Dense i.e. Costly.
==> Mainly used classification between two classes
==> It suffers from Vanishing gradient problem.
ReLu is the most used activation function. The range of ReLu is from (0 to infinity). But, the issue is negative values become zero immediately which decreases the ability to map the negative values appropriately.
Advantage:
==> Activation is sparse and efficient.
Drawback:
==> It should only be used within Hidden layers of a Neural Network Model.
==> The dying ReLu problem, i.e. for negative value, gradient will be zero, that means those neurons which go into that state will stop responding to variations in error.
Okay, which one do we use?
==> If you know the function you are trying to approximate has certain characteristics, you can choose an activation function which will approximate the function faster leading to faster training process.
==> If you don’t know the nature of the function you are trying to approximate, then maybe i would suggest start with ReLu and check with others.
It totally depends on the neural network architecture. In one of my research work hyperbolic tangent activation function results more accuracy than ReLu activation function.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size