In deep learning, activation function enables a deep neural model to learn, understand and perform complicated task by introduce non-linearity properties into the network. Activation functions are essential for deep learning methods to learn and perform complex tasks such as image classification.
Here are some points:
1. I do believe no one is using ReLU any more. Many companies are using LeakyReLU. Why?
2. I do believe many companies are using XGBoost instead of Keras and TensorFlow. Why?
Ernest Bonat, Ph.D.
Senior Software Engineer
Senior Data Scientist
I agree with Daniel: "However, if you are using some NN-framework as Tensorflow, Pytorch, Keras, etc, they already have a lot of different activation functions implemented efficiently and you just need to call them. In fact, this is the most efficient way to build your NN, not only for the activation functions but because everything you need is already implemented."
This very low level detail will certainly have been optimized by hardware instruction like SSE, SIMD, gpu, etc. in the most recent frameworks like TensorFlow, Microsoft Cognitive Toolkit, etc.
- https://www.microsoft.com/en-us/cognitive-toolkit/
- https://www.tensorflow.org/
If you are re-implementing this, you will likely forfeit ALL the advantages of industrial frameworks like GPU optimization, cloud computing, robustness and the benefits of their close relation with hardware manufacturers like Intel, NVidia, etc.
Unless you have profiled the code of some of these frameworks implementing your algorithm and found the activation function to be the real bottleneck by a big factor, it seems like a huge waste of time going into that direction, time that could be much better spent improving the algorithm, improving data preprocessing, modifying your code to use cloud computing, etc.
Regarding ReLU vs Leaky ReLU: standard rectified linear units have a problem called “dying”, namely the product of the input to a ReLU with negative weights being zero. Leaky ReLU tries to fix this problem. It has a small negative slope (around 0.01) of the left side where x
- Badges
- Science topic
- Similar topics
- Computer Science
- Images