I am designing a Neural Network with two hidden layers for a regression problem in Python. While all inputs are positive, there are supposed to be negative values in the output. The model was unable to predict the negative values when I used ReLU as the activation function, and the losses stagnated over the course of epochs when I used tanh as the activation function in all my layers. I didn't get any additional benefit from using Leaky ReLU or PReLU. Are there any other activation functions that can be used?
The Activation function for the bottom layers does not matter for regression.
All you need to do is use a linear activation in the classification layer to be able to predict values in all ranges. Then you can choose other activations based on the performance.
When building a model and training a neural network, the selection of activation functions is critical. Experimenting with different activation functions for different problems will allow you to achieve much better results.
CHOOSING THE RIGHT ACTIVATION FUNCTION
- The basic rule of thumb is if you really don’t know what activation function to use, then simply use RELU as it is a general activation function and is used in most cases these days.
- If your output is for binary classification then, the sigmoid function is a very natural choice for the output layer.
For example, the following keras code is used for regression task.
- model = Sequential()
- model.add(Dense(12, input_dim=5, kernel_initializer='normal', activation='relu'))
- model.add(Dense(8, activation='relu'))
- model.add(Dense(1, activation='linear'))
- model.summary()
- model.compile(loss='mse', optimizer='adam', metrics=['mse','mae'])
7 Types of Neural Network Activation Functions: How to Choose?
https://missinglink.ai/guides/neural-network-concepts/7-types-neural-network-activation-functions-right/
https://www.geeksforgeeks.org/activation-functions-neural-networks/the
https://datascienceplus.com/keras-regression-based-neural-networks/
I am agree with Ibrahim's detailed answer, however., I would like to suggest you some more literature: https://www.analyticsvidhya.com/blog/2020/01/fundamentals-deep-learning-activation-functions-when-to-use-them/
https://medium.com/@snaily16/what-why-and-which-activation-functions-b2bf748c0441
- Badges
- Science topic