Quick Answer:

MLP with Backprop: => use tanh

A silghtly more detailed answer:

First of all

you would have to specify what type of neural network we are dealing with.

Let's assume: a Multi-Layer-Perceptron (arguing would be different if i am wrong)

Second,

we would have to take a look at the learning paradigm:

Let's assume: Backpropagation of Error

Then,

we would have to get a closer look at the details of the problem:

Do you plan to use the MLP for regrression tasks (function approximation)?

or do you want to apply it on pattern recognition?

The different tasks can be facilitated by the correct choice of the transfer function in the output layer.

Task:

pattern recognition

in pattern recognition the desired output values are typically discrete,

most of the time they are even binary

In this case, you can use the logistic transfer function (sometimes called Fermi function, or just "sigmoid function),

it is generating output values in intervall from 0.0 to 1.0.

The hyperbolic tangent (tanh) can be used as well.

tanh is a "sigmoid function" as well. generating values between -1.0 and +1.0.

Task:

regression / function approximation.

you CAN use logistic function and tanh, the learning mechanism should be capable of adjusting the weights, to reduce the error.

It has been reported in the literature, that a linear transfer function (IDENTITY) is sometimes superior to the sigmoid functions.

When, you use a linear transfer function in the OUTPUT layer (ONLY THERE),

then you must guarantee, that your teacher values are bounded within small limits (e.g. between -1.0 and +1.0) and, that your learning rate for the output layer is small as well.

So, if you can use different transfer functions in the different layers, you have a choice for the output layer.

For the hidden layers, you MUST use a nonlinear transfer function (otherwise you end up with only linear separable solutions).

If you use Backpropagation of Error you should use a transferfunction that is:

differentiable, smooth, monotonic, and bounded.

(thus, a step function is not a good advice, and a non-monotonic function e.g. witha a hump neither.)

For the experts:

yes, i know that there are easy extensions to BP, to deal with all these extras

Typical possibilities for the trasnfer function:

logistic function or hyperbolic tangent (tanh)

Advice:

use the hyperbolic tangent for the hidden layer.

Why:?

the logistic function will generate values close to 0.0 if the argument of the function is substantially negative.

Thus, the output of this very hidden neuron will be close to zero, and thus lowering the learning rate for all subsequent weights!!!

Thus, it will almost stop learning.

The tanh function, will in the same situation generate a value close to -1.0, and thus will maintain learning.

As a rule of thump:

If all layers need the same transfer function (depends on simulation tool)

take the tanh

If you can chosse different transfer functions, and different leaning rates for the different layers:

choose tanh for all layers if you do pattern recognition / classification

choose tanh for the hidden layers, and tanh or linear for the output layer if you want to do regression / function approximation.

Take a look into the free textbook on neural networks

by David Kriesel

http://www.dkriesel.com/en/science/neural_networks

There, chapter 5.7.3

Quick Answer:

MLP with Backprop: => use tanh

A silghtly more detailed answer:

First of all

you would have to specify what type of neural network we are dealing with.

Let's assume: a Multi-Layer-Perceptron (arguing would be different if i am wrong)

Second,

we would have to take a look at the learning paradigm:

Let's assume: Backpropagation of Error

Then,

we would have to get a closer look at the details of the problem:

Do you plan to use the MLP for regrression tasks (function approximation)?

or do you want to apply it on pattern recognition?

The different tasks can be facilitated by the correct choice of the transfer function in the output layer.

Task:

pattern recognition

in pattern recognition the desired output values are typically discrete,

most of the time they are even binary

In this case, you can use the logistic transfer function (sometimes called Fermi function, or just "sigmoid function),

it is generating output values in intervall from 0.0 to 1.0.

The hyperbolic tangent (tanh) can be used as well.

tanh is a "sigmoid function" as well. generating values between -1.0 and +1.0.

Task:

regression / function approximation.

you CAN use logistic function and tanh, the learning mechanism should be capable of adjusting the weights, to reduce the error.

It has been reported in the literature, that a linear transfer function (IDENTITY) is sometimes superior to the sigmoid functions.

When, you use a linear transfer function in the OUTPUT layer (ONLY THERE),

then you must guarantee, that your teacher values are bounded within small limits (e.g. between -1.0 and +1.0) and, that your learning rate for the output layer is small as well.

So, if you can use different transfer functions in the different layers, you have a choice for the output layer.

For the hidden layers, you MUST use a nonlinear transfer function (otherwise you end up with only linear separable solutions).

If you use Backpropagation of Error you should use a transferfunction that is:

differentiable, smooth, monotonic, and bounded.

(thus, a step function is not a good advice, and a non-monotonic function e.g. witha a hump neither.)

For the experts:

yes, i know that there are easy extensions to BP, to deal with all these extras

Typical possibilities for the trasnfer function:

logistic function or hyperbolic tangent (tanh)

Advice:

use the hyperbolic tangent for the hidden layer.

Why:?

the logistic function will generate values close to 0.0 if the argument of the function is substantially negative.

Thus, the output of this very hidden neuron will be close to zero, and thus lowering the learning rate for all subsequent weights!!!

Thus, it will almost stop learning.

The tanh function, will in the same situation generate a value close to -1.0, and thus will maintain learning.

As a rule of thump:

If all layers need the same transfer function (depends on simulation tool)

take the tanh

If you can chosse different transfer functions, and different leaning rates for the different layers:

choose tanh for all layers if you do pattern recognition / classification

choose tanh for the hidden layers, and tanh or linear for the output layer if you want to do regression / function approximation.

Take a look into the free textbook on neural networks

by David Kriesel

http://www.dkriesel.com/en/science/neural_networks

There, chapter 5.7.3

Thank you Nils,

but your answer is too long. can you be brief ? I have a MLP with backpropagation of error. I want to use it for different tasks, I mean for pattern recognition, classification, and regression ? So, just be brief please or answer me in three point. (notice the diff between hidden and outer layers too.)

Quick answer:

MLP with BP

=> hyperbolic tangent

suitable for hidden and output layers

suitable for pattern recognition, classification, regression, function approximation

i wish you success for your work

thank you

Nils

Generally the most important differences are a. smooth continuously differentiable like tanh and logistic vs step or truncated b. competitive vs transfer c. sigmoid vs radial d. symmetric (-1,+1) vs asymmetric (0,1)

Generally the differentiable requirement is needed for hidden layers and tanh is often recommended as being more balanced. The 0 for tanh is at the fastest point (highest gradient or gain) and not a trap, while for logistic 0 is the lowest point and a trap for anything pushing deeper into negative territory. Radial (basis) functions are about distance from a typical prototype and good for convex circular regions about a neuron, while the sigmoid functions are about separating linearly and good for half spaces - and it will require many for good approximation to a convex region, with circular/spherical regions being worst for sigmoids and best for radials.

Generally, the recommendation is for tanh on the intermediate layers for +/- balance, and suit the output layer to the task (boolean/dichotomous class decision with threshold, logistic or competitive outputs (e.g. softmax, a self-normalizing multiclass generalization of logistic); regression tasks can even be linear). The output layer doesn't need to be continuously differentiable. The input layer should be normalized in some way, either to [0,1] or better still standardization or normalization with demeaning to [-1,+1]. If you include a dummy input of 1 then normalize so ||x||p = 1 you are dividing by a sum or length and this magnitude information is retained in the dummy bias input rather than being lost. If you normalize over examples, this is technically interfering with your test data if you look at them, or they may be out of range if you don't. But with ||2 normalization such variations or errors should approach the normal distribution if they are effects of natural distribution or error. This means that they with high probability they won't exceed the original range (probably around 2 standard deviations) by more than a small factor (viz. such overrange values are regarded as outliers and not significant).

So I recommend unbiased instance normalization or biased pattern standardization or both on the input layer (possibly with data reduction with SVD), tanh on the hidden layers, and a threshold function, logistic function or competitive function on the output for classification, but linear with unnormalized targets or perhaps logsig with normalized targets for regression.

Thank you to everyone on this thread for their contributions. I have been working on using the MLP for function approximation (MLP-FA) problem. I started from the scratch and have been learning from experience. 

I have found out that what @Nils Goerke suggested is true for the MLP-FA problem. I started off with the logistic function and noticed a very slow change in error after some iterations. And in most cases, it would overshoot. But on using the tanh function, I have notice a great change in the network performance.

However, I actually would like to confirm something. Can I modify my error function (performance function) to be the mse of a function of the network output. To be simply put, I am training a network whose output is the elevation and azimuth angle of the spherical plane. I would like to stop training if the dot-product angle (inner angle) between the vector pointing at the target direction (\theta, \phi) and the vector pointing at the output direction (\hat{\theta}, \hat{\phi}). 

Would this be reasonable? 

I worked out the performance function and it contains sine and cosine components. From your experiences, what effect do you think this would have on the MLP training?

Thank You.its informative. 

Hello everyone, I would ask if anyone can help. I need to use deep belief network with backpropagation for time series prediction. In this case what should be the best transfer function I can use!. Thanks in advance 

logsig pureline transfer function