An artificial neuron performs a non-linear function denoted by sigma, on the weighted sum of its multiple input signals plus a constant denoted by theta.
What's the role of sigma? Is there any study on parametric functions sigma(lambda) for a variable parameter lambda, as the Feigenbaum cascades study the dependency on f(lambda) of families of functions f parametered by lambda, to generate sequences u(n+1)=f(lambda)(u(n))?
The sigmoid function in deep learning transform the sum of the weighted input into a new dimensional-space locates in [-1,1] nonlinearly . This transformation provides new possibilities for the samples to be regressed or classified. If the activiation function was a linear function, not the sigmoid function, the deep network becomes a deep MLP. There are also a few other nonlinear activation function like RELU, HyperbolicTangent function et al.
Hi Renaud,
If I understand you correctly, you are trying to model a dynamical system. Recurrent neural networks have proven their worth in this respect:
If you're not so interested in model persistence and the issues such as trying to capture short-term memory by a network, and you are more interested in the nonlinear regression aspect of the response of sigmoid functions, I would be tempted to train a simple back-propagation neural network (BPNN) with vector inputs for u(n) and lambda, and vector outputs u(n+1), setting the activation function of hidden layer neurons to the sigmoid function S(x) = 1 / (1 + e^(-x)) biased by theta.
You could then experiment with various network architectures (e.g. varying the number of hidden layers, hidden neurons in each hidden layer and cross-validation using various proportions of training data to be used for network learning and simulation validation. Optimising the architecture in this way is to be recommended. Here is an example from a recent-ish paper:
Taylor, M., Kazadzis, S., Tsekeri, A., Gkikas, A., & Amiridis, V. (2014). Satellite retrieval of aerosol microphysical and optical parameters using neural networks: a new methodology applied to the Sahara desert dust peak. Atmospheric Measurement Techniques, 7(9), 3151-3175.
Hope this helps,
Mike
The sigmoid function in deep learning transform the sum of the weighted input into a new dimensional-space locates in [-1,1] nonlinearly . This transformation provides new possibilities for the samples to be regressed or classified. If the activiation function was a linear function, not the sigmoid function, the deep network becomes a deep MLP. There are also a few other nonlinear activation function like RELU, HyperbolicTangent function et al.
.
varying the sigmoid parameter of a NN, feed the output into the input and observe what happens has been the game of the town in the 90's !
.
for instance
x(t+1) = th( a*th(a*x(t)+a/2) - a*th(a*x(t)-a/2) - a )
leads to a clean "logistic-like" Feigenbaum bifurcation diagram
.
see :
A Field Guide to Dynamical Recurrent Networks
John F. Kolen, Stefan C. Kremer
Wiley & Sons (2001)
pp-68sq
.
same example in :
Recurrent networks: State machines or iterated function systems
John F Kolen
Proceedings of the 1993 Connectionist Models summer school
.
Thanks Fabrice Clerot , these references are very much appreciated!
The activation function is the nonlinear transformation that is done over the input signal. This transformed output is then sent to the next layer of neurons as input. Activation function is ReLU – Rectified Linear Unit. And the other activation functions are tanh, sigmoid, leaky ReLU, Maxout, and ELU.
ReLU function is widely used as activation function in neural networks. The advantage of ReLU is that it does not activate all neurons at the same time. From the image for ReLU function, it converts all negative inputs to zero and the neuron does not get activated. This makes it very computational efficient as few neurons are activated per time. It does not saturate at the positive region. ReLU converges six times faster than tanh and sigmoid activation functions.
A variety of sigmoid function includes the logistic and hyperbolic tangent functions as the activation function of artificial neurons. Sigmoid curves are also common in statistics as cumulative distribution functions (which go from 0 to 1), such as the integrals of the logistic distribution, the normal distribution.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence