Hi,

It will be wise to start with the simplest of architectures such as the MLP with a single hidden layer and make simulations by comparing to another classifier for the baseline as this is a justification for which you are using machine learning.

For more details and in formation about this subject i suggest you to see links and attached file on topic.

Article Weight Initialization Possibilities for Feedforward Neural N...

Article Interval Based Weight Initialization Method for Sigmoidal Fe...

Article Comparison of linear regression and artificial neural networ...

Mohamed-Mourad Lafifi thank you for your answer. I have tried MLP and simple feed forward network architecture, but since my signals vary in time these architectures did not model my system well. I was thinking about networks similar to LTSM or NARX neural network models.

Kimia Hadian, have a look at Kevin Gibson's blog (https://rkevingibson.github.io/blog/neural-networks-as-ordinary-differential-equations/) and linked references.

Usage of Neural Network for Non-Linear system is not useful at all; unless you have trend analysis of the input patterns. Why? Because..ANN will simply try to give approximations based on trained sequences. Thus, for untrained sequences the performance is generally low (unless you have huge data-set to cater to several varying conditions and parametric association).

For Non-linear systems its better to opt for Fuzzy method. Its fairly easy to build prototype and test your system. Also, you wouldn't give guess approximations like in the case of neural network.

Check out this video to jump-start with fuzzy logic based technique: https://www.youtube.com/watch?v=O348HnWPm7A&ab_channel=MATLAB

However, if you still want to stick with neural network then kindly check out cascaded neural network and Long Short Term Memory Network. That will do provided you have huge data-set.

Hi Kimia,

I also had to train a network to fit a dynamical nonlinear system. I tried using LSTM and Echo state networks. They work decently even if training the LSTM is a bit cumbersome since you need to manipulate your data to perform a batch truncated backpropagation, while the Echo state networks just require a least square because of how they are built.

If you know the states of your system I would suggest you to build your own architecture, for example your network could take as inputs the current system state and the process inputs, generating the next system state (it's an approximation of the state update formula x(t+1) = f(x,u)). You could generate your next state using a MLP or you could exploit the knowledge of your system for example adding explicitly in f() the relation between speed and position of the mass, boiling down the problem to just identify the velocity getting the position with Euler or Tustin approximation. I tried this to identify the model of a double pendulum and it worked great.

Using this architecture you also have the advance of having a insight of what the actual network state is representing (position and velocity), this will help in case you are using your model for control.

It always remains the problem of the state initialization if your state is not completely measurable. In case your system is asymptotically stable you could use a state washout technique in which you don't consider the first samples of your training sequence in the cost and use them just to let the network state to settle at some input dependent point. In this way the wrong state initialization should not affect the learned gain.

I'm always available in case of questions.

Hope to be helpful.

I would recommend invariant feature transform (sift) with LTSM for your issue

RNNs are used if you need to memorize previous state of the system for the present state of the system. Maybe you could use a combination of Neural Networks, predict the Non-linear part and linear part.

Dear Kimia,

Several neural-networks approaches are proposed in the literature. I have been working with one structure proposed by H. M. Romero-Ugalde reaching a good performance. These are several papers using those neural-networks:

1. "Neural network design and model reduction approach for black box nonlinear system identification with reduced number of parameters"

2. "Computational cost improvement of neural network models in black box nonlinear system identification"

3. "Balanced simplicity–accuracy neural network model families for system identification"

4." Solving fractional differential equations of variable-order involving operators with Mittag-Leffler kernel using artificial neural networks"

5. "Robust Estimation of Balanced Simplicity-Accuracy Neural Networks-Based Models".

Best regards

Thank you all for your answer.

RBF can perform good under several scenarios

Hi Kimia

Different neural network architectures do not have significant impact on the performance of the neural network. Try data pre-processing and analysis before applying ANN.