The state of art is H-infinity control and filtering and geometric conditions to covert a nonlinear system into a form suitable for estimation.  

Try the book in the link below,  it has excellent examples on exactly what you are looking for.  

http://www.springer.com/engineering/control/book/978-0-85729-521-7

You are right: there has been no great advance since the first predictor/corrector estimators (the Kalman filter and the Luenberger observer) were proposed. All other general recursive methods share the same structure but differ in the correction gain computation technique. Among them are the extended Kalman filter, iterated Kalman filter, unscented ("sigma-point") Kalman filter, cubature Kalman filter, particle filter, H-infinity filter, etc. None of them guarantees convergence for all nonlinear systems.

It is even possible to imagine a system (with a multimodal state vector distribution), for which the notion of a single "state estimate" is meaningless. For such systems, there exist more general methods based on the Fokker-Planck-Kolmogorov equation, but they are much more complex.

My personal experience is that it is often better to investigate the system in depth and to invent an ad hoc estimator which is very simple (though not general) and has a clear physical meaning.

Article A Simple Observer for Gyro and Accelerometer Biases in Land ...

I strongly recommend using PSO(Particel swarm optimization) for parameters  estimation. It was originallly designed and introduced by Eberhart and Kennedy in 1995.

The PSO is a population search algorithm based on the simulation of the social behavior of birds, bees or a school of fishes. Each individual within the swarm is represented by a vector in multidimensional search space. This vector has also one assigned vector which determines the next movement of the particle and is called the velocity vector. The PSO algorithm also determines how to update the velocity of a particle. Each particle updates its velocity based on current velocity and the best position it has explored so far; And also based on the global best position explored by swarm. All of the particles have the fitness values which are evaluated by the objective function to be optimized, and have velocities which direct the flying of the particles. The objective function is to minimize the integral squared error involved in the analysis system.

The optimization method can be identified as an effective way to find unknown parameters of a gray system. Also, more complex multi-parameter can be derived under the measred reference data. The PSO algorithm can be easily implemented and find resources on the website.

I agree with Choi Hyungjoo, PSO will work like a charm, though it's necessary to define all boundaries conditions as a penalty function otherwise it may get stuck on bounds (heuristic method) I would also recommend you Square Root Unscented Kalman Filter proposed by Rudolf van der Merwe and Eric A. Wan. It's definitely better - stable version of the Unscented Kalman Filter. I have been using both: PSO and SRUKF for quadrotor model parameters estimation.

Thank you all for answers.

Before I have used PSO for nonlinear optimization. However I never saw an example which PSO is used for parameter estimation in a nonlinear dynamical system. Could anyone provide a specific example for that?

Hello. Without being more specific it's not possible to answer your question. You say the system is highly nonlinear, so at the very least you would need an extended Kalman filter or a variant thereof (iterated EKF, unscented KF). This assumes that you have white process and measurement noise and they are mutually independent. Also one supposes there is no finite-dimensional filter for your problem. You could also try Gaussian sum filters, which are good for modelling multi-modal state distributions. A particle filtering/Monte Carlo technique might work better than EKF based methods since you are essentially trying to evaluate a complicated integral over the state-space to get the PDF of the next state (assuming a recursive estimator). I have seen examples where particle filters are very efficient. For continuous state spaces (with finite-dimensional states - i.e. not a PDE), you may have to resort to the Fokker-Planck-Kolmogorov equation, as noted by one of the respondants (though I'm not sure if this applies if the process noise is not white) and use general nonlinear filtering theory (see Jazwinski "Stochastic Processes and Filtering Theory"). All of this naturally assumes that you know what measurement sequence applies to the state that satisfies the (stochastic) dynamical equation. If you don't (as is the case in radar, sonar, etc), you need to include data association and your filter gets even more complicated. Although this is generally not relevant in control theory, it is relevant for general estimation theoretic problems. For the latter, you may find this review article (from a target tracking perspective) of interest.

I just re-read your question and you seem to want to do parameter estimation rather than state estimation. Have you tried something simple, like iterated least squares?

Article Taxonomy of multiple target tracking methods

In the references below you can find our solution to a problem of estimating parameters of a nonlinear system.

V. PETRIDIS, E. PATERAKIS, A. KEHAGIAS “ A Hybrid Neural-Genetic Multi-Model Parameter Estimation Algorithm” IEEE Transactions on Neural Networks, Special issue on hybrid systems , Vol. 9, No. 5, pp. 862-876, September 1998.

V.PETRIDIS and A.KEHAGIAS “Predictive Modular Neural Networks: Application to Time Series “Kluwer Academic Publishers, 314 pages, Boston 1998.

Have you considered perhaps using the Nelder-Mead algorithm and a cost-function?

Thank you all who shared his/her opinion and knowledge. I appreciate all the knowledge and experience.

I think I need to be a little more specific. For my Phd thesis I am searching of application of estimation methods to estimate contact conditions between wheelset and rail. I am currently working on wheelset dynamical model, and I know that there exists some fairly new research about it.

In the publication below

"Hussain, I., Mei, T. X., & Ritchings, R. T. (2013). Estimation of wheel–rail contact conditions and adhesion using the multiple model approach. Vehicle System Dynamics, 51(1), 32-53."

They already applied "Kalman Filtering" for estimation of contact conditions based on the dynamic model of a wheelset. However in this work they just linearised the wheelset dynamics around operating point and then applied conventional Kalman Filter. Besides in order to estimate contact conditions Kalman Filter is not enough, so that they used Fuzzy Logic to interpret residuals of Kalman Filter.

I just wanted to know if such kind of thing can be done without linearising the dynamical model and without using post processing of data (i.e. fuzzy logic)?

If someone knows such a method (I think I can do it using an unscented Kalman Filter, but I am not sure) and if he/she can share it with me, it will be greatly appreciated.

Thank you again.

I do not know the dynamics of your problem. I would suggest the following unless you have specific reasons to use some kind of kalman filtering.

If you need state estimation  try particle filters. In case of parameter estimation we found  that the methods in the reverences in my previous answer are effective even in case of highly nonlinear systems.

Hello again. Now that you mention your problem area, it seems that you are dealing with a system that is what I would call multi-modal. I imagine that wheel–rail contact includes different dynamics when the wheel is rolling or sliding or a bit of both. For this type of estimation problem, where it is desired to estimate the mode of the system, as well as perhaps its state, there is quite a large literature under the heading of hybrid systems. Different approaches, mostly Kalman filter based, include multiple model, GPB (generalised pseudo-Bayesian), and interacting multiple model (IMM) algorithms. As the problem of manoeuvring target tracking is a hybrid system estimation problem, these approaches have been developed quite a lot in this context. You can find some references in the survey paper below. I am sure that more has been done since then, but it is somewhere to start. Just thinking about the nonlinearity aspect - it may be that it is more important to model the modal transitions of your system than the fact that in each mode it may be nonlinear. In any case, the latter nonlinearities can be dealt with by using a mode-conditioned EKF. The IMM in particular requires you to specify the transition probabilities for the modes of the system - you can just make these uniform if you don't know them, or use average sojourn time information if you have it. BTW there should be no need to use fuzzy logic to solve a hybrid system estimation problem.

Technical Report A Survey of Manoeuvring Target Tracking Methods

Please see the report arXiv:1503.04313v1.pdf

I agree with Vasiliy that there haven't been any significant advances in terms of the basic state estimation principles. However, recent trends in applications, such as Cyber-Physical or Networked Systems, IoT, Smart Grids, Big Data, Industry 4.0 etc. have motivated new research into "distributed" or "partition-based" state estimation algorithms. For someone who finds this question on his search for interesting research directions, I mention this here and attach a paper and a link to a project, both containing useful references.

https://www.researchgate.net/project/State-and-Parameter-Estimation-for-Large-Scale-and-Distributed-Systems

Article An iterative partition-based moving horizon estimator with c...

Please see the reports arXiv:1503.04313v1.pdf, arXiv:1505.07201v1.pdf and arXiv:1505.07208v1.pdf. and the two papers in SADHANA December 2016 issue..

Please see the reports arXiv:1503.04313v1.pdf, arXiv:1505.07201v1.pdf and arXiv:1505.07208v1.pdf. and the two papers in SADHANA December 2016 issues.

Maybe you can consider the recursive least squares algorithm (RLS). RLS is the recursive application of the well-known least squares (LS) regression algorithm, so that each new data point is taken in account to modify (correct) a previous estimate of the parameters from some linear (or linearized) correlation thought to model the observed system. The method allows for the dynamical application of LS to time series acquired in real-time. As with LS, there may be several correlation equations with the corresponding set of dependent (observed) variables. For RLS with forgetting factor (RLS-FF), acquired data is weighted according to its age, with increased weight given to the most recent data.

Years ago, while investigating adaptive control and energetic optimization of aerobic fermenters, I have applied the RLS-FF algorithm to estimate the parameters from the KLa correlation, used to predict the O2 gas-liquid mass-transfer, hence giving increased weight to most recent data. Estimates were improved by imposing sinusoidal disturbance to air flow and agitation speed (manipulated variables). Simulations assessed the effect of numerically generated white Gaussian noise (2-sigma truncated) and of first order delay. This investigation was reported at (MSc Thesis):

Thesis Controlo do Oxigénio Dissolvido em Fermentadores para Minimi...

I am not sure if this is what you are looking for, but it might give you inspiration:

http://www.physicsjournal.net/article/view/30/3-3-11