Hi Mohammad, as you said, the sliding mode is a good suggestion, but in its simple form, it may not be able to handle too complicated systems. in such cases, I have seen people sometimes use hierarchical controllers or even integrate other controllers with the MPC to make it more robust.

anyway, Kalman-based methods are also applicable in such cases.

Dear Mohammad Javad Mirzaei

Bayesian optimization and Bayesian inference can predict the behavior of a black-box system, where there is no information about how the system evolutions over time due to unavailability of ODEs or expensiveness of function evaluations.

Please see the Wikipedia URLs:

https://en.wikipedia.org/wiki/Bayesian_optimization

https://en.wikipedia.org/wiki/Bayesian_inference

There is also a relevant question posted in ResearchGate as:

https://www.researchgate.net/post/How_can_we_find_the_optimal_weights_for_a_cost_function

Sensor-based INDI(Incremental Nonlinear Dynamic Inversion) is a control method that feedback the measured angular acceleration from IMU sensor and is a very robust control method for model uncertainties as it does not depend on the model.

But, it is known that the sensor-based INDI control using the angular acceleration measured from the sensor is significantly affected by the time delay of the measured angular acceleration.

Hybrid INDI control, which uses a mixture of model and sensor-based INDI, compensates for these shortcomings and is a control method that can be selectively applied according to aircraft nonlinear characteristics and flight conditions

Hi Mohammad Javad Mirzaei for estimation of state of a nonlinear system you can use High Gain Observer. If you know some of the states or the out of the system you can apply High Gain Observer. For linear System we can use linear observer or Kalman filter but in nonlinear case these two are mostly failed. So, its better to use high gain observer.

While talking about sliding mode controllers. SMC is robust controller and have good results in most cases. If you the estimate the nonlinear system state using HGO then you can also use SMC in conjunction with HGO to get good results. For more detail you can read this paper which discuss both HGO and SMC and SMC is used in conjunction with HGO. May be this paper will help you.

[1]. Ullah, Hameed, et al. "Robust Output Feedback Control of Single-Link Flexible-Joint Robot Manipulator with Matched Disturbances Using High Gain Observer." Sensors 21.9 (2021): 3252.

And for High Gain Observer you read Hassan Khalil papers and Books. You can find it here.

https://www.egr.msu.edu/~khalil/

The Time-Delay Control approach based on input-output linearization would be another strategy to control systems with unknown dynamics. while TDC has properties similar to the incremental control, which stills relies partially on model information, tuning for feedback gains in TDC is comparatively simplified than Incremental NDI.

If sliding mode is an option, it could also be integrated into an incremental framework, so that model dependency could be fairly reduced.

See model-free control (intelligent PID) by Fliess, which approximate the model in a data-driven manner (so-called ultra-local model):

M. Fliess and C. Join, "Intelligent PID controllers," 2008 16th Mediterranean Conference on Control and Automation, 2008, pp. 326-331, doi: 10.1109/MED.2008.4601995.

Reinforcement learning / approximate dynamic programing would be another alternative which could provide (near to) optimal control performance.