In a higher-order sliding mode control design for a system with relative degree two, how can system stability be proved in the presence of actuator saturation?
If you are using the quasi-continuous HOSM, you can overcome the problem of actuator saturation by artificially increasing the order of the controllers. The needed higher order derivatives of the sliding variable can be robustly and exactly obtained using the numerical higher-order sliding mode differentiators.
See my paper "Arbitrary-Order Sliding-Mode-Based Homing-Missile Guidance for Intercepting Highly Maneuverable Targets"
If the problem is actuator saturation then what is the exact technique used "i don't know". But once we've gone through the same situation in a process control problem. Sometimes the system dynamics are very slow and the controller try to strive the surface to zero as quickly as possible which causes actuator saturation. We handled the problem by bringing the second order sliding mode into the loop at times when sufficient output has been produced by some constant (educated guess) actuator action.
The stability problem is not that much severe for second order sliding modes. Since your system has a relative degree two with respect to the switching manifold so as obvious choice can be the real twisting algorithm which we don't need to prove mathematically for a specific system.