You can always prove local exponential (un)stability of an equilibrium point by looking at the eigenvalues of the tangent linearization. Apart from that there is no general constructive method for proving stability. For instance you know thare exists a Lyapunov function, but you cannot in general compute it by a constructive algorithm.
A very good book on the subject is "Nonlinear systems" (3rd edition) by H. Khalil.
You can prove the stability of a system by observing its time response to delta pulse. If the output grows indefinitely with time, the system will be unstable. From the point of view of the anal sis you can piece wise linearize the system and apply on each part the stabilty criterion.
Hello. As has already been said, local stability of an equilibrium point can be established through examination of the eigenvalues of the linearized system. This is of course very important since you can get an idea of the behaviour of your system around the equilibrium of interest. On the other hand, there are two (general) approaches for tackling stability questions for nonlinear systems: 1) stability analysis by Lyapunov's Second theorem and thus, construction of lyapunov functions (see for example the book by H. Khalil); 2) the dual to Lyapunov's 2nd theorem and construction of density functions (see the works by Anders Rantzer). The second one has been drawing increasing attention and there has been shown for some examples to provide an easier way to establishing stability properties. Moreover, the two approaches can also be combined.
First you find the impulse response of your system in time domain. If your system is unknown the pass the impulse as an input. Then measure the output response. If it is finite then it will be a stable system. If it is infinite then it is unstable system