First, transform the nonlinear system into a linear system. Next, calculate the eigenvalues at the equilibrium point. If all eigenvalues have a negative part and the equilibrium is hyperbolic. Then the system is stable.
Otherwise, you have to apply the bifurcation or center manifold theory.
For example, x''+sin(x)=0 can be written as, x"+x=0. Put x'=y, then x"=y'.
Firstly, try to change the second-order nonlinear system of equations into a first-order linear/nonlinear system of equations, then calculate the eigenvalues of the jacobian matrix at an equilibrium point, if all the eigenvalues have a negative real part, implies the system is stable at that equilibrium point otherwise unstable.