It depends of which kind of stability you want to investigate.
(i) Local stability => Search for singular points and then, linearize the system around these points
(ii) Global stability => Search for a Poincarré-Bendixon domain and/or a Lyapunov function
(iii) Lyapunov exponent is a quantity that characterizes the rate of separation of infinitesimally close trajectories, it is often use to ivestigate if the system is chaotic or not.
(iV) I recommend you the two book (The first chapter of the first book talk about your question).
A-
Mathematical Biology: I. An Introduction (Interdisciplinary Applied Mathematics) (Pt. 1) Hardcover
by James D. Murray (Author)
B-
Differential Equations and Mathematical Biology Hardcover – February 26, 2003
by D.S. Jones (Author), B.D. Sleeman (Author), D. S. Jones (Author)
A compact and straightforward description is found in the works of Gabor Stepan. He focuses on the milling process but the outlined method is applicable in general. I suggest to look right into his PhD thesis at http://galilei.mm.bme.hu/~inspi/