What are the nature of the functions f(x) here? I presume these are non-linear? It would be interesting to see a typical system you are trying to understand.
you can have: just use a polynomial form for f, with zero constant and linear term, then x1=...=xn=0 is an equilibrium and Jacobian matrix is identically zero matrix! For this non-hyperbolic point, you can't use linear stability condition. I think (but I am not sure for higher dimension) this form maybe exhibit a singularity at finite time.
Dear Sajad, this happens when the Jacobi matrix at X* is similar to either some upper-triangular, or lower-triangular marix with a zero diagonal. An example: the zero equilibrium,
note that in usual bifurcation point, this phenomena doesn't happen. what happened in bifurcation point, is about real part of eigenvalue not eigenvalue!
Take the Lure' system $\dot{x}=Ax +b f(c^{T}x)$, where $A$ is a nilpotent matrix and $f$ is such that the system has dominating linear part, e.g., $f(y)=y^{3}$. Recall that a nilpotent matrix has null eigenvectors only.
Yes, a dynamical system can have an equilibrium point with all eigenvalues are zero.
Although this case was not dealt in the famous Ljapunov's article ("Problème de la stabilité du mouvement", Annales de la Faculté des Sciences de Toulouse, vol. 9, 1907), in 1954 an unknown manuscript was discovered and published in 1963 by State Univ. Press, Leningrad under the title "Issledovanie odnogo iz osobennyk slucaev zadaci ob ustoicivosti dvizeniya" (Study of a particular case of the problem of motion stability). This text is related to your question.
The elements of this study were translated in English "Stability of motion", Academic Press, Mathematics in Science and Engineering, Vol. 30, 1966, with what interests you pp.128-202.
In fact the general case of a Dim n autonomous ODE, n>2, is not completely dealt. Ljapunov mentions that in this case we meet peculiar difficulties, and so he solves the problem completely for n=2 (pp. 131-202) for the standard equation (pp. 131-184):
dx/dt=y+X , dy/dt=Y
where the series expansions of X and Y do not contain terms of second lower order.
All the possible cases are summarized pp. 178-180, with the stability conditions of the point (0;0).
An attempt of dealing the equivalent problem for Dim 2 maps (the two eigenvalues are equal to +1) is given pp. 239-245 of the book "Chaotic Dynamics. From the one-dimensional endomorphism to the two-dimensional diffeomorphism" by Christian Mira (World Scientific, 1987), and also the bifurcations by crossing through this case pp. 245-251.