Do you have any symmetry in your system? Is it time reversible? Also, do you have "symmetric" periodic solutions, "in the mirror" or something alike? Do you have a system "kinetic+potential"?

Great thanks for the answer from Cristina. The hamiltonian system can be normalized as 

H(x,y,px,py,s)=1/2 px^2 + 1/2 py^2 + 1/2kx(s)x^2 +1/2ky(s)y^2 - K ln (x+y) + epsilon_x^2/x^2 +epsilon_y^2/y^2, where Kx(s) and Ky(s) are periodic function of s, K and epsilon_x, epsilon_x are constant. 

Are epsilon_x, epsilon_y small? are they different? Are all K,  epsilon_x, epsilon_y varying parameters, of K is fixed? In any case, here are some references which I think might be helpful:

 Sanders, J. A.; Verhulst, F., Averaging Methods in Nonlinear Dynamical Systems. Berlin-Heidelberg-New York-Tokyo, Springer-Verlag 1985.

 https://math.uc.edu/~meyer/siam08.pdf

 Also, check:

 http://eprints.ma.man.ac.uk/1299/

Here, for my case, the value of epsilon_x an epsilon_y is constant and can not be considered as small. The value K is also not small.  This means the nonlinear term of force can not be considered as perturbation.  The period function of Kx(s) and Ky(s) are usually period step function and satisfy the basic equation Ky(s)=-Kx(s) or Ky(s)=Kx(s).

For the first case, Ky(s)=-Kx(s), it means quadrupole channel,  and for the second case Ky(s)=Kx(s), it represent solenoid channel. These two channel are the basic cases in accelerator physics.  

The dynamics equation derived from the hamilton H(x,y,px,py,s)=1/2 px^2 + 1/2 py^2 + 1/2kx(s)x^2 +1/2ky(s)y^2 - K ln (x+y) + epsilon_x^2/x^2 +epsilon_y^2/y^2 is called envelope equation (we focused on x and y in real space which represent the beam size or envelope).  Because of  the Hamiltonian is periodic, there exists  only one period solution for  the system, which also means there is only a group of  initial value of  (x,y,px,py) _0 that can make the solution of (x,y) repeats perfectly with the same period as kx(s). 

In the classic dynamics system, the stability of the system is determined by the eigenvalue of Jacobi Matrix in one period. 

So let us go back to the original question. The 4*4 matrix has 4 eigenvalues. The instability occurs when the Arg |Lambda_i|=Arg |Lambda_j|=180 or Arg |Lambda_i|+Arg |Lambda_j|=360, which is also called as resonances conditions. 

In my calculation calculation, in solenoid channel  Ky(s)=Kx(s), the stability is only excited by the condition  Arg |Lambda_i|=Arg |Lambda_j|=180. When the condition 

|Lambda_i|+Arg |Lambda_j|=360 is satisfied, nothing happens. 

In quadrupole channel, Ky(s)=-Kx(s), the instability happens only when |Lambda_i|+Arg |Lambda_j|=360. 

 The phenomenon remind me of M. G. Krein's theory.  

 Sometime, when the eigenvalues collide, the system gets unstable, and sometimes, it does not. 

Or do you have any other good ways to explain this ?

I'll tell you what I know.

If a Hamiltonian is invariant with respect to to some group action (e.g. it is rotational invariant, or it is invariant with respect to a discrete group action, say Z_2), then such symmetries have non-trivial effects on the bifurcation of equiibria or relative equilibria (rotational equilibria) and their stability, as external parameters vary (your K, epsilon_x, or epsilon_y). In combination with time-reversibility, such symmetries also effect the bifurcation and stability of periodic orbits; the characteristic exponents (your eigenvalues) sometimes collide on the circle and then stay there, or split, making the periodic solution unstable. For certain classes of Hamiltonian systems, there are criteria which tell you what kind of bifurcation to expect. For instance, Hamiltonian-Hopf bifurcation theory for equilibria of Hamiltonian systems at a Krein collision.

 Now, with respect to you particular system, there are things which confuse me. First, your system is not symmetric with respect to any spatial symmetry unless epsilon_x=epsilon_y. Second, your statement "Because of the Hamiltonian is periodic, there exists only one period solution for the system, which also means there is only a group of initial value of (x,y,px,py) _0 that can make the solution of (x,y) repeats perfectly with the same period as kx(s)" is also confusing, since I am not aware of such a statement. As far as I know, the existence of isolated periodic solutions is a calculus of variations problem (functional analysis) which is a kind of non-trivial. My guess is that you mean something else, and we just have a hard time communicating.

  To conclude, if you want to continue this discussion, feel free to email me.