1. you need to transform tracking problem into stabilisation one. i.e express the system into error between system and desired states. Then apply SMC. Alone x1-x1d isn't enough, you need (x2-x2d) , (x3-x3d) etc.

2. if nonlinearity is actuator saturation, i suggest read the book chapter:

Deepak Fulwani, B. Bandyopadhyay, "Design of Sliding Mode Controller with Actuator Saturation ", Advances in Sliding Mode Control: Control, Theory and Implementation, Eds. B. Bandyopadhyay, S. Janardhanan, Sarah K. Spurgeon in Lecture Notes in Control and Information Sciences, Springer-Verlag,

http://link.springer.com/chapter/10.1007/978-3-642-36986-5_10

3. If nonlinearities are function of states then better transform it to normal form and see to it if you can apply SMC.

As Dr. Paril said, the problem is because the lack of stability. The most important feature of any control technique is the guarantee of stability. Then one can discuss about optimization. First, be sure about the stability, and then, execute PSO for optimizing the parameters.

The nonlinear ccf is normally known as the "regular form".  Converting a system from one from to an other doesn't affect the properties of the system however conversion to CCF or OCF  or regular form makes our calculations easy.

The sliding surface you've designed does not guarantee that forcing S = 0 means e1 =0, rather it means that the sum X2+X3+ C(X1d-X1)=0.

You'd better take s = e1 and do stability analysis, hopefully it works.