I think gamultiobj is able to handle nonlinear constraints, but it may take a long time for each iteration. Besides, it takes long time to generate the initial population even if only linear constrants are applied. A better solution is that you may try to use some customer algorithms to handle these linear or nonlinear constraints.
I don't know the case in MATLAB 2014, I am using R2017a, you may find from the attached link that nonlcon is the function for the user to set nonlinear constraints. https://it.mathworks.com/help/gads/gamultiobj.html
I installed Matlab R2015b I found that it has the possibility to handle nonlinear constraint using optimization toolbox, witch is not possible in previous Matlab versions.
I run the optimization problem and I got a Pareto front with just one solution, what do you think about that?
the Pareto solution file is attached in this message
thank you so much for your help, I have one last question if you don"t mind,
I used the weighted sum method in Matlab with the Fmincon solver. I got a Pareto front with both dominated and non dominated solution. my question is how can do to eliminate the dominated one and get the Pareto front with just non dominated solution?