PSO involves an objective function (cost function) that the user supplies as a MATLAB function, which is to be either minimized or maximized. A simple thing to try is program a penalty in the objective function. That is, whenever the design variables fall outside of your constraints, return the function value as something larger that it could otherwise be, This assumes you're trying to minimize the objective function. If you're trying to maximize the objective function, return a value smaller than it could otherwise be. This will introduce discontinuities in the objective function, but PSO won't care about that.
There may be other more elegant solutions, but the above is something easy you can try, By the way, you might also consider just using another optimization package that handles constraints, Check out http://www4.ncsu.edu/~ctk/imfil.html for example.
There are different options to solve a constrained optimization algorithms. The penalty function methods are simpkest and the most commonly used.
To gain a good insight, I recommend reading the excellent article written by Prof. Coelho and colleagues. This paper provides a comprehensive survey of the most popular constraint-handling techniques currently used with evolutionary algorithms (please refer to the link and attached file).
1.In constrained optimization problems , points satisfying Kuhn-Tucker conditions are likely conditions for optimal solution.
2.Transformation methods are the simplest and most widely used for dealing with constraints. Constrained problem is transformed in to unconstrained one by adding penalty terms for each constraint violation. Some functions worthwhile are, parabolic penalty, infinite barrier penalty, log penalty,inverse penalty,and bracket operator penalty.
3. The bottleneck with penalty functions is that the penalized function may often go distorted becoming non-amenable for solution. The solution is method of multipliers which uses a fixed penalty parameter for each constraint.
For further details, refer,
Kalyanmoy Deb, Optimization for Engineering Design, Prentice Hall of India, 2004.
You can use constraint handling techniques such as : penalty functions; special representations and operators; repair algorithms; separate objective and constraints; and hybrid methods. Penalty function approach is very common.