Dear David

This is linear system and the so called parametric optimization problem. Use Hurwitz criterion for finding stability region in parameter (K1, K2) space and then put these constraints as nonlinear constraints in fmincon. For example if the closed loop transfer function has the form L(s)/M(s) where M(s)=s^2+bs+c, then matrix A-KC is stable if and only of b>0 and c>0. Generally if the cost function has the form \int\limits_0^{\infty} \e(t)^2dt, where e(t)=w(t)-y(t) is an output error, then one can find the solution analytically.

Regards

Piotr

As any other NLP solver it does not know how your OCP looks like. So it's up to you to transcribe the problem, which means properly transforming cost function, differential equations, and path constraints into the format required by the solver.

Have a look at transcription methods, like multiple shooting or pseudospectral methods.

If the system is second order and the cost function is quadratic then It is also possible to solve suitable linear quadratic (LQ) problem.