Otsu's thresholding algorithm is implemented in the function graythresh(), from the Image Processing Toolbox. It is implemented in pure matlab. The code can be inspected by
>edit graythresh
and is really simple to follow the implementation.
You need to look for optimisation algorithms. Have a look in the matlab contributed. There are some that can estimate thge optimal paramets for an energy functional such as the snake.