As I understood the question, the matrices A and B are as follows:

A={ (x1,y1,1), (x2,y2,1), ......, (xN,yN,1) }

B={ (x1', y1'), (x2', y2'), ........, (xN', yN') }

Consider only the first point. So, for a specific value of Sx, Sy, r, tx and ty, A*T will give you a 2d point which may be different than (x1', y1') because of error.

Generalizing this for each point, B-A*T will give you a matrix of N 2d points which are the differences between the expected result and the actual outcome for specific values of parameters. It can also be referred to as distances between two points in 2d space. So, squaring and adding them is the total square error. If we denote total square error by E, then

E=sum(sum((B-A*T).^2));

For varying values of parameters, this is the function you want to minimize. So it becomes your objective function. Your variables are Sx, Sy, r, tx and ty with the bounds of the variables given. Now you can use "optimtool" GUI of MATLAB. Total square error will be calculated in the function file. The GUI gives you a choice for algorithm. Any algorithm which is applicable for constrained nonlinear optimization, such as fmincon, ga, etc. will give you the optimum point.

Of course, you can also minimize mean square error, where the average of total square error is considered. But it will not affect the optimum point.

I hope that I have explained properly the point.

As suggested above, apply least square approximation for minimum error.

Thanks for responses, and sorry for my bad English!!, Yes I have already tested "optimtool" functions on Matlab (may be applied on any non linear function), however, the results are mostly local optimum. I was hopping for more specific solution to the problem with numericals approaches, here I have downed the problem as, above there is a little mistake (between sx and sy):

[Xi Yi]  and [Xi' Yi'] are sets of centred 2d point, so we do not bother with translation (tx, ty)

[Xi' Yi'] = [Xi Yi] * |Sx    0| * | cos(r)  -sin(r)|   = [Xi Yi] * Sc * Rot

                           |0    Sy|   | sin(r)    cos(r)|     

The result may be threshold after, to satisfy constrains on r, sx and sy.

Rot matrix is orthonormal, and "pinv" is pseudo inverse matrix

Sc*Rot=pinv([Xi Yi])*[Xi' Yi']

use of SVD decomposition:

Sc*Rot=V*S*D    , where V and D are orthonormal matrix:

Here we can assume :  Rot=D  and Sc=V*S, however  Sc is diagonal matrix,

Then, is there any approximation that may estimate Sc (diagonal matrix) to minimize error between [Xi' Yi'] and transformed [Xi Yi].

Thanks :)