Not all matrices are 'rotation' matrices (standard math terminology is unitary matrix). Therefore, it is unclear what 'converting' to a rotation matrix means in the first place. If the matrix is not unitary, all the procedures you outline simply destroy any relationship between the original matrix and the matrix you obtain after manipulation. Symmetric matrices can be diagonalized by a unitary matrix, so in that case you have a single unitary matrix that is associated to the original one, but even there one can hardly talk about 'converting' if that means writing the same thing in a different way.

Addendum to the previous answer: for a generic square matrix one can also consider QR decomposition, so that A=Q*R where Q is unitary and R triangular. If A is an invertible matrix, Q is uniquely identified.

The application that requires conversion is in kinematics. You start by tracking a 3-point rigid cluster in 3D space. That gives you a 3x3 matrix of 3D coordinates. But what you want is to know the orientation difference of one cluster to another -- movement across joint. So, you need to convert these 3x3 matrices of 3D coordinates to an orthonormal rotation matrix.

Sorry, not being familiar with the specific problem I may be saying something stupid, but should not one then consider the difference between two such subsequent 3x3 matrices and find the 'closest' rotation matrix to that? some experimental error will have to be allowed for..and also translations of the 3-point clusters. Maybe one should just try to compute a least square fit between these data (differences of subsequent coordinate matrices) and a rotation matrix.

There is no unique "canonical conversion" of some 3x3 matrix to a rotation matrix exist. You can consider columns of some rotation matrix as the orthonormal basis of space. Than any process of converting some matrix to rotation matrix may be viewed as the process of orthonormalisation of basis. However depending on the algorithm and order of vectors you use, you obtain different results. That is why you have more than one solution to your particular problem.

Try using polar decomposition of a matrix. It may not give you a direct/expected result, but it is a natural approach.

Thanks, but polar decomposition does not produce an orthogonal matrix from the example I have given.

@ Dr Greiner: It is a classical result (see Ky Fan, AJ Hoffman: Some metric inequalities in the space of matrices, 1955) that the closest unitary matrix to a given nonsingular matrix A, with SVD A=USV', is given by its polar decomposition, UV' * VSV' = QP, where Q is unitary and VSV' is symmetric positive definite, i.e., UV' is the best unitary approximation to A, minimizing the Euclidean operator norm ||A-Q||2 over all unitary matrices Q. This "explains" your "removal" of singular values.

This gives you the best approximation in the unitary group, but if you als want a rotation matrix, you want an approximation in a subset of the unitary group. You want a unitary matrix with determinant +1 (i.e. a rotation, as opposed to reflections, which have determinant -1). This subgroup is the special orthogonal group, SO(3).

Now, if you have a real 3x3 matrix, and compute the polar decomposition of A, you will find that UV' either has determinant +1 or -1. Suppose A=QP with det(Q) = -1. Then Q is a reflector. But the polar decomposition of -A = -QP, where P is still symmetric positive definite, and your unitary matrix is -Q, with

det(-Q) = -det(Q) = +1,

so -Q is a rotation.

I am not sure whether this is helpful (i.e., whether this simple trick is permissible in your application), but using UV' or -UV' is in my view the logical first choice, rather than other factorizations.

The matrix obtained from the SVD (or the polar decomposition) is the one closest to the original matrix, in a least-squares sense. What you get by G-S orthogonalization is only an approximation.