I know in 2-D, there are five degree of freedom for a grain boundary, therefore there should be a 5by5 rotation matrix for it. Determining this 5*5 matrix is my target.
5 parameters you need to describe a flat grain boundary in 3D, not in 2D.
A. Khorashadizadeh, D. Raabe, S. Zaefferer, G.S. Rohrer, A.D. Rollett, M. Winning, "Five-parameter grain boundary analysis by 3D EBSD of an ultra fine grained CuZr alloy processed by equal channel angular pressing," Advanced Engineering Materials, 13 (2011) 237-244.
There are several papers from Greg Rohrers group from Carnegie Mellon University, Pittsburgh describing this. As far as I know he even has free software which can be downloaded.
The rotation matrix is always a 3x3 matrix. In 3D space in the general case you require at least 8 parameters to describe comprehensively a GB, 3 Euler angles (that can be transformed in a rotation matrix), one vector (3 components) to describe the lattice translation of one grain with respect to the other and one unit vector (3 components: 2 independent and 1 dependent of the other two) to describe the normal vector of the grain boundary plane. By the way, in 2D you require 4 parameters: one angle for misorientation, 1 vector (2 components) for translation and one vector (2 component: only one independent) for the grain boundary normal.
I already realized it is always a 3*3 rotation matrix, now I am dealing with this issue that how can I get that rotation matrix for a hcp crystal system either for 3D or 2D based on those 8 or 5 parameters respectively…I mean how these parameters would be related to that actual rotation matrix for hcp crystal system...