I know in 2-D, there are five degree of freedom for a grain boundary, therefore there should be a 5by5 rotation matrix for it. Determining this 5*5 matrix is my target.
5 parameters you need to describe a flat grain boundary in 3D, not in 2D.
A. Khorashadizadeh, D. Raabe, S. Zaefferer, G.S. Rohrer, A.D. Rollett, M. Winning, "Five-parameter grain boundary analysis by 3D EBSD of an ultra fine grained CuZr alloy processed by equal channel angular pressing," Advanced Engineering Materials, 13 (2011) 237-244.
There are several papers from Greg Rohrers group from Carnegie Mellon University, Pittsburgh describing this. As far as I know he even has free software which can be downloaded.
The rotation matrix is always a 3x3 matrix. In 3D space in the general case you require at least 8 parameters to describe comprehensively a GB, 3 Euler angles (that can be transformed in a rotation matrix), one vector (3 components) to describe the lattice translation of one grain with respect to the other and one unit vector (3 components: 2 independent and 1 dependent of the other two) to describe the normal vector of the grain boundary plane. By the way, in 2D you require 4 parameters: one angle for misorientation, 1 vector (2 components) for translation and one vector (2 component: only one independent) for the grain boundary normal.
I already realized it is always a 3*3 rotation matrix, now I am dealing with this issue that how can I get that rotation matrix for a hcp crystal system either for 3D or 2D based on those 8 or 5 parameters respectively…I mean how these parameters would be related to that actual rotation matrix for hcp crystal system...
If you have two crystals, each with a given orientation in rotation matrix representation R1 and R2. The misorientation M or rotation matrix for the grain boundary results from: M=R2 R1^(-1). The other parameters such as translation vector and grain boundary plane do not affect the rotation matrix. This matrix is solely defined by the orientations of the adjacent grains to the GB. In the literature, there are several methods to calculate the misorientation, I suggest to utilize quaternions. You can refer to the work by Grimmer. Hexagonal can be a little tricky but in reciprocal space the calculation is straightforward.