I suppose you already know where to check the matrix data up in HFSS solution data.
Note also in HFSS project tree, mode 1 is always along y axis while mode 2 is always along x axis even though you rotate the calculation box, meaning no need to have boundaries align along x-y axis.
Rxx is the reflexion coefficient of the same mode (co pol).
Rxy is the reflexion coefficient from one mode to the other one ( cross pol ).
Txx is the transmission coefficient of the same mode ( co pol ).
Txy is the transmission coefficient from one mode to the other one ( cross pol ) .
Assuming the incident field is polarized along x-axis: