I do know the formula for BetaP(degree), but can't understand how do I get S12, S21, and S22 of the structure from HFSS. What exact simulation procedure do I follow?
You have to provide electric and magnetic boundaries to your unit cell along two of the axis and wave propagation vector (waveguide ports) along the third. Check the attached article, it will definitely help you.
Model the structure as shown in the above PDF I.e. PEC on either side, PMC on other two opposite boundaries and wave port on remaining two opposite boundaries.
Note that the boundaries with wave port is perpendicular to the metallic strip placed at the bottom of the metamaterial structure.
To get the s parameters, after analysing/simulation,
Go to HFSS- results-create modal solution data report-rectangular plot
In electromagnetics, the permittivity, permeability, and conductivity define the material’s characteristics. The extraction of these values for different frequencies defi nes the propagation profi le of the material at that frequency. In the extraction method used in this work, the refractive index and impedance of the material are used to extract the permittivity and permeability of the material under test. To extract these material parameters ( ε and µ ), consider the unit element of a metamaterial with lattice vectors in all three dimensions. Appropriate boundary conditions and excitations are assigned to the different surface of the three-dimensional unit element to simulate the periodic metamaterial and excitation of this metamaterial to extract the S parameters.
There are certain formulas to calculate S parameters manually. Refers this paper attached with.