You should use Floquet boundary conditions at the axes in which the metamaterial is stratified (or assumed to be infinitely periodic) and use open boundary conditions at where the radiation occurs (and where the material is not assumed to be infinite): e.g. if you are trying to simulate a plane wave along x-axis that is impinging on a metamaterial slab, you may use Floquet boundary conditions for y and z axes, where the metamaterial would be assumed to be infinitely periodic. Nevertheless, in such cases, you should take the finite thickness of the slab into account and use open boundary conditions for x axes to simulate radiation.

If the metamaterial structures are periodic like FSS, then you have to use floquet port. Else you can excite the structure by assigning wave port to two faces of the excitation box along with PerfectE and PerfectH on other sides. Kiran Kumari

I want to use metamaterial like as the image attached.

I am confused about how to apply boundary conditions in CST while designing a unit cell. Is it (unit cell along the x and Y axis and open add space along the z-axis) or is it ( magnetic field (Ht=0) along z-axis, electric field (Et=0) along y-axis and open along x-axis) in this case open boundary is the only option not the floquent mode.