Just model the waveguide with waveguide ports at each end. S21 will give exactly what you want. It will be in dB, for the length that you choose to model. It is simple to turn this to the attenuation constant which is in nepers/metre.
Divide the dB by the length, to give dB per metre, then divide by 20log10(e) = 8.886 (approx).
The S11 magnitude in dB gives you alpha, in dB, which you can convert to nepers, as I described above. If you make the guide 1 m long this will be per metre, which is the right units.
You can also plot the phase in degrees, which over one metre gives you degrees per metre. If you multiply this by 2 x pi / 360 this will be radians per metre. This is beta.
If you make the guide 1/8.686 metres long, the S11 dB plot will be in nepers per metre any you won't need to multiply it.
If you make the guide 2 x pi / 360 metres long the phase plot will be in radians per metre and you won't need to multiply it.
Hi Malcolm White - Thanks a lot for sharing that. I was actually asking for the reference from a paper where the method of extracting the attenuation constant from the S-Parameter has been performed.