How are you feeding it and how are you terminating it?  Are there any discontinuities along the waveguide?

If you have a continuous waveguide, properly sourced and properly terminated, your S11 seems too good to be true, but not unreasonable.  It should be low because there is nothing to produce reflections.

The small and fast ripples on the measurement typically indicate a reflection on a cable -or in your case, wave guide, though I have no experience with that. These (suspected) reflections eat away at the dynamic range of your measurement. Under these conditions, measuring 120dB return loss seems very spectacular. As for the large variations in the measurement, most likely the material you use has some resonances where it becomes completely transparent. (Obviously I may be completely wrong in all these assumptions)

To get more confidence in your setup, you could measure a known load. Assuming you perform some kind of calibration, you could re-measure the calibration loads again AFTER the calibration. The return loss you get from your useful measurement can normally not be better than the return loss you measure from your calibration load. Good luck with your research!

Dear Raj,

Its very simple to understand. 

S11 defines the matching, if you have perfect impedance match at input, u can achieve the S11 which u think of at least in simulation. 

The more practical scenario would be to look for a a S21 in the range of 0,01 dB or less, where the loss tangent of your dielectric will come into picture. 

The circular waveguide which you have chosen is dominant or overmoded ??

The line "Material = PEC with Vacuum" gives away that these results are from a simulation. If you are getting -120dB S11 (+120dB return loss to be pedantic) I'm guessing you are simulating a simple straight piece of perfectly terminated guide. From basic transmission line and waveguide theory, S11=0 (-infinity dB) for such a case.

The difference between PTFE and vacuum may be due the simulator imperfections - but still so close to a perfect match to be acceptable.

If, on the other hand these are accurate measurements, how are you doing them because they are far better than anything I have seen?

Good observation, this explains a lot. If indeed this is a simulation, the result makes perfect sense. I should have known better than to assume this was an actual measurement. 120dB dynamic range would have been spectacular indeed...

In simulations you can have any match.

Adam rightly said, in simulation you can have any match. But to throw more let, any thing better than 27dB return loss is enough. Simulation gives you data of mathematical calculation and can be of the value. As a matter of fact perfect matching is with zero reflection or infinite return loss. The actual measurement will have its own measuring limit/accuracy while in simulation there is no such limit, it has to simply calculate mathematical figure. VSWR=1.01 and VSWR=1.001 makes hardly any difference in reflected power but in terms of S11 it creates huge difference of 20dB.