If i am designing a RF switch, and simulations show -64dB reflection coefficient at the output port (S(5,5))= -64dB, is it possible to have such a value ? or should it be around -10dB ?
It is possible but don't rely on getting it deliberately, unless you tune it when it is in place in the circuit, and even then you will probably lose it when the temperature changes. 20 dB is possible with care and 30 dB perhaps if it really matters. Good coaxial connectors are only specified to be 26dB or 30 dB for example, so something else will be worse than 30 dB and effort put into achieving 60 dB will be wasted, unless it really is an exceptional system it is going into.
You can get via the simulation tools values which looks very satisfactory but the problem of realization and the dimensions is realizable or not??? . Usually the life cycle between the CAD and realization should completed to verify the design targets.
The simulator is saying the voltage reflection coefficient is almost zero (6.31e-4). Therefore the ReturnLoss is -64dB, Like Malcom stated, you wont actually measure anything to that level of precision in real life. A really good 50 Ohm load may be around -20dB whereas -40dB is common for a precision cal kit. In summary, it just means you have a perfect simulated match. If the Return Loss is -64dB over a large frequency band, its pretty much 50+j0 Ohms.