How to find reflection and transmission coefficient for TEM waves for different frequencies ,What should be the port settings and boundary conditions ?
I'm sorry, but your question is much too general. I recommend to read the book "Time-Harmonic Electromagnetic Fields" from Roger F. Harrington (Wiley Interscience and IEEE Press) first, especially the second chapter "Introduction to Waves".
It is very general and and even electromagnetic starts from this topic onwards. Refer any book, like "Jordan and Balmain" for your reference and there you can find different relations and equations for transmission line for TEM mode. Use any HFSS vedio from an internet, for your design help.