I expect CST is right. Did your different feeds have different impedances? Waveguide ports can have patterns you don't expect - is the field of the waveguide port what you expected it would be? Is the field shape in the device the same when you use a waveguide port as when you use a coaxial port?

Is your coaxial feed properly matched?

You can't just use a coaxial feed somewhere where a waveguide feed worked, you have to make sure the transformer between the two works properly.

I attached my simulation file in my question, as u can see I use a 50-ohm wave port and 50 ohm SMA connector, in fact, I use SMA connector imported of Antana magus and I calculate the impedance of the connector...

if u couldn't open files I attach pictures of my simulation and result and reference impedance of SMA connector ...

so "Is your coaxial feed properly matched? " is YES

I can't open you .rar file, so something else would be helpful, like a .cst file

these my 3 test structure

CST 2019

I have CST 2017 and can't read your files. It will have to be pictures instead, unless you send me a list of your parameters - I have a history list but no parameters, so it fails to build the model.

these files that I sent them in the rare file before

In both of the sma versions you have a poor match to the coaxial adaptor (in the connector edge waveguide) or the coaxial cable (in the connector sma waveguide. Both of them have periodic changes in the s-parameters that show there are two reflections, probably separated by the length of the microstrip, that get worse with increasing frequency. Your junction between coax and microstrip in both cases is not good enough at the higher frequencies. You need to improve it by changing the geometry. Bear in mind that a return loss of -3 dB is caused by a -9dB reflection at each end, and a return loss of -6 dB is caused by a reflection of -12 dB at each end. You need to improve the reflection at each end of the microstrip.

There is no problem with match in the simple waveguide, except that perhaps the input port may not be wide or high enough, so that at low frequencies the match of the mode pattern that will actually go on the line and what the port will support is not good, but the ports have a match of -22 dB each end even at the bottom frequency. The period of the ripple gives you the separation of the reflections that are causing it, work out a half wavelength (on the transmission line) at the frequency difference between nulls, and this is the separation between the two reflections. It is almost certainly the ;length of the microstrip.

Thanks for sharing the problem Hossein Ghani and thanks Malcolm White for such an elaborated answer.