The (waveguide) port solves a 2D problem prior to the 3D. In this sense, the port mode solver finds out the modes that propagate into the structure first, by seeing the port dimensions and its attached boundary conditions. Later the same mode pattern is applied to the complete structure, when the 3D evaluation takes place.
If your port size is too large the solver may find an incorrect mode (eg. a TE instead of a TEM). If its too small the area is not large enough to sample the fields and compute their ratio E/H (which results in the impedance mode), providing then an incorrect answer. The best practice is to (1) know previously what kind of mode will propagate and compare it to the virtual result and (2) compare the computed impedance to some analytical formulation.