A waveguide polarizer has been created in Comsol ,which can make TE pass/TM absorb. But I don't know how to excite TE and TM model in a specific way, as is shown in following picture, I excite TE mode in this way, I'm not sure if it is correct. If I use rectangular port, TM mode will occur a error when I do simulation, anyone can give me some ideas?
Firstly, it seems from the diagram that you are talking about rectangular waveguide (the matter would be quite different in circular waveguide). So presuming rectangular waveguide there is an easy way to ensure a TE mode since the TE10 mode is the dominant mode - it has the lowest cut-off frequency. So apply a signal of frequency above the TE10 mode cut-off frequency but below the cut-off frequency of higher modes. This ensure propagation in only the TE10 mode. As for a TM mode, it is a little more difficult as the TM mode of the lowest cut-off frequency is the TM11 mode. If you use a frequency above the TM11 cut-off then several TE modes will also be supported, which is not what you want. You could introduce a polariser within your waveguide to promote the TM11 mode and suppress other modes. This is usually made from a series of conducting wires in a pattern that meets the boundary conditions (E-field must be perpendicular to the surface of a conductor, H-field must run parallel to the surface of a conductor) of the TM11 mode but not other modes. Looking at the TM11 mode field patterns such a polariser could be constructed from wire loops within the waveguide (in the sectional plane) that are circular in the centre of the guide but become more rectangular towards the edges of the guide, i.e. parallel to the H-field loops. The polarisation purity is probably about 25dB at best and would need to be considerably better than the polarisation purity of the polariser you have designed so that you get a true indication of what your polariser can achieve. Hope this helps.
As it seems to be an optical waveguide, the rectangular port is not the correct choice (it is for metallic microwave waveguides). You have to choose the numerical port and also include the cladding into the port definition, not just the core. Then let the boundary mode analysis find the first few modes and choose the mode you want to excite. For the given waveguide the TE mode should be the one with the highest effective refractive index. For the first TM mode you'll have to look at the mode distribution. The Comsol support is very helpful for that...
First, set the 'Transform' to 'Effective mode index' and the 'Desired number of modes' in the boundary mode analysis to anything higher than 1, as can be seen in the first picture. The value in 'Search for modes around:' should be higher than the maximum effective refractive index. In my first attempts I started with 3.5.
After calculating the boundary mode analysis (only the first one), make 3D Plot group with a surface plot in it. If you select the proper solution, you can choose from the found modes (10 in my example). They are sorted from the lowest to the highest effective refractive index. You can draw the 'Normal boundary mode electric field' (2. picture) or the 'Tangential boundary mode electric field norm' (3. picture) to find out, which is the desired mode. The second and third pictures are for the mode with the highest effective refractive indes, which is the TE mode in this case. The next lower refractive index has it's tangential field perpendicular to that, as shown in the 4. picture. This should be the TM mode.
Of course also higher modes are found, which look like the example in the 5. picture. This field distribution is of course not what we want.
For the simulations set the desired number of modes back to '1' in the boundary mode analysis step and the value for 'Search for modes around' to approximately the effective refractive index you found before.