As is shown in attachment, there is hybrid plasmonic waveguide (HPW) and dielectric waveguide (DW), I want know how to calculate mode overlap ratio to evaluated the mode distribution similarity of two polarizations (TE and TM) in the HPW and DW
This looks fairly complicated. I think you need to start simpler, and investigate how you can calculate overlap in general of two functions along a line. The number one mistake I see people do when it comes to simulations is to start out to complicated and then keep revolving around this particular setup. Use a first principles approach, start simple, and get some successes along the way.