For end but coupling, the coupling efficiency to each mode in a multimode waveguide can be obtained using power overlap integrals. 

After you have the coupling efficiency to each mode, you can also calculate the proportion between different modes.

During propagation this proportion may change, because of power exchanges between modes as they propagate along same path.

Maybe this examples help : http://www.ee.washington.edu/people/faculty/lin_lih/EE539Sp08/Waveguide%20coupling.pdf (Lih Y. Lin)

All modes of a perfect waveguide are orthogonals to each other. This means that you don't have any power exchange during the propagation if the waveguide is perfect.

With the power overlap integrals, we can calculated the power fraction of an input beam coupled for each modes of a multimode waveguide. Modification of the beam shape during the propagation is due to the relative phase shift induce by the fact that the modes have not the same effective index, but we don't have any coupling between LP01 and LP11 because the modes are orthogonals and overlap integral is equal to 0.

Yes, that's what I say in my first paragraph, the coupling efficiency into to each mode of a multimode waveguide can be obtained using power overlap integrals. So it depends on what source is coupled at the input of said waveguide - maybe his source also is multimode. The software "BPM-CAD" used to comprise a Mode Analyser module, allowing to calculate this, with the option to scan lateral and height offsets.

Regarding the propagation issue, I am assuming that Jian's question is about a real optical fibre, which isn't perfect and where energy transfers between modes may occur.

I really appreciate  your help. Different refractive indices corresponding to different mode field situation when I use COMSOL to analysis the mode field of SHC-PCF. In the simulation, the source is a single wavelengh, and the mode field situation of SHC-PCF is very complex. Maybe we can calculated the power fraction of an input beam coupled for each modes of a multimode waveguide with the power overlap integrals. But in the experiment, the soure is multimode.  Compared with COMSOL, "BPM-CAD" is a better choice for optic waveguide, and I will try it. 

Thanks again for the PPT, which is a very good information for me.

I really appreciate  your help. Different refractive indices corresponding to different mode field situation when I use COMSOL to analysis the mode field of SHC-PCF. In the simulation, the source is a single wavelengh, and the mode field situation of SHC-PCF is very complex. Maybe we can calculated the power fraction of an input beam coupled for each modes of a multimode waveguide with the power overlap integrals. But in the experiment, the soure is multimode.  Compared with COMSOL, "BPM-CAD" is a better choice for optic waveguide, and I will try it. 

Thanks again for the PPT, which is a very good information for me.

I really appreciate  your help. Different refractive indices corresponding to different mode field situation when I use COMSOL to analysis the mode field of SHC-PCF. In the simulation, the source is a single wavelengh, and the mode field situation of SHC-PCF is very complex. Maybe we can calculated the power fraction of an input beam coupled for each modes of a multimode waveguide with the power overlap integrals. But in the experiment, the soure is multimode.  Compared with COMSOL, "BPM-CAD" is a better choice for optic waveguide, and I will try it. 

Thanks again for the PPT, which is a very good information for me.

these links might also be helpfull for getting started

https://en.m.wikipedia.org/wiki/Mode_coupling

https://www.rp-photonics.com/mode_coupling.html

In a guide to guide coupling device exange of power between the different modes of the individual waveguides is described by the coupled mode theory that is an approximation to the first order. A rigourous way to model exchange of power is to consider the even and odd modes of the twin waveguides. The power of the input waveguide is coupled equally to the even and odd modes of the twin waveguides (the sum of the 2 modes gives power only in the first waveguide of the twin configuration because amplitude of the even and odd modes are opposite in the second waveguide). The power during the propagation does not change. As  the 2 modes have not the same propagation constant, a phase shift occurs during the propagation. If the phase shift is equal to pi, sommation of the profil of the even and odd modes gives only power in the second waveguide because amplitudes of the 2 modes are in opposite way in the first waveguide. In this case overlapping with the mode of the single waveguide at the output. In this case, power exchange occurs because at the input and the output, you don't have the same waveguides. For an invariant waveguiding structure, we don't have any exchange of power between the different modes. As the effective indexes are not the same, the phase shift induces a change of  the total field profil during the propagation that can gives power exchange at the output of the device because the overlapping is made with an other structure that does not have the same modes. When we perform calculation with the mode matching method, we calculated transfert matrix with the overlapping of the modes of 2 different waveguides at an interface and a propagation matrix that is only the phase shift induced by the propagation. Multiplication of the matrix guives propagation of power in the complete structure.

Regards,