As the power in waveguide, splits in different EM modes, is it possible to measure them separately?

To measure power transfer of a specific mode, you'll need to suppress the unwanted modes using techniques such as making cuts and slits vertical to the current direction of those unwanted modes. Another method of suppression is to launch the waveguide in ways that encourage only the mode you are measuring. So, the first step is to know the wall current directions of various modes, and fields configuration at the launch area.

A system was built at SLAC National Accelerator Laboratory to do such a measurement. It is described in :

"Evaluation of the TE12 mode in circular waveguide for low-loss, high-power rf transmission",PHYSICAL REVIEW SPECIAL TOPICS - ACCELERATORS AND BEAMS, VOLUME 3, 082001 (2000)

We were faced with a very similar question many years ago (according the propagation of beam-excited field power in particle accelerator structures) and developed a measurement scheme for determining the multimodal S-matrices of single devices in the beam line. There is a brief explanation here:

H.-W. Glock; U. van Rienen. An Iterative Algorithm to Evaluate Multimodal S-Parameter Measurements. IEEE Transactions on Magnetics, Vol. 36, (2000): pp. 1841-1845

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=877803&tag=1

(if you have access to this journal)

and a little bit of background reading is e.g. here:

http://linac96.web.cern.ch/Linac96/Compendium/PAPERS/THURSDAY/THP28.PDF

and some more details you may find here:

http://publikationen.ub.uni-frankfurt.de/files/3323/Report-1.pdf

Admittedly this may not exactly fit your situation. Please let us know, whether you have possibilities to insert a dedicated coupling device in your setup (as we had).

If yes, you may calibrate it either numerically in order to get its coupling properties against both polarizations of TE11, TM01 and the two polarizations of TE21 (you mentioned three modes). Or you could apply some experimental procedure as we did (with a lot of effort and difficulties to distinct between polarizations).

In either case you need to have either several coupling antennas with coaxial connectors to (sequentially) connect a power detector (or better a spectrum analyzer or even better a network analyzer, if you are able to drive your system), or you have to modify the coupling of a single port in a defined manner (we used the movement of the coupling device in the waveguide).

Modifying your device by slots is well suited to suppress unwanted mode types, but it introduces reflections which change your results significantly. It is not the same system after sloting the waveguide.

Furthermore: Are we dealing with transients (bad, since broad banded) or steady-state oscillating fields, may be only at a single frequency?

You opened a generally difficult question; maybe you could provide some details?

Regards.

I would take into account that different modes have different cutoff frequencies, hence, they propagate at different angles. About 30 years ago Manfred Thumm, when he was still working at Stuttgart University, developed a so-called k^z-spectrometer, which allowed him to accurately measure the mode content of millimeter-wave radiation propagating through his circular waveguide.