Depends on your intended application, waveguide material and a lot of other factors. For some applications the shape of the waveguide cross-section does not matter, as long as sizes and materials are optimised.

Here are well written articles about waveguides:

http://en.wikipedia.org/wiki/Waveguide_(electromagnetism)

http://en.wikipedia.org/wiki/Waveguide_(optics)

Since fields of propagating waveguide modes in the fiber are described by Bessel functions, so circular symmetry is better to provide a lower propagation loss. The corners of the rectangular waveguide will cause the transition of the part of propagating power into higher-order modes which are quickly disappear. However, the hollow-inside rectangular waveguides are typically used for transmission of high-power optical radiation.

What spectral range do you have in mind? Optical, microwave, etc? For example, optical waveguides that are used in integrated optical components (sometimes called "photonic integrated circuits") are mostly based on a rectangular geometry (due to the ease of fabrication and processing steps - etching, depositing, etc.). You can also find rectangular waveguides used at microwave frequencies. So, various shapes can be used in principle... 

 Optical. Yes, optical waveguides in integrated optics (photonic) could be rectangular or planar based on connivance for manufacturing. But there are small dimensions for wave’s propagation in integrated optics, so additional losses are neglected in comparison with fiber optics. The comers produce changes of propagated power into high-order modes for any waveguide in optics, microwave and etc., so waves like circular symmetry because Bessel and exponent don’t like cornes.  

Before giving an answer, you have to specify at least the frequency of your application...

Прямоугольный волновод широко используется в реальной технике. Это связано с тем, что его можно изгибать. Если в волноводе бежит плоская волна, то никаких потерь в углах волновода нет. С моей точки зрения самые интересные одномерно- периодические волноводы. Они могут состоять из отдельных не связных между собой частей. Это волноводы, которые совсем не похожи на привычные волноводы.

Corners and any boundaries produce the wave diffractions which don’t correspond to waveguide profile, it tends to losses. The fiber optical able can be bended as the planar waveguide.

There is Sergey's translation: The rectangular waveguide is widely used in real technique. This is due to the fact that it can be bent. If the waveguide is running flat wave, then there are no losses in the corners of the waveguide not. From my point of view the most interesting one-dimensional - periodic waveguides. They may consist of individual communication between the parts. It waveguides, which are not similar to the usual waveguides.

Tamara V. Tulaikova is wrong. There is no inherent loss just because the waveguide cross section is not circular. It is just we can not solve the Maxwell's equation in a closed form solution other than circular cross section. But nature does not care about our inability to solve this mathematically and light can still propagate without loss in a waveguide with non-circular cross section.

No, I am right. Ken forget about diffraction mechanism for wave losses then wave meets the corners.