There are several measures available to reduce back pressure in a muffler system, e.g. the combination of reflextion volums of different size, the application of specific shapes of the tube ends, double-wall sheet metals, as well as the application of damping materials in specific areas of the muffler.
Calculation methods:
- 1D-consideration by use of streamline & volume based simulation according to the Bernoulli equation. Software: e.g. AVL BOOST
- 3D-consideration by use of finite-element based approaches in computational fluid dynamics simulation (CFD). Software: e.g. AVL FIRE, Fluent, Star CC
To reduce back pressure in a muffler, a combination of both, 1D-simulation and 3D-consideration makes sense. 1D simulation to layout the volumes in the muffler chamber and the lengths of the connection tubes. 3D CFD simulation to get detail information about the flow characteristics at inlet- and outlet areas of the connection tubes and the behavior of damping materials.
You can use 1D Transfer Matrix Method (TMM)to compute the Transmission Loss. Matlab is an excellent tool for this. You can read about the TMM method from Acoustics of Ducts and Mufflers by Prof Munjal from IISc Bangalore.
I will be able to help you if you can share your muffler geometry.