Both Mellin transform and MGFs are useful tools for analyzing the performance of communication systems. However, the choice depends mostly on the fact that which of these (or some other) tools result in the easier tractable analysis (or solvable integrals, most of the time). The MGF methods are more popular for analysing the error performance (this is one of the examples) over wireless channels, Mellin transform is often used for analyzing the delay-constrained performance. For example, in the following articles, the Mellin transform is used for delay-limited performance analysis:
Article Network-Layer Performance Analysis of Multihop Fading Channels
Conference Paper Delay Analysis for Wireless Fading Channels with Finite Bloc...
Preprint Delay Violation Probability and Age of Information Interplay...
Conference Paper QoS provisioning in large wireless networks
Article Delay Performance of Wireless Communications With Imperfect ...
Article Delay Performance of the Multiuser MISO Downlink under Imper...
Article Delay Violation Probability and Effective Rate of Downlink N...