The matlab is suitable a mathematical environment that can be used to model any system as it is a mathematical solver. You can find many tool boxes under matlab such digital signal processing tool box, filter design too box, communication systems tool box and so on and many more. The opnet and ns3 are software suits to model and analyse the communication networks. In the sense they contain the physical layer, the data link layer , the network layer , the transport layer, the session layer,the presentation and application layers. Many network protocols are modeled in these network analyzers software.
MATLAB is best for simulation of Wireless Communication Systems. It has a wide variety of toolboxes that i quite helpful for design and performance analyses of individual components as well as exhaustive and unified communication system. NS3 and OPNET are suitable for Wireless Networks for carrying out simulations of adhoc routing protocols etc. You can effectively simulate any Wireless Communication System in MATLAB and see its performance using stanardard terms that is to say BER, Eye diagram, PDF, CDF, Power Spectral Density, Constellation Diagram etc. A novice can start with basic MATLAB tutorials and practice MATLAB guides of famous toolboxes pertinent to Comms like Digital Signal Processing Toolbox, Communication System Toolbox, Signal Processing Toolbox etc. NS# and OPNET are useful for simulation at data link, network and transport layers.
Modeling, simulation, and analysis of wireless communication systems. For Modeling and analysis of the TCP/OSI Layer Protocol I will advise NS3 or OPNET. But for Simulation I consider MATLAB the most appropriate.
I was very fond of SPW by Synopsys, but looks like they dont make it any more. It was a complex-envelope simulation tool, excellent for modeling non-linearities, such as amplifiers and TWTAs. I miss it very much. After using Simulink to try to replace it, I find Simulink not that great. Matlab, on the the hand is a great deal of work and and hard to manage. So I have not found anything I really like.
After trying various tools, we now work with NetSim in our university. It has the most extensive wireless coverage - 5G, IoT, MANET, VANETs, LTE, WiFi, etc. Their tech support team is responsive and knowledgeable.
NetSim (https://tetcos.com/) has a lot of features and is much easier to handle, at least when compared to open source tools. Also, our students can download and run it from home.
OPNET isn't any longer under active development. I would suggest NetSim or NS3. NetSim, while not free is discounted for EDU, and has a nice user interface (read documentation at: https://tetcos.com/netsim-documentation.html). NS3 is free but there is no GUI and the user should be familiar with programming.