Instead of using MATLAB, you can use Network Simulator 2 which is an open source software. It has almost all kind of wired and wireless routing protocols available. You have to learn OTcl language in order to perform simulation using existing routing modules available. In case, If you want to develop your own routing protocol, you have to learn C++ then. It is free and you need a Linux machine to install it.

I used Network Simulator 2 to compare some Ad hoc routing protocols.  You can install it either on linux or on Windows by using Cygwin.

This a very open question, and in an area that was a hot trend a few years ago. The first thing you need to do is identify your research goal. Read a few survey papers (I  have added a few links, but you should do your own searches in Google Scholar after reading those)  and if you aim to be your phD topic you need to identify a new area or context i which you are applying these protocols, i.e. a sensor network with constrains, a limited budget, or a reliability issue etc.  Sensor networks for autonomous cars could be a good topic. 

Pay attention when you read the papers at how they evaluate their routing proposals and what network configurations they use for testing.  NS2 was the most common choice for simulation when I looked at this topic a few years ago. 

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1020231&tag=1

http://epubs.surrey.ac.uk/2124/1/SRF002122.pdf

http://dl.acm.org/citation.cfm?id=778450

Routing algorithms for Ad-Hoc system like OLSR, DSR, DSDV, DSR are available at Network Simulator-2. You may use this Open Source software for simulation.

Thank you Karamdeep S. Parmar, Jihen Bokri, Jusak Jusak.............and a special thanks to Cruz Izu for her publications.................

As I know for NS2 I have to learn OTcl language, but I have very limited time to learn and explore new software or language, so if anyone can guide me to simulate these protocols using MATLAB then I will be very grateful to him/her.

Apart from that if anyone of you have used OMNET++ then also your comments or suggestions are welcome.

Suresh Network simulator is the best place where you can compare MANET protocol, It is open source, easier to understand and install too. You can subscribe to their forum to get help from the other people working in ns, [email protected] ;

Thank you Ranjana Pathak.......

You can compare using the NS-2 simulator. And you can use the common measures following : the delay, throughput, control overhead and packet delivery ratio.

There have been numerous simulations involving major MANET protocols. Event driven simulations are mostly from NS2. Unfortunately NS2 carries some fake asumptions that make difficult the comparison with real life situations. Eg about mobility: the classic random waypoint with no speed lower bound asymptotically leads to a frozen network (thus favoring reactive protocols). I think there have been simulations of OLSR in scenario tested in real life and in good accordance with them. About MATLAB or other formal math sofware, they are used to test specific features in very dense situations untestable with event driven, eg the distance stretch of AODV or the  multipoint relay overhead gain in OLSR broadcast.

Many AdHoc wireless routing protocols are popular and hot topic for research (WRP, CGSR, AODV, LAR, ZRP etc.). It is compared on the basis of end to end delay, throughput, reliability, new path set up time  and many more.

As many people suggested, I also recommend NS2. It is the best one compare to MATLAB specially for this domain.

Thank you Mohamed Amine Ferrag, Philippe Jacquet and Arjav Bavarva...................

Download source code of OLSR protocol in NS2, all right reserved by Wireless Sensor Networks Laboratory of Iran. Zip and PDF

http://wsnlab.org/downloads/