One recommendation is to use MatLab/Simulink. Here's a link to an example: https://www.researchgate.net/publication/260735849_Simulation_Platform_for_Quadricopter_Using_MatlabSimulink_and_X-Plane.
Another program you can use is LabVIEW. Here's a link to case study using this simulation software: http://sine.ni.com/cs/app/doc/p/id/cs-12188.
Hope this helps,
Max
Conference Paper Simulation Platform for Quadricopter: Using Matlab/Simulink ...
One widely used communication protocol out there for UAVs is Mavlink. Several autopilots use it, e.g. Pix4 and APM. You can use the simulation frameworks that these autopilots provide to simulate the exchange of mavlink messages, e.g., https://pixhawk.org/users/hil This might or might not be appropriate for your problem. It really depends which communication layer you want to simulate.