Look at the following discussions:

• https://www.researchgate.net/post/Interfacing_AI_ML_models_with_NetSim
• https://www.researchgate.net/post/How_can_I_effectively_simulate_closed-loop_control_of_5G_Radio_Access_Network_RAN_functions
• https://www.researchgate.net/post/Applying_Machine_learning_in_5G_simulations

1. Suggestions for projects involving DRL in NetSim:

• Scheduling/Resource Allocation
• Traffic Prediction and Management
• Network Slicing
• Handover Optimization
• Interference Management
• Network Attacks and Countermeasures

2. A brief set of steps to integrate RL Algorithms with NetSim:

• Identifying the states, actions, and reward functions.
• Identify the integration points.
• Writing the Deep RL algorithm using TensorFlow Keras or PyTorch.
• Using the built-in NetSim-Python interface to pass states and rewards from NetSim, and to obtain the actions from the Python RL program.
• If the Python Deep RL algorithm is too slow, then consider rewriting it natively in C and linking it to NetSim.

Thank you for your detailed responses, Abhishek Tayal Sai R N

I am considering scheduling strategies, for a scenario where I wish to optimize network performance across two different types of service nodes. For example, the goal could be to maximize throughput for eMBB nodes while simultaneously ensuring that the URLLC nodes adhere to delay bounds. How could I go about this?

The constrained optimization problem could be something like this:

Objective: Maximize the total aggregated throughput across all eMBB nodes and across all slots

Constraints:

You would need to implement the solution to this problem in the scheduler in NetSim.

This looks like a delay constrained throughput maximization problem. The delay constraints make the problem a complex Markov Decision Problem (MDP) involving an average cost. A possible solution is to use algorithms based on Lyapunov drift and Lyapunov optimization, in conjunction with Q-learning.

It would be easier to start with a problem with a scheduling or interference management problem where the MDP has pure rewards and no cost terms, i.e., the value iteration step to update values of state-action pairs only involves a reward term. The loop would be: obtain the states from NetSim, pass the action based on the output of your Python Q-learning program, and then obtain the reward and the next state from NetSim.

Also see

• https://www.researchgate.net/post/How_can_connect_a_deep_neural_network_to_NetSim_5G_simulator_and_train_it
• https://www.researchgate.net/post/How_can_I_apply_Reinforcement_learning_for_5G

Sai R N Ashish Bagwari Abhishek Tayal Ananya Hegde

Thank you for your responses

I have been able to implement RL within NetSim to optimize throughput (of eMBB UEs) under delay constraints (for URLLC UEs). I am currently trying with a simple case of 2 eMBB UEs and 1 delay sensitive UE.

I am able to get the average throughput for each eMBB UE and average queue length for the delay sensitive UE. I have used two scale reinforcement learning with stochastic approximation to do this.

Could someone recommend a couple of effective non-RL queuing policies for benchmarking my algorithm?

Paarth Dwivedi

This is a basic policy you can try for benchmarking.

Define the Lyapunov function

L(t) = 1/2 (Q(t) - T)^2

where Q(t) is the current queue length of the URLLC node and T is your queue length (delay) target

Then using the drift plus penalty you get

Min (Q(t) - T)(Q(t) + A(t) - R_urllc(t)) - V(a1*R_embb1 + a2*R_embb2)

At each time slot t, given the rates of the nodes based on their channel condition, and the arrival rate A(t) to the URLLC node, schedule the UE which minimizes the above function. V is a control parameter trading off between queue stability and throughput.