Hello,
After reading the infamous publication "More Bang for your Bucks", I developed a workstation of my own using Xeon CPU (>3GHz, E5 series) along with one RTX and one GTX GPUs. In case of single runs, for approx 60k atom systems, I am getting 140-150ns/day.
Problem starts when I'm trying to run two simulations in parallel without overscribing (16 threads). I am even not going beyond 8 threads.
For single run, PME/PP ratio is around 1.04-1.05 and load imbalance is around 2-3% with DLB on. Fourier spacing kept at 0.10 and cut offs at 1.0 nm.
Is there any specific reason for this? Is there any way to solve this?
Because the GPU consumes a lot of overhead time in "context switching" and for "unloading and loading" data from the two concurrent simulations.
The best solution is to refrain from running two simulations using the same GPU.
Either each simulation is attached to a separate GPU (or more), or divide the time between them in chunk basis (e.g. day 1 for simulation 1, day 2 for simulation 2, and repeat).
1. There will be loss of computational power if you use two different types of cards. (I guess its clearly mentioned in the paper you mentioned.)
2. try to run the simulation in one card instead of mixing it. OPtimise the load for both cards for your CPU>
Souparno Adhikary Optimal performance typically requires pinning mdrun lines to specific CPU cores and GPUs. You can think of it a sort-of "divide and conquer" approach. I'd recommend running your simulations in parallel on separate GPUs because crosstalk between them is likely slow.
I'm guessing here but I think you could run at least 4 simulations (2 per GPU) with that setup before you start losing performance. Here's an example part of an old script I use to run 4 simulations on a Broadwell node with 32 cores (0-31) and 2 GPUs (0,1).
##########################################################
gmx mdrun -deffnm production -ntmpi 2 -ntomp 4 -gputasks 00 -pin on -pinoffset 0 -nb gpu -nstlist 120 -pme gpu -npme 1 &
gmx mdrun -deffnm production -ntmpi 2 -ntomp 4 -gputasks 00 -pin on -pinoffset 8 -nb gpu -nstlist 120 -pme gpu -npme 1 &
gmx mdrun -deffnm production -ntmpi 2 -ntomp 4 -gputasks 11 -pin on -pinoffset 16 -nb gpu -nstlist 120 -pme gpu -npme 1 &
gmx mdrun -deffnm production -ntmpi 2 -ntomp 4 -gputasks 11 -pin on -pinoffset 24 -nb gpu -nstlist 120 -pme gpu -npme 1 &
wait
#########################################################
I think the standard flags/options here are: "-pin on", -pme gpu, -nb gpu, "-npme 1", and a -nstlist at/over 100. I'd start by benchmarking short runs with different combinations of the -ntmpi, -ntomp, -pinoffset, and -gputasks flags.
JD Tamucci , This approach is not working... Performance comes down to 40-50ns/day and I'm not oversubscribing.
Nikhil Maroli , I usually put the PME calculations in the slower card and the faster ones in the RTX so that the PME/F ratio remains around 1.00.
If I put the whole simulation in the RTX card, card usage increases to >90% but no significant increase in speed.
I just want to know now, how to run two different simulations in parallel.
I'm sorry to hear you're having trouble optimizing your performance. I can't directly help without trying your setup myself. But, as I understand it, you have 8 cores (16 threads) on the Xeon E5 and 2 GPUs, correct? So, if I were in your situation, I'd use GROMACS 2019 and try something like this.
##########################################################
#run on first GPU and first two cores, 0 & 1
gmx mdrun -deffnm production -ntmpi 1 -ntomp 2 -gputasks 00 -pin on -pinoffset 0 -nb gpu -bonded gpu -nstlist 120 -pme gpu &
# run on first GPU, and second two cores, 2 & 3
gmx mdrun -deffnm production -ntmpi 1 -ntomp 2 -gputasks 00 -pin on -pinoffset 2 -nb gpu -bonded gpu -nstlist 120 -pme gpu &
#run on second GPU, and third two cores, 4 & 5
gmx mdrun -deffnm production -ntmpi 1 -ntomp 2 -gputasks 11 -pin on -pinoffset 4 -nb gpu -bonded gpu -nstlist 120 -pme gpu &
#run on second GPU, and fourth two cores, 6 & 7
gmx mdrun -deffnm production -ntmpi 1 -ntomp 2 -gputasks 11 -pin on -pinoffset 6 -nb gpu -bonded gpu -nstlist 120 -pme gpu &
wait
##########################################################
Optimal performance takes rigorous benchmarking of all the logical options. I just write a script which tests them all by running them for 15000-ish steps each and compare the output.
- Badges
- Science topic
- Similar topics
- Biological Science
- Kinesiology
- Running