For background, I am trying to run two independent and separate .conf files each on a different GPU. It is NAMD molecular dynamics simulations I am running.
I login correctly and everything and specific this to request: qrsh -l gpu,A100,cuda=2,h_rt=12:00:00. I am running via remote server to a HPC cluster.
When I go to run my files: CUDA_VISIBLE_DEVICES= 0 ~/bin/namd3 +p1 +setcpuaffinity +devices 0 config.conf > log.log or CUDA_VISIBLE_DEVICES= 1 ~/bin/namd3 +p1 +setcpuaffinity +devices 0 config.conf > log.log. (+p1 was done cause its GPUresident in NAMD. not important to the issue at hand however).
However, this slows down both simulations very much. Normally, it takes 1 hour to run a .conf file on a single A100 GPU card. But when I run both at the same time, even though they are on different A100 GPUs it slows down immensely and say it will take 4.5 hours to finish each one.
I am hoping to find a solution to this because it should finish in an 1 hour total because each sim is one different GPUs and they aren't interfering with one another. This was a GPU usage graph. The CUDA_VISIBLE_DEVICES doesn't fix didn't work. Below is my GPU usage during the two sims running on separate GPUs.
Aarav Singh One possible cause is contention of RAM / VRAM, e.g. with only one GPU loaded, the machine can focus all PCI resources (and RAM) to that GPU's tasks. With two, you now introduce contention. You can test this by using gpu monitoring tools that capture GPU to host transfers.
If the topology (how many GPUs share the same bus) is under your control, you can (if the machine isn't otherwise loaded) try to change the GPU_ID (e.g. 0 & 3 vs 0 & 1) to see if that makes a difference.
If the tasks are independent, but the data is shared, your processes may be limited by IO (writing to disk/caching).
- Badges
- Science topic
- Similar topics
- Biological Science
- Biochemistry
- Enzymology
- Enzymes