You need to differ between two schemes of parallelisation, which is the underlying idea of supercomputing and also GPU computing. Parallelisation in principle describes the use of multiple computing units on only one problem to achieve an acceleration. This can be the cores of a processor, multiple processors, multiple computers with a fast network or even GPUs.

Away from that, when you do such a parallelisation you can either do:

1. Shared Memory

Computing units share the same main memory. (Single node, e.g. OpenMP; often states as SMP or Threading). Here usually few processes are launched that use multiple cores and show a CPU usage > 100%. Btw. a GPU is such a SMP system.

2. Distributed Memory

Computing units don't share the same Memory, but are connected with a network. (Multiple nodes (computers), e.g. MPI). Here many processes are started with a CPU usage usually ~100%. Btw, multiple GPUs are such a system, the network here is then the PCIE bus,

Away from GPUs Gromacs is capable of both schemes in the 1st place and you usually combine them for best performance. This is because SMP cannot use multiple nodes and MPI is inefficient with many communication endpoints. So when using Gromacs on multiple nodes, you need at least one process per node, and COULD use the rest of the cores with SMP (-ntomp). BUT, as gromacs pointed out having 20 OpenMP workers is not very fast usually. ADDITIONALLY you would need at least one process per GPU, since one process cannot use multiple GPUs in gromacs. Therefore you are basically supposed to launch 2 processes per node with each having 10 threads.

How you specifically do this in your environment is up to u. Usually the MPI implementation is aware of SLURM's (the queueing system) allocation, so specifiying this information at submit time (sbatch) is enough.

How I would do it:

Jobscript snippet:

[...]

mpirun gmx_mpi mdrun -ntomp $SLURM_CPUS_PER_TASK

[...]

At submit time:

sbatch -N 2 --ntasks-per-node=2 -c 10 -p partition ./script.sh

Maybe telling the same to srun works aswell ,altough it seems a little bit dirty to me, using srun at all.

srun -N 2 --ntasks-per-node=2 /applic/applications/gromacs/2016.4/bin/gmx_mpi mdrun -v -deffnm step7_production -pin on -nb gpu -ntomp 10

Let us know ;)

Dear Norman, Many thanks for your explanation.

I did changes according to your suggestion, but it failed with an error of Invalid numeric value "tasks-per-node=2" for number of tasks.

I did remove #SBATCH --ntasks=20 line from the script file and run the simulation. Is this the reason why it shows the error.

Looking forward to hear your expert opinion.

I think you should still keep the #SBATCH --ntasks=20 but also the parameter for srun is --ntasks-per-node (note the two - in front).

Hello Norman,

Greetings of the day!

I set up the script file according to your suggestion and submitted the job, but it fails again.

Note: Your choice of number of MPI ranks and amount of resources results in using 10 OpenMP threads per rank, which is most likely inefficient. The optimum is usually between 2 and 6 threads per rank. Please let me know how to solve this problem. I have attached to script file and error file.

Looking forward to hearing from you.

Well, break down the Threading by incresing the number of Processes then. So e.g. running 4 processes per node, with each having 5 Threads.

You just need to follow all the boundary conditions:

1. Gromacs want you to have between 2 and 6 Threads not more.

2. Your machines have 20 Cores and 2 GPUs

3. You need at least one Process per GPU, but total number of Processes should be a multiplier of GPUs if higher.

This results in only few reasonable setups for gromacs e.g.

--ntasks-per-node=4 and -ntomp 5 (uses all 20 cores)

--ntasks-per-node=6 and -ntomp 3 (looses 2 cores)

--ntasks-per-node=8 and -ntomp 2 (looses 4 cores)

--ntasks-per-node=10 and -ntomp 2 (uses all 20 cores)

--ntasks-per-node=20 and -ntomp 1 (uses all 20 cores)

You could benchmark which of these is the fastest, the difference might be huge.

Dear Norman,

Thanks for your help.

I am succeeded running MD simulation on GPU.

I used --ntasks-per-node=4 and -ntomp 5

it run but have higher load imb: force 9.4%

how to solve this one now.

Regards,

Changdev

If anyone is struggling with getting GPU acceleration for GROMACS, there are pre-built workstations & servers available.

https://www.exxactcorp.com/GROMACS-Certified-GPU-Systems