My cluster has > 3500 cores, uses linux and infiniband connectivity. What is the most systematic way to determine the best values for NPAR, NCORE, NSIM and KPAR?? I am using the most recent version of VASP. I am not sure how to go about optimizing these 4 tags. In my submission script, I set the number of cores, and then the others of course in the INCAR. My cluster has nodes with 28 cores per node. Here are stats if that helps: https://doit.txstate.edu/rc/leap.html. Thanks in advance.
Mohammad Saeed Bahramy
What is the size of your system in terms of the number of bands, number of k-points etc.? Knowing the size of the occupied bands, you can make a rough estimate for the number of the computing cores, needed. Let's say, for example, your calculation involves 42 valence bands. If you choose 56 cores on a system with 28 cores per node, VASP by default sets NBANDS=56 and thus adds 14 empty bands. This can be considered a good choice if this number of empty bands meets your needs. Any further increase in the number of cores will proportionally lead to an increase of empty bands, adversely affecting the calculation's overall performance. So the first and foremost point is to choose the optimal number of cores for your calculation. The number of cores chosen should be an integer multiple of the cores in each node. In your case, this means you should go for 28, 56, 84, .... To reduce the inter-node communications through the physical Infiniband connections, choose an NPAR that divides cores on each node into integer number of pools. For example, on your system with 28 cores per node, you can experiment with NPAR=2, 4, 7 and 14 and see which one works the best for you. On my machine with 16 cores per node, I've noticed that I get the best performance with NPAR=4. Note that this process could also be done using NCORE=total number of cores/NPRAR (in your case, for example NCORE=28), but that should basically do the same thing as NPAR. I personally prefer playing with NPAR than NCORE. If your calculation is running over many nodes, set LPLANE=.FALSE. to reduce load balancing. Optionally, you may try to parallelise the k-points as well using KPAR. Just make sure to choose a number that is a factor of your total number of cores.
To get the best performance from a VASP calculation, you need to consider a few things.
6 votes
6 thanks
- Badges
- Science topic
- Similar topics
- Phytochemistry
- Extracts
More Eric Welch's questions See All
Similar questions and discussions