I saw that there are many ways to make VASP calculations faster. I have some questions regarding these methods.
(1)Set ALGO=F. Basically, this uses Blocked Davidson algorithm in the first five steps and uses RMM-DIIS. Question: Is this safe to be used? I read from a tutorial handout that RMM-DIIS can skip some of the eigenstates. I am not sure what that mean though.
(2)Set LREAL=A I heard this will affect accuracy. How will I know if the accuracy effect is not acceptable? Is there some convergence/benchmarking tests like for ENCUT and kpoints to see if its effect on accuracy is unacceptable?
(3) Use NSIM = 4. I haven't read much about this. Any experience on using this option?
(4) Use mixing tags. How do you know how to play around and set the values of IMIX, INIMIX, MAXMIX, AMIX, BMIX, AMIX MAG, BMIX MAG, AMIN, MIXPRE, and WC tags?
Thank you very much,
1) Algo = Fast is used by a lot of people, I used to use it as well but switched back in the end, I believe I saw some inconsistencies in my calculation. If this really is an issue I'm not sure, though standard Davidson is safest.
2) I'd really just leave LREAL on false, but if your systems get too large you can always explicitly test it
3) mixing tags I would not touch unless there really are electronic convergence issues. Then you can switch to linear mixing (see manual) or tinker a bit. Such badly converging cases can also be an indication of a k-mesh which is not dense enough (and can thus actually be slower at lower accuracy)
4) We typically also use LASPH=.TRUE. (cheap and adds some corrections to energy) and PREC=Accurate (does not slow the calculation too much, but should be tested) to keep the calculations at an accurate level
5) In the previous points I've been slowing things down... now to comment on speeding things up. You have two types of parallellization, band and k-point. K-point parallellization (KPAR) has not existed too long in VASP, it is however nearly serial and thus fastest. Band parallellization has been there much longer (NCORE/NPAR) works well but does not scale too far. These settings can depend on the processor so you really need to benchmark a bit. NSIM can help a bit but for us never changed much. The optimal settings of course depends on the kind of computing power you have available (and the size of the system you want to calculate).
If you have a small system with lots of k-points you can probably do 1 k-point per core. For normal/large systems I typically do 1 k-point per node and then within a node I use band parallellization. On our recent cluster the optimal setting is NCORE=16 (for a 16-core node), basically NCORE*NPAR = total cores so if you set this parameter to the number of cores in a node it'll do 1 band per node, minimizing communication. On our older clusters NCORE=1 worked better.
Also enable LPLANE=.TRUE. (or test) this is a tag which controls how much data is exchanged between nodes.
To summarize:
-If you have multiple k-points, use KPAR, set the number to be able to divide your irreducible kpoints (amount in IBZKPT, run vasp and kill it at the start if you don't know how many there are). For instance if you have 20 k-points KPAR can be 1, 2, 4, 5, 10, 20
Reducing the K-Points will not affect much the structural and electronic calculations while optical calculations must be calculated in a well made momentum space mesh.
In general, vasp can be made faster using parallel flags NPAR and NSIM, and the right combination of cores and nodes.
it takes time, efforts and patience to adjust them optimally, but these two parameters affect vasp parallization most.
Check out this webpage for various comparisons, and tips of choosing parallelization parameters.
https://www.nsc.liu.se/~pla/blog/2015/01/12/vasp-how-many-cores/
https://www.nsc.liu.se/~pla/blog/2012/09/26/vaspkpar/
Thanks for the reply. Am I right in thinking that these tags should not affect the values of converged energies? So therefore, I can use different NPAR and NSIM tags and still safely compare the systems/subtract energies? I use different kinds of computers so I might need to set them differently.
Parallelization flags will not, in general, affect total energies, or any other physical quantities calculated therein. Unless of course, there is wrong doing in setting up the cluster.
Best way to check is to have a set of small benchmark calculations, that calculate the relevant physical quantities. e.g. Total Energy, a quick optimization, band structure, etc.
Run these benchmarks over different clusters. They should give same answers within acceptable error limits.
Changing parallelization on same cluster, should definitely not change the answers.
1) Algo = Fast is used by a lot of people, I used to use it as well but switched back in the end, I believe I saw some inconsistencies in my calculation. If this really is an issue I'm not sure, though standard Davidson is safest.
2) I'd really just leave LREAL on false, but if your systems get too large you can always explicitly test it
3) mixing tags I would not touch unless there really are electronic convergence issues. Then you can switch to linear mixing (see manual) or tinker a bit. Such badly converging cases can also be an indication of a k-mesh which is not dense enough (and can thus actually be slower at lower accuracy)
4) We typically also use LASPH=.TRUE. (cheap and adds some corrections to energy) and PREC=Accurate (does not slow the calculation too much, but should be tested) to keep the calculations at an accurate level
5) In the previous points I've been slowing things down... now to comment on speeding things up. You have two types of parallellization, band and k-point. K-point parallellization (KPAR) has not existed too long in VASP, it is however nearly serial and thus fastest. Band parallellization has been there much longer (NCORE/NPAR) works well but does not scale too far. These settings can depend on the processor so you really need to benchmark a bit. NSIM can help a bit but for us never changed much. The optimal settings of course depends on the kind of computing power you have available (and the size of the system you want to calculate).
If you have a small system with lots of k-points you can probably do 1 k-point per core. For normal/large systems I typically do 1 k-point per node and then within a node I use band parallellization. On our recent cluster the optimal setting is NCORE=16 (for a 16-core node), basically NCORE*NPAR = total cores so if you set this parameter to the number of cores in a node it'll do 1 band per node, minimizing communication. On our older clusters NCORE=1 worked better.
Also enable LPLANE=.TRUE. (or test) this is a tag which controls how much data is exchanged between nodes.
To summarize:
-If you have multiple k-points, use KPAR, set the number to be able to divide your irreducible kpoints (amount in IBZKPT, run vasp and kill it at the start if you don't know how many there are). For instance if you have 20 k-points KPAR can be 1, 2, 4, 5, 10, 20
Thank you for your very comprehensive answer. From what I read, parallelization is dependent on the machine you are using and not on the system you are simulating. Does this mean that if the parallelization tags are benchmarked for a specific system and cluster, it should work for other crystal systems as long as I use the same computer cluster?
The results are transferrable to an extent. Basically you will have linear scaling up to a point and then you will get this kind of saturation effect, where scaling further won't get you much gain. Where this saturation begins depends on the size of your system.
Running a 1-atom cell on 10 nodes is not that useful, but a 50 atom cell is. This typically won't affect settings like NCORE and such. Basically you should keep in mind that parallel calculations always lose some efficiency, so it is better to do 5 calculations on 1 node than 1 calculation on 5 nodes. However, to get your result in a reasonable time you usually do want to parallellize for larger systems (you typically don't have as many of them either).
If you run high accuracy calculations you do need to watch out with memory, some parallellization routines end with collecting all the data on one node, which can push it over the memory limit, in these cases it may for instance be better to allocate more nodes per k-point, which is slightly less efficient, but uses less memory ( not using all cores per node, but using all memory is also an option in these cases). This often happens at the end of an SCF cycle, where it prepares for writing the CHGCAR and WAVECAR (even if you tell VASP not to write them to disk...).
A small addendum to the remark Michael made:
The combination of NPAR and KPAR is very powerfull, but , as said, will depend on your system size. The impact can be huge (more than an order of magnitude) as I saw during scaling tests on a HPC machine:
https://dannyvanpoucke.be/scaling-vasp-breniac-en/
* the first image shows the effect on a 2 atoms system.
* the second image is for a 70 atom system (the number next to the data indicate the NPAR value)
The almost serial scaling of KPAR is shown in the third image(dashed curves) where I just increased the number of k-points, KPAR and nodes such that k-points=KPAR=nodes. The solid curves are for a fixed number of Kpoints and KPAR=#nodes.
Danny
Hi, In the following link, this question is comprehensively answered and the point is cleared with some examples and I highly suggest to all VASP users.
https://www.youtube.com/watch?v=8srGJMRPYoY
https://bit.ly/2UtvbHE
Cheers,
- Badges
- Science topic