Hello,
I constantly submit parallel jobs in a shared HPC cluster for my OpenFOAM flow simulations and like everyone else, I have to decide how many cores I need. Our cluster runs constantly over head and my jobs often wait in the queue. I know that it heavily depends on geometry, solver and system specifications, but are there any ways to make a rough estimate of "optimum number of cores" before losing speedup, based on the number of cells and -be it OpenFOAM's- solver?
Thanks in advance,
Huseyin
Just as you have pointed out, it totally depends on the solver, mesh and hardware availability. Hence, It varies for each user. The best way to go about it is to conduct a HPC scaling study before commencing intense simulation work.
Conventionally, this done by calculating the time taken for a fixed number of iterations/time-steps(say 100) by holding your mesh count constant(assuming you have a converged grid) and then increasing the number of cores gradually and recording the time taken for the number of time-steps. You'll find that there'll be a particular number of cores,N, beyond which, adding more cores does not improvise your simulation speed. Ideally you'd aim to use this number of cores(N) to run your simulation.
Hope this helps.
Cheers
Just as you have pointed out, it totally depends on the solver, mesh and hardware availability. Hence, It varies for each user. The best way to go about it is to conduct a HPC scaling study before commencing intense simulation work.
Conventionally, this done by calculating the time taken for a fixed number of iterations/time-steps(say 100) by holding your mesh count constant(assuming you have a converged grid) and then increasing the number of cores gradually and recording the time taken for the number of time-steps. You'll find that there'll be a particular number of cores,N, beyond which, adding more cores does not improvise your simulation speed. Ideally you'd aim to use this number of cores(N) to run your simulation.
Hope this helps.
Cheers
+1 for Arun Prabhakar . Also, you mentioned that your jobs often wait in the queuye. Queue managers usually take into account the number of cores, amount of memory and wall time simulation required in order to find your place in the submission queue. The more resources you require, the least priority you'll be given.
This is very physics and solver dependent. Essentially it depends on how much numerical work the solver needs to spent per mesh cell, i.e. how many differential and algebraic equations are specified in your model to be solved in each mesh cell. For just noncompressible turbulent fluid flow without chemical reactions the limit of efficient parallelization is about 40.000-50.000 mesh cells per CPU core. But as mentioned, this limit is not a universal law, it is just a number which is determined from performance testing and personal experience. You can take one of your cases and just run an own parallel performance scaling, to find it out more precisely for your specific cases. It might be depending on your cluster hardware and parallel interconnect network (GE vs. Infiniband) as well.
Regards,
Dr. Th. Frank.
A book on this subject might be helpful. Shamoon Jamshed (2014) "Using HPC for COMPUTATIONAL FLUID DYNAMIC: A Guide to High Performance Computing for CFD Engineers".
Cheers,
Biao
- Badges
- Science topic