I am simulating a multiphase flow using STAR-CCM+. I was running some tests solving the problem on one processor and I obtained residuals around 1e-5. I ran the exact same case on a cluster on 16 processors and I was able to obtain residuals of about 1e-9 for the same number of iterations.
I am really surprised by the difference which is quiet significant. Does anyone have a clue about the source of this discrepancy? Could it be the MPI?
PS: An other guy in my lab just had the same thing happening with a completely different application.
HI Jonathan,
Yes it is possible although i was face the same thing so i figured it out that its possible to get the variation in-fact if you solve same problem in older version of software then it might be not converge so don't panic about this so much perhaps in some problem if you directly run that case in to parallel mode then not going to converges very well like as an example in solar loads problem its necessary to put up the case in to sequential mode initially if you put it on parallel it will not going on an appropriate way.
regards
S. Shukla
The Faster convergence may be due to the parallel processing. In fact, you can do a exercise by reducing the number of processors in turn (say, 3) and record the number of iterations/time for a chosen target residual. It will give you a good idea about your machine performance.
Ok, so it is not an uncommon phenomenon.
It surely comes from the paralell processing and if I have time this week, I will do Arshad's test to know how the phenomenon evolves with the number of processors.
So it brings an other question: which solution would be more likely to be the best? I would say that it is the sequential solution given that there is no interference of the MPI . What do you think about that?
Actually, the choice may get complicated depending on the type of problem we are solving, the software tool etc. In fact, I think there may be convergence behavior/ differences that can be affected by parallel processing.
A direct answer to the question is quite uncertain !
You can try and let us know your experience on a test and validation case.
Parallel computation can affect the residual, but I think it not due to the use of parallel computers, per se. Instead it is from the growth of noise that one will observe differences using different system software, say compilers ( PGI vs INTEL); or different system hardware, say MPI interconnect. Assuming everything is IEEE compliant, these differences will be tiny, at the level of precision being used. Nevertheless one can see differences, for example in elliptic solutions using Krylov solvers. With petascale computations significant improvements may occur using quad precision vs double precision. Furthermore, regards the effect of the number of cores vs. residual, modern machines (eg. CRAY XT-6) show no such effect. 10-15 years ago, yes you would see it. I remember this particularly on a machine in which MPI double precision was reduced one bit in library for mathematical functions. There is also possibility that the compiler optimizations have done something not quite right... you could try reducing optimization level to see if that makes a difference.
I would not rule out the possibility that the difference is caused by the difference in discrete equation system between the sequential and parallel case. I do not have any idea, how parallelisation has been implemented in STAR-CCM. However, if the parallelisation involves domain decomposition, the implicit coupling between the domains is broken in parallel case and the equation system changes at the domain boundaries. Depending on the problem, the convergence characteristics may be very sensitive to the boundary conditions.
I finally got some time to run some test with a small case from the tutorials (However I don't know if my conclusions would the same with heavy calculations).
It appeared that the major difference between sequential/ paralell comes from the node on which it was running in sequential. An other fact to take into account is that the job was affected to the login node by mistake. So perhaps there is some precison limitationson it.
Also, by curiosity, I ran a batch of tests on 0.5,1, 2, 3 nodes (8 cores each) and I obtained converged residuals from using at least 1 complete node (even if the case with 0.5 node was very close too).
Thank for the help provided by everyone!
Most welcome !
Its good to hear that your confusion is fix it up.
good luck
Interesting question, we have a similar issue: usually we get higher residuals on the University grid computing infrastructure, even with the same number of CPUs. I think this is due to the computer architecture. For instance, a run performed on an i7 CPU with 8 cores has smaller residuals (at least one order of magnitude) than a run performed with 8 CPUs on the computing grid. Also, residuals tend to be even bigger with the coupled implicit solver...
On one of my cases, I got bigger residuals with the coupled implicit solver than the segregated solver. This effect seems to be normal as I look in the internet. It looks like this type of solver need a smaller timestep to be stable. In my case, a Lagrangian - Eulerian multiphase flow, using this method was really difficult and time consuming regarding the segregated solver.
When you talk about the i7 CPU,is it installed on a local computer?
With mixed precision residuals might drop to 1e-5 and with double precision it might drop to 1e-9 due to a smaller round off error. But how much they drop is case dependent and influenced by factors such as the physics of the problem and the grid quality etc. Based on your description and since your mate in the same lab experience the same thing I would bet my two cents that your are using mixed precision in serial but double precision on the cluster.
- Badges
- Science topic