It appears that your CFD partial differential equations are solved using mesh partitioning, potentially data partitioning. A reason why solution fails at a specific number of cpus is that one partition's system has a much higher condition number than its adjacent ones (which do not have to be geometrically adjacent). I have encountered this problem with my codes, and my current hack is by trail and error.

Thanks, Sanjay Kharche. Not an answer I was hoping for, but one I was half expecting.

So by partitioning the domain/mesh, parts of my geometry may become unstable. By your experience, would you recommend partitioning even (2,4,8 etc) or uneven? I would expect uneven partitioning to be more unstable, or not?

Not necessarily. I use an odd number of CPUs (for nefarious reasons, not recommended) which works okay. Does your platform have preconditioning - I found that reducing the complete sys matrix's stiffness often helps. I do it at low level code and use CG/BIC-STAB, but a good starting point may be Jacobi preconditioner.

Does your simulation work on your chosen number of procs without any "activity" (null solution, very smooth known solution)? Also, have you quantified scaling of your actual simulation? Answers to q's like this may help understand problem better.

I am using ANSYS CFX. I am not sure if I can precondition my solution.

If you are using incompressible equations or compressible equations at low Mach number for the multi-phase flow model, then the solver uses a preconditioning technique. Therefore, the iterative solution of the resulting sparse matrix depends on the number of the multi grids/multi blocks and hence the number of processors. You may need to use an under-relaxation for the iterative solution to guarantee convergence for any number of processors. You may also change the sparse matrix iterative solver.

There is another aspect of applying parallelization to a CFD problem.

1) Imagine a boundary value problem in serial solution mode. In this case the linearized algebraic equation solver can determine within one single iteration a first iterative solution on the full domain, which is affected already by all applied boundary conditions on the domain boundaries.

2) Now imagine, that your domain is partitioned in N partitions in length and solved on N processors. In the first iteration of the lin. alg. eq. solver the solution on the first partition is knowing only its own applied boundary conditions, while the boundary conditions on the other N-1 partitions cannot affect the solution on the 1. partition, because the ghost cell value son the partition boundary are being lagged by one iteration (i.e. they get exchanged between processors at the end of the first iteration). Now on the 2nd iteration the lin. alg. eq. solver "feels" the influence of the boundary conditions on the 1st and 2nd partition. Etc. That means for a parallel computation, that it takes N iterations of the lin. alg. eq. solver until the solutions on all partitions are getting influenced by all applied boundary conditions, if the mesh is partitioned just along one selected coordinate direction. By using e.g. Metis partitioner this is going faster of course. But you can imagine this as a kind of implicite underrelaxation with respect to boundary conditions being applied. This can lead to the fact, that a flow simulation with soe extreme applied boundary conditions is "exploding" immediately if one tries to solve it in serial mode, while it is converging if "sufficiently" parallelized. Just because the occurring information lagging on the partition boundaries during the solution process. This of course depends on how the inter-process communication is handled in the CFD solver and may vary from one CFD solver to another.

Regards,

Dr. Th. Frank.

Thank you Thomas Frank for your very informative input!

From your experience: Which approach would you suggest initially when the solver is crashing in the first iteration of my transient multiphase simulation (Bubble Column with GENTOP concept) without further information on the error (e.g. error message of ANSYS CFX Solver: Fatal Overflow of linear solver).

The standard partitioning I am currently using with CFX is MeTiS. Would it be more productive to change the type of partitioning or change the number of partitions?

Kind regards,

Thomas Bernstein

Metis provides the best partitioning of both geometric meshes and algebraic matrices. One of the other options your software may offer is box partitioning. Metis based partitioning aims at extreme load balancing and may produce (unlikely) partition edges that give rise to observed problem. It will be interesting to know if your problem works on any number of procs if you used box partitioning (not expected).

Thomas Bernstein : Just one question: is your case with heat/mass transfer or is it an adiabatic case (i.e. const. temperature problem)?

Besides any parallelization questions or strategies, it is of highly crucial importance for non-adiabatic multiphase flows to start (if possible) from a thermal equilibrium flow situation. If the flow is at T=0s in a non-equilibrium state, than sudden mass/heat transfers start to take place comparable to steam explosion or shock condensation phenomena in corresponding experiments. Those starting physical processes can be quite violent and a CFD solver is struggling to survive these situations.

Following strategies can be applied for observed issues to start with a transient simulation:

1) start with a substantially smaller timestep and ramp the timestep up to the desired value as a linear ramp over - let's say - 100 timesteps

2) Apply a better and more suitable initialization of your flow field, e.g. from a precursor steady-state simulation of the comparable physics.

3) Maybe a partitioning from the inlet towards the outlet using the partitioning along the corresponding coordinate direction can help, not sure. But maybe worth trying.

4) Eventually it behaves better with a rather larger then smaller core count due to the given reason in my previous post.

5) In CFX you can apply underrelaxation of e.g. pressure, temperature and fluid densities on the CCL level. To my knowledge it is not possible in the GUI. Maybe you ask Thomas Höhne on how to do that.

6) If you are using IAPWS for water/water steam, than you should use IAPWS tables for an as narrow as ever possible range of temperatures and pressures and at least 500 points in the IAPWS tables for sufficient table accuracy. The CFX default value is much too low for achieving good convergence.

So far my experiences with that.

Best regards,

Dr. Th. Frank.

Dear Thomas Frank my case is an adiabatic case for now. Later on we want to implement heat / mass transfer so thank you for the initial insight into the upcoming set of problems.

For now I applied a directional partitioning in the flow direction. which seemed to work fine with more partitions.

Thank you dear colleagues for your input.

I will keep you updated on my further progress!