Domain decomposition in space has diverse approaches for optimal load balancing to simulate massive CFD problem in parallel. However, the time-evolution being serial seems to be a very big restraint in the run-time of the simulations.
I understand, that if we have time parallelism we have a matrix in front of the du/dt vector, which we need to invert. Are there any studies where this matrix inversion has been done in parallel ?
A Lawrence Livermore study reflects that a parallel multigrid solver has been implemented:
https://computation.llnl.gov/projects/parallel-time-integration-multigrid/strand2d-pit.pdf
# But, then the question is why are there so limited studies?
# What was the real bottleneck? Was it stability ? Not good schemes developed? I do not seem to grasp that.
#And why has been time parallelism been ignored completely in most CFD books?
PS: For statistically stationary turbulent flows where there are spatial symmetries, there are cunning ways to generate multiple snapshots after a simulation is complete. That may work for the mean calculation (it essentially does the job of time-parallelism, i.e. quicker data generation but it is not time-parallel) but may potentially inject artificiality to turbulent structures when doing conditional average/spectra.
I am not sure to understand correctly the key of your question. If you assume that the time is like a spatial direction you can see a fundamental issue in this hypothesis: the hyperbolic/parabolic character when time is present. That means that the functional state at a certain time depends only on the previous states. A different case would be an equation that has an elliptic character in time (I am not sure but that could be the case of the general relativity theory) where a partition in time could be possible.
The paper you linked does not perform a really time domain decomposition as one would do for the 3D spatial domain. At least this is what I understand...
I am not sure to understand correctly the key of your question. If you assume that the time is like a spatial direction you can see a fundamental issue in this hypothesis: the hyperbolic/parabolic character when time is present. That means that the functional state at a certain time depends only on the previous states. A different case would be an equation that has an elliptic character in time (I am not sure but that could be the case of the general relativity theory) where a partition in time could be possible.
The paper you linked does not perform a really time domain decomposition as one would do for the 3D spatial domain. At least this is what I understand...
Dear Professor Denaro..That's a valuable point regarding a hyperbolic/ parabolic trait. Thanks for pointing that out...
Short answer: if you do that you will run out of memory really really fast, and not even for a good reason.
Long answer: In CFD we don't store every time step data. In simulation, time should be discretized but it's discretized differently than in space, let's say time discreatization is easier (space discretization have non local effect: in incompressible flow, information of pression is instaneously transmit to the whole domain so your points in different sides of domaine is linked.)
Because time discreatization have none of those weirdness, we can store 1 time step, may be 2, 3, 4, ..., depend on your time discretization scheme, and delete from memory all the previous one, it's has no effect. And we don't have influence from the future to the present (unfortunately, I really like guys from the future messing with the pass) . So we can virtually get rid of 1 dimensional. That's huge. In a realistic CFD simulation where you got 1 million to 100 billion point in space, and 1000 to 1 million iterations, do the math and you will see no RAM can handle that.
Saying in other words what the other answers to this question convey: I think the key problem with parallel-in-time simulations of the Navier-Stokes equations is causality. Each flow state has to follow deterministically from previous flow states; so how could one compute something going on later in time without knowing already what happens now. Now, apparently this is not a problem for linear equations, for which (I believe) radial basis functions in both time and space have been used for parallelization in both the temporal and spatial domains. But, of course, the Navier-Stokes equations are nonlinear. The only way (that I can think of at this point) to solve this problem, is to iteratively solve the Navier-Stokes equations in time. As far as I'm aware, the following two works aim at that:
https://www.researchgate.net/publication/281896270_A_Block_Krylov_Subspace_Implementation_of_the_Time-Parallel_Paraexp_Method_and_its_Extension_for_Nonlinear_Partial_Differential_Equations
https://www.researchgate.net/publication/324880639_An_Exponential_Time_Integrator_for_the_Incompressible_Navier--Stokes_Equation
As one of the other contributors to this question has remarked, though: for fully 3D Navier-Stokes computations, (such) parallel-in-time methods can be expected to have a prohibitive computational cost.
Temporal parallel computation requires all spatio-temporal information are available/calculated simultaneously, and different temporal information are decomposed. One possibility of this is to use FFT in time, so that the governing equation for different frequencies can be solved independently. For saturated turbulence, an artificial periodicity T and time step dt need to be defined (similar to computation domain and grid size for the conventional fully developed channel flow). This value of T needs to be sufficiently large so that the artificial periodicity is not affecting the physics, and dt should be small so that essential small-temporal-scale is resolved. (sorry I am thinking of DNS)
I can think of one application:
For the 2D vortex shedding motion behind a cylinder, where the periodicity is a constant (not turbulence), and applying FFT in time is possible. However, it is noted that the nonlinear term can couple information from different frequencies. So an iterative solver is still needed.
This idea is much more expensive compared to the conventional time stepper method. The only advantage I can think of that it can achieve a spectral accuracy in time. Why this would be so important is beyond my comprehension.
- Badges
- Science topic
- Similar topics
- Vectorization