After reading the proceeding: "Best Practices in Running Collaborative GPU Hackathons: Advancing Scientific Applications with a Sustained Impact" I came across the paper: "Porting the MPI Parallelized LES Model PALM to Multi-GPU Systems – An Experience Report". It is evident that GPU's are Single Instruction, multiple threads and therefore it is not a cache-optimized computer architecture. My experience is that unless you changed the whole kernel the Hybrid approach MPI+GPU affects the performance a lot. However, there is a boom in HPC with GPU's. In some instances, MPI alone performs better than the hybrid since we do not need to move information back and forth to the GPU and CPU, respectively. We have different good practices but there is no a standard or a reference that we can always take to the bank. The picture becomes even worse when high order schemes are used.
1) Which is your common approach for this issues in CFD?
2) Should we keep the structure of the source code as functional programming, imperative programming with a bunch of do .. end do in each subroutine that affects the performance a lot?
3) Should we use data region (where the data remains local in the GPU) where we packed all the computations even though we are hurting the readability of the source code?
4) Should we update the ghost cells in every time step?
Again the focus is only CFD
While there is no universal answer to your question about hybrid computation, more and more people consider that the directive approach (OpenACC / OpenMP) to be the nominal way to go for porting large legacy codes, mostly because OpenAcc / OpenMP directives can be added incrementally in the code.
An interesting emerging alternative, which is probably more adapted to new codes aiming both at performance and portability across architectures (x86_64 multicore, nvidia GPU, AMD GPU, ARM, ...) is to use a "smart" library like Kokkos (developped at Sandia NL) or RAJA (devel at Livermore NL) or others to implement shared memory parallelism. These libraries provides "generic" concepts to express parallel loops (for, reduce, and so on). These libraries focus on node-level parallelism, and need to be used in combination with MPI for large scale distributed computation.
Another major advantage of these libraries, when compared to the directive approach, is that, and this is especially true for Kokkos, is that they also provide "architecture-aware" data containers. E.g. Kokkos provides arrays (called Kokkos::View) for CPU/GPU/... which allows you to access data like this data(i,j,k), very intuitively, and simplify the memory management when dealing with GPUs. Note that Kokkos library is a core building piece of the large parallel linear algebra package Trilinos.
I think that portability and more precisely "performance portability" is a keep concept today, that is being able to develop codes that can run efficiently on multiple platforms, not just one.
We have already discussed this topic elsewhere, if I remember correctly. However, here they are my two cents:
1) Achieving (and mantaining!) the same MPI performances in a hybrid and consistent (in the sense that everything is done with the same approach) setup is hard if not impossible. People at PETSc don't like it, so who am I to disagree?
2) Scientific software is quite different from other software as you either get a right answer or you don't. Mantaining the source code is thus of primary importance, in my opinion even above performances (within certain limits). If you start changing language at every season, you are not using your time efficiently, as the next gen hardware might give you without effort that 10% you hardly gained by restructuring the whole code.
4) In MPI you have to update ghost cells at every iteration and/or time step, otherwise you fastly loose control as the suitability of not doing it depends from the case and partitioning; also, it is nice to not have the code behavior to strongly depend on the parallelization.
Yes Paolo Lampitella you and I have discussed this before and again thanks for your contribution to this topic. However, I am very worried because my impression is that if you are not using GPU's you are not in the game of CFD. Again, that is my impression based on the last conferences I have attended/presented my work. I've seen a lot of folks from NASA, Wright Patterson that are extremely focused on GPU's, even more than anything else.
I agree with you that we do scientific computation but it seems that the CFD is allowing other new computational trends such as AI among others, to influence our goals and objectives. For example, last meetings one professor has insisted too much on the idea of GPU's because funds are available for that. But my question is: What about the physics? I clearly understand the issues with HPC and I do my best to come up with efficient implementation, but some of them affect the maintainability of the implementation.
Finally, here is another catch, cost! Yes, HPC systems are extremely expensive, but, if you add the extra cost of GPU's for each node the cost goes through the roof. Thus, we only have a couple of blades with GPU's or if we are a lucky guy you might have a dozen of them. Your point about updating the Ghost cells at every time step makes perfect sense for a time-dependent solution, like LES/DNS among others.
O P Gupta that is not necessary anymore. You can use OpenACC which is a library that simplifies and takes the heavy weight off your shoulders.
Thanks Krzysztof Jurczuk it is very interesting your experience. However, I do not visualize folks from NASA or AFRL using Spark and leaving behind MPI. MPI is not hard at all once you really understand the fundamentals!
> cc -c empty.c
does not give an error, though does nothing useful.
> cc empty.c
predictably issues an error: undefined reference to `main'
Note that the corresponding Fortran compiler seems a bit more informative:
> ftn -c empty.f
ftn-1391 crayftn: WARNING in command line
Source file "empty.f" contains no Fortran statements.
Technically the empty source file is an error since a Fortran program unit is required to have an "end" statement.
Ignore the previous answer - posted to the wrong question. Sorry.
For Hybrid HPC calculation I would use Fortran + OpenMP. The OpenMP facility has taken over the role of OpenACC for the most part, and seems to be the method preferred going forward. I prefer Fortran because the SPMD parallel execution semantics are built into the language, hence MPI is no longer needed.
Bill Long I disagree atlghou I think that can be very useful and solve a lot of problem we see currently. if we had a huge blade with Enugu ram memory then we can get rid of MPI. I don’t think that will see that soon in the market. In the meantime there is no way we can bypass MPI.
Sorry Bill Long but I still disagree. How will you manage a memory bound problem ? I do not what version of Fortran you are referring to, but the one I used Fortran 90, requires MPI. Otherwise, you will run out of memory rather quickly. !
Bill Long thanks for the clarification and to be honest this is the first time I hear about that. I wonder, why NASA and big research institutions haven’t adopted that ? Why CFD companies haven’t adopted that ?
While there is no universal answer to your question about hybrid computation, more and more people consider that the directive approach (OpenACC / OpenMP) to be the nominal way to go for porting large legacy codes, mostly because OpenAcc / OpenMP directives can be added incrementally in the code.
An interesting emerging alternative, which is probably more adapted to new codes aiming both at performance and portability across architectures (x86_64 multicore, nvidia GPU, AMD GPU, ARM, ...) is to use a "smart" library like Kokkos (developped at Sandia NL) or RAJA (devel at Livermore NL) or others to implement shared memory parallelism. These libraries provides "generic" concepts to express parallel loops (for, reduce, and so on). These libraries focus on node-level parallelism, and need to be used in combination with MPI for large scale distributed computation.
Another major advantage of these libraries, when compared to the directive approach, is that, and this is especially true for Kokkos, is that they also provide "architecture-aware" data containers. E.g. Kokkos provides arrays (called Kokkos::View) for CPU/GPU/... which allows you to access data like this data(i,j,k), very intuitively, and simplify the memory management when dealing with GPUs. Note that Kokkos library is a core building piece of the large parallel linear algebra package Trilinos.
I think that portability and more precisely "performance portability" is a keep concept today, that is being able to develop codes that can run efficiently on multiple platforms, not just one.
Julio Mendez, there are several reasons for not adopting (yet) the Fortran parallel syntax.
The prominent one is certainly priorities. If you have a working parallel implementation and, as human being, a limited amount of time, you certainly have more interesting things to do than replace your parallel implementation. In addition, the Fortran one is also not quite similar to the MPI one, and requires getting used to it (more importantly, it's not something you would put in a separate module, you should typically program everything from scratch with that).
The second one is availability. The standard has been there for a while, but the actual implementation in GCC and other compilers has started more recently.
The third one is, in my pinion, performance assessment. There is no large availability of performance benchmarks or best practices.
Finally, people (including you, an actual Fortran programmer, if you allow me) are not typically aware of the Fortran evolving standard (did you know we also have assumed rank and assumed type declarations now?) and Fortran in itself is not much used and, as consequence, teached anymore. Long story short: people don't know and nobody tells them.
Thanks Paolo Lampitella for your contribution I agree with you ! Pierre Kestener you point of view is very valuable too. I think I 'll wait until a standard comes out and it is world wide embraced.
- Badges
- Science topic