Well, the most common model used today is MPI+X. MPI is used for distributed memory communication and X is one of the shared-memory methods (OpenMP, OpenACC, CUDA or OpenCL).

I would recommend a recent presentation by Bill Gropp on "MPI+X on The Way to Exascale"

http://wgropp.cs.illinois.edu/bib/talks/tdata/2017/ISC-mpix-exascale.pdf

Well, the most common model used today is MPI+X. MPI is used for distributed memory communication and X is one of the shared-memory methods (OpenMP, OpenACC, CUDA or OpenCL).

I would recommend a recent presentation by Bill Gropp on "MPI+X on The Way to Exascale"

http://wgropp.cs.illinois.edu/bib/talks/tdata/2017/ISC-mpix-exascale.pdf

Vladimir Puzyrev thank you for your answer. Using MPI+X makes the development of parallel algorithms even more challenging (based on the presentation that you posted). One of the main reasons why algorithms do not use the available parallel features is the complexity involved in re-writing an algorithm into parallel. Thereafter, OMP was presented as a solution to this problem but has overhead problems and other issues that relate to scalability, etc. So my personal opinion is that one of these methods (that you mentioned) will definitely prevail but the one that will do so will be most probably the one that will offer simplicity when writing the code and be intergraded with the ability to build efficient parallel code without the need to write numerous commands at the development level.

I think for large scale parallel computing MPI will be there for the near future and possibly for a long time. Along with that GPUs are becoming part of high performance computing and so CUDA, OpenMP, OpenACC are being used. But basically I wanted to say that for large scale computing MPI will continue to be used heavily. There are many third party software (molecular dynamics for e.g. ...AMBER, NAMD etc.) available now a days in CUDA and give excellent performance on GPUs. In this case you are not developing the code, but using it for your research (say if you are a biochemist). In addition individual researchers are also developing large scale simulation codes for GPUs.

Thank you for sharing your view on the matter with us, Amitava Majumdar. 

As I see it, one solution will not prevail on others since science faces a large variety of challenges and needs. Indeed MPI+x remains a kind of standard. GPU is also very interesting for pure calculation, but poorly adresses the needs for computations manipulating large amount of input data. Some other applications can largely benefit from grid computing approaches, like the Berkeley initiative BOINC, especially for astronomers (but not only). And with the rise of IoT, the significant increase of distributed and linked data, we can imagine that large scale computation will more and more involve significant data volumes distributed around the globe. Maybe the future is in a large ecosystem of interoperable technologies ?

Thank you Jerome for sharing your view on the matter with us.

With the current trend in technology and in the computational science, I say it is possible

bigdata analytical tools, software and hardware. robotic analyst, we do not use human to do maths anymore, we use robots, application and AI/BI. do a deep literature review to find out. God bless your hard work.

I think, with the AI, Multi-core computing hardware it may possible to compute large scale numerical problem. However, large problems area also becoming complex. Space, time, diversity always dominates and possess challenge for creation of another NextGen AI,..

If human can not understand something, Jennifer, it is of no use for him.

The sentence "One of the main reasons why algorithms do not use the available parallel features is the complexity involved in re-writing an algorithm into parallel" reminds me to what prof. El-Ghazali Talbi said: Think parallel while you write your code. Hence, the coders must be aware of the architecture they use, which is today (at least) multicore.

Thank you Tatjana for your comment. It's true that when you code something new then you can apply this much easier. The main issue lies within the "trillions" of lines that are already programmed and work fine because of endless hours of testing and debugging. In many cases the solution method applied needs to be entirely rewritten from scratch.

I understood that the question was related to the future of large scale problems, and I think that applies mostly to new things rather than existing one. On the other hand, it sounds like the amount of existing code is so huge that just to parallelize it requires at least next 10 years. I believe that with shared memory paradigms we address needs of the largest number of users: posix, openmp, cuda...

Lastly, if we continue to write sequential code, completely disregarding hardware architecture, then I believe that the future looks confusing.

Tatjana,

the problem is that not all problems can be parallelized.

Really, almost all problems have at least a parallelizable portion, so it´s important to desing the parallel solutions using the Ian Foster Methodology to separate parallel´s process. We should know deeply the theory of parallel algorithm design to solve heterogeneous problems more efficiently.

Several replies mention MPI. The real answer is SPMD (Single Program Multiple Data) programming for distributed-memory systems. SPMD is the scheme demonstrated to scale. MPI is one example of this model. But there are other libraries, such as SHMEM. And, perhaps easier to use, are languages with this model built into the syntax, such as UPC++, CoarrayC++, and standard Fortran. MPI is the "legacy" option and will be around for a long time because of inertia of existing codes. But there are other, easier to use options available for new codes.