1) If you are beginner, all of them are going to take almost similar time to learn and start programming.

2) It will always depend on your problem, what should be the best way to accelerate your program.

3) In general, cuda is much much more effiecient than MPI and OpenMP, but it has its limitations too.

In overall, cuda is more efficient and  also beneficial for career point of view , nvidia is growing exponentially.. :)

Gourav, can you take a super short survey regarding this issue? It would really help my research for my thesis.

Thank you in advance. 

Survey link:

https://docs.google.com/forms/d/1-wksY4O7AhO3rwloGmz07iMvctiyABZ9Ufo7xINY3I0/viewform?usp=send_form

It depends, if you want to do parallel programs in large scale installations (hundreds / thousands of cores)  MPI would be most suitable, because it can use network communication . OpenMP would be suitable if your parallel application can run ok in a multicore machine (actually 60 cores is the most you can get on Intel machines, I think). CUDA is fast, but only if your do a lot of parallel processing on matrices.

CUDA can be very fast, but for some kind of applications. Data transfer in CUDA is often the bottleneck.

MPI is suitable for cluster environment and large scale network of computers.

OpenMP is more suitable for multi-core systems. So it's speed depends on the number of cores.

I prefer CUDA in compare to OpenMP. But MPI's applications are often different than the other two.

I think CUDA is more beneficial for job opportunities.

There is no one answer to this question. MPI is your only option from these three if your software should run on a distributed system, e.g. a cluster. If you want to only run your software on a single computer MPI is not worth the overhead. Most importantly you should consider that each MPI process allocates the same amount of memory, whereas with OpenMP uses a shared memory model.

I personally think that OpenMP is the easiest one to get started with. In the beginning you'll only need very few directives to parallelize some of your for-loops. It is hard, though, to understand the full OpenMP standard and it is also hard to get the best performance out of this.

CUDA is a completely different model, but I don't think it is really hard to learn. Because it is different from CPU programming there is good introductory material. For setting up a CUDA kernel most is copy and past which makes this step quite easy. Kernel programming is a little harder. But, also here it is easy to get your first kernel running. It is hard, as with the other two, to finally tweak the kernel for best parallel performance.

Now, I would say that it depends on the job which of these is most beneficial to you. For cluster software you cannot get around MPI. For smaller scientific software you'll  need OpenMP. If you want to find a job where CUDA is a requirement you really need to be an expert. I think there are already many good CUDA programmers out there and there are not that many jobs that have CUDA as an important requirement. So, you'll need to be better than the others. In most jobs that are not in research directly, other methods for parallelization are more important. Regular desktop software usually uses libraries to start threads or asynchronous computations. With C++11 this is already built into the STL. This will become an important skill for a lot of the software (but it depends if this is the kind of software you want to program for). Also important in these areas might be OpenCL and not CUDA because it is portable. There will be a lot of areas where OpenCL is more important than CUDA. Especially if you want to write code for the Mac platform OpenCL is of interest.

I think, the best thing you can do is learn MPI plus one of the other. Hybrid parallelization is becoming more and more important. Our simulation software was parallelized in MPI and OpenMP separately, i.e. you could choose only one of these. Because of memory constraints on our clusters we went for a hybrid simulation using both MPI and OpenMP at the same time--use MPI to communicate between computers and OpenMP for threads on the same computer using shared memory.

In another project I am using OpenCL because we expect that the CPU will not be powerful enough for realtime visualization. The software is supposed to run on consumer-grade hardware. CUDA is not an option since it is not portable. Furthermore, the advantage of OpenCL is that at least in theory we have to write the source code only once to be able to run on both the CPU and the GPU (pick the CPU if no appropriate GPU is available).

As others have said, depends on the application. In my case, I prefer CUDA because my application (Lattice Boltzmann Method) perfectly scaled using this interface. I also have used OpenMP and MPI, where the performance was agreeable, but the architecture of GPUs is formidable and the performance is very high.

I feel that the method of parallelization depends on your application and the target hardware. As mentioned in the above replies, MPI is most certainly used for distributed computing using clusters. OpenMP uses the shared memory model and can be useful to parallelize some important for-loops in your application. If you have a cluster for computing which hosts multiple-cores per node, then you can also consider the MPI-OpenMP solution. This uses MPI for data distribution across nodes and OpenMP for on-node parallelization. Such solutions have been tested effectively for Sparse-Matrix Multiplications and other HPC applications.

In regards with CUDA, it is meant for GPUs and that too from Nvidia. So if you are going to be fixed to the same platform, then CUDA is the way to go for GPGPU computing. CUDA offers a lot of tuned libraries for GPU computing. The recent support for dynamic parallelism and unified virtual memory on CUDA GPUs makes them support applications with irregular structures such as graph algorithms.

But for general, code-portable GPU programming, I would rely on OpenCL. It supports far many GPUs and also the accelerators such as Intel Xeon Phi. The OpenCL 2.0 offers similar programming features as CUDA 6. The OpenCL community lags behind in developing OpenCL tuned libraries. This is difficult as different architectures are supported by OpenCL and may not have the same tuning parameters. OpenCL based GPU/accelerator vendors have to provide these libraries. 

I want to add an extra note to the answers. Parallel code in OpenCL is portable, not the performance (specially between CPU and GPU). So OpenMP and OpenCL (CUDA for NVIDIA) is a better solution instead of solely OpenCL in a single processing node. For NVIDI GPUs, CUDA code performs better. Some hybrid programming frameworks are suggested by the research community. 

I would say "none of the above".  While MPI is the only option of the three choices that works for distributed-memory programming, it is not the only option available to programmers.  Others include SHMEM, UPC, Charm++, Coarray-C++, and Fortran 2008.  I find the parallel programming syntax built into Fortran the easiest to use.  While this has mainly been in the HPC domain, the upcoming GCC 5.0 release (4.10 dev) will include support and hence be available to anyone in a free compiler.

A good reference regarding the question:

A Survey of Parallel Programming Models and Tools in the Multi and Many-Core Era

Javier Diaz, Camelia Mun˜oz-Caro, and Alfonso Nin˜o

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 23, NO. 8, AUGUST 2012

OpenMP :  Limited to shared memory or distributed shared memory.

             I've found that OpenMP performance is generally poor when programmed

         the "easy" way.  If you pay attention to data placement, and use local variables everywhere you can, you can beat MPI on a virtual shared memory machine.  However, the program is quite ugly, and is essentially the same as an MPI program.

         So for most users, OpenMP is eay to use, but performs poorly in most cases.  Since OpenMP only performs well if you program it like MPI, you might as use MPI.

MPI:  Best performance in general, but hardest to program.  The biggest disadvantage

   over OpenMP is that you must program the entire problem in parallel at omnce.

   With OpenMP, you can work one section of the program at a time, parallelizing incrementally.

Cuda:  This is a specialized language for a specific vendor.  I can't recall how many of these I have learned and forgotten.  These get a lot devotees, then the hardware goes away, and your knowledge is useless.

(Remember Cell Processors, FPGAs, Digital Signal Processors,  MasPar, Thinking Machines ... ? )

  I agree with Bill Long that a high level parallel language is probably better,

E.g. the Coarray;s in Fortran 2008, or shared arrays in UPC. 

You can program at a high level, and  your program won't be useless when your current hardware goes away.  Other than that, use MPI.

@James: FPGAs and DSPs are not totally gone. Sure, a lot of people will stop writing very complicated VHDL or Verilog code for FPGAs, but now it is also possible to use OpenCL for this (though I've never tried it). DSPs were always a special market. You will not find them in a regular desktop PC. But, in audio some companies are still betting on DSPs and are still selling external hardware for your PC because it is faster (it is only faster because their own software is slow on a CPU, but the competition knows how to write fast software on the CPU).

I am also not sure if CUDA is a good comparison to e.g. the Cell processor. CUDA uses hardware that a lot of people already have: it runs on your regular NVIDIA GPU. I don't expect the hardware to go away any time soon. The only thing that could stop CUDA in its tracks right now is portability. As soon as everybody thinks they need to include GPGPU in their software some might go for portability and thus pick OpenCL instead. But for now there are a lot more highly tuned libraries out there for CUDA than there are for OpenCL.

@Simon:  The comment in DSPs was not about the purpose for which they were intended, but there was a movement to write software libraries for them and to use them as a coprocessor.  I find this very similar to the situation using GPUs for General Purpose computing.  I believe that GPUs will probably stay around, but that GPGPUs with double precision hardware will not be around for long, as their functionality will be moved into the CPU, much as MMX and late SSE hardware became part of the CPU.

GPUs will go back to being a relatively cheap add-on using only single precision and being dedicated to graphics, or whatever output medium there is in the future,

I've just read that Intel is now also providing better support for OpenCL 2.0. Both with current AMD CPUs and the new Intel CPUs the on-chip GPUs have a shared memory with the CPU (with Intel it is the L3 cache) and you can even share the pointers (i.e. they are the same for regular CPU computation and OpenCL computation). This makes OpenCL particularly interesting for short parallel computations; especially when the data does not have to be copied at all.

If this thing takes off if would be really beneficial to learn OpenCL. If might become the language how to parallelize on the CPU--and you get the GPU (even though it is not optimized for it) for free. And CUDA is definitely not competing here. OpenMP, however, tries to do something similar. The Intel compiler has now support for OpenMP 4.0. If I am not mistaken this includes directives for GPU acceleration. I give OpenCL a better chance for good performance because you will design algorithms for parallel hardware specifically instead of writing a few directives around loops like with OpenMP (although you could use OpenMP differently, most people won't).

CUDA suitable for SIMD-architecture (only). It is not suitable for all algorithms.

But GPU with CUDA gives good efficiency then cluster-system with MPI. 

There are also libraries that combine CUDA with OpenMP or MPI.

for beginners in parallel programming,OpenMP is easy and best .cuda is well suited /efficient for large and complex problem.If you looking for performance of your application then go with hybrid i.e OpenMP+ MPI+CUDA. 

without MPI will not get optimal performance in distributed sysytem

Followup to C. Naikodi - without MPI or some other SPMD programming system (Fortran, UPC, SHMEM, ...)  you will not get optimal performance on  distributed memory system.  A good implementation of modern Fortran can usually outperform MPI.

I prefer you CUDA. It is extended version of C++. 

For distributed system MPi+Cuda  is efficient one 

I prefer CUDA for GPU programming

CUDA is well adapted for GPU and OpenMP for CPU

It depends on the concerned application

THRUST/Cuda is very simple and I feel like writing classical C++.

Article Study of runtime performance for Java-multithread PSO on mul...

It is however dependent on the problem statement.

Unless any restrictions on resources one can systematically widen the horizon with a hybrid technique using either purely CPU dependent MPI+OpenMP or CPU-GPU interface with MPI+CUDA.

As many have responded quickly that CUDA is a way good approach for high demanding computational algorithms but let me emphasise on the fact that it is not the full picture.

At a hardware GPU architecture, one should know that "all blocks launched do not go onto work parallely." That means, at max a certain amount of blocks with fit into the streaming multiprocessor goes into kind of parallel task (because in the end it depends on how the warps work on one single instruction)

Thus, one can see that taking a big chunk of data, load it to HBM won't make the best parallel technique.

The better use it to hybridise it with MPI where smaller chunks of data are fed to various slaves and then use GPGPUs for the purpose.