At the implementation stage of a machine learning model, is there rule of thumb to prefer MPI against GPGPU architecture as the architecture?
Generally, GPUs processing power can be exploited more efficiently for problems where the algorithm operating on the data can be straightforwardly parallelized, e.g., vector to vector operations (element-wise additions/multiplications). On the other hand, when the parallel algorithm is implemented by a small number of parallel threads/processes and they operate on large chunks of data, then you probably should opt for MPI.
In general, GPU-based programming needs careful assesment of the tradeoff of speedup vs. data transfer because transferring data from and to the GPU usually takes some time.
You might want to take a look at OpenCL which combines multicore and GPU programming into one framework.
Generally, GPUs processing power can be exploited more efficiently for problems where the algorithm operating on the data can be straightforwardly parallelized, e.g., vector to vector operations (element-wise additions/multiplications). On the other hand, when the parallel algorithm is implemented by a small number of parallel threads/processes and they operate on large chunks of data, then you probably should opt for MPI.
While it is true that the CPU-GPU is slow to send data, with a decent sized GPU Memory in general a lot of the data can be stored on the device. But the more important problem is that there is a difference between the parallelism of the MPI and of the GPU. The GPU is a SMID architecture and thus could be ideally suited for approaches such as SVM but MPI has more versatility and can handle larger chunks of data by each process. Moreover these days MPI that contains GPU is quite common so the most improtant is to know what algorithm is to use and then directly match the hardware on it
On a single computing platform, the GPUs are good when the data can be segmented into chunks and the same operations are required to be performed independently on each chunk (data parallelism on an SPMD architecture). Multithreading is good when the job is better parallelized by separating it into tasks that can be run independently on the same data (task parallelism on a shared memory architecture).
It is possible that a given job/algorithm can have elements of both task parallelism and data parallelism, as well as sections that are inherently sequential in nature.
I think it mainly depends on the algorithm you want to implement. If you can give us a clue on the method, we might be more precise.
The effect of multithreading in the computational time depends on how well your processes can be distinctively separated and as a consequence running independently in your threads. It will better to present your ideas here in order to help you find the best solution for your problem.
Thanks a lot friends!
I've seen a lot of very interesting comments. In the specific case, we have implemented two machine learning algs. One is a pure GA with an trigonometric expansion as fitness and the threshold MSE as fitness evaluation (if you're interested please have a look to Cavuoti et al. 2012 "Genetic Algorithm Modeling with GPU Parallel Computing Technology", which I've loaded in my researchgate pub list (because it is in press)).
The second was a MLP trained by a GA, replacing back propagation rule.
In the first case the GPU version was built wrapping the serial version (originally written in C++), through Thrust library and extern call mechanism to bypass C++ limitations of CUDA. The speedup was incredible (200X).
In the second case, we developed the GPU version from scratch, duplicating CUDA routines to be able to run on multi-thread CPU for comparison. One year before I've also written the C++ serial version of MLPGA. Initially the performances were surprising: Both new CPU multi-thread and GPU versions were faster than the older C++. But, incredibly, the CPU one was faster of GPU version!!
That's the reason why I launched the discussion here...
From my experience (not so deep) MPI seems better than GPU when the flow host-device has to be massively employed. For example a hybrid model like MLP+GA on massive datasets requires an intensive use of host-device exchange and it affects dramatically the speedup. Moreover, the dynamic memory allocation functions provided by CUDA reveal worst performances than the same C functions (such as malloc against cudaMalloc), when the allocated size grows up to 1 MB per call. Probably in such cases MPI is the solution...
@Berlich
I hope that you do not mind a quick question:
When you say
"We found that, in order to get a significant speedup in the cluster environment without infiniband interconnect, each compute node needs to run for at least 3-4 seconds without interaction with the server."
do you mean that with p compute nodes the time to distribute evaluations and collect the results is roughly 3-4*p seconds, and that this is covered by the speedup of splitting the evaluations into p parallel tasks each running for at least 3-4 seconds? If so, can you say how the overhead grows with p?
- Badges
- Science topic