My C++ app is already parallelized in linux using MPI. The process is splitted in N cpus, but some of the data is repeated in each process, and for technical reasons this data cannot be easily split over MPI. How can I sue MPI_PUT and MPI_GET to reduce the total memory consumption of my program? Can you help me with a simple example?
Hi Mohamadreza, it seems weird to me to use MPI_PUT/MPI_GET to solve your problem. MPI communication is slow and using it for memory access will probably slow things down. Memory access for different MPI processes has to be serialized because over one connection there can be only one communication at a time. And I would expect that all processes would require communication at the same time. This would mean your application will not scale well to more processes. For a program that is almost serialized because the processes are always waiting using MPI does not make sense anymore.
Here are two suggestions how you could reduce the memory problem: (1) use a hybrid approach with MPI and OpenMP, (2) use memory mapped files. I'll explain in the following what I mean.
If you have a memory problem because several MPI processes are running on the same physical computer then other parallelization methods based on shared memory work a lot better. In this case I suggest OpenMP. You can still use MPI for communication between different computers, but on the same computer OpenMP will use threads instead of processes for parallelization. Data that is read-only can be easily shared between threads. Most people only know about using OpenMP for parallelization of for-loops. Since you already have a working parallelization based on MPI it would make more sense to use OpenMP tasks to distribute work the same way as with MPI. OpenMP and MPI can also be combined in a single application. Just make sure you are using MPI_THREAD_INIT instead of MPI_INIT.
A second possibility is to use memory mapped files. You can have all your data stored to disk (in a binary format) instead of loading everything into memory. With memory mapped files you can map a chunk of the file to memory and use normal pointers to get the information. You would have to remap portions of your file every now and then based on the region of the file you want to access. In general I would expect file access to be faster than MPI communication over a network. The Boost library provides a class for memory mapped files that is quite easy to use and portable between different operating systems.
- Badges
- Science topic