Not familiar with "OMP"? What does that stand for?

If you are dealing with only one computer and that computer has 16 cores, why not just spawn threads? Threads share the same program memory.

While MPI spawns processes that do not share the same program memory, each process can certainly spawn threads. Each thread shares the same program memory assigned to their spawning process.

In both cases, processes that share and processes that do not share the same program memory require synchronization. Within threads, access to memory has to be managed so that memory being accessed by one process is not, at the same time, being modified by another process.

Let's continue to discuss what you are trying to accomplish. Glad to help out.

Peter Raeth

Thank you !

I mean OMP (Open multi-processing).

When I try to parallelize with OpenMP, synchronization is not allowed inside 'for loop'.

Let's say I have 100x100 mesh grid and I choose pixels and calculate something. The problem is that each calculation have affect on the next calculation which means that, each pixel has its own probability to be chosen and calculation changes pixel's probability. When I parallelize it, I choose 16 pixels(#of cores) at once and calculate them(this change probability). At the next step, another 16 pixels are chosen depending on the new probability. This is why I need to synchronize inside 'for loop'.

But the problem is that it is not allowed. Once threads spawn and go through loop, it never stops or synchronized in the middle of loop.

Code looks like this

Thank you again!!

My best experiences have been with the MPICH version of MPI: https://www.mpich.org. I don't see any reason why synchronization inside a for-loop would be a problem with that tool. If you like, I can set up a simple case here and give it a whirl, based on your code segment above.

It is worth saying that MPICH/MPI is designed for processes that do not share program memory. Threads share program memory. Your code segment mentions threads. Which do you prefer?

I need is 'shared memory' and 'multiprocessing' , thread !

TLDR: Use OpenMP for easy coding in a single node. Use MPI for multinode computing.

I would suggest using OpenMP if you're not planning to scale to more than a single node. OpenMP makes the coding easier by allowing you to parallelize your C/C++ or Fortran code using pragmas (the code looks just like your serial code but with some minimal extra lines). If you want the easiest path to get a code working in the shortest amount of time, that is the way to go. It is supported by pretty much all compilers. Now, if you're planning to extend the code to multiple nodes, then MPI is the way to go. You can have hybrid code that uses MPI and OpenMP together, but that is more cumbersome to manage. So I would suggest to commit to just one or the other depending on what you want to do long term.

Ok. So MPI is not the right solution. Let me put an example together that might shed some light.

Peter Raeth Hi Peter! I would actually argue MPI is the way to go. Performance wise is comparable to OpenMP (or pthreads) and you get the scalability as an added bonus. I think once people get a parallel code working, they just push it until their system is not enough, then they have to refactor it again to allow for multinode computing. Why not just do multinode from the get go? It takes a little bit more work, but the pay off in the long run is more significant. I suggested OpenMP assuming the poster just wants something quick and dirty that has acceptable performance, but if they're willing to go the extra mile then MPI is the best choice all around.

MPI does not offer shared program memory for ALL processes. The need is for ALL processes to share the same program memory, as I understand things. There is only one node, which has 16 cores. The example I am working on uses language-standard threads and mutexes.

Ultimately, it is true that MPI could spawn the resulting program on multiple nodes and a component of the datablock sent to each node's main process. A coordinator process would then accumulate final results.

At issue is the case when a required pixel exists on another node. Then, some means is needed to access that pixel for reading and writing, all the while, in any case, protecting from simultaneous writes or reading while writing.

True, the program by default won't support shared memory and explicit communication using the MPI functions would be needed. But that is the beauty of it! Because the library is agnostic of the underlying hardware, it will work right away on 1 node or 100 nodes (assuming things are coded correctly). With OpenMP or threads the code becomes limited by the number of cores on the computer. 16 cores may be enough for Taehyun Kim but it is a short sighted solution, in my opinion. I've never heard anyone say "I'm comfortable with 16 cores even if I have access to hundreds or more". The usefulness of HPC is in being able to run larger problems with more computing resources. I'm not saying threads aren't the right option, but in my experience with general scientific simulations (e.g., materials, fluids, hydrodynamis, etc) it is an insufficient tool.

As the project proceeds and more equipment becomes available, a network of machines can be added. Each machine can start by reporting its number of cores and its computational ability. Then the program can be spawned on each and a chip sent. That will be no problem. The software is easily expandable. For now, I am concentrating on the basic algorithm and single-node use. That does not preclude use on a network of machines.

To that end, attached is a zip file that illustrates two basic approaches for mutexes and matrix manipulation via shared memory. See the .txt file at the zip root. Wanted to visualize the result but I have to work further with OpenCV, an open-access image analysis tool. Let's discuss further since the attached is just a framework and likely does not reflect the entire project's needs. I tried to follow the framework. Much more can be done.

MPI, in contrast to OPENMP, does not provide shared memory communication, since MPI is a parallel platform for communication between various machines. MPI enables the transfer of data across threads using the message passing paradigm. Several commands, such as send(), receive(), scatter(), and so on, have been added to facilitate data transmission across threads running on separate computers.

Each node's MPI process can spawn threads. Those threads do share memory. If a unique chip is sent to each node and each process is guaranteed to process a unique pixel, the threads still play. At issue is when calculations on one pixel depend on the contents of other pixels and whether those pixels exist on other nodes or the same node. There are open-source tools that simulate internode shared memory.

Attached is an expansion of the previous code for one machine processing an RGB image. See the Notes file. Going to wait now for further input from Taehyun Kim.

Thank you so much all of you to consider my issue, Peter Raeth , Pedro Bello-Maldonado and Javad Dogani .

Actually, I have made parallelized code, MPI version. But tremendous data are transferred so that computation time costs a lot... So I tried shared memory(I think this is the best) but there is still a problem; synchronization is not possible inside a loop. I thought this is technical problem, not intrinsic. So I moved on to find shared memory with MPI or other stuff. But it seems that's not what I've been looking for.

It seems conclusion by your helps seems to be finding another way of using multi-thread for my code, eventhough Pedro suggests new aspect of MPI but it seems this is beyond my ability...

I will read files attached by Peter and post my example or pseudo code.

I have one more question about multithreading.

Is is possible to access a certain array with multiple thread simultaneously??

I mean there are many elements in an array like array[0], array[3] .... and each element assign its own memory. I don't mean access to a same element. I mean access to array of different elements.