I have a problem when coding using OpenMp on NUMA machines. The code does not scale well, because of having memory allocated on some nodes and accessed by other nodes.
I tried a lot of things but all tricks did not work as expected. It would be good if some one can lead me to a good article/book for beginners.
I don't have a reference for you, just 4 hints:
Use private variables whenever possible.
For shared variables, most openmp implementations adopt a first touch
allocation policy, so memory page for an array is allocated on the
memory closest to the processor that first writes to that portion of the array.
So it is best to allocate within a parallel do/for loop which access the data in
the same way that you will later.
Parallel dos/fors are best across rows of a fortran
2D matrix or rows of a 2D C matrix. This is because you are not accessing
adjacent elements on different processors which can lead to false sharing in cache lines and a similar problem in pages.
Use memory and processor affinity where possible to keep data next to the processor that used it earlier.
@James - I think you meant to say *columns* of a 2D C matrix.
Yes, columns in C. Multi-tasking is getting harder these days.
Thanks, Jerry
The first touch policy is not a property of the OpenMP implementation but of the OS Kernel. Our book (http://goo.gl/QSwJ2x) contains a comprehensive chapter on ccNUMA optimizations.
It depends of your program.
If it does not scale well it means that your problem is memory bound. The peak performance of your program is limited by memory bandwidth and not by processing power.
See the youtube video of Tim Mattson (4:30 for schedule clause).
https://www.youtube.com/watch?v=jdYHVeh2wEI
For more details see OpenMP documentation.
https://www.youtube.com/watch?v=jdYHVeh2wEI
https://software.intel.com/en-us/articles/openmp-thread-affinity-control
https://gcc.gnu.org/onlinedocs/libgomp/GOMP_005fCPU_005fAFFINITY.html
- Badges
- Science topic
- Similar topics
- Filing