You generally don't need to do this. Modern MPIs are node-aware and will

use shared memory when possible.

There can be a slight advantage in memory use with the hybrid code when you

have a large array that needs to be read by all the OpenMP threads. Then you do not

have to replicate it, all threads can share the same array, reducing the memory footprint. No control of access to the array is needed when the array is only

read, not written, within a parallel region.

Using MPI, I would use either the -rr (round robin) or the -perhosts option

to mpirun.

For running 12 MPI processes on 6 nodes each with 16 cores, you wioulod request 6 nodes with all 16 cores on the scheduler reservation, but in the mpirun command you would issue

mpirun --perhost 2 -np 12 ./program

to start two MPI processes per node with process 0 and 1 on the first node,

2,3 on the next nodes, etc.

mpirun --rr -np 12 ./program

would also place 2 processes per node, but in a different order:

0,6 on the first node, etc.

OpenMPI also has similar flags.

You might look into process and memory pinning which can be another reason

to use hybrid with 2 MPI processes on a two CPU socket node configuration.

Thank you, James. I didn't know about the perhost and round robin options of MPI.

The reason to use a hybrid approach is indeed the memory footprint. In our current cluster we have lots of old nodes with only 8GB of RAM and a couple of newer nodes. This leaves only 2GB per process on a quad-core machine. But, also for the newer nodes the RAM per core is not sufficient to launch one MPI process per core.

Unfortunately, the perhost option would not always work, and I am not sure about the round robin option. Our cluster has older machines with only 4 cores and newer ones with 12 cores (+ Hyperthreading). When I request 4 cores per node I might get several reservations on the newer nodes (up to 6 nodes with 4 cores each on a single physical computer). I'm not exactly sure how the perhost and round robin options work, but my guess is that they take the host name to decide what "perhost" means.

Since we deliver the software to a few clients the software itself should be smart enough and decide how it should be configured in hybrid mode. And our clients also have different kinds of computers in their cluster. This is why I chose this rather complicated approach. (In my defense, I didn't expect MPI barrier to burn CPU cycles.)

Do you use Torque or PBS as the scheduler,

if so do you use Maui or pbs_sched as the scheduler part of Torque?

you can issue

ps -aef | grep pbs

and

ps -aef | grep maui

on the head nofe of the cluster to see about this.

Do the requests look like:

-lnodes=4,ppn=12

?

We are using Torque as resource manager and MOAB for the job scheduling.

Requests look indeed like

-l nodes=4:ppn=12

Usually, I also request an interactive shell with the -I (

The question about the sceduler was an attempt to see if there was a way to

request the nodes such that you knew which nodes you needed.

That is not germane if what you really want is software distributed to run uner

other schedulers.

For Intel MPI, you can set the environmental variable

I_MPI_WAIT_MODE=off

this should lessen the impact of polling on CPU time.

I don't think that OpenMPI has such an option, at least one that works.

If you just want some MPI processes to wait at the end of the program just beofre

the MPI_Finalize, you could use a do while loop with an a call to the C sleep(5) command followed by an iwait in it, though you would need a fortran 2003 or

better compiler and need to use the C bindings that the f2003 standard introduced.

Examples at:

http://software.intel.com/sites/products/documentation/hpc/composerxe/en-us/2011Update/fortran/lin/bldaps_for/common/bldaps_interopc.htm

I would just like to point out that this problem is very much depending on architectures: whatever general solution you would find that works fine on one architecture will have problems on another one.

My personal approach here, would be to keep the code general, that is, do not rely on the fact that it has to be used in Hybrid mode. Then, you could decompose to your liking. As James pointed out, the memory should not be so much of a problem, because MPI uses the shared memory anyhow. So, choosing however many threads suit your architecture best should be the solution. Then, the only issues you are left with are

(1) the placement of the MPI-Tasks and

(2) the placement of the OpenMP Tasks

Concerning (1)

I would think that it is hard, to find a generally (independent of MPI-Version, Batch-Scheduler and Hardware) working way to do this. Here, in my opinion, the load-leveler is the best way to go. But it would involve asking your customer to pass proper options to the load-leveler

Concerning (2)

I think, this is even harder to figure out generally, because here the load-leveler and MPI-Library will have to play perfectly together in terms of (a) knowing what you want, that is, parsing the options properly and (b) knowing what your system looks like. Even the optimum in terms of performance is not always clear, but in mostly the round-robin

using every n-th available core for an MPI-task, where n is the number of OpenMP threads works fine. This, however, depends on how the cores have to access the network: Often CPUs are built such that some core have faster access to the network. These, of course, should be used for MPI then...

@James

I didn't know about I_MPI_WAIT_MODE yet. But, from the manual I take it that it should be set to 'on'. 'off' is the default value and will always do active polling. There is one restriction, though: It does not support InfiniBand devices.

I have been looking into a similar option as you are describing, but using mostly portable FORTRAN for sleeping: We are using the Intel compiler and also the software has to run with Windows, not just Linux. Here is what I do:

do

call SLEEPQQ(60000) ! sleep one minute

call MPI_IPROBE(MPI_ANY_SOURCE, MPI_ANY_TAG, &

MPI_COMM_WORLD, flag, tmp, ierror)

if (flag .eq. .true.) exit

end do

call MPI_Finalize()

Then the master process just needs to send any message and the sleeping thread will eventually finish. This works. But, I am curious if there is a more elegant solution to this problem that does not involve active polling.

@Cedrick

MPI can only use shared memory for communication which will make communication faster. But, because of the programming model each process will have to load the same data. There is no shared memory for data in MPI. This is the advantage of OpenMP which will decrease memory footprint. And this is the only reason why we consider a hybrid approach.

We will always keep the pure MPI version and pure OpenMP version of our software as well.

BTW, I know about processor pinning. But, this is not the problem I am concerned with. It is quite easy to figure out for Intel MPI and Intel OpenMP. The real question is how do I make my application smart enough to configure itself for hybrid use. I thought about this a lot and I think that it is definitely necessary to first start one MPI process per core. Then I need to figure out how many MPI processes should participate in the computation and which ones those are. These MPI processes will later split work on OpenMP threads. The only question is what to do with the remaining MPI processes that are not necessary for computation.

Simon MPI _WAIT_MODE is more likely for asynchronius call.

Of course it depends on a task how one can use mixed MPI/OpenMP techniques on a Fortran, for example you can do Jacobi iterations method for every three strings with OpenMP parallel than passing it to MPI to proceed in parallel.

!$OMP& REDUCTION(+:x)

!$omp do i 2,N-1

x(i,j) = 0.25*(x(i-1,j-1)+x(i-1,j+1)+x(i+1,j-1)+x(i+1,j+1))

and then do like metioned in previous comment. I am not the best in MPI, you know.