Having worked on Blue Gene, I can tell you that much time/effort was spent to minimize any latency issues with regards to MPI - in fact the machine was designed with MPI explicitly in mind. As I ported MPI programs from other platforms for potential customers and/or RFPs, the performance was always as good (or better) even though we had to run significantly more MPI tasks - because of the hardware limitations of Blue Gene.

It's always a good idea to have some key parameters of your problem tunable - say, number of cores to divide your grid on to, or some other geometric/topological parameter, that may affect the performance. Then you just make small time footprint version of your code (running a couple of minutes, for example), and on any new supercomputer you just optimise the parameter for best performance, or lowest cost or whatever you want using a small pre-computing.

MPI does have some topology routines that you can use to

create MPI communicators that are intended to be useful to MPI routines.

I'm not sure how much they are used.

I do know that Cray has made a great effort to optimize their MPI

collective routines, I assume that IBM and other major vendors have also,

but do not have direct experience.

Our group has reported any performance problems to vendors in terms of

1) we see this performance in your library routine, but

2) we can obtain better performance using an alternate version of

the collective routine written in terms of point-to-point routines.

One way to get good performance for collectives involving large messages

among large numbers of processors is to pipeline the process, and use the fact

that communication connection are generally bi-directional.

E.g.

To broadcast a large array of size NxN to p processors from process 0 to all others

1) Send the first row block (or column block for Fortran) from 0 to 1

2) Send 2nd block from 0 to 1

3) Simultaneously on proc 1, send 1st block , recv 2nd block withe sendrecv

4) Keep adding blocks to pipeline on proc 0 until all blocks have been sent,

5) Keep adding new processes to pipline till it is full.

This makes a broadcast which is O(1) in terms of the number of processors,

assuming

the message size is large enough so that the p communication latencies

is small compared to the time to send a block.

communication between proc k and k+1 is the same for all processes and

all adjacent processes can communicate in parallel. (A communication ring would suffice.) Latency goes up, due to the piplining, so this is not appropriate

for small messages.

This can all be coded in a MPI_Bcast_alternate subroutine/function and

you can compare it with your current MPI_Bcast

I did discover this on my own, but later found the result already published

for the Intel Delta machine.

A similar technique can be applied to all collective routines which send large

of data.

N.B. If you have a realy complex geometry and/or complex topology of computing nodes interconnection it's a good idea to utilize spetial library which optimizes your data exchange costs. I've used this one when faced to problem of this kind - http://www.labri.fr/perso/pelegrin/scotch/

For best performance, you should design your application as two-level (MPI as high-level, OpenMP as low-level of paralelism). On the low-level, you can use all features of shared memory (including mutexes, critical sections and so on). You should balance carefully the number of threads per one MPI node (it is usually not equal to the number of cores!). And then you should consider the influence of mapping of data among MPI nodes:

a) if the data are local (with small distance between MPI nodes), it usually implies that you should use point-to-point routines.

b) If the data are not local (with high distance between MPI nodes), it usually implies that you should use collective routines.

There are some additional optimizations: you can hide (some part of) latencies caused by MPI routines by clever usage of non-blocking version of these routines and so on.

Whether you can retain you application performance when moving from one platform to another depends on several factor. First, which functions are you using? The "usual suspects" are commonly optimized well, especially on large-scale systems like Blue Gene and Cray. On those platform, the regular collectives are usually tuned to the underlying hardware, and even some of the Cartesian topology functions might be tuned for specific cases.

If you walk off the beaten path, though, it depends whether your MPI implementer already had the time and manpower to invest in tuning that specific function. Sometimes it may be faster to pad data structures to enable use of well-tuned common functions than to use a "default" implementation of an uncommon counterpart. In the end, it will always be a combination of application, MPI implementation, hardware platform, and how well those play together.

Depending on the platform you are running, you also may experience different sweet spots in terms of number of processes and/or number of threads per process for your application. In general, a common thing to look for in MPI-parallelized applications is a decent load-balancing and the reduction of explicit and implicit process synchronization where possible.

There are some common optimizations for MPI programs.

1) reduce the number of synchronization points

2) prefer non-blocking versions of operations

3) overlap communication and computation

4) avoid the cumulation of communication (it is better if the communication load has a uniform distribution over time)

Their effect depend on the algorithm and on the topology, but these optimization hints are very general and some of them cannot be applied to the given algorithm

Thank you for opening this question.

I have experience with BlueGene/P (BG/P), especially with MPI collective communication operations (MPI broadcast and reduce) and parallel matrix multiplication algorithms, such as SUMMA and HSUMMA (I am not sure if in your question you are talking about SUMMA matrix multiplication algorithm or not, but I think my answer would be helpful.). In addition, my research is about optimization of MPI collectives in topology-oblivious way.

I would like to mention only two special cases without  going into much details of different broadcast algorithm selections on BG/P (more info can be found in IBM Redbooks). Most of the optimizations of the collectives on BlueGene/P are specific to its architectures:

1) MPI broadcast operation is most efficient if the communicator is MPI_COMM_WORLD. In this case BG/P MPI implementation will use physical tree topology and of course will beat any other algorithm. However, in the real world numerical libraries MPI_COMM_WORLD is hardly used as the part of the main communication operations.

2) If the communicator is rectangular then multi-color rectangle algorithm will be used for the broadcast operation. Usually this algorithm is the second most effective, especially for large messages.  However, in order to get rectangular communicator it is required to map the processes to the physical hardware carefully. Moreover, it highly depends on the allocated BG/P partition. Again, this case is also quite restrictive.

In some other cases, BG/P uses broadcast algorithms from MPICH, such as binomial and scatter-allgather based algorithms. Of course their efficiency again depends on the message size, topology and mappings of the processes to the physical hardware.

Therefore, in our research we try to introduce more universal approach to these kind of optimizations. We redesign MPI collective algorithms and parallel matrix multiplication algorithms (HSUMMA) in a hierarchical way. The hierarchy here is introduced into the communication patterns of the algorithms at the application/algorithm level without taking topology into account.

You can check our papers for more information and I would be happy to discuss more on this.

Best regards,

Khalid