MPI¶

MPI Version¶

This module implements the C-API for the MPI 1.1 standard. It currently does not support the following routines

MPI_Op_create
MPI_Op_free
MPI_Errhandler_create
MPI_Errhandler_free
MPI_Keyval_create
MPI_Keyval_free

since all of these are built around user-defined handlers, that is not supported.

This module also includes a few functions that are in later MPI standards. These are

MPI_Init_thread (and related constants)
MPI_Ibarrier

Compiler wrappers¶

Using MPI requires pointing the compiler to the location of the MPI headers, and including the appropriate MPI libraries. This is often done by using compiler wrappers like "mpicc". Setting CHPL_TARGET_COMPILER=mpi-gnu and CHPL_TARGET_CPU=none will get Chapel to use mpicc and assume that it works like gcc does.

Note that on Cray systems, this is not necessary, since the default compilers include the necessary MPI information.

Interoperability Modes¶

There are two modes in which Chapel and MPI can interoperate with one another:

CHPL_TARGET_COMPILER=cray-prgenv-gnu
CHPL_TASKS=fifo
CHPL_COMM=none
MPICH_MAX_THREAD_SAFETY=multiple

CHPL_TARGET_COMPILER=cray-prgenv-{gnu, intel}
CHPL_TASKS={qthreads, fifo}   # see discussion below
CHPL_COMM={ugni, gasnet}
CHPL_COMM_SUBSTRATE={aries, mpi}   # if CHPL_COMM=gasnet
MPICH_MAX_THREAD_SAFETY=multiple
AMMPI_MPI_THREAD=multiple         # if CHPL_COMM_SUBSTRATE=mpi

Qthreads and MPI¶

Since MPI is not natively Qthread-aware, some care is required to avoid deadlocks. This section describes current recommendations on using CHPL_TASKS=qthreads and the MPI module.

We assume that CHPL_COMM is either ugni or gasnet+aries. We do not recommend using the MPI module with the gasnet+mpi communication backend and qthreads.

1. Use non-blocking calls whenever possible. Note that this also requires using MPI_Test instead of MPI_Wait. For convenience, we provide wrappers for a subset of the MPI blocking calls that are implemented with non-blocking calls and correctly yield tasks while waiting.
2. Any blocking calls (including third-party libraries) must be preceded with a call to Barrier.
3. Blocking calls must be serialized; concurrent blocking calls can result in deadlocks.

Configurations Constants¶

This module uses two boolean config constants: autoInitMPI and requireThreadedMPI. Refer to their documentation for more information on when to use them.

Communicators¶

The GASNet runtime, and therefore Chapel, makes no guarantees that the MPI ranks will match the GASNet locales. This module creates a new MPI communicator CHPL_COMM_WORLD that ensures that this mapping is true. Note that if numLocales is 1, CHPL_COMM_WORLD is identical to MPI_COMM_WORLD, which is the desired behaviour for SPMD mode programs.

MPI Module Documentation¶

config const autoInitMPI = true¶

Automatically initializes MPI (if not already initialized by the runtime), and shuts it down as well.

config const requireThreadedMPI = true¶

Ensures that MPI is running in MPI_THREAD_MULTIPLE mode, and will abort if not. This is not necessary if you are running SPMD mode, but is likely to be necessary if you run in multilocale mode.

proc CHPL_COMM_WORLD¶

Define a new communicator that directly maps to Chapel locales. This is just a reordering of MPI_COMM_WORLD, which you are, of course, free to continue to use. This just guarantees that locale ids and MPI ranks agree.

proc initialize()¶

Helper routine that also sets process IDs (world rank) and number of processes (world size)

proc commRank(comm: MPI_Comm = MPI_COMM_WORLD): c_int¶

Wrapper to get the rank of the communicator comm. Defaults to MPI_COMM_WORLD, if no communicator is passed in.

proc commSize(comm: MPI_Comm = MPI_COMM_WORLD): c_int¶

Wrapper to get the size of the communicator comm. Defaults to MPI_COMM_WORLD, if no communicator is passed in.

proc Wait(ref request: MPI_Request, ref status: MPI_Status): c_int¶

Drop in replacement for MPI_Wait, implemented with non-blocking MPI calls.

This is simply implemented as a while loop that continually calls MPI_Test. The loop will yield, allowing other tasks to run.

proc Wait(ref request: MPI_Request): c_int

Overloaded version of Wait, which ignores the returned status

proc Barrier(comm: MPI_Comm): c_int¶

Drop in replacement for MPI_Barrier, with non-blocking MPI calls.

This is implemented by a call to MPI_Ibarrier, followed by a call to Wait above. The returned value of MPI_Status is ignored.

proc Send(ref buf, count: c_int, datatype: MPI_Datatype, dest: c_int, tag: c_int, comm: MPI_Comm): c_int¶

Drop in replacement for MPI_Send, with non-blocking MPI calls.

This is implemented by a call to MPI_Isend, followed by a call to Wait above. The returned value of MPI_Status is ignored.

proc Recv(ref buf, count: c_int, datatype: MPI_Datatype, source: c_int, tag: c_int, comm: MPI_Comm, ref status: MPI_Status): c_int¶

Drop in replacement for MPI_Recv, with non-blocking MPI calls.

This is implemented by a call to MPI_Irecv, followed by a call to Wait above. The returned value of MPI_Status is ignored.

record MPI_Status¶

A wrapper around MPI_Status. Only the defined fields are exposed

var MPI_SOURCE: c_int¶

var MPI_TAG: c_int¶

var MPI_ERROR: c_int¶

proc MPI_Status.getCount(tt: MPI_Datatype)¶

Get the count from a status object

type MPI_Aint = c_ptrdiff¶

type MPI_Group¶

type MPI_Comm¶

type MPI_Datatype¶

type MPI_Request¶

type MPI_Op¶

const MPI_THREAD_SINGLE: c_int¶

const MPI_THREAD_FUNNELED: c_int¶

const MPI_THREAD_SERIALIZED: c_int¶

const MPI_THREAD_MULTIPLE: c_int¶

type MPI_Errhandler = opaque¶

const MPI_SUCCESS: c_int¶

const MPI_ERR_BUFFER: c_int¶

const MPI_ERR_COUNT: c_int¶

const MPI_ERR_TYPE: c_int¶

const MPI_ERR_TAG: c_int¶

const MPI_ERR_COMM: c_int¶

const MPI_ERR_RANK: c_int¶

const MPI_ERR_REQUEST: c_int¶

const MPI_ERR_ROOT: c_int¶

const MPI_ERR_GROUP: c_int¶

const MPI_ERR_OP: c_int¶

const MPI_ERR_TOPOLOGY: c_int¶

const MPI_ERR_DIMS: c_int¶

const MPI_ERR_ARG: c_int¶

const MPI_ERR_UNKNOWN: c_int¶

const MPI_ERR_TRUNCATE: c_int¶

const MPI_ERR_OTHER: c_int¶

const MPI_ERR_INTERN: c_int¶

const MPI_PENDING: c_int¶

const MPI_ERR_IN_STATUS: c_int¶

const MPI_ERR_LASTCODE: c_int¶

const MPI_BOTTOM: opaque¶

const MPI_PROC_NULL: opaque¶

const MPI_ANY_SOURCE: opaque¶

const MPI_ANY_TAG: opaque¶

const MPI_UNDEFINED: opaque¶

const MPI_BSEND_OVERHEAD: opaque¶

const MPI_KEYVAL_INVALID: opaque¶

const MPI_ERRORS_ARE_FATAL: MPI_Errhandler¶

const MPI_ERRORS_RETURN: MPI_Errhandler¶

const MPI_MAX_PROCESSOR_NAME: c_int¶

const MPI_MAX_ERROR_STRING: c_int¶

const MPI_CHAR: MPI_Datatype¶

const MPI_SHORT: MPI_Datatype¶

const MPI_INT: MPI_Datatype¶

const MPI_LONG: MPI_Datatype¶

const MPI_UNSIGNED_CHAR: MPI_Datatype¶

const MPI_UNSIGNED_SHORT: MPI_Datatype¶

const MPI_UNSIGNED: MPI_Datatype¶

const MPI_UNSIGNED_LONG: MPI_Datatype¶

const MPI_FLOAT: MPI_Datatype¶

const MPI_DOUBLE: MPI_Datatype¶

const MPI_LONG_DOUBLE: MPI_Datatype¶

const MPI_BYTE: MPI_Datatype¶

const MPI_PACKED: MPI_Datatype¶

const MPI_FLOAT_INT: MPI_Datatype¶

const MPI_DOUBLE_INT: MPI_Datatype¶

const MPI_LONG_INT: MPI_Datatype¶

const MPI_2INT: MPI_Datatype¶

const MPI_SHORT_INT: MPI_Datatype¶

const MPI_LONG_DOUBLE_INT: MPI_Datatype¶

const MPI_LONG_LONG_INT: MPI_Datatype¶

const MPI_UB: MPI_Datatype¶

const MPI_LB: MPI_Datatype¶

const MPI_COMM_WORLD: MPI_Comm¶

const MPI_COMM_SELF: MPI_Comm¶

const MPI_COMM_NULL: MPI_Comm¶

const MPI_IDENT: c_int¶

const MPI_CONGRUENT: c_int¶

const MPI_SIMILAR: c_int¶

const MPI_UNEQUAL: c_int¶

const MPI_TAG_UB: c_int¶

const MPI_IO: c_int¶

const MPI_HOST: c_int¶

const MPI_WTIME_IS_GLOBAL: c_int¶

const MPI_MAX: MPI_Op¶

const MPI_MIN: MPI_Op¶

const MPI_SUM: MPI_Op¶

const MPI_PROD: MPI_Op¶

const MPI_MAXLOC: MPI_Op¶

const MPI_MINLOC: MPI_Op¶

const MPI_BAND: MPI_Op¶

const MPI_BOR: MPI_Op¶

const MPI_BXOR: MPI_Op¶

const MPI_LAND: MPI_Op¶

const MPI_LOR: MPI_Op¶

const MPI_LXOR: MPI_Op¶