Using Chapel with GASNet

What is GASNet?

GASNet is a one-sided communication and active message library being developed by Lawrence Berkeley National Laboratory and UC Berkeley. For details, refer to the GASNet website.

Setting CHPL_COMM_SUBSTRATE

Users can set CHPL_COMM_SUBSTRATE to indicate the GASNet conduit that they wish to use. Conduits are alternative implementations of the GASNet library. GASNet uses different conduits to support different networks. Novice users can leave this unset and Chapel will make a choice for them.

Most settings for CHPL_COMM_SUBSTRATE rely on the particular network hardware. The options include:

ibv

OpenIB/OpenFabrics Verbs for InfiniBand (see Using Chapel with InfiniBand)

udp

UDP - portable conduit, works on any network with a TCP/IP stack (see Using the Portable UDP Conduit)

mpi

MPI - portable conduit, works on any network with MPI 1.1 or newer (see Using the GASNet MPI Conduit)

smp

Runs multiple co-locales on a single shared-memory machine (see Multilocale Execution with Shared-Memory)

ofi

OpenFabrics Interface (libfabric) for Slingshot and Omni-Path.

ucx

Unified Communication X for InfiniBand. Chapel does not actively support this substrate and its use is not recommended.

See the GASNet website for more information on each of these conduits.

Current defaults are:

CHPL_TARGET_PLATFORM	CHPL_COMM_SUBSTRATE
cray-cs	ibv
pwr6	ibv
other	udp

Setting CHPL_GASNET_SEGMENT

Users can set CHPL_GASNET_SEGMENT to choose a memory segment to use with GASNet. A GASNet segment is a region of memory that is expected to be used for remote memory access. The GASNet library works to make memory in this segment available for accelerated memory access supported directly by network hardware. The options are:

everything

All memory is available for remote memory access. This option is not supported with the smp or ucx substrates. If using the ofi substrate, this option is not supported when targeting the cxi provider (Slingshot).

fast

A limited portion of memory is available and optimized for fastest remote memory access

large

As with fast, but a larger amount of memory is available for communication

Each choice of segment has different tradeoffs. For the fast segment, the Chapel heap is entirely in memory that can be directly accessed over the network, but the drawback is that the size of the heap must be specified at program start-up. For everything, accessing memory over the network is generally slower, but there is no need to worry about the size of the heap or whether some memory is registered with the network. The large segment offers a compromise between these two options, where some portion of the heap will be available for faster network access.

Current defaults are:

CHPL_COMM_SUBSTRATE	CHPL_GASNET_SEGMENT
ibv	large
ofi	fast (only when the CHPL_TARGET_PLATFORM is a hpe-cray-ex or hpe-cray-xd)
smp	fast
ucx	fast
other	everything

Multilocale Execution with Shared-Memory

When running Chapel programs with CHPL_COMM=gasnet, you can use the smp conduit to have multilocale execution on a single shared-memory machine. This emulates a multinode environment by partitioning the machine’s resources (similar to Co-locales). This allows you to run Chapel programs that are designed for multinode execution without needing a distributed system or network support. To do this, set the environment variable CHPL_COMM_SUBSTRATE to smp before running your Chapel program:

export CHPL_COMM=gasnet
export CHPL_COMM_SUBSTRATE=smp

Using Chapel in this way will partition the machine’s resources into multiple locales.

The smp conduit is designed for shared-memory execution and is very similar to Co-locales. Notably, when specifying the number of locales the following commands are equivalent:

./myprogram -nl 4
./myprogram -nl 1x4
./myprogram -nl 4x1

All of these will result in 4 locales running on a single node/machine, partitioning the resources of that node between them.

Emulating Distributed Execution with the UDP Conduit

While the UDP conduit is primarily intended for portable multilocale execution over ethernet, it can also be used to run multiple locales locally on a single node by oversubscribing the machine’s resources.

With a build of Chapel with CHPL_COMM=gasnet and CHPL_COMM_SUBSTRATE=udp, you can run a Chapel program on multiple locales on a single machine with the following environment variable settings:

export GASNET_SPAWNFN=L
export GASNET_ROUTE_OUTPUT=0
export GASNET_MASTERIP=127.0.0.1
export GASNET_WORKERIP=127.0.0.0
export CHPL_RT_OVERSUBSCRIBED=yes

See the documentation for the UDP conduit for more details on what these environment variables do.

When should I use CHPL_COMM_SUBSTRATE=udp vs CHPL_COMM_SUBSTRATE=smp?

While both the udp and smp conduits can be used to run multilocale Chapel programs on a single node or shared memory machine (e.g. laptops or workstations), there are a few key differences:

• udp creates multiple locales on a single node that oversubscribe the machine’s resources, so each locale is sharing the same resources. smp creates multiple co-locales which partition the machine’s resources between them, so each locale has its own dedicated resources.

• udp (by default with CHPL_GASNET_SEGMENT=everything) implements communication between locales over the TCP/IP network, so all traffic is routed through the network stack. smp always uses shared memory for communication between co-locales, which can be more efficient and requires less configuration.

• udp always requires a working network configuration to start jobs and can sometimes require additional configuration tweaks to successfully spawn jobs in the presence of firewalls or misconfigured DNS services. smp does not rely on any network services.

Using the GASNet MPI Conduit

To use MPI directly with GASNet, you must set the CHPL_COMM_SUBSTRATE environment variable to mpi and rebuild Chapel. This will configure Chapel to use the MPI conduit of GASNet, which allows you to run Chapel programs using the MPI library for communication between locales.

For best practices about how to configure/use GASNet to avoid such conflicts with MPI, please see the GASNet docs for the MPI Conduit (also available at $CHPL_HOME/third-party/gasnet/gasnet-src/mpi-conduit/README).

Troubleshooting

For CHPL_COMM=gasnet runs, you may also want to consider setting one or both of:

export GASNET_SPAWN_VERBOSE=1
export GASNET_VERBOSEENV=1

where the first prints more information about GASNet’s job launch actions, and the second is used to audit environment variable settings.

When running CHPL_COMM=gasnet programs using the udp conduit, we’ve had best results with console I/O using:

export GASNET_ROUTE_OUTPUT=0