Chapel Launchers

When compiling Chapel programs for multiple locales, a launcher binary is typically created that will execute the appropriate command(s) to get your program started. For example, when compiling for multiple locales, typically two binaries will be generated by the compiler (e.g., a.out and a.out_real). The first binary contains code to get your program up and running on multiple locales while the second contains your actual program code.

The goals of the launcher binary are:

  1. to wrap details of job startup in a portable way so that new users can quickly get Chapel programs up and running on an unfamiliar platform.
  2. to perform command-line parsing and error checking prior to waiting in a queue or firing off a parallel job in order to save time and resources related to simple errors/typos in the command line.
  3. to preserve Chapel’s global-view programming model by permitting the user to run their program using a single binary (corresponding to the single logical task that executes main()) without getting bogged down in questions of numbers of nodes, numbers of cores per node, numbers of program instances to start up, etc.
  4. if necessary, to coordinate runtime functional activity, such as I/O.

Executing a Chapel program using the verbose (-v) flag will typically print out the command(s) used to launch the program.

Executing using the help (-h/--help) flag will typically print out any launcher-specific options in addition to the normal help message for the program itself.

Currently supported launchers include:

Launcher Name Description
amudprun GASNet launcher for programs running over UDP
aprun Cray application launcher using aprun
gasnetrun_ibv GASNet launcher for programs running over Infiniband
gasnetrun_mpi GASNet launcher for programs using the MPI conduit
lsf-gasnetrun_ibv GASNet launcher using LSF (bsub) over Infiniband
pbs-aprun Cray application launcher using PBS (qsub) + aprun
pbs-gasnetrun_ibv GASNet launcher using PBS (qsub) over Infiniband
slurm-gasnetrun_ibv GASNet launcher using SLURM over Infiniband
slurm-srun native SLURM launcher
none do not use a launcher

A specific launcher can be explicitly requested by setting the CHPL_LAUNCHER environment variable. If left unset, a default is picked as follows:

  • if CHPL_PLATFORM is cray-xc, cray-xe, cray-xk, or cray-xt:

    If CHPL_LAUNCHER
    both aprun and srun in user’s path none
    aprun in user’s path aprun
    srun in user’s path slurm-srun
    otherwise none

  • otherwise, if CHPL_COMM is gasnet:

    If CHPL_LAUNCHER
    CHPL_COMM_SUBSTRATE=ibv gasnetrun_ibv
    CHPL_COMM_SUBSTRATE=mpi gasnetrun_mpi
    CHPL_COMM_SUBSTRATE=mxm gasnetrun_ibv
    CHPL_COMM_SUBSTRATE=udp amudprun
    otherwise none

  • otherwise CHPL_LAUNCHER is set to none

If the launcher binary does not work for your system (due to an installation-specific configuration, e.g.), you can often use the -v flag to capture the commands that the launcher executes on your behalf and customize them for your needs.

Forwarding Environment Variables

Chapel launchers generally arrange for environment variables to be forwarded to worker processes. However, this strategy is not always reliable. The remote system may ovrride some environment variables, and some launchers might not correctly forward all environment variables. In particular, the amudprun launcher will not forward environment variables containing backticks (`).

_real binary suffix for execution

In order to support profiling tools that produce new binaries for the launcher to execute, the suffix of the real binary executed by the launcher may be changed with the CHPL_LAUNCHER_SUFFIX environment variable. If this variable is unset, the suffix defaults to “_real”, matching the compiler’s output.

Bypassing the launcher

If the Chapel launcher capability fails you completely, set CHPL_LAUNCHER to none, recompile, and execute the a.out binary according to the following rules using tools and queueing mechanisms appropriate for your system:

  • on most systems, the number of locales should be equal to the number of nodes on which you execute which should be equal to the number of copies of a.out (or a.out_real using a launcher) that you are running.
  • some queueing systems require you to specify the number of cores to use per node. For best results, you will typically want to use all of them. All intra-node parallelism is typically implemented using Chapel’s threading layer (e.g., pthreads), so extra copies of the binary are not required per core.

SLURM notes

Prerequisites:

To use native SLURM, set:

export CHPL_LAUNCHER=slurm-srun

This will happen automatically if srun is found in your path, but not when both srun and aprun are found in your path.

To use SLURM using GASNet over Infiniband, set:

export CHPL_LAUNCHER=slurm-gasnetrun_ibv
export CHPL_COMM=gasnet
export CHPL_COMM_SUBSTRATE=ibv

To have GASNet use mpirun to launch your program, set:

export GASNET_IBV_SPAWNER=mpi

At this point, rebuild your Chapel runtime (‘make’ from $CHPL_HOME), and compile your program.

Settings:

Before running, you will need to set the amount of time to request from SLURM. For example, the following requests 15 minutes:

export CHPL_LAUNCHER_WALLTIME=00:15:00

If needed, you can request a specific node feature from SLURM by putting it in the CHPL_LAUNCHER_CONSTRAINT environment variable. For example, to use nodes with the ‘cal’ feature (as defined in the slurm.conf file), set:

export CHPL_LAUNCHER_CONSTRAINT=cal

If this environment variable is undefined, SLURM may use any node in the computer.

If the environment variable CHPL_LAUNCHER_USE_SBATCH is defined then sbatch is used to launch the job to the queue system, rather than running it interactively as usual. In this mode, the output will be written by default to a file called <executableName>.<jobID>.out. The environment variable CHPL_LAUNCHER_SLURM_OUTPUT_FILENAME can be used to specify a different filename for the output.

Other SLURM variables will have an impact; for example, the SLURM partition can be set to ‘debug’ with the command:

export SLURM_PARTITION=debug

Additional launchers

In addition to the launchers listed above there are several others that are not actively maintained but may still work.

Launcher Name Description
loadleveler launch using IBM loadleveler (still needs refining)
marenostrum launch using MareNostrum’s mnsubmit script
mpirun launch using mpirun (no mpi comm currently)

These launchers are the default for the following configurations:

If CHPL_LAUNCHER
CHPL_PLATFORM=marenostrum marenostrum