Sanitizers

Sanitizers are a compiler feature that support instrumenting programs to do dynamic analysis to catch many classes of bugs at runtime.

AddressSanitizer (ASan) is a fast memory error detector. It consists of a compiler instrumentation module and a run-time library. ASan is similar to valgrind (Valgrind) in that it can help identify memory errors. ASan is much faster than valgrind, but does require recompilation. Note that only GCC and Clang support sanitizers.

How-to

To use AddressSanitizer with Chapel-generated executables only:

export CHPL_MEM=cstdlib
export CHPL_TASKS=fifo
export CHPL_LLVM=none   # or set CHPL_TARGET_COMPILER to gnu, clang, etc.
export CHPL_SANITIZE_EXE=address
export ASAN_OPTIONS="use_sigaltstack=0,detect_leaks=0"

cd $CHPL_HOME
make
chpl <program.chpl>
./<program>

To use it with both the Chapel compiler and its generated executables:

export CHPL_MEM=cstdlib
export CHPL_TASKS=fifo
export CHPL_LLVM=none
export CHPL_HOST_MEM=cstdlib
export CHPL_SANITIZE=address
export ASAN_OPTIONS="use_sigaltstack=0,detect_leaks=0"

cd $CHPL_HOME
make
chpl <program.chpl>
./<program>

To get better stack traces when optimizations are enabled:

export DEBUG=1

Note

Non-chplenv environment variables aren’t propagated by paratest. So, to turn off leak checking, it is necessary to either pass -env ASAN_OPTIONS="use_sigaltstack=0,detect_leaks=0" or to include export ASAN_OPTIONS="use_sigaltstack=0,detect_leaks=0" in .bashrc or the equivalent.

Limitations

AddressSanitizer should be able to detect almost all memory errors for single locale configurations using CHPL_COMM=none. However, ASan cannot detect all classes of memory errors in multilocale configurations and the extent of the support depends on the CHPL_COMM setting. As a specific example some configurations will not be able to detect invalid remote reads/writes since ASan only knows about local memory. In the following program the invalid write to the array may not be detected for configurations that use RDMA to perform writes.

var A: [1..10] int;
on Locales[numLocales-1] {
  A[11] = 11; // Out of bounds write
}

This particular example would be caught by Chapel bounds checking, which are enabled by default but disabled with --fast. To get bounds checking and optimizations you can use --fast --bounds-checks or --fast --checks.

CHPL_COMM=gasnet with CHPL_COMM_SUBSTRATE=udp should be able to detect most memory errors since remote reads/writes are performed with active messages instead of RDMA.

CHPL_COMM=ugni without hugepages can detect some, but not all invalid remote reads/writes, but comes with a large performance cost. Hugepages can be unloaded with module unload $(module -t list 2>&1 | grep craype-hugepages)

Configuration Limitations

The above options are needed because not all third-party libraries support sanitizers. In particular:

  • Sanitizer instrumentation is added by the C compiler, so LLVM compilations don’t currently work

  • Sanitizers hook into the system allocator, so using jemalloc is not supported

  • qthreads performs task-switching in user-space, which throws off stack frame tracking. We expect to be able to resolve this in the future.

  • By default the gcc address sanitizer will enable leak checking, but Chapel intentionally leaks some memory in the runtime, so we disable that tracking for now. See Debugging Chapel Programs for more info about debugging memory leaks in Chapel.

  • An upstream bug can result in false-positives for some gcc versions. use_sigaltstack=0 works around this.

Other Sanitizers

Currently, only AddressSanitizer has been tested, but the options passed to CHPL_SANITIZE are passed directly to the backend compiler’s -fsanitize= option, so other sanitizers can be enabled. e.g. to use memory sanitizer and address sanitizer set CHPL_SANITIZE=address,memory