.. index:: single: debug single: debugging single: debugging; gdb single: debugging; lldb .. _readme-debugging: ========================= Debugging Chapel Programs ========================= This document discusses support for debugging your Chapel program and a set of experimental settings to enable task monitoring and memory tracking. .. contents:: ------------------- Running in gdb/lldb ------------------- The compiler-generated executable has a ``--gdb`` flag that can be used to launch the program within a ``gdb`` session. A similar flag, ``--lldb``, exists to launch the program within a ``lldb`` session. For best results, you should read :ref:`this section ` to build Chapel and build your application. Running in gdb/lldb with a launcher ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When using almost any launcher, you can launch ``gdb`` by setting the environment variable ``CHPL_COMM_USE_GDB`` when running the program. This will open up a separate terminal emulator window for each locale, each running the debugger on that locale's program instance. On the Mac OS X (darwin) platform, you can launch ``lldb`` instead, by setting the ``CHPL_COMM_USE_LLDB`` environment variable. This works in all of these launchers:: amudprun aprun gasnetrun_ibv gasnetrun_mpi gasnetrun_ofi mpirun4ofi pbs-aprun smp The default terminal emulator program is ``xterm``, but by setting the environment variable ``CHPL_COMM_DBG_TERM=urxvt`` you can force use of ``urxvt`` instead. Whichever terminal emulator is used must be in your ``PATH`` on the compute node or an error will result. Note that it is the user's responsibility to make sure things are set up so the terminal emulator run in the target environment can open its display window in the launch environment. Using a graphical debugger ~~~~~~~~~~~~~~~~~~~~~~~~~~ The Chapel VSCode extension provides a graphical debugging experience with either ``gdb`` or ``lldb``. See :ref:`vscode-debugging` for more information. The `Debugger.breakpoint` statement ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The :chpl:mod:`Debugger` module provides a parenless function called :chpl:proc:`~Debugger.breakpoint`. When the code is compiled and run with debug symbols, i.e. ``-g``, the attached debugger will automatically stop at calls to this function as a breakpoint. .. _readme-debugging-bkc: --------------------------------------- Best Known Configuration - LLVM backend --------------------------------------- When debugging Chapel using the default LLVM backend, make sure to use the ``-g`` flag to generate debug symbols in the executable. This will allow you to use a debugger like ``gdb`` or ``lldb`` to step through the code and inspect variables. There are a few flags that may be useful for debugging with the LLVM backend. For example, the following is the current best configuration which results in generated code with the best possible debug information: .. code-block:: bash chpl -g --preserve-inlined-line-numbers \ --no-return-by-ref --no-inline --no-copy-propagation If you intend to use ``gdb`` to debug your Chapel program, you may also want to use the ``--ccflags -gdwarf-4``, as ``gdb`` has been found to have issues with DWARF 5 debug information generated by the LLVM backend. .. code-block:: bash chpl -g --ccflags -gdwarf-4 --preserve-inlined-line-numbers \ --no-return-by-ref --no-inline --no-copy-propagation The following table summarizes the flags that can be used to improve the debuggability of the generated executable: =================================== ========================================= Flag Description =================================== ========================================= ``-g`` Generate debug symbols in the executable ``--ccflags -gdwarf4`` Use DWARF 4 debug information (useful for ``gdb``) ``--preserve-inlined-line-numbers`` When code gets inlined (e.g. replacing a function call with the function body) maintain the filename and line number information of the original function. ``--no-return-by-ref`` Don't use an extra reference argument when compiling a Chapel function that returns a record. ``--no-inline`` Avoid inlining in many cases. ``--no-copy-propagation`` Disables a Chapel optimization pass that can make debugging harder. =================================== ========================================= ------------------------------------ Best Known Configuration - C backend ------------------------------------ It may be useful to debug Chapel programs using the C backend. The current best practice for debugging Chapel source code with the C backend is to use a series of flags to improve the debuggability of the generated executable. While debugging with just ``-g`` is possible, these flags result in generated code with debug information that most closely matches the Chapel source code. This can be done in two steps. 1) Build the compiler with ``CHPL_TARGET_COMPILER`` set to ``gnu`` (or ``clang`` if on Mac): .. code-block:: bash CHPL_TARGET_COMPILER=gnu make # On MacOS # CHPL_TARGET_COMPILER=clang make 2) Build the executable from Chapel source code: .. code-block:: bash chpl -g --target-compiler=gnu --preserve-inlined-line-numbers \ --no-munge-user-idents --no-return-by-ref \ --no-inline --no-copy-propagation # On MacOS # chpl -g --target-compiler=clang --preserve-inlined-line-numbers \ # --no-munge-user-idents --no-return-by-ref --no-inline \ # --no-copy-propagation For more details on these settings, read the rest of this section. Building the Compiler ~~~~~~~~~~~~~~~~~~~~~ For best results while debugging, we recommend building the compiler with ``CHPL_TARGET_COMPILER`` set to ``gnu`` (or ``clang`` if on Mac). See :ref:`readme-chplenv` for more information on building the compiler. With two invocations of the build command, both backends can be built. First execute ``make`` (which uses the LLVM backend by default) and then execute ``CHPL_TARGET_COMPILER=gnu make``. This will keep the default as LLVM and allow switching to the C backend as needed for debugging. This can be done for a particular invocation of the compiler with ``chpl --target-compiler=gnu ...``. Building the Application ~~~~~~~~~~~~~~~~~~~~~~~~ The following flags can be useful for making the generated C more amenable to debugging. Any of them can be omitted as desired. =================================== ========================================= Flag Description =================================== ========================================= ``-g`` Generate debug symbols in the executable ``--target-compiler=gnu`` Target the C backend with GCC ``--target-compiler=clang`` Target the C backend with Clang ``--savec `` Write out the generated C to a given directory ``--preserve-inlined-line-numbers`` When code gets inlined (e.g. replacing a function call with the function body) maintain the filename and line number information of the original function. ``--no-munge-user-idents`` Don't munge user identifiers (e.g. variable or function names). Munging typically prevents conflicts with identifiers in external code but makes debugging harder. ``--no-return-by-ref`` Don't use an extra reference argument when compiling a Chapel function that returns a record. ``--no-inline`` Avoid inlining in many cases. ``--no-copy-propagation`` Disables a Chapel optimization pass that can make debugging harder. =================================== ========================================= Notes on munging '''''''''''''''' The utility of using a debugger with Chapel depends greatly on your familiarity with the Chapel generated code. However, if your program is crashing or running into a runtime error, you can often determine where that is taking place by looking at a stack trace within ``gdb``. When debugging Chapel, it is useful to know that in generating its code, the Chapel compiler renames user identifiers. By default, the Chapel compiler munges all user identifiers, such that a variable named ``x`` would be code generated as ``x_chpl``. This munging can be controlled using the ``--[no-]munge-user-idents`` flag (see the ``chpl`` man page for more information). In some cases, additional munging may be required or applied that cannot be turned off. The net effect of this is that Chapel variables can often be inspected using ``p`` *name*\ ``_chpl`` (or ``p`` *name*\ ``_chpl`` in cases where the compiler has further renamed the variable). If the ``--no-munge-user-idents`` flag is used, ``p`` *name* or ``p`` *name*\ ```` should work in most cases. See :ref:`more-munging-info` for more information on munging. Over time, we plan to improve our ability to debug the generated code for a Chapel program. If you find yourself debugging the generated code a lot and need help or have requests for better support, please let us know so that we can prioritize accordingly. ------------------------------- Tracking and Reporting on Tasks ------------------------------- For certain tasking layers, Chapel supports an experimental capability for tracking the status of tasks, primarily designed for use in a single-locale execution. To enable this capability, your program must be compiled with the ``--task-tracking`` flag. The feature itself is enabled at execution time by setting the boolean environment variable ``CHPL_RT_ENABLE_TASK_REPORTING`` to any of the values "1", "yes", or "true". If this is done, then when ```` is entered while a program is executing, a list of pending and executing tasks will be printed to the console, giving an indication of which tasks are at which source locations. This is only supported with ``CHPL_TASKS=fifo``. Note that task tracking adds a fair amount of runtime overhead to task-parallel programs. ------------------------------------------- Configuration Constants for Tracking Memory ------------------------------------------- Chapel supports a number of configuration constants related to dynamic memory allocation for the compiler-generated executable, currently designed for use primarily by the development team to track memory usage in tests. For full information on these configuration constants consult :chpl:mod:`MemDiagnostics`. A brief synopsis of these configuration constants is as follows: --memTrack turn on memory tracking and enable reporting --memStats call ``printMemAllocStats()`` on normal termination --memLeaksByType call ``printMemAllocsByType()`` on normal termination --memLeaks call ``printMemAllocs()`` on normal termination --memMax=int set maximum level of allocatable memory --memThreshold=int set minimum threshold for memory tracking --memLog=string file to contain all memory reporting --memLeaksLog=string if set, append final stats and leaks-by-type here