Using Non-Chapel Dependencies

Mason allows for specification of external, non-Chapel dependencies through two mediums, Spack and pkg-config. The following two sections document how to use mason external and mason system to interface with the Spack and pkg-config packages respectively.

Using System Dependencies

System dependencies are packages that are found on your system through pkg-config. To use this functionality of Mason, users must have pkg-config installed.

Mason interfaces with pkg-config through the mason system command.

mason system search will print all the current packages installed and available for use in a Mason package. To examine the .pc file of a particular package, use mason system pc <package> where <package> is replaced with the particular package you are looking for. Here is an example of a workflow for creating a Mason package with openssl which has already been installed.

First, search to see that it is installed with mason system search openSSl which outputs:

$ mason system search openssl
openssl               OpenSSL - Secure Sockets Layer and cryptography libraries and tools

To find out more about the package, since it is in fact installed on my system, use the mason system pc command as follows

$ mason system pc openssl

------- openSSL.pc -------

prefix=/usr
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: OpenSSL
Description: Secure Sockets Layer and cryptography libraries and tools
Version: 0.9.8zh
Requires:
Libs: -L${libdir} -lssl -lcrypto -lz
Cflags: -I${includedir}

-------------------

Use the mason add --system command to add the dependency to the Mason.toml of the package.

$ mason add --system openSSL@0.9.8zh
Adding system dependency openSSL version 0.9.8zh

The Mason.toml now looks like:

[brick]
chplVersion = "1.18.0"
license = "None"
name = "myPackage"
version = "0.1.0"

[system]
openSSL = "0.9.8zh"

Now, upon calling mason build or mason run --build, Mason will go get openssl and include it in the package so that it can be used as a dependency.

Using Spack Dependencies

Mason users can interface with Spack, a package manager geared towards high performance computing, through the mason external command. For a list of available Spack packages, see: packages.

This interface is analogous to the previous example except when a package is missing, user’s can download that package through the Spack integration. The following is a workflow of finding, installing, and adding a Spack dependency to a Mason Package.

Setting up Spack backend

First, the Spack backend must be installed. Users can have mason install Spack or point mason to an existing spack installation.

This command will install Spack into $MASON_HOME/spack and set it up so that it can be used by Mason. It should be noted that this command pulls from the master branch of spack for setting up the spack registry at $MASON_HOME/spack-registry:

mason external --setup

Alternatively, users can set $SPACK_ROOT to their own spack installation:

export SPACK_ROOT=/path/to/spack

Searching Spack packages

Let’s use openSSL as an example since we used it in the system example. mason external search openSSL will search for the package and produce the following output:

$ mason external search openSSL
==> 2 packages.
openssl  r-openssl

Since there are two types of the package listed, we need to figure out which one to use. To find out more about a package, use mason external info <package> as follows:

$ mason external info openssl
Package:   openssl

Description:
OpenSSL is an open source package that provides a robust, commercial-
grade, and full-featured toolkit for the Transport Layer Security (TLS)
and Secure Sockets Layer (SSL) protocols. It is also a general-purpose
cryptography library.

Homepage: http://www.openssl.org

Tags:
  None

Preferred version:
  1.0.2k    http://www.openssl.org/source/openssl-1.0.2k.tar.gz

Safe versions:
  1.1.0e    http://www.openssl.org/source/openssl-1.1.0e.tar.gz
  1.1.0d    http://www.openssl.org/source/openssl-1.1.0d.tar.gz
  1.1.0c    http://www.openssl.org/source/openssl-1.1.0c.tar.gz
  1.0.2k    http://www.openssl.org/source/openssl-1.0.2k.tar.gz
  1.0.2j    http://www.openssl.org/source/openssl-1.0.2j.tar.gz

Variants:
 None

Installation Phases:
 install

Build Dependencies:
 zlib

Link Dependencies:
 zlib

Run Dependencies:
 None

Virtual Packages:
 None

Installing Spack packages

The correct package has been found, but not yet installed. Let’s fix that. We know that we want the preferred version which is 1.0.2k. The command to install openssl version 1.0.2k would be:

mason external install openssl

Since the version was left out, version 1.0.2k is used because Mason will always take the preferred version. This is a case where Spack’s spec expression syntax can be used to specify exactly which package is desired. For example, another way to install openSSL would be:

mason external install openssl@1.0.2k

which simply specifies the exact version that we want. If we wanted to specify which compiler the package was built with:

mason external install openssl@1.0.2k%gcc

Mason will infer which compiler, in the case that the compiler is left out of the spec, by looking at the environment variable CHPL_TARGET_COMPILER. For more information on how to use spec expressions, use the command mason external --spec which will output the following:

 spec expression syntax:

package [constraints] [^dependency [constraints] ...]

package                           any package from 'spack list'

constraints:
  versions:
    @version                      single version
    @min:max                      version range (inclusive)
    @min:                         version <min> or higher
    @:max                         up to version <max> (inclusive)

  compilers:
    %compiler                     build with <compiler>
    %compiler@version             build with specific compiler version
    %compiler@min:max             specific version range (see above)

  variants:
    +variant                      enable <variant>
    -variant or ~variant          disable <variant>
    variant=value                 set non-boolean <variant> to <value>
    variant=value1,value2,value3  set multi-value <variant> values

  architecture variants:
    target=target                 specific <target> processor
    os=operating_system           specific <operating_system>
    platform=platform             linux, darwin, cray, bgq, etc.
    arch=platform-os-target       shortcut for all three above

  cross-compiling:
    os=backend or os=be           build for compute node (backend)
    os=frontend or os=fe          build for login node (frontend)

  dependencies:
    ^dependency [constraints]     specify constraints on dependencies

examples:
    hdf5                          any hdf5 configuration
    hdf5 @1.10.1                  hdf5 version 1.10.1
    hdf5 @1.8:                    hdf5 1.8 or higher
    hdf5 @1.8: %gcc               hdf5 1.8 or higher built with gcc
    hdf5 +mpi                     hdf5 with mpi enabled
    hdf5 ~mpi                     hdf5 with mpi disabled
    hdf5 +mpi ^mpich              hdf5 with mpi, using mpich
    hdf5 +mpi ^openmpi@1.7        hdf5 with mpi, using openmpi 1.7
    boxlib dim=2                  boxlib built for 2 dimensions
    libdwarf %intel ^libelf%gcc
        libdwarf, built with intel compiler, linked to libelf built with gcc
    mvapich2 %pgi fabrics=psm,mrail,sock
        mvapich2, built with pgi compiler, with support for multiple fabrics

Resuming the example, the result of the install given openssl as the sole argument would output the following:

$ mason external install openssl
==> Installing zlib
==> Fetching http://zlib.net/fossils/zlib-1.2.11.tar.gz
==> Staging archive: /$HOME/.mason/spack/var/spack/stage/zlib-1.2.11-cpdvq4e7otjepbwdtxmgk5bzszze27fj/zlib-1.2.11.tar.gz
==> Created stage in /$HOME/.mason/spack/var/spack/stage/zlib-1.2.11-cpdvq4e7otjepbwdtxmgk5bzszze27fj
==> No patches needed for zlib
==> Building zlib [Package]
==> Executing phase: 'install'
==> Successfully installed zlib
Fetch: 4.84s.  Build: 4.24s.  Total: 9.08s.

==> Installing openssl
==> Fetching http://www.openssl.org/source/openssl-1.0.2k.tar.gz
==> Staging archive: /$HOME/.mason/spack/var/spack/stage/openssl-1.0.2k-fwnsee6qcvbbgvmgp3f5hio6dwg6nh2d/openssl-1.0.2k.tar.gz
==> Created stage in /$HOME/.mason/spack/var/spack/stage/openssl-1.0.2k-fwnsee6qcvbbgvmgp3f5hio6dwg6nh2d
==> No patches needed for openssl
==> Building openssl [Package]
==> Executing phase: 'install'
==> Successfully installed openssl
Fetch: 3.37s.  Build: 3m 11.76s.  Total: 3m 15.13s.
######################################################################## 100.0%
######################################################################## 100.0%

As shown, Mason not only goes and gets the package specified, but also all of the dependencies of the package specified. Packages are installed into unique directories such that it is impossible for package namespaces to collide. Each dependency is downloaded distinctly for a package so no previous installs will be broken by installing new packages. This way, multiple versions and builds of a package can be installed on a system and used without clashing with each other.

Specifying Spack packages in the manifest file

Now that the correct package is installed, add it to the Mason.toml as follows:

$ mason add --external openssl@1.0.2k
Adding external dependency with spec openssl@1.0.2k

The Mason.toml now looks like:

[brick]
chplVersion = "1.18.0"
license = "None"
name = "myPackage"
version = "0.1.0"

[external]
openSSL = "1.0.2k"

To ensure the package is installed on the system, run mason external find which will list all of the current Spack packages installed on system. For example:

==> 2 installed packages.
-- darwin-sierra-x86_64 / clang@9.0.0-apple ---------------------
openssl@1.0.2k  zlib@1.2.11

Now, everything necessary to use openssl in a Mason package has been done. Upon building, Mason will retrieve the necessary files and file locations for building myPackage with openssl.