StencilDist¶

Usage

use StencilDist;

or

import StencilDist;

Warning

StencilDist is unstable and may change in the future

record stencilDist: writeSerializable¶

The stencilDist distribution is a variant of the blockDist distribution that attempts to improve performance for stencil computations by reducing the amount of communication necessary during array accesses. From the user’s perspective, it behaves very similarly to blockDist in terms of reads, writes, and iteration.

This distribution reduces communication by creating read-only caches for elements adjacent to the block of elements owned by each locale. This documentation may refer to these cached regions as ‘ghost cells’ or ‘fluff’. This approach can avoid many fine-grained GETs and PUTs when performing a stencil computation near the boundary of the current locale’s chunk of array elements. The user must manually refresh these caches after writes by calling the updateFluff method. Otherwise, reading and writing array elements behaves the same as a block-distributed array.

The indices are partitioned in the same way as the blockDist distribution.

The stencilDist initializer is defined as follows:

proc stencilDist.init(
  boundingBox: domain(?),
  targetLocales: [] locale  = Locales,
  dataParTasksPerLocale     = // value of  dataParTasksPerLocale      config const,
  dataParIgnoreRunningTasks = // value of  dataParIgnoreRunningTasks  config const,
  dataParMinGranularity     = // value of  dataParMinGranularity      config const,
  param rank                = boundingBox.rank,
  type  idxType             = boundingBox.idxType,
  fluff: rank*idxType       = makeZero(rank),
  periodic: bool            = false)

The fluff argument indicates the requested number of cached elements in each dimension. If an element of fluff is greater than zero, the user can use indices outside of boundingBox to index into the array. If the domain is not strided, you can consider indices for dimension i to be:

boundingBox.dim(i).expand(fluff(i))

If the domain is strided:

const bb = boundingBox.dim(i);
bb.expand(fluff(i) * abs(bb.stride));

The same logic is used when determining the cached index set on each locale, except you can imagine boundingBox to be replaced with the returned value from localSubdomain.

The periodic argument indicates whether or not the stencil distribution should treat the array as a discrete chunk in a larger space. When enabled, the ghost cells outside of boundingBox will contain values as if the array was replicated on all sides of the space. When disabled, the outermost ghost cells will be initialized with the default value of the element’s type. The periodic functionality is disabled by default.

Note

Note that this domain does not currently handle indices outside of the expanded bounding box, so a user must manually wrap periodic indices themselves.

Iterating directly over a stencil-distributed domain or array will only yield indices and elements within the boundingBox.

Convenience Factory Methods

It is common for a stencilDist-distributed domain or array to be declared using the same indices for both its boundingBox and its index set (as in the example using Space above). It is also common to not override any of the other defaulted initializer arguments. In such cases, factory procedures can be used for convenience and to avoid repetition.

use StencilDist;

var BlockDom1 = stencilDist.createDomain({1..5, 1..5});
var BlockArr1 = stencilDist.createArray({1..5, 1..5}, real);
var BlockDom2 = stencilDist.createDomain(1..5, 1..5);
var BlockArr2 = stencilDist.createArray(1..5, 1..5, real);

The helper methods on stencilDist have the following signatures:

proc type createDomain(dom: domain, targetLocales = Locales, fluff, periodic = false)¶: Create a stencil-distributed domain. The provided domain is used as the boundingBox.

proc type createDomain(rng: range(?)..., targetLocales = Locales, fluff, periodic = false): Create a stencil-distributed domain from a series of ranges. The ranges are also used to construct the boundingBox.

proc type createArray(dom: domain, type eltType, targetLocales = Locales, fluff, periodic = false)¶: Create a default-initialized, stencil-distributed array whose indices match those of the given domain.

proc type createArray(rng: range(?)..., type eltType, targetLocales = Locales, fluff, periodic = false): Create a default-initialized, stencil-distributed array using a domain constructed from the series of ranges.

proc type createArray(dom: domain, type eltType, initExpr, targetLocales = Locales, fluff, periodic = false)

Create a stencil-distributed array whose indices match those of the given domain.

The array’s values are initialized using initExpr which can be any of the following:

a value coercible to eltType — all elements of the array will be assigned with this value
an iterator expression with compatible size and type — the array elements will be initialized with the values yielded by the iterator
an array of compatible size and type — the array will be assigned into the distributed array

proc type createArray(rng: range(?)..., type eltType, initExpr, targetLocales = Locales, fluff, periodic = false)

Create a stencil-distributed array using a domain constructed from the series of ranges.

The array’s values are initialized using initExpr which can be any of the following:

a value coercible to eltType — all elements of the array will be assigned with this value
an iterator expression with compatible size and type — the array elements will be initialized with the values yielded by the iterator
an array of compatible size and type — the array will be assigned into the distributed array

proc createDomain(dom: domain(?)): Create a stencil-distributed domain over an existing blockDist by copying the index space from the passed domain.

proc createDomain(rng: range(?)...): Create a stencil-distributed domain from a series of ranges over an existing blockDist.

Note that the fluff argument in the above methods defaults to a tuple of n zeros, where n is the domain’s rank or the number of provided ranges.

Updating the Cached Elements

Once you have completed a series of writes to the array, you will need to call the updateFluff method to update the cached elements for each locale. Here is a simple example:

use StencilDist;

const Space = {1..10, 1..10};
const Dist = new stencilDist(boundingBox=Space, fluff=(1,1));
const D = Dist.createDomain(Space);
var A : [D] int;

forall (i,j) in D with (ref A) do
  A[i,j] = i*10 + j;

// At this point, the ghost cell caches are out of date

A.updateFluff();

// ghost caches are now up-to-date

After updating, any read from the array should be up-to-date. The updateFluff method does not currently accept any arguments.

Reading and Writing to Array Elements

The stencilDist distribution uses ghost cells as cached read-only values from other locales. When reading from a stencil-distributed array, the distribution will attempt to read from the local ghost cache first. If the index is not within the cached index set of the current locale, then we default to a remote read from the locale on which the element is located.

Any write to array data will be applied to the actual element, the same as if you were using a Block-distributed array.

Modifying Exterior Ghost Cells

Updating the outermost ghost cells can be useful when working with a periodic stencil-distributed array. If your array contains position information, you may want to modify the ghost cells to ‘wrap’ around the physical space correctly.

You can currently do this with the boundaries() iterator on a stencil-distributed array. This iterator yields a tuple where the first component is the ghost cell element to be modified, and the second component is a tuple indicating the side on which this ghost cell lives. This direction tuple will contain values in the range -1..1.

The release benchmark ‘miniMD’ contains an example of how one might use this iterator.

Warning

There is a known issue with this iterator where the program will fail to compile if the array element is not an array or a class.