Zarr

Usage

use Zarr;

or

import Zarr;

Support for distributed reading and writing of Zarr stores. Support is limited to v2 Zarr arrays stored on local filesystems. NFS is not supported. The module uses c-blosc to compress and decompress chunks. Zarr specification: https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html

config param zarrProfiling = false

Turns on/off profiling of Zarr IO

iter zarrProfilingResults() throws

Returns a map of profiling results for Zarr IO operations. The keys are the names of the operations and the values are the total time spent in each operation across all threads. Requires that zarrProfiling be set to true.

record zarrMetadataV2
var zarr_format : int
var chunks : list(int)
var dtype : string
var shape : list(int)
record zarrMetadataV3

Unused until support is added for v3.0 stores

var zarr_format : int
var node_type : string
var shape : list(int)
var data_type : string
var dimension_names : list(string)
proc getLocalChunks(D: domain(?), localD: domain(?), chunkShape: ?dimCount*int) : domain(dimCount)

Returns the domain of chunks that the calling locale is responsible for

proc readChunk(param dimCount: int, chunkPath: string, chunkDomain: domain(dimCount), ref arraySlice: [] ?t) throws

Reads a chunk from storage and fills arraySlice with its corresponding values.

Arguments:

  • dimCount – Dimensionality of the array being read.

  • chunkPath – Relative or absolute path to the chunk being read.

  • chunkDomain – Array subdomain the chunk contains.

  • arraySlice – Reference to the portion of the array the calling locale stores.

Throws:

Error – If the decompression fails

proc writeChunk(param dimCount, chunkPath: string, chunkDomain: domain(dimCount), ref arraySlice: [] ?t, bloscLevel: int(32) = 9) throws

Updates a chunk in storage with a locale’s contribution to that chunk. The calling function is expected to manage synchronization among locales. If the locale contributes the entire chunk, it will immediately compress and write the chunk’s data. If the contribution is partial, it decompresses the chunk, updates the necessary values, then compresses and writes the chunk to storage.

Arguments:

  • dimCount – Dimensionality of the array being written.

  • chunkPath – Relative or absolute path to the chunk being written.

  • chunkDomain – Array subdomain that the chunk contains.

  • arraySlice – The portion of the array that the calling locale contributes to this chunk.

  • bloscLevel – Compression level to use. 0 indicates no compression, 9 (default) indicates maximum compression. Values outside of this range will be clipped to a value between 0 and 9.

Throws:

Error – If the compression fails

proc readZarrArray(directoryPath: string, type dtype, param dimCount: int, bloscThreads: int(32) = 1) throws

Reads a v2.0 zarr store from storage, returning a block distributed array. Each locale reads and decompresses the chunks with elements in its subdomain. This method assumes a shared filesystem where all nodes can access the store directory.

Arguments:

  • directoryPath – Relative or absolute path to the root of the zarr store. The store is expected to contain a ‘.zarray’ metadata file

  • dtype – Chapel type of the store’s data

  • dimCount – Dimensionality of the zarr array

  • bloscThreads – The number of threads to use during decompression (default=1)

proc writeZarrArray(directoryPath: string, ref A: [?domainType] ?dtype, chunkShape: ?dimCount*int, bloscThreads: int(32) = 1, bloscLevel: int(32) = 9) throws

Writes an array to storage as a v2.0 zarr store. The array metadata and chunks will be stored within the directoryPath directory, which is created if it does not yet exist. The chunks will have the dimensions given in the chunkShape argument. This function writes chunks in parallel, and supports distributed execution. It assumes a shared filesystem where all nodes can access the store directory.

Arguments:

  • directoryPath – Relative or absolute path to the root of the zarr store. The directory and all necessary parent directories will be created if it does not exist.

  • A – The array to write to storage.

  • chunkShape – The dimension extents to use when breaking A into chunks.

  • bloscThreads – The number of threads to use during compression (default=1)

  • bloscLevel – Compression level to use. 0 indicates no compression, 9 (default) indicates maximum compression.