IO

Support for a variety of kinds of input and output.

Input/output (I/O) facilities in Chapel include the types file and channel; the constants stdin, stdout and stderr; the functions open, file.close, file.reader, file.writer, channel.read, channel.write, and many others.

I/O Overview

A file in Chapel identifies a file in the underlying operating system. Reads and writes to a file are done via one or more channels associated with the file. Each channel uses a buffer to provide sequential read or write access to its file, optionally starting at an offset.

For example, the following program opens a file and writes an integer to it:

// open the file "test-file.txt" for writing, creating it if
// it does not exist yet.
var myFile = open("test-file.txt", iomode.cw);

// create a writing channel starting at file offset 0
// (start and end offsets can be specified when creating the
// channel)
var myWritingChannel = myFile.writer();

var x: int = 17;

// This function will write the human-readable text version of x;
// binary I/O is also possible.
myWritingChannel.write(x);

// Now test-file.txt contains:
// 17

Then, the following program can be used to read the integer:

// open the file "test-file.txt" for reading only
var myFile = open("test-file.txt", iomode.r);

// create a reading channel starting at file offset 0
// (start and end offsets can be specified when creating the
// channel)
var myReadingChannel = myFile.reader();

var x: int;

// Now read a textual integer. Note that the
// channel.read function returns a bool to indicate
// if it read something or if the end of the file
// was reached before something could be read.
var readSomething = myReadingChannel.read(x);

writeln("Read integer ", x);
// prints out:
// 17

Design Rationale

Since channels operate independently, concurrent I/O to the same open file is possible without contending for locks. Furthermore, since the channel (and not the file) stores the current file offset, it is straightforward to create programs that access the same open file in parallel. Note that such parallel access is not possible in C when multiple threads are using the same FILE* to write to different regions of a file because of the race condition between fseek and fwrite. Because of these issues, Chapel programmers wishing to perform I/O will need to know how to open files as well as create channels.

I/O Styles

Reading and writing of Chapel's basic types is regulated by an applicable iostyle. In particular, the I/O style controls whether binary or text I/O should be performed. For binary I/O it specifies, for example, byte order and string encoding. For text I/O it specifies string representation; the base, field width and precision for numeric types; and so on. Each channel has an associated I/O style. It applies to all read/write operations on that channel, except when the program specifies explicitly an I/O style for a particular read or write.

See the definition for the iostyle type. This type represents I/O styles and provides details on formatting and other representation choices.

The default value of the iostyle type is undefined. However, the compiler-generated constructor is available. It can be used to generate the default I/O style, with or without modifications. In addition, the function defaultIOStyle will return the default I/O style just as new iostyle() will.

The I/O style for an I/O operation can be provided through an optional style= argument in a variety of places:

Note that file.reader, or file.writer will copy the file's I/O style if a style= argument is not provided. Also note that I/O functions on channels will by default use the I/O style stored with that channel.

A channel's I/O style may be retrieved using channel._style and set using channel._set_style. These functions should only be called while the channel lock is held, however. See Synchronization of Channel Data and Avoiding Data Races for more information on channel locks.

Note

iostyle is work in progress: the fields and/or their types may change. Among other changes, we expect to be replacing the types of some multiple-choice fields from integral to enums.

As an example for specifying an I/O style, the code below specifies the minimum width for writing numbers so array elements are aligned in the output:

stdout.writeln(MyArray, new iostyle(min_width=10));

I/O facilities in Chapel also include several other ways to control I/O formatting. There is support for formatted I/O with channel.readf and channel.writef. It is possible to write data to strings (see "The write and writeln Methods on Strings" in the Chapel language specification) which can then be further modified or combined programmatically. Lastly, record or class implementations can provide custom functions implementing read or write operations for that type (see "The readThis, writeThis, and readWriteThis Methods" in the Chapel language specification).

Files

There are several functions that open a file and return a file including open, opentmp, openmem, openfd, and openfp.

Once a file is open, it is necessary to create associated channel(s) - see file.reader and file.writer - to write to and/or read from the file.

Use the file.fsync function to explicitly synchronize the file to ensure that file data is committed to the file's underlying device for persistence.

To release any resources associated with a file, it is necessary to first close any channels using that file (with channel.close) and then the file itself (with file.close).

Functions for Channel Creation

file.writer creates a channel for writing to a file, and file.reader create a channel for reading from a file.

Synchronization of Channel Data and Avoiding Data Races

Channels (and files) contain locks in order to keep their operation safe for multiple tasks. When creating a channel, it is possible to disable the lock (for performance reasons) by passing locking=false to e.g. file.writer(). Some channel methods - in particular those beginning with the underscore - should only be called on locked channels. With these methods, it is possible to get or set the channel style, or perform I/O "transactions" (see channel._mark). To use these methods, first lock the channel with channel.lock(), call the methods you need, and then unlock the channel with channel.unlock(). Note that in the future, we may move to alternative ways of calling these functions that guarantee that they are not called on a channel without the appropriate locking.

Besides data races that can occur if locking is not used in channels when it should be, it is also possible for there to be data races on file data that is buffered simultaneously in multiple channels. The main way to avoid such data races is the channel.flush synchronization operation. channel.flush will make all writes to the channel, if any, available to concurrent viewers of its associated file, such as other channels or other applications accessing this file concurrently. See the note below for more details on the situation in which this kind of data race can occur.

Note

Since channels can buffer data until channel.flush is called, it is possible to write programs that have undefined behaviour because of race conditions on channel buffers. In particular, the problem comes up for programs that make:

  • conncurrent operations on multiple channels that operate on overlapping regions of a file
  • where at least one of the overlapping channels is a writing channel
  • and where data could be stored more than one of the overlapping channel's buffers at the same time (ie, write and read ordering are not enforced through channel.flush and other mean such as sync variabless).

Note that it is possible in some cases to create a file that does not allow multiple channels at different offsets. Channels created on such files will not change the file's position based on a start= offset arguments. Instead, each read or write operation will use the file descriptor's current position. Therefore, only one channel should be created for files created in the following situations:

  • with openfp
  • with openfd when provided a non-seekable system file descriptor

Performing I/O with Channels

Channels contain read and write methods, which are generic methods that can read or write anything, and can also take optional arguments such as I/O style or to return an error instead of halting. These functions generally take any number of arguments. See:

In addition, there are several convenient synonyms for channel.write and channel.read:

Sometimes it's important to flush the buffer in a channel - to do that, use the .flush() method. Flushing the buffer will make all writes available to other applications or other views of the file (ie, it will call e.g. the OS call pwrite). It is also possible to close a channel, which will implicitly flush it and release any buffer memory used by the channel. Note that if you need to ensure that data from a channel is on disk, you'll have to call channel.flush or channel.close and then file.fsync on the related file.

Functions for Closing Channels

A channel must be closed in order to free the resources allocated for it, to ensure that data written to it is visible to other channels, or to allow the associated file to be closed.

See channel.close.

It is an error to perform any I/O operations on a channel that has been closed. It is an error to close a file when it has channels that have not been closed.

In the future we plan to implement reference counting for files and channels. Each file and channel will be closed automatically when no references remain to it. For example, if only a local variable refers to a channel, the channel will be closed when that variable goes out of scope.

The ability for the program to close a file or a channel explicitly will remain available.

The stdin, stdout, and stderr Channels

Chapel provides the predefined channels stdin, stdout, and stderr to access the corresponding operating system streams standard input, standard output, and standard error.

stdin supports reading; stdout and stderr support writing.

All three channels are safe to use concurrently. Their types' kind argument is dynamic

Error Handling

Most I/O routines accept an optional error= argument. If that argument is used, instead of halting when an error is encountered, the function will return the error code.

These error codes are stored with the type SysBasic.syserr. Success is represented by SysBasic.ENOERR. The error codes and their meaning are described in SysBasic. Some of these error codes that are commonly used within the I/O implementation include:

An error code can be converted to a string using the function Error.errorToString.

Ensuring Successful I/O

It is possible - in some situations - for I/O to fail without returning an error. In cases where a programmer wants to be sure that there was no error writing the data to disk, it is important to call file.fsync to make sure that data has arrived on disk without an error. Many errors can be reported with typical operation, but some errors can only be reported by the system during file.close or even file.fsync.

When a file (or channel) is closed, data written to that file will be written to disk eventually by the operating system. If an application needs to be sure that the data is immediately written to persistent storage, it should use file.fsync prior to closing the file.

Correspondence to C I/O

It is not possible to seek, read, or write to a file directly. Create a channel to proceed.

channel.flush in Chapel has the same conceptual meaning as fflush() in C. However, fflush() is not necessarily called in channel.flush(). Unlike fsync(), which is actually called in file.fsync() in Chapel.

The iomode constants in Chapel have the same meaning as the following strings passed to fopen() in C:

  • iomode.r "r"
  • iomode.rw "r+"
  • iomode.cw "w"
  • iomode.cwr "w+"

However, open() in Chapel does not necessarily invoke fopen().

Bytes Type

A Buffers.bytes object is just some data in memory along with a size. Bytes objects are reference counted, and the memory will be freed when nothing refers to the bytes object any more.

Buffers

A Buffers.buffer stores some number subsections of bytes objects. It is efficient to go to a particular offset in a buffer, and to push or pop bytes objects from the beginning or end of a buffer.

Buffers are used internally in each channel.

Formatted I/O

See below for a sample-based overview of the format strings. Further below, we describes the format string syntax in detail. Finally, we demonstrate the functionality with example function calls.

Overview of Format Strings

In a manner similar to C's 'printf' and 'scanf', the IO package includes channel.writef and channel.readf functions. These functions take in a format string and some arguments. The string.format method is also available and is loosly equivalent to C's 'sprintf'. For example, one might do:

writef("My favorite %s is %i\n", "number", 7);

var s:string = "My favorite %s is %i".format("number", 7);
writeln(s);

// prints:
// My favorite number is 7
// My favorite number is 7

The following sections offer a tour through the conversions to illustrate the common cases. A more precise definition follows in the "Format String Syntax in Detail" section below.

In this file, we use "integral" to refer to the Chapel types int or uint and "floating-point" to refer to real, imaginary, or complex, of any bit width.

Formatted I/O for C Programmers

This table is designed to help C programmers understand the equivalent Chapel format specifiers.

C Chapel Meaning
%i %i an integer in decimal
%d %i an integer in decimal
%u %u an unsigned integer in decimal
%x %xu an unsigned integer in hexadecimal
%g %r real number in exponential or decimal (if compact)
%7.2g %7.2r real, 2 significant digits, padded to 7 columns
%f %dr real number always in decimal
%7.3f %7.3dr real, 3 digits after ., padded to 7 columns
%e %er real number always in exponential
%7.3e %7.3er real, 3 digits after ., padded to 7 columns
%s %s a string without any quoting

Unlike in C, a value of the wrong type will be cast appropriately - so for example printing 2 (an int) with %.2dr will result in 2.00. Note that %n and %t are equivalent to %r for real conversions and %i for numeric conversions; so these are also equivalent to %i %d or %g in C. Also note that Chapel format strings includes many capabilities not available with C formatted I/O routines - including quoted strings, binary numbers, complex numbers, and raw binary I/O.

Generic Numeric Conversions

%{##.###}

decimal number padded on the left with spaces to 2 digits before the point, rounded to 3 after. Works with integral, real, imaginary, or complex arguments.

In all cases, the output is padded on the left to the total length of the conversion specifier (6 in this example). The output can be longer, when needed to accomodate the number.

%{##}
integral value padded out to 2 digits. Also works with real, imaginary or complex numbers by rounding them to integers. Numbers with more digits will take up more space instead of being truncated.

In both conversions above, an imaginary argument gets an 'i' afterwards and the entire expression is padded out to the width of ##### digits. For example:

writef("|${#####}|\n", 2.0i);
     // outputs:
     //   |   2i|

writef("|%{#####.#}|\n", 2.0i);
     // outputs:
     //   |   2.0i|

Complex arguments are printed in the format a + bi, where each of a and b is rounded individually as if printed under that conversion on its own. Then, the formatted complex number is padded to the requested size. For example:

writef("|%{#########}|\n", 1.0+2.0i);
     // outputs:
     //   |   1 + 2i|

writef("|%{#########.#}|\n", 1.0+2.0i);
     // outputs:
     //   | 1.0 + 2.0i|

See # Specifiers for more details on this conversion type.

%n
a "number" - equivalent to one of %i, %u, %r, %m, or %z below, depending on the type
%17n
a number padded out to 17 columns
%.4n
a number with 4 significant digits or a precision of 4

Integral Conversions

%i or %di
a signed integer in decimal, possibly negative (note - when reading an %i, - is allowed)
%u or %du
an unsigned decimal integer (note - when reading a %u, - is not allowed)
%bi
a binary signed integer
%bu
a binary unsigned integer
%@bu
a binary unsigned integer prefixed with 0b
%oi
an octal signed integer
%ou
an octal unsigned integer
%@ou
an octal unsigned integer prefixed with 0o
%xu
a hexadecimal unsigned integer
%xi
a hexadecimal signed integer
%@xu
a hexadecimal unsigned integer prefixed with 0x
%Xu
a hexadecimal unsigned integer in uppercase
%@Xu
a hexadecimal unsigned integer prefixed with 0X and uppercase
%17i
a decimal integer padded on the left with spaces to 17 columns (That is, it is right-justified in a 17-column field. Padding width is ignored when reading integers)
%*i
as with %17i but read the minimum width from the preceding argument
%017i
a decimal integer padded on the left with zeros to 17 columns
%-17i
a decimal integer left-justified (padded on the right) to 17 columns
%+i
a decimal integer showing + for positive numbers
% i
a decimal integer with a space for positive numbers
%|4i
output 4 raw, binary bytes of the passed integer in native endianness
%<4i
output 4 raw, binary bytes of the passed integer little endian
%>4i
output 4 raw, binary bytes of the passed integer big endian
%<8i
output 8 raw, binary bytes of the passed integer little endian (byte widths of 1, 2, 4, and 8 are supported for integral conversions)

Real Conversions

%r
a real number with decimal or exponential notation, where exponential is chosen if the decimal version would be too long
%6r
as with %r but padded on the left to 6 columns (ie right-justified)
%-6r
as with %r but padded on the right to 6 columns (ie left-justfied)
%.4r
as with %r but with 4 significant digits
%.*r
as with %.4r but with significant digits read from preceding argument
%6.4r
as with %r but padded on the left to 6 columns and with 4 significant digits
%*.*r
as with %6.4r but read minimum width and significant digits from preceding arguments
%dr
a real number in decimal notation, e.g. 12.34
%6dr
a decimal number padded on the left to 6 columns (right-justified)
%.4dr
a decimal number with 4 digits after the radix point
%6.4dr
a decimal number padded on the left to 6 columns and with 4 digits after the radix point (width and precision are ignored when reading numbers in readf)
%er
a real number in exponential notation, e.g. 8.2e-23
%Er
like %er but with the 'e' in uppercase, e.g. 8.2E-23
%.4er
exponential notiation with 4 digits after the period, e.g. 8.2000e-23
%xer
hexadecimal number using p to mark exponent e.g. 6c.3f7p-2a
%|4r
emit 4 raw, binary bytes of the passed number in native endianness
%<8r
emit 8 raw, binary bytes of the passed number in little endian
%<4r
emit 4 raw, binary bytes of the passed number in little endian (< | and > are supported for widths 4 or 8)

Complex and Imaginary Conversions

%m
an imaginary number, like a real with %r but ends with an i
%z
print complex number with %r for each part in the format a + bi
%@z
print complex number with %r for each part in the format (a,b)
%6z
as with %z but pad the entire complex number out to 6 columns
%6.4z
print a and b 4 significant digits and pad the entire complex number out to 6 columns
%dz
print a and b with %dr
%ez
print a and b with %er
%|4m
same as %|4r
%|8z
emit 8 raw, binary bytes of native-endian complex (a,b are each 4 bytes)
%<16z
emit 16 raw, binary bytes of little-endian complex (a,b each 8 bytes)

String and Bytes Conversions

%s
a string. When reading, read until whitespace. Note that if you want to be able to read your string back in, you should use one of the quoted or encoded binary versions (see below), since generally with %s it's not clear where the string ends.
%c
a single Unicode character (argument should be a string or an integral storing the character code)
%17s
  • when writing - a string left padded (right justified) to 17 columns
  • when reading - read up to 17 bytes or a whitespace, whichever comes first, rounding down to whole characters
%-17s
  • when writing - a string right padded (left justified) to 17 columns
%.17s
  • when writing - a string truncated to 17 columns. When combined with quoting strings, for example %.17"S, the conversion will print ... after a string if it was truncated. The truncation includes leaving room for the quotes and - if needed - the periods, so the shortest truncated string is ""... Generally, you won't be able to read these back in.
  • when reading - read exactly 17 Unicode code points
%|17s
  • when writing - emit string but cause runtime error if length does not match
  • when reading - read exactly 17 bytes (error if we read < 17 bytes)
%|*s
as with %17s but the length is specified in the argument before the string.
%"S
use double-quotes to delimit string
%'S
use single-quotes to delimit string
%cS
use any character (c) to delimit string
%{(S)}
quoted string, starting with (, ending with ), where the parens could be replaced by arbitrary characters
%*S
quoted string, the arg before the string to specifies quote character
%|0S
write a string null-terminated or read bytes until a null-terminator
%|*S
means read bytes until a terminator byte. The terminator byte is read from the argument before the string.
%|1S %|2S %|4S and %|8S
work with encoded strings storing a length and then the string data. The digit before S is the number of bytes of length which is by default stored native endian. <, |, > can be used to specify the endianness of the length field, for example %<8S is 8 bytes of little-endian length and then string data.
%|vS
as with %|1S-%|8S but the string length is encoded using a variable-length byte scheme (which is always the same no matter what endianness). In this scheme, the high bit of each encoded length byte records whether or not there are more length bytes (and the remaining bits encode the length in a big-endian manner).
%|*vS or %|*0S
read an encoded string but limit it to a number of bytes read from the argument before the string; when writing cause a runtime error if the string is longer than the maximum.
%/a+/
where any regular expression can be used instead of a+ consume one or more 'a's when reading, gives an error when printing, and does not assign to any arguments (note - regular expression support is dependent on RE2 build; see Regexp)
%/(a+)/
consume one or more 'a's and then set the corresponding string argument to the read value
%17/a+/
match a regular expression up to 17 bytes (note that %.17/a+/, which would mean to match 17 characters, is not supported).
%/*/
next argument contains the regular expression to match

General Conversions

%t
read or write the object according to its readThis/writeThis routine
%jt
read or write an object in JSON format using readThis/writeThis
%ht
read or write an object in Chapel syntax using readThis/writeThis
%|t
read or write an object in binary native-endian with readThis/writeThis
%<t
read or write an object little-endian in binary with readThis/writeThis
%>t
read or write an object big-endian in binary with readThis/writeThis

Note About Whitespace

When reading, \n in a format string matches any zero or more space characters other than newline and then exactly one newline character. In contrast, " " matches at least one space character of any kind.

When writing, whitespace is printed from the format string just like any other literal would be.

Finally, space characters after a binary conversion will be ignored, so that a binary format string can appear more readable.

Format String Syntax in Detail

Chapel's format strings are simpler than those in C in one way: it is no longer necessary to specify the types of the arguments in the format string. For example, in C the l in %ld is specifying the type of the argument for integer (decimal) conversion. That is not necessary in Chapel since the compiler is able to use type information from the call.

Format strings in Chapel consist of:

  • conversion specifiers e.g. "%xi" (described below)

  • newline e.g. "\n"

    • when writing - prints a newline
    • when reading - reads any amount of non-newline whitespace and then exactly one newline. Causes the format string not to match if it did not read a newline.
  • other whitespace e.g. " "

    • when writing - prints as the specified whitespace
    • when reading - matches at least one character of whitespace, possibly including newlines.
  • other text e.g. "test"

    • when writing - prints the specified text
    • when reading - matches the specified text

# Specifiers

All # specifiers must be enclosed in %{} syntax, for example %{#} is the shortest one, and %{#.#} is a more typical one. The integer portion of the number will be padded out to match the number of # s before the decimal point, and the number of # s after the decimal point indicate how many digits to print after the decimal point. In other words, display how many digits to use when printing a floating-point number by using the # symbol to stand for digits. The fractional portion of the number will be rounded appropriately and extra space will be made if the integer portion is too small:

writef("n:%{###.###}\n", 1.2349);
     // outputs:
     // n:  1.235

This syntax also works for numbers without a decimal point by rounding them appropriately.

A # specifier may start with a ..

writef("%{.##}\n", 0.777);
     // outputs:
     //  0.78

% Specifiers

Specifiers starting with % offer quite a few options. First, some basic rules.

%%
means a literal %
\n
means a literal newline
\\
means a single backslash
%{}
curly braces can wrap a % or # conversion specifier. That way, even odd specifiers can be interpreted unambiguously. Some of the more complex features require the use of the %{} syntax, but it's always acceptable to use curly braces to make the format string clearer. Curly braces are required for # conversion specifiers.

In general, a % specifier consists of either text or binary conversions:

%
[optional endian flag (binary conversions only)]
[optional flags]
[optional field width or size in bytes]
[optional . then precision]
[optional base flag]
[optional exponential type]
[conversion type]

Going through each section for text conversions:

[optional flags]
@
means "alternate form". It means to print out a base when not using decimal (e.g. 0xFFF or 0b101011); and it will format a complex number with parens instead of as e.g. 1.0+2.0i
+
means to show a plus sign when printing positive numbers
0
means to pad numeric conversions with 0 instead of space
" "
(a space) leaves a blank before a positive number (in order to help line up with negative numbers)
-
left-justify the converted value instead of right-justifying. Note, if both 0 and - are given, the effect is as if only - were given.
~
when reading a record or class instance, skip over fields in the input not present in the Chapel type. This flag currently only works in combination with the JSON format. This flag allows a Chapel program to describe only the relevant fields in a record when the input might contain many more fields.
[optional field width]

When printing numeric or string values, the field width specifies the number of columns that the conversion should use to display the value. It can be *, which means to read the field width from an integral argument before the converted value.

For string conversions in readf (%s %" %' %//), the field width specifies the maximum number of bytes to read.

For numeric conversions in readf, the field width is ignored.

[optional . then precision]

When printing floating point values, the precision is used to control the number of decimal digits to print. For %r conversions, it specifies the number of significant digits to print; for %dr or %er conversions, it specifies the number of digits following the decimal point. It can also be *, which means to read the precision from an integral argument before the converted value.

For textual string conversions in writef, (%s %" %'), the precision indicates the maximum number of columns to print - and the result will be truncated if it does not fit. In readf for these textual string conversions, the precision indicates the maximum number of characters (e.g. Unicode code points) to input.

The precision is silently ignored for integral conversions (%i, %u, etc) and for %// conversions.

[optional base flag]
d
means decimal (and not exponential for floating-point)
x
means lower-case hexadecimal
X
means upper-case hexadecimal
o
means octal
b
means binary
j
means JSON-style strings, numbers, and structures
h
means Chapel-style strings, numbers, and structures
'
means single-quoted string (with \ and ')
"
means double-quoted string (with \ and ")
[optional exponential type]
e
means floating-point conversion printing exponential -12.34e+56
E
means floating-point conversion printing uppercase exponential -12.34E+56
[conversion type]
t
means type-based or thing - uses writeThis/readThis but ignores width and precision
n
means type-based number, allowing width and precision
i
means integral conversion
u
means unsigned integral conversion
r
means real conversion (e.g. 12.23)
m
means imaginary conversion with an i after it (e.g. 12.23i)
z
means complex conversion
s
means string conversion
S
means a quoted string conversion
{cS}
means string conversion with quote char c
{*S}
means string conversion with quote char in argument before the string
{xSy}
means string conversion with left and right quote chars x and y
/.../
means a regular expression (for reading only)
{/.../xyz}
means regular expression with flags xyz
c
means a Unicode character - either the first character in a string or an integral character code

For binary conversions:

[optional endian flag]
<
means little-endian
>
means big-endian
|
means native-endian
[optional size in bytes]

This is the number of bytes the format should read or write in this conversion. For integral conversions (e.g. %|i) it specifies the number of bytes in the integer, and 1, 2, 4, and 8 are supported. For real and imaginary conversions, 4 and 8 are supported. For complex conversions, 8 and 16 are supported. The size in bytes is required for binary integral and floating-point conversions.

The size can be *, which means that the number of bytes is read from the argument before the conversion.

For strings, if a terminator or length field is specified, exactly this number is the maximum size in bytes; if the terminator or length is not specified, the string must be exactly that size (and if the argument is not exactly that number of bytes it will cause an error even when writing).

[conversion type]
t
means type-based or thing - to read or write with readThis/writeThis
n
means type-based number (size is not mandatory)
i
means integral. Note that the size is mandatory for binary integral conversions
u
means unsigned integral. Note that the size is mandatory for binary integral conversions
r
means real. Note that the size is mandatory for binary real conversions
m
works the same as r for binary conversions
z
means complex. Note that the size is mandatory for binary complex conversions
s
  • means string binary I/O
  • %|17s means exactly 17 byte string
0S/1S/2S/4S/8S
  • mean encoded string binary I/O:
  • %|0S means null-terminated string
  • %{|S*} means next-argument specifies string terminator byte
  • %|1S means a one-byte length and then the string
  • %|2S means a two-byte length and then the string
  • %|4S means a four-byte length and then the string
  • %|8S means an eight-byte length and then the string
  • %|vS means a variable-byte-encoded length and then the string
c
means a Unicode character - either the first character in a string or an integral character code

Formatted I/O Examples

writef("%5i %5s %5r\n", 1, "test", 6.34);
     // outputs:
     //    1  test  6.34

writef("%2.4z\n", 43.291 + 279.112i);
     // outputs:
     // 43.29 + 279.1i

writef("%<4u", 0x11223344);
     // outputs:
     // (hexdump of the output)
     // 4433 2211
writef("%>4u", 0x11223344);
     // outputs:
     // (hexdump of the output)
     // 1122 3344
writef("%<4i %<4i", 2, 32);
     // outputs:
     // (hexdump of the output -- note that spaces after
     //  a binary format specifier are ignored)
     // 0200 0000 2000 0000


writef("%|0S\n", "test");
     // outputs:
     // (hexdump of the output)
     // 7465 7374 000a
writef("%|1S\n", "test");
     // outputs:
     // (hexdump of the output)
     // 0474 6573 740a
writef("%>2S\n", "test");
     // outputs:
     // (hexdump of the output)
     // 0004 7465 7374 0a
writef("%>4S\n", "test");
     // outputs:
     // (hexdump of the output)
     // 0000 0004 7465 7374 0a
writef("%>8S\n", "test");
     // outputs:
     // (hexdump of the output)
     // 0000 0000 0000 0004 7465 7374 0a
writef("%|vS\n", "test");
     // outputs:
     // (hexdump of the output)
     // 04 7465 7374 0a

writef('%"S\n', "test \"\" \'\' !");
     // outputs:
     // "test \"\" '' !"
writef("%'S\n", "test \"\" \'\' !");
     // outputs:
     // 'test "" \'\' !'
writef("%{(S)}\n", "test ()", "(", ")");
     // outputs:
     // (test (\))


writef("%40s|\n", "test");
writef("%-40s|\n", "test");
     // outputs:
     //                                     test|
     // test                                    |

writef("123456\n");
writef("%6.6'S\n", "a");
writef("%6.6'S\n", "abcdefg");
writef("%.3'S\n", "a");
writef("%.3'S\n", "abcd");
     // outputs:
     // 123456
     //    'a'
     // 'a'...
     // 'a'
     // ''...


var s:string;
var got = readf(" %c", s);
// if the input is " a", "\na", "  a", etc, s will contain "a"
// if the input is "b", got will be false and s will contain ""

var s:string;
var got = readf("\n%c", s);
// if the input is "\na", or " \na", s will contain "a"
// if the input is "b", got will be false and s will be ""

var got = readf("%/a+/");
// if the input is "a" or "aa" (and so on), got will return true
// if the input is "c" got will be false

var s:string;
var got = readf("%/a(b+)/", s);
// if the input is "c" got will be false and s will be ""
// if the input is "ab", got will be true and s will be "b"
// if the input is "abb", got will be true and s will be "bb"

IO Functions and Types

enum iomode { r = 1, cw = 2, rw = 3, cwr = 4 }

The iomode type is an enum. When used as arguments when opening files, its constants have the following meaning:

  • iomode.r - open an existing file for reading.
  • iomode.rw - open an existing file for reading and writing.
  • iomode.cw - create a new file for writing. If the file already exists, its contents are removed when the file is opened in this mode.
  • iomode.cwr - as with iomode.cw but reading from the file is also allowed.
enum iokind { dynamic = 0, native = 1, big = 2, little = 3 }

The iokind type is an enum. When used as arguments to the channel type, its constants have the following meaning:

  • iokind.big means binary I/O with big-endian byte order is performed when writing/reading basic types from the channel.
  • iokind.little means binary I/O with little-endian byte order (similar to iokind.big but with little-endian byte order).
  • iokind.native means binary I/O in native byte order (similar to iokind.big but with the byte order that is native to the target platform).
  • iokind.dynamic means that the applicable I/O style has full effect and as a result the kind varies at runtime.

In the case of iokind.big, iokind.little, and iokind.native the applicable iostyle is consulted when writing/reading strings, but not for other basic types.

There are synonyms available for these values:

param iodynamic = iokind.dynamic

A synonym for iokind.dynamic; see iokind

param ionative = iokind.native

A synonym for iokind.native; see iokind

param iobig = iokind.big

A synonym for iokind.big; see iokind

param iolittle = iokind.little

A synonym for iokind.little; see iokind

enum iostringstyle { len1b_data = -1, len2b_data = -2, len4b_data = -4, len8b_data = -8, lenVb_data = -10, data_toeof = -65280, data_null = -256 }

This enum contains values used to control binary I/O with strings via the str_style field in iostyle.

  • iostringstyle.len1b_data indicates a string format of 1 byte of length followed by length bytes of string data.
  • iostringstyle.len2b_data indicates a string format of 2 bytes of length followed by length bytes of string data.
  • iostringstyle.len4b_data indicates a string format of 4 bytes of length followed by length bytes of string data.
  • iostringstyle.len8b_data indicates a string format of 8 bytes of length followed by length bytes of string data.
  • iostringstyle.lenVb_data indicates a string format of a variable number of bytes of length, encoded with high-bit meaning more bytes of length follow, and where the 7-bits of length from each byte store the 7-bit portions of the length in order from least-significant to most-significant. This way of encoding a variable-byte length matches Google Protocol Buffers.
  • iostringstyle.data_toeof indicates a string format that contains string data until the end of the file
  • iostringstyle.data_null indicates a string that is terminated by a zero byte. It can be combined with other numeric values to indicate a string terminated by a particular byte. For example, to indicate a string terminated by $ (which in ASCII has byte value 0x24), one would use the value iostringstyle.data_null|0x24.
  • A positive and nonzero value indicates that a string of exactly that many bytes should be read or written.
proc stringStyleTerminated(terminator: uint(8))

This method returns the appropriate iostyle str_style value to indicate a string format where strings are terminated by a particular byte.

Arguments:terminator -- a byte value that the strings will be terminated by
Returns:a value that indicates a string format where strings are terminated by the terminator byte. This value is appropriate to store in iostyle.str_style.
proc stringStyleNullTerminated()

This method returns the appropriate iostyle str_style value to indicate a string format where strings are terminated by a zero byte.

proc stringStyleExactLen(len: int(64))

This method returns the appropriate iostyle str_style value to indicate a string format where strings have an exact length.

proc stringStyleWithVariableLength()

This method returns the appropriate iostyle str_style value to indicate a string format where string data is preceeded by a variable-byte length as described in iostringstyle.

proc stringStyleWithLength(lengthBytes: int)

This method returns the appropriate iostyle str_style value to indicate a string format where string data is preceeded by a lengthBytes of length. Only lengths of 1, 2, 4, or 8 are supported; if this method is called with any other length, it will halt with an error.

const IOHINT_NONE = 0: c_int

IOHINT_NONE means normal operation, nothing special to hint. Expect to use NONE most of the time. The other hints can be bitwise-ORed in.

const IOHINT_RANDOM = QIO_HINT_RANDOM

IOHINT_RANDOM means we expect random access to a file

const IOHINT_SEQUENTIAL = QIO_HINT_SEQUENTIAL

IOHINT_SEQUENTIAL means expect sequential access. On Linux, this should double the readahead.

const IOHINT_CACHED = QIO_HINT_CACHED

IOHINT_CACHED means we expect the entire file to be cached and/or we pull it in all at once. May request readahead on the entire file.

const IOHINT_PARALLEL = QIO_HINT_PARALLEL

IOHINT_PARALLEL means that we expect to have many channels working with this file in parallel. It might change the reading/writing implementation to something more efficient in that scenario.

record iostyle

The iostyle type represents I/O styles defining how Chapel's basic types should be read or written.

See The stdin, stdout, and stderr Channels.

var binary: uint(8) = 0

Perform binary I/O? 1 - yes, 0 - no. This field is ignored for iokind values other than dynamic.

var byteorder: uint(8) = iokind.native: uint(8)

What byte order should we use when performing binary I/O? This field is ignored for iokind values other than dynamic. It should be set to a value in iokind.

var str_style: int(64) = -10

What string format should we use when writing strings in binary mode? See iostringstyle for more information on what the values of str_style mean.

var min_width_columns: uint(32) = 0

When performing text I/O, pad out to this many columns

var max_width_columns: uint(32) = max(uint(32))

When performing text I/O, do not use more than this many columns

var max_width_characters: uint(32) = max(uint(32))

When performing text I/O, do not use more than this many characters

var max_width_bytes: uint(32) = max(uint(32))

When performing text I/O, do not use more than this many bytes

var string_start: style_char_t = 34

What character do we start strings with, when appropriate?

var string_end: style_char_t = 34

What character do we end strings with, when appropriate?

var string_format: uint(8) = 0

How should we format strings when performing text I/O?

  • QIO_STRING_FORMAT_WORD means string is as-is; reading reads until whitespace.
  • QIO_STRING_FORMAT_BASIC means only escape string_end and \ with \
  • QIO_STRING_FORMAT_CHPL means escape string_end \ ' " \n with \ and nonprinting characters c = 0xXY with \xXY
  • QIO_STRING_FORMAT_JSON means escape string_end " and \ with \, and nonprinting characters c = \uABCD
  • QIO_STRING_FORMAT_TOEND means string is as-is; reading reads until string_end
  • QIO_STRING_FORMAT_TOEOF means string is as-is; reading reads until end of file
var base: uint(8) = 0
var point_char: style_char_t = 46
var exponent_char: style_char_t = 101
var other_exponent_char: style_char_t = 112
var positive_char: style_char_t = 43
var negative_char: style_char_t = 45
var i_char: style_char_t = 105
var prefix_base: uint(8) = 1
var pad_char: style_char_t = 32
var showplus: uint(8) = 0
var uppercase: uint(8) = 0
var leftjustify: uint(8) = 0
var showpoint: uint(8) = 0
var showpointzero: uint(8) = 1
var precision: int(32) = -1
var realfmt: uint(8) = 0

Formatting of real numbers:

  • 0 means print out 'significant_digits' number of significant digits (%g in printf)
  • 1 means print out 'precision' number of digits after the decimal point (%f)
  • 2 means always use exponential and 'precision' number of digits (%e)
var complex_style: uint(8) = 0
var array_style: uint(8) = 0
var aggregate_style: uint(8) = 0
var tuple_style: uint(8) = 0
proc defaultIOStyle(): iostyle
Returns:the default I/O style. See iostyle and I/O Styles
proc iostyle.native(str_style: int(64) = stringStyleWithVariableLength()): iostyle

Get an I/O style indicating binary I/O in native byte order.

Arguments:str_style -- see iostringstyle - which format to use when reading or writing strings. Defaults to variable-byte length.
Returns:the requested iostyle
proc iostyle.big(str_style: int(64) = stringStyleWithVariableLength()): iostyle

Get an I/O style indicating binary I/O in big-endian byte order.

Arguments:str_style -- see iostringstyle - which format to use when reading or writing strings. Defaults to variable-byte length.
Returns:the requested iostyle
proc iostyle.little(str_style: int(64) = stringStyleWithVariableLength()): iostyle

Get an I/O style indicating binary I/O in little-endian byte order.

Arguments:str_style -- see iostringstyle - which format to use when reading or writing strings. Defaults to variable-byte length.
Returns:the requested iostyle
proc iostyle.text(): iostyle

Get an I/O style indicating text I/O.

Returns:the requested iostyle
type iohints = c_int

A value of the iohints type defines a set of hints about the I/O that the file or channel will perform. These hints may be used by the implementation to select optimized versions of the I/O operations.

The iohints type is implementation-defined. The following iohints constants are provided:

  • IOHINT_NONE defines an empty set, which provides no hints.
  • IOHINT_RANDOM suggests to expect random access.
  • IOHINT_SEQUENTIAL suggests to expect sequential access.
  • IOHINT_CACHED suggests that the file data is or should be cached in memory, possibly all at once.
  • IOHINT_PARALLEL suggests to expect many channels working with this file in parallel.

Other hints might be added in the future.

The following binary operators are defined on iohints:

  • | for set union
  • & for set intersection
  • == for set equality
  • 1= for set inequality

When an iohints formal has default intent, the actual is copied to the formal upon a function call and the formal cannot be assigned within the function.

The default value of the iohints type is undefined.

record file

The file type is implementation-defined. A value of the file type refers to the state that is used by the implementation to identify and interact with the OS file.

When a file formal argument has default intent, the actual is copied to the formal upon a function call and the formal cannot be assigned within the function.

The default value of the file type does not represent any OS file. It is illegal to perform any I/O operations on the default value.

proc file.check()

Halt if a file is invalid

proc file.close(out error: syserr)

Close a file.

In order to free the resources allocated for a file, it must be closed using this method.

It is an error to perform any I/O operations on a file that has been closed. It is an error to close a file when it has channels that have not been closed.

Closing a file does not guarantee immediate persistence of the performed updates, if any. In cases where immediate persistence is important, file.fsync should be used for that purpose prior to closing the file. In particular, even though closing the file might complete without errors, the data written might not persist in the event of a severe error like running out of storage space or power loss. See also Ensuring Successful I/O.

In the future, we hope to automatically close files when the file variable goes out of scope and all channels using that file are closed. The ability for a program to close a file will remain available.

Arguments:error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
proc file.fsync(out error: syserr)

Sync a file to disk.

Commits file data to the device associated with this file. Data written to the file by a channel will be committed only if the channel has been closed or flushed.

This function will typically call the fsync system call.

Arguments:error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
proc file.getPath(out error: syserr): string

Get the path to an open file.

Note that not all files have a path (e.g. files opened with openmem), and that this function may not work on all operating systems.

The function Path.file.realPath is an alternative way to get the path to a file.

Arguments:error -- argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
proc file.tryGetPath(): string

Get the path to an open file, or return "unknown" if there was a problem getting the path to the open file.

proc file.path: string

Get the path to an open file. Halt if there is an error getting the path.

proc file.length(): int(64)

Get the current length of an open file. Note that the length can always change if other channels, tasks or programs are writing to the file.

Returns:the current file length
proc open(out error: syserr, path: string = "", mode: iomode, hints: iohints = IOHINT_NONE, style: iostyle = defaultIOStyle(), url: string = ""): file

Open a file on a filesystem or stored at a particular URL. Note that once the file is open, you will need to use a file.reader or file.writer to create a channel to actually perform I/O operations

Arguments:
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
  • path -- which file to open (for example, "some/file.txt"). This argument is required unless the url= argument is used.
  • iomode -- specify whether to open the file for reading or writing and whether or not to create the file if it doesn't exist. See iomode.
  • hints -- optional argument to specify any hints to the I/O system about this file. See iohints.
  • style -- optional argument to specify I/O style associated with this file. The provided style will be the default for any channels created for on this file, and that in turn will be the default for all I/O operations performed with those channels.
  • url -- optional argument to specify a URL to open. See Curl and HDFS for more information on url= support for those systems. If HDFS is enabled, this function supports url= arguments of the form "hdfs://<host>:<port>/<path>". If Curl is enabled, this function supports url= starting with http://, https://, ftp://, ftps://, smtp://, smtps://, imap://, or imaps://
Returns:

an open file to the requested resource. If the error= argument was provided and the file was not opened because of an error, returns the default file value.

proc openfd(fd: fd_t, out error: syserr, hints: iohints = IOHINT_NONE, style: iostyle = defaultIOStyle()): file

Create a Chapel file that works with a system file descriptor Note that once the file is open, you will need to use a file.reader or file.writer to create a channel to actually perform I/O operations

The system file descriptor will be closed when the Chapel file is closed.

Note

This function can be used to create Chapel files that refer to system file descriptors that do not support the seek functionality. For example, file descriptors that represent pipes or open socket connections have this property. In that case, the resulting file value should only be used with one channel at a time. The I/O system will ignore the channel offsets when reading or writing to files backed by non-seekable file descriptors.

Arguments:
  • fd -- a system file descriptor (obtained with Sys.sys_open or Sys.sys_connect for example).
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
  • hints -- optional argument to specify any hints to the I/O system about this file. See iohints.
  • style -- optional argument to specify I/O style associated with this file. The provided style will be the default for any channels created for on this file, and that in turn will be the default for all I/O operations performed with those channels.
Returns:

an open file using the specified file descriptor. If the error= argument was provided and the file was not opened because of an error, returns the default file value.

proc openfp(fp: _file, out error: syserr, hints: iohints = IOHINT_NONE, style: iostyle = defaultIOStyle()): file

Create a Chapel file that works with an open C file (ie a FILE*). Note that once the file is open, you will need to use a file.reader or file.writer to create a channel to actually perform I/O operations

Note

The resulting file value should only be used with one channel at a time. The I/O system will ignore the channel offsets when reading or writing to a file opened with openfp.

Arguments:
  • fp -- a C FILE* to work with
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
  • hints -- optional argument to specify any hints to the I/O system about this file. See iohints.
  • style -- optional argument to specify I/O style associated with this file. The provided style will be the default for any channels created for on this file, and that in turn will be the default for all I/O operations performed with those channels.
Returns:

an open file that uses the underlying FILE* argument. If the error= argument was provided and the file was not opened because of an error, returns the default file value.

proc opentmp(out error: syserr, hints: iohints = IOHINT_NONE, style: iostyle = defaultIOStyle()): file

Open a temporary file. Note that once the file is open, you will need to use a file.reader or file.writer to create a channel to actually perform I/O operations.

The temporary file will be created in an OS-dependent temporary directory, for example "/tmp" is the typical location. The temporary file will be deleted upon closing.

Temporary files are always opened with iomode iomode.cwr; that is, a new file is created that supports both writing and reading.

Arguments:
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
  • hints -- optional argument to specify any hints to the I/O system about this file. See iohints.
  • style -- optional argument to specify I/O style associated with this file. The provided style will be the default for any channels created for on this file, and that in turn will be the default for all I/O operations performed with those channels.
Returns:

an open temporary file. If the error= argument was provided and the file was not opened because of an error, returns the default file value.

proc openmem(out error: syserr, style: iostyle = defaultIOStyle())

Open a file that is backed by a buffer in memory that will not persist when the file is closed. Note that once the file is open, you will need to use a file.reader or file.writer to create a channel to actually perform I/O operations.

The resulting file supports both reading and writing.

Arguments:
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
  • style -- optional argument to specify I/O style associated with this file. The provided style will be the default for any channels created for on this file, and that in turn will be the default for all I/O operations performed with those channels.
Returns:

an open memory file. If the error= argument was provided and the file was not opened because of an error, returns the default file value.

record channel

A channel supports either sequential reading or sequential writing to an underlying file object. A channel can buffer data. Read operations on the channel might return old data. Write operations might not have an immediate effect. Use channel.flush to control this buffering.

The channel type is implementation-defined. A value of the channel type refers to the state that is used to implement the channel operations.

When a channel formal has default intent, the actual is copied to the formal upon a function call and the formal cannot be assigned within the function.

The default value of the channel type is not associated with any file and so cannot be used to perform I/O.

The channel type is generic.

param writing: bool

writing is a boolean indicating whether the channels of this type support writing (when true) or reading (when false).

param kind: iokind

kind is an enum iokind that allows narrowing this channel's I/O style for more efficient binary I/O.

param locking: bool

locking is a boolean indicating whether it is safe to use this channel concurrently (when true).

record ioChar

Represents a Unicode code point. I/O routines (such as channel.read and channel.write) can use arguments of this type in order to read or write a single Unicode code point.

var ch: int(32)

The code point value

record ioNewline

Represents a newline character or character sequence (ie \n). I/O routines (such as channel.read and channel.write) can use arguments of this type in order to read or write a newline. This is different from 'n' because an ioNewline always produces an actual newline, but in some cases writing \n will produce an escaped string (such as "\n").

When reading an ioNewline, read routines will skip any character sequence (including e.g. letters and numbers) to get to the newline character unless skipWhitespaceOnly is set to true.

var skipWhitespaceOnly: bool = false

Normally, we will skip anything at all to get to a n, but if skipWhitespaceOnly is set, it will be an error if we run into non-space charcters other than n.

record ioLiteral

Used to represent a constant string we want to read or write.

When writing, the ioLiteral is output without any quoting or escaping.

When reading, the ioLiteral must be matched exactly - or else the read call will return an error with code SysBasic.EFORMAT.

var val: string

The value of the literal

var ignoreWhiteSpace: bool = true

Should read operations using this literal ignore and consume whitespace before the literal?

proc writeThis(f)
record ioBits

Represents a value with a particular bit length that we want to read or write. The I/O will always be done in binary mode.

var v: uint(64)

The bottom nbits of v will be read or written

var nbits: int(8)

How many of the low-order bits of v should we read or write?

proc channel.lock(out error: syserr)

Acquire a channel's lock.

Arguments:error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
proc channel.unlock()

Release a channel's lock.

proc channel.offset(): int(64)

Return the current offset of a channel. Note that other operations on the channel (e.g. by other tasks) might change the offset. If you are doing another operation on the channel based upon the current offset, you should use channel.lock, channel._offset, and channel.unlock to prevent race conditions.

Returns:the current offset of the channel
proc channel.advance(amount: int(64), ref error: syserr)

Move a channel offset forward.

For a reading channel, this function will consume the next amount bytes. If EOF is reached, the channel position may be left at the EOF.

For a writing channel, this function will write amount zeros - or some other data if it is stored in the channel's buffer, for example with channel._mark and channel._revert.

Arguments:error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
proc channel._offset(): int(64)

For a channel locked with channel.lock, return the offset of that channel.

proc channel._mark(): syserr

mark a channel - that is, save the current offset of the channel on its mark stack. This function should only be called on a channel that is already locked with with channel.lock.

The mark stack stores several channel offsets. For any channel offset that is between the minimum and maximum value in the mark stack, I/O operations on the channel will keep that region of the file buffered in memory so that those operations can be un-done. As a result, it is possible to perform I/O transactions on a channel. The basic steps for an I/O transaction are:

  • lock the channel with channel.lock (or work on an already-locked channel)
  • mark the current position with channel._mark
  • do something speculative (e.g. try to read 200 bytes of anything followed by a 'B')
  • if the speculative operation was successful, commit the changes by calling channel._commit
  • if the speculative operation was not succesful, go back to the mark by calling channel._revert. Subsequent I/O operations will work as though nothing happened.
  • unlock the channel with channel.unlock if necessary

Note

Note that it is possible to request an entire file be buffered in memory using this feature, for example by marking at offset=0 and then advancing to the end of the file. It is important to be aware of these memory space requirements.

Returns:an error code, if an error was encountered.
proc channel._revert()

Abort an I/O transaction. See channel._mark. This function should only be called on a channel that has already been locked and marked. This function will pop the last element from the mark stack and then leave the previous channel offset unchanged.

proc channel._commit()

Commit an I/O transaction. See channel._mark. This function should only be called on a channel that has already been locked and marked. This function will pop the last element from the mark stack and then set the channel offset to the popped offset.

proc channel._style(): iostyle

Return the current style used by a channel. This function should only be called on a locked channel.

proc channel._set_style(style: iostyle)

Set the style associated with a channel. This function should only be called on a locked channel.

proc openreader(out err: syserr, path: string = "", param kind = iokind.dynamic, param locking = true, start: int(64) = 0, end: int(64) = max(int(64)), hints: iohints = IOHINT_NONE, url: string = ""): channel(false, kind, locking)

Open a file at a particular path or URL and return a reading channel for it. This function is equivalent to calling open and then file.reader on the resulting file.

Arguments:
  • err -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
  • path -- which file to open (for example, "some/file.txt"). This argument is required unless the url= argument is used.
  • kind -- iokind compile-time argument to determine the corresponding parameter of the channel type. Defaults to iokind.dynamic, meaning that the associated iostyle controls the formatting choices.
  • locking -- compile-time argument to determine whether or not the channel should use locking; sets the corresponding parameter of the channel type. Defaults to true, but when safe, setting it to false can improve performance.
  • start -- zero-based byte offset indicating where in the file the channel should start reading. Defaults to 0.
  • end -- zero-based byte offset indicating where in the file the channel should no longer be allowed to read. Defaults to a max(int) - meaning no end point.
  • hints -- optional argument to specify any hints to the I/O system about this file. See iohints.
  • url -- optional argument to specify a URL to open. See Curl and HDFS for more information on url= support for those systems. If HDFS is enabled, this function supports url= arguments of the form "hdfs://<host>:<port>/<path>". If Curl is enabled, this function supports url= starting with http://, https://, ftp://, ftps://, smtp://, smtps://, imap://, or imaps://
Returns:

an open reading channel to the requested resource. If the error= argument was provided and the channel was not opened because of an error, returns the default channel value.

proc openwriter(out err: syserr, path: string = "", param kind = iokind.dynamic, param locking = true, start: int(64) = 0, end: int(64) = max(int(64)), hints: iohints = IOHINT_NONE, url: string = ""): channel(true, kind, locking)

Open a file at a particular path or URL and return a writing channel for it. This function is equivalent to calling open with iomode.cwr and then file.writer on the resulting file.

Arguments:
  • err -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
  • path -- which file to open (for example, "some/file.txt"). This argument is required unless the url= argument is used.
  • kind -- iokind compile-time argument to determine the corresponding parameter of the channel type. Defaults to iokind.dynamic, meaning that the associated iostyle controls the formatting choices.
  • locking -- compile-time argument to determine whether or not the channel should use locking; sets the corresponding parameter of the channel type. Defaults to true, but when safe, setting it to false can improve performance.
  • start -- zero-based byte offset indicating where in the file the channel should start writing. Defaults to 0.
  • end -- zero-based byte offset indicating where in the file the channel should no longer be allowed to write. Defaults to a max(int) - meaning no end point.
  • hints -- optional argument to specify any hints to the I/O system about this file. See iohints.
  • url -- optional argument to specify a URL to open. See Curl and HDFS for more information on url= support for those systems. If HDFS is enabled, this function supports url= arguments of the form "hdfs://<host>:<port>/<path>". If Curl is enabled, this function supports url= starting with http://, https://, ftp://, ftps://, smtp://, smtps://, imap://, or imaps://
Returns:

an open reading channel to the requested resource. If the error= argument was provided and the channel was not opened because of an error, returns the default channel value.

proc file.reader(out error: syserr, param kind = iokind.dynamic, param locking = true, start: int(64) = 0, end: int(64) = max(int(64)), hints: iohints = IOHINT_NONE, style: iostyle = this._style): channel(false, kind, locking)

Create a channel that supports reading from a file. See I/O Overview.

The start= and end= arguments define the region of the file that the channel will read from. These are byte offsets; the beginning of the file is at the offset 0. The defaults for these arguments enable the channel to access the entire file.

A channel will never read beyond its maximum end position. In addition, reading from a channel beyond the end of the underlying file will not extend that file. Reading beyond the end of the file or beyond the end offset of the channel will produce the error EEOF (and return false in many cases such as channel.read) to indicate that the end was reached.

Arguments:
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
  • kind -- iokind compile-time argument to determine the corresponding parameter of the channel type. Defaults to iokind.dynamic, meaning that the associated iostyle controls the formatting choices.
  • locking -- compile-time argument to determine whether or not the channel should use locking; sets the corresponding parameter of the channel type. Defaults to true, but when safe, setting it to false can improve performance.
  • start -- zero-based byte offset indicating where in the file the channel should start reading. Defaults to 0.
  • end -- zero-based byte offset indicating where in the file the channel should no longer be allowed to read. Defaults to a max(int) - meaning no end point.
  • hints -- provide hints about the I/O that this channel will perform. See iohints. The default value of IOHINT_NONE will cause the channel to use the hints provided when opening the file.
  • style -- provide a iostyle to use with this channel. The default value will be the iostyle associated with this file.
proc file.lines(out error: syserr, param locking: bool = true, start: int(64) = 0, end: int(64) = max(int(64)), hints: iohints = IOHINT_NONE, in local_style: iostyle = this._style)

Iterate over all of the lines in a file.

Arguments:error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:an object which yields strings read from the file
proc file.writer(out error: syserr, param kind = iokind.dynamic, param locking = true, start: int(64) = 0, end: int(64) = max(int(64)), hints: iohints = IOHINT_NONE, style: iostyle = this._style): channel(true, kind, locking)

Create a channel that supports writing to a file. See I/O Overview.

The start= and end= arguments define the region of the file that the channel will write to. These are byte offsets; the beginning of the file is at the offset 0. The defaults for these arguments enable the channel to access the entire file.

When a channel writes to a file, it will replace file data that was previously stored at the relevant offset. If the offset is beyond the end of the file, the file will be extended.

A channel will never write beyond its maximum end position. It will extend the file only as necessary to store data written to the channel. In other words, specifying end here does not impact the file size directly; it impacts only the section of the file that this channel can write to. After all channels to a file are closed, that file will have a size equal to the last position written to by any channel.

Arguments:
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
  • kind -- iokind compile-time argument to determine the corresponding parameter of the channel type. Defaults to iokind.dynamic, meaning that the associated iostyle controls the formatting choices.
  • locking -- compile-time argument to determine whether or not the channel should use locking; sets the corresponding parameter of the channel type. Defaults to true, but when safe, setting it to false can improve performance.
  • start -- zero-based byte offset indicating where in the file the channel should start writing. Defaults to 0.
  • end -- zero-based byte offset indicating where in the file the channel should no longer be allowed to write. Defaults to a max(int) - meaning no end point.
  • hints -- provide hints about the I/O that this channel will perform. See iohints. The default value of IOHINT_NONE will cause the channel to use the hints provided when opening the file.
  • style -- provide a iostyle to use with this channel. The default value will be the iostyle associated with this file.
proc channel.readwrite(x)

For a writing channel, writes as with channel.write. For a reading channel, reads as with channel.read. Stores any error encountered in the channel. Does not return anything.

proc <~>(ch: channel, x)

The <~> operator

This <~> operator is the same as calling channel.readwrite, except that it returns the channel so that multiple operator calls can be chained together.

Returns:ch
proc <~>(r: channel, lit: ioLiteral)

Overload to support reading an IO.ioLiteral without passing ioLiterals by reference, so that

reader <~> new ioLiteral("=")

works without requiring an explicit temporary value to store the ioLiteral.

proc <~>(r: channel, nl: ioNewline)

Overload to support reading an IO.ioNewline without passing ioNewline by reference, so that

reader <~> new ioNewline("=")

works without requiring an explicit temporary value to store the ioNewline.

proc channel.readWriteLiteral(lit: string, ignoreWhiteSpace = true)

Explicit call for reading or writing a literal as an alternative to using IO.ioLiteral.

proc channel.readWriteNewline()

Explicit call for reading or writing a newline as an alternative to using IO.ioNewline.

proc channel.binary(): bool

Returns true if this channel is configured for binary I/O.

proc channel.error(): syserr

Return any saved error code.

proc channel.setError(e: syserr)

Save an error code.

proc channel.clearError()

Clear any saved error code.

proc channel.writeBytes(x, len: ssize_t)

Write a sequence of bytes.

proc channel.read(ref args ...?k, out error: syserr): bool

Returns true if we read all the args, false if we encountered EOF (or possibly another error and didn't halt)

proc stringify(args ...?k): string

Creates a string representing the result of writing the arguments.

Writes each argument, possibly using a writeThis method, to a string and returns the result.

proc channel.read(ref args ...?k, style: iostyle, out error: syserr): bool

Read values from a channel. The input will be consumed atomically - the channel lock will be held while reading all of the passed values.

Arguments:
  • args -- a list of arguments to read. Basic types are handled internally, but for other types this function will call value.readThis() with a Reader argument as described in the specification.
  • style -- optional argument to provide an iostyle for this read. If this argument is not provided, use the current style associated with this channel.
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:

true if the read succeded, and false on error or end of file.

proc channel.readline(arg: [] uint(8), out numRead: int, start = arg.domain.low, amount = arg.domain.high-start, out error: syserr): bool

Read a line into a Chapel array of bytes. Reads until a \n is reached. The \n is consumed but not returned in the array.

Arguments:
  • arg -- A 1D DefaultRectangular array which must have at least 1 element.
  • numRead -- The number of bytes read.
  • start -- Index to begin reading into.
  • amount -- The maximum amount of bytes to read.
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:

true if the bytes were read without error.

proc channel.readline(ref arg: string, out error: syserr): bool

Read a line into a Chapel string. Reads until a \n is reached. The \n is included in the resulting string.

Arguments:
  • arg -- a string to receive the line
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:

true if a line was read without error, false on error or EOF

proc channel.readstring(ref str_out: string, len: int(64) = -1, out error: syserr): bool

read a given number of bytes from a channel

Arguments:
  • str_out -- The string to be read into
  • len -- Read up to len bytes from the channel, up until EOF (or some kind of I/O error). If the default value of -1 is provided, read until EOF starting from the channel's current offset.
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:

true if we read something, false on EOF or error

proc channel.readbits(out v: integral, nbits: integral, out error: syserr): bool

Read bits with binary I/O

Arguments:
  • v -- where to store the read bits. This value will have its nbits least-significant bits set.
  • nbits -- how many bits to read
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:

true if the bits were read without error, false on error or EOF

proc channel.writebits(v: integral, nbits: integral, out error: syserr): bool

Write bits with binary I/O

Arguments:
  • v -- a value containing nbits bits to write the least-significant bits
  • nbits -- how many bits to write
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:

true if the bits were written without error, false on error

proc channel.readln(ref args ...?k, style: iostyle, out error: syserr): bool

Read values from a channel and then consume any bytes until newline is reached. The input will be consumed atomically - the channel lock will be held while reading all of the passed values.

Arguments:
  • args -- a list of arguments to read. This routine can be called with zero or more such arguments. Basic types are handled internally, but for other types this function will call value.readThis() with a Reader argument as described in the specification.
  • style -- optional argument to provide an iostyle for this read. If this argument is not provided, use the current style associated with this channel.
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:

true if the read succeded, and false on error or end of file.

proc channel.read(type t)

Read a value of passed type. Halts if an error is encountered.

Note

It is difficult to handle errors or to handle reaching the end of the file with this function. If such cases are important please use the channel.read returning the values read in arguments instead.

For example, the following line of code reads a value of type int from stdin and uses it to initialize a variable x:

var x = stdin.read(int)
Arguments:t -- the type to read
Returns:the value read
proc channel.readln(type t)

Read a value of passed type followed by a newline. Halts if an error is encountered.

Note

It is difficult to handle errors or to handle reaching the end of the file with this function. If such cases are important please use channel.readln instead.

Arguments:t -- the type to read
Returns:the value read
proc channel.readln(type t ...?numTypes)

Read values of passed types followed by a newline and return a tuple containing the read values. Halts if an error is encountered.

Arguments:t -- more than one type to read
Returns:a tuple of the read values
proc channel.read(type t ...?numTypes)

Read values of passed types and return a tuple containing the read values. Halts if an error is encountered.

Arguments:t -- more than one type to read
Returns:a tuple of the read values
proc channel.write(args ...?k, style: iostyle, out error: syserr): bool

Write values to a channel. The output will be produced atomically - the channel lock will be held while writing all of the passed values.

Arguments:
  • args -- a list of arguments to write. Basic types are handled internally, but for other types this function will call value.writeThis() with the channel as an argument.
  • style -- optional argument to provide an iostyle for this write. If this argument is not provided, use the current style associated with this channel.
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:

true if the write succeded

proc channel.writeln(args ...?k, style: iostyle, out error: syserr): bool

Write values to a channel followed by a newline. The output will be produced atomically - the channel lock will be held while writing all of the passed values.

Arguments:
  • args -- a variable number of arguments to write. This method can be called with zero or more arguments. Basic types are handled internally, but for other types this function will call value.writeThis() with the channel as an argument.
  • style -- optional argument to provide an iostyle for this write. If this argument is not provided, use the current style associated with this channel.
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:

true if the write succeded

proc channel.flush(out error: syserr)

Makes all writes to the channel, if any, available to concurrent viewers of its associated file, such as other channels or other applications accessing this file concurrently. Unlike file.fsync, this does not commit the written data to the file's device.

Arguments:error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
proc channel.assertEOF(error: string)

Assert that a channel has reached end-of-file. Halts with an error message if the receiving channel is not currently at EOF.

Arguments:error -- an optional string argument which will be printed out if the assert fails. The default prints "Not at EOF".
proc channel.close(out error: syserr)

Close a channel. Implicitly performs the channel.flush operation (see Synchronization of Channel Data and Avoiding Data Races).

Arguments:error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
proc channel.isclosed()

Return true if a channel is currently closed.

record ItemReader

Wrapper class on a channel to make it only read values of a single type. Also supports an iterator yielding the read values.

type ItemType

What type do we read and yield?

param kind: iokind

the kind field for our channel

param locking: bool

the locking field for our channel

var ch: channel(false, kind, locking)

our channel

proc read(out arg: ItemType, out error: syserr): bool

read a single item, returning an error

proc read(out arg: ItemType): bool

read a single item, halting on error

iter these()

iterate through all items of that type read from the channel

proc channel.itemReader(type ItemType, param kind: iokind = iokind.dynamic)

Create and return an ItemReader that can yield read values of a single type.

record ItemWriter
type ItemType

What type do we write?

param kind: iokind

the kind field for our channel

param locking: bool

the locking field for our channel

var ch: channel(true, kind, locking)

our channel

proc write(arg: ItemType, out error: syserr): bool

write a single item, returning an error

proc write(arg: ItemType): bool

write a single item, halting on error

proc channel.itemWriter(type ItemType, param kind: iokind = iokind.dynamic)

Create and return an ItemWriter that can write values of a single type.

const stdin: channel(false, iokind.dynamic, true) = AppendExpr.Call08

standard input, otherwise known as file descriptor 0

const stdout: channel(true, iokind.dynamic, true) = AppendExpr.Call08

standard output, otherwise known as file descriptor 1

const stderr: channel(true, iokind.dynamic, true) = AppendExpr.Call08

standard error, otherwise known as file descriptor 2

proc write(args ...?n)

Equivalent to stdout.write. See channel.write

proc writeln(args ...?n)

Equivalent to stdout.writeln. See channel.writeln

proc read(ref args ...?n): bool

Equivalent to stdin.read. See channel.read

proc readln(ref args ...?n): bool

Equivalent to stdin.readln. See channel.readln

proc readln(type t ...?numTypes)

Equivalent to stdin.readln. See channel.readln for types

proc read(type t ...?numTypes)

Equivalent to stdin.read. See channel.read for types

Delete a file. This function is likely to be replaced by FileSystem.remove.

Arguments:
  • path -- the path to the file to remove
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
proc unicodeSupported(): bool
Returns:true if this version of the Chapel runtime supports UTF-8 output.
proc channel.writef(fmtStr: string, args ...?k, out error: syserr): bool

Write arguments according to a format string. See Formatted I/O.

Arguments:
  • fmt -- the format string
  • args -- the arguments to write
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
proc channel.readf(fmtStr: string, ref args ...?k, out error: syserr): bool

Read arguments according to a format string. See Formatted I/O.

Arguments:
  • fmt -- the format string
  • args -- the arguments to read
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:

true if all arguments were read according to the format string, false on EOF. If the format did not match the input, returns false with error=EFORMAT or halts if no error argument was provided.

proc writef(fmt: string, args ...?k): bool

Call stdout.writef; see channel.writef.

proc readf(fmt: string, ref args ...?k): bool

Call stdout.readf; see channel.readf.

proc channel.skipField(out error: syserr)

Skip a field in the current aggregate format. This method is currently only supported for JSON format and returns ENOTSUP for other formats. In other formats, it may not be possible in general to know when a field ends.

The field skipped includes a field name and value but not a following separator. For example, for a JSON format channel, given the input:

"fieldName":"fieldValue", "otherField":3

this function will skip to (but leave unread) the comma after the first field value.

Arguments:error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
proc string.format(args ...?k, out error: syserr): string

Return a new string consisting of values formatted according to a format string. See Formatted I/O.

Arguments:
  • this -- the format string
  • args -- the arguments to format
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:

the resulting string

proc channel.extractMatch(m: reMatch, ref arg, ref error: syserr)

Sets arg to the string of a match.

Assumes that the channel has been marked before where the captures are being returned. Will change the channel position to just after the match. Will not do anything if error is set.

Arguments:
  • m -- a Regexp.reMatch storing a location that matched
  • arg -- an argument to retrieve the match into. If it is not a string, the string match will be cast to arg.type.
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
proc channel.search(re: regexp, ref captures ...?k, ref error: syserr): reMatch

Search for an offset in the channel matching the passed regular expression, possibly pulling out capture groups. If there is a match, leaves the channel position at the match. If there is no match, the channel position will be advanced to the end of the channel (or end of the file).

Arguments:
  • re -- a Regexp.regexp record representing a compiled regular expression.
  • captures -- an optional variable number of arguments in which to store the regions of the file matching the capture groups in the regular expression.
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:

the region of the channel that matched

proc channel.match(re: regexp, ref captures ...?k, ref error: syserr): reMatch

Match, starting at the current position in the channel, against a regexp, possibly pulling out capture groups. If there was a match, leaves the channel position at the match. If there was no match, leaves the channel position where it was at the start of this call.

Arguments:
  • re -- a Regexp.regexp record representing a compiled regular expression.
  • captures -- an optional variable number of arguments in which to store the regions of the file matching the capture groups in the regular expression.
  • error -- optional argument to capture an error code. If this argument is not provided and an error is encountered, this function will halt with an error message.
Returns:

the region of the channel that matched

iter channel.matches(re: regexp, param captures = 0, maxmatches: int = max(int))

Enumerates matches in the string as well as capture groups.

Yields tuples of Regexp.reMatch objects, the 1st is always the match for the whole pattern.

At the time each match is returned, the channel position is at the start of that match. Note though that you would have to use channel.advance to get to the position of a capture group.

After returning each match, advances to just after that match and looks for another match. Thus, it will not return overlapping matches.

In the end, leaves the channel position at the end of the last reported match (if we ran out of maxmatches) or at the end of the channel (if we no longer matched)

Holds the channel lock for the duration of the search.

Arguments:
  • re -- a Regexp.regexp record representing a compiled regular expression.
  • captures -- an optional compile-time constant representing the number of captures to be yielded in tuple elements.
  • maxmatches -- the maximum number of matches to report.
Yields:

tuples of Regexp.reMatch objects, where the first element is the whole pattern. The tuples will have 1+captures elements.

proc file.getchunk(start: int(64) = 0, end: int(64) = max(int(64))): (int(64), int(64))

Returns (chunk start, chunk end) for the first chunk in the file containing data in the region start..end-1. Note that the returned chunk might not cover all of the region in question.

Returns (0,0) if no such value exists.

Arguments:
  • start -- the file offset (starting from 0) where the region begins
  • end -- the file offset just after the region
Returns:

a tuple of (chunkStart, chunkEnd) so that the bytes in chunkStart..chunkEnd-1 are stored in a manner that makes reading that chunk at a time most efficient

proc file.localesForRegion(start: int(64), end: int(64))

Returns the 'best' locale to run something working with the region of the file in start..end-1.

This must return the same result when called from different locales. Returns a domain of locales that are "best" for the given region. If no locales are "best" we return a domain containing all locales.

Arguments:
  • start -- the file offset (starting from 0) where the region begins
  • end -- the file offset just after the region
Returns:

a set of locales that are best for working with this region

Return type:

domain(locale)