ZMQ

Usage

use ZMQ;

Lightweight messaging with ZeroMQ (or ØMQ)

This module provides high-level Chapel bindings to the ZeroMQ messaging library.

Dependencies

The ZMQ module in Chapel is dependent on ZeroMQ. For information on how to install ZeroMQ, see the ZeroMQ installation instructions.

Note

Chapel's support for ZeroMQ is a work in progress and may not (yet) support the full functionality of the ZeroMQ C API.

Note

Chapel's ZMQ module was developed for compatibility with ZeroMQ v4.x.

Using ZMQ in Chapel

Contexts

In ZeroMQ proper, a context is an opaque, thread-safe handle to manage ØMQ's resources within a process. Typically, a process will allocate only one context, although more than one context per process is allowable [ref].

In Chapel, a Context is a reference-counted wrapper around the underlying ØMQ context. To create a context, it is sufficient to write:

var context: Context;

Sockets

In ZeroMQ, a socket is an opaque handle to an asynchronous, message-based communication channel that is "typed" to provide one of a series of common communication patterns (i.e., the socket type).

In Chapel, as with a Context, a Socket is a reference-counted wrapper around the underlying ØMQ socket. Sockets are created via Context.socket() and maintain a reference to the parent context, so that the parent context may go out of scope and the context will not be reclaimed while any sockets are still in use.

Note

As with ØMQ's C API, a Socket object is not thread safe. That is, a Socket object should not be accessed concurrently by multiple Chapel tasks. (This may change in a future ZMQ module.)

A Socket may be one of the socket types in the following list of compatible pairs of socket types [ref]:

// create a PUB socket
var context: Context;
var socket = context.socket(ZMQ.PUB);

Sending and Receiving Messages

Sending a message is as simple as passing the object to send as the argument to Socket.send(). Receiving a message requires that the type to be received be passed as the argument to Socket.recv(). In either case, if the object sent or type to be received cannot be serialized by ZMQ, the following error shall be produced at compile time.

// send or receive an int
var val = 42;
socket.send(val);
val = socket.recv(int);

// error: Type "Foo" is not serializable by ZMQ
class Foo { var val: int; }
socket.recv(Foo);

Multilocale Support

Chapel's ZMQ module supports multilocale execution of ZeroMQ programs. The locale on which the Context object is created sets the "home" locale for all sockets created from this context and all operations performed on the socket. For example, a send call on a socket from a locale other than the home locale will migrate a task to the home locale that will remotely access the data to send it over the socket.

Examples

Example: "Hello, World"

This "Hello, World" example demonstrates a PUSH-PULL socket pair in two Chapel programs that exchange a single string message.

// pusher.chpl
use ZMQ;
config const to: string = "world!";
var context: Context;
var socket = context.socket(ZMQ.PUSH);
socket.bind("tcp://*:5555");
socket.send(to);
// puller.chpl
use ZMQ;
var context: Context;
var socket = context.socket(ZMQ.PULL);
socket.connect("tcp://localhost:5555");
writeln("Hello, ", socket.recv(string));

Implementation Notes

As noted previously, the ZMQ module is a work in progress. The implementation notes below describe some of how the ZMQ module is implemented and how future versions may expose more features native to ZeroMQ.

Serialization

In Chapel, sending or receiving messages is supported for a variety of types. Primitive numeric types and strings are supported as the foundation. In addition, user-defined record types may be serialized automatically as multipart messages by internal use of the Reflection module. Currently, the ZMQ module can serialize records of primitive numeric types, strings, and other serializable records.

Note

The serialization protocol for strings changed in Chapel 1.16 in order to support inter-language messaging through ZeroMQ. (See notes on interoperability below.) As a result, programs using ZMQ that were compiled by Chapel 1.15 or earlier cannot communicate using strings with programs compiled by Chapel 1.16 or later, although such communication can be used between programs compiled by the same version without issue.

Prior to Chapel 1.16, ZMQ would send a string as two multipart messages: the first sent the length as int; the second sent the character array (in bytes). It was identified that this scheme was incompatible with how other language bindings for ZeroMQ serialize and send strings.

As of Chapel 1.16, the ZMQ module uses the C-level zmq_msg_send() and zmq_msg_recv() API for Socket.send() and Socket.recv(), respectively, when transmitting strings. Further, ZMQ sends the string as a single message of only the byte stream of the string's character array. (Recall that Chapel's string type currently only supports ASCII strings, not full Unicode strings.)

Interoperability

The ZeroMQ messaging library has robust support in many programming languages and Chapel's ZMQ module intends on providing interfaces and serialization protocols suitable for exchanging data between Chapel and non-Chapel programs.

As an example (and test) of Chapel-Python interoperability over ZeroMQ, the following sources demonstrate a REQ-REP socket pair between a Chapel server and a Python client using the PyZMQ Python bindings for ZeroMQ.

  begin {
    var client = spawn(["./wrapper.sh", "client.py"]);
    client.communicate();
  }

// server.chpl
use ZMQ;

var context: Context;
var socket = context.socket(ZMQ.REP);
socket.bind("tcp://*:5555");

for i in 0..#10 {
  var msg = socket.recv(string);
  writeln("[Chapel] Received message: ", msg);
  socket.send("Hello %i from Chapel".format(i));
}
# client.py
import zmq

context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")

for request in range(10):
    message = "Hello %i from Python" % request
    print("[Python] Sending request: %s" % message)
    socket.send_string(message)
    message = socket.recv_string()
    print("[Python] Received response: %s" % message)

Tasking-Layer Interaction

As noted previously, a Socket object is not thread safe. As the mapping of Chapel tasks to OS threads is dependent on the Chapel tasking layer and may be cooperatively scheduled, a Socket object should not be accessed concurrently by multiple Chapel tasks.

The ZMQ module is designed to "play nicely" with the Chapel tasking layer. While the C-level call zmq_send() may be a blocking call (depending on the socket type and flag arguments), it is desirable that a semantically-blocking call to Socket.send() allow other Chapel tasks to be scheduled on the OS thread as supported by the tasking layer. Internally, the ZMQ module uses non-blocking calls to zmq_send() and zmq_recv() to transfer data, and yields to the tasking layer via chpl_task_yield() when the call would otherwise block.

Limitations and Future Work

Currently, the ZMQ module does not provide interfaces for working with ZeroMQ message objects, handling errors, or making explicitly non-blocking send/receive calls. These are active-but-incomplete areas of work that are intended to be supported in future Chapel releases.

One interaction of these features is worth noting explicitly: because multipart messages are used to automatically serialize non-primitive data types (e.g., strings and records) and a partially-sent multi-part message cannot be canceled (except by closing the socket), an explicitly non-blocking send call that encountered an error in the ZeroMQ library during serialization would not be in a recoverable state, nor would there be a matching "partial receive".

References

const PUB = ZMQ_PUB

The publisher socket type for a publish-subscribe messaging pattern.

const SUB = ZMQ_SUB

The subscriber socket type for a publish-subscribe messaging pattern.

const REQ = ZMQ_REQ

The requester socket type for a paired request-reply messaging pattern.

const REP = ZMQ_REP

The replier socket type for a paired request-reply messaging pattern.

const PUSH = ZMQ_PUSH

The pusher socket type for a pipeline messaging pattern.

const PULL = ZMQ_PULL

The puller socket type for a pipeline messaging pattern.

const SUBSCRIBE = ZMQ_SUBSCRIBE

The Socket.setsockopt() option value to specify the message filter for a SUB-type Socket.

const UNSUBSCRIBE = ZMQ_UNSUBSCRIBE

The Socket.setsockopt() option value to remote an existing message filter for a SUB-type Socket.

const LINGER = ZMQ_LINGER

The Socket.setsockopt() option value to specify the linger period for the associated Socket object.

proc version: (int, int, int)

Query the ZMQ library version.

Returns:An (int,int,int) tuple of the major, minor, and patch version of the ZMQ library.
record Context

A ZeroMQ context. See more on using Contexts. Note that this record contains private fields not listed below.

proc init()

Create a ZMQ context.

proc socket(sockType: int): Socket

Create a Socket of type sockType from this context.

Arguments:sockType : int -- The ZMQ socket type to be created; e.g., PUB, PUSH, etc.
record Socket

A ZeroMQ socket. See more on using Sockets. Note that this record contains private fields not listed below.

proc close(linger: int = unset)

Close the socket.

Arguments:linger : int -- Optional argument to specify the linger period for the socket prior to closing. If -1, then the linger period is infinite; if non-negative, then the linger period shall be set to the specified value (in milliseconds).
proc bind(endpoint: string)

Bind the socket to the specified local endpoint and accept incoming connections.

proc connect(endpoint: string)

Connect the socket to the specified endpoint.

proc setsockopt(option: int, value: ?T)

Set socket options; see zmq_setsockopt

Arguments:
proc send(data: ?T)

Send an object data on a socket.

Arguments:data -- The object to be sent. If data is an object whose type is not serializable by the ZMQ module, a compile-time error will be raised.
proc recv(type T): T

Receive an object of type T from a socket.

Arguments:T -- The type of the object to be received. If T is not serializable by the ZMQ module, a compile-time error will be raised.
Returns:An object of type T